This week’s conference on ‘Systemic approaches in evaluation’, hosted by GIZ in Frankfurt showcased some useful approaches to evaluation (which will be made available on the conference website soon). It also highlighted some common myths about systems approaches to evaluation:
1. Systems approaches are about including everything
This is impossible. And trying to do it is not useful. One of the benefits of systems approaches is paying careful attention to how boundaries are drawn. One particular systems method, Critical Systems Heuristics, includes specific attention to this issue.
2. Systems approaches are more expensive than traditional studies
This myth follows on from a misconception that systems approaches require measuring everything. Some systems approaches need lots of data, specialist expertise and specialist software. Other systems approaches involve applying some ideas which don’t require specialist expertise or additional expense.
3. Systems approaches are about complexity, which is about making things very, very complicated
Frameworks that distinguish between simple, complicated, and complex aspects of programs and situation (such as Glouberman and Zimmerman, and Kurtz and Snowden) make important distinction between complicated and complex.
If it is possible to identify all the components and their relationships, at least for experts, then it is more useful to classify this as complicated. The word ‘complex’ is reserved for interventions and situations where it is not possible, even for experts, to know in advance what will emerge.
4. Systems approaches are always quantitative/always qualitative
Many people think that systems dynamics is the only systems approach, which uses quantitative data to model situations or interventions.
But sometimes people think that systems approaches are always qualitative. Some of this confusion might arise from the Latin American approach to evaluation called ‘systematization‘, which is a participatory, largely qualitative approach to organizational learning and process improvement.
Some systems approaches use qualitative data and some use quantitative data. Some are based in a post-positivist epistemology, some in a constructivist epistemology, and some in a pragmatic approach which seeks to understand both objective reality and people’s subjective response to it.
5. Systems approaches focus on processes not on outcomes and impacts
Some systems approaches do focus on processes and developing better evaluation questions. But other approaches focus particularly on outcomes and impacts.
Michael Patton, in his book Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, shows how using systems concepts is not an excuse for dodging hard questions about impact, and in fact require careful attention to evidence about impact.
6. Systems approaches are only about evaluation for learning, not evaluation for accountability
Systems approaches are often used to support organizational learning. But they can also provide better evidence for evaluations that are about accountability, by highlighting different perspectives about what constitute relevant impacts (positive or negative) or through better understanding of how impacts have been achieved.
7. Systems approaches are always participatory
Like all evaluations, evaluations that use systems approaches can often be enhanced by appropriate and effective participation of relevant stakeholders, but it is not an essential component of all approaches.
One of the clear messages is that there is incredible diversity in systems approaches, concepts, methods and tools. Bob Williams and Richard Hummelbrunner, in their book Systems Concepts in Action: A Practitioner’s Toolkit, and in Bob’s brief article, show how different systems concepts and approaches are relevant to different types of evaluation questions.
Listen to the next discussion you hear about using systems thinking or systems approaches in evaluation and check if any of these myths are in play. Or send us new ones to add to the list.
Evaluation and complexity – Patricia Rogers
Conference on evaluation and complexity – Patricia Rogers