Evaluation and complexity

I’ve recently been running workshops on purposeful program theory in Australia, New Zealand and the United States. It’s been a great treat to explore with so many different people how we might develop, represent and use program theory for policies and programs that have significant complicated or complex aspects

It’s an issue I’ve been puzzling over for some time now, drawing on the work of Glouberman and Zimmerman, Kurtz and Snowden, and Patton who distinguish between what is complicated (multiple components, requiring expertise and co-ordination) and complex (emergent, where cause and effect are understandable only in retrospect).

The term ‘complex’ gets thrown around a lot. Sometimes it is used as a fancy or derogatory way of saying ‘complicated’. Sometimes it’s used as an excuse for not providing transparent explanations or doing adequate planning. And given this it is understandable that some politicians, policymakers and funders are growing impatient when the term is used.

But since it is clear that not everything can be effectively planned, managed or evaluated as if it were simple, or even complicated, it is important for us to be able to explain both what we mean by complexity, and how we might support effective management of complexity.

Dave Snowden has a memorable explanation of chaotic, ordered and complex systems, applying them to the planning of a children’s party. The big implications for evaluation are::

  1. it needs to help to articulate boundaries – perhaps in the form of guiding principles within which decisions and action can be taken
  2. it needs to observe and report what happens when there is a probe (an initial action) – this rapid feedback is essential to inform adaptive management

Related posts:

Launch of Developmental Evaluation

Evaluation Revisited Conference

1 comment to Evaluation and complexity

  • There is an aspect of “complexity” that I’d like to toss into the conversation. A great deal of the discussion about complexity has the flavor of unpredictability, which is another way of tapping notions such as path dependence and changing fitness landscapes. All true, but let’s not forget something else. In my business (and yours too I bet) the defining characteristic is not unpredictability but stability. Programs, systems, organizations — they all seem impervious to evaluation. You can whack them on the head continually with data, and no change happens. One way to think about this is to say that the entities in question are so deep in a stable attractor that they don’t even know there is anywhere else to exist. Another is to assume that they don’t change because it is in fact adaptive for them not to, i.e. they are well evolved to thrive in their environment. Or one could look at the internals of the system and observe the cross linkages and mutually supporting elements that keep the system chugging along. Or, one might think of the constituents of the system as autonomous agents acting on decision rules whose higher level consequences (emergent behavior) are predictable. All of these are elements of complex adaptive systems (and in many ways, different ways of saying the same thing.) In any case, if we are going to apply concepts of complexity to evaluation, I think we should consider how the whole range of complex system behavior is relevant to what we do.