How good is a “good” outcome?

Earlier in the week, I passed on a quote from a review of Ziliak and McCloskey’s (2008) book The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives asserting that:

… many researchers are so obsessed with statistical significance that they neglect to ask themselves whether the detected discrepancies are large enough to be of any subject-matter significance

From a review by Olle Häggström in the “commentary” column of
Notices of the Journal of the American Mathematical Society

Reflecting on this quote, I posed the question:

How good is a “good” outcome for a program, policy, initiative, product or service?

I’ve been puzzling over this one a lot recently, particularly with respect to outcomes that involve a shift or change, and it seems to me to come down to a few key considerations:

  1. Life impact/practical significance – is the experienced change enough to actually make a noticeable impact on people’s lives, as opposed to just a tiny blip on the radar of life?
  2. Destination – does the shift get the impactees/recipients/consumers where they need to go with repect to this outcome? Does it eliminate the most important sources of disadvantage or dysfunction? How far does it get with respect to fulfilling aspirations (as opposed to merely meeting needs or filling gaps)?
  3. Relative magnitude of shift – is the shift about as substantial as it gets, relative to what other efforts have achieved? Or pretty weak in the scheme of things?

For example, I’ve recently been looking at initiatives in schools that are aimed specifically at students achieving below curriculum expectations in literacy. In translating the above concepts for the specific topic/program to interpret outcomes for students, we get:

  1. Is student progress in literacy being accelerated substantially relative to the progress we would normally expect to see without any special initiative or approach? Is the magnitude of the acceleration big enough that students, teachers, parents and others would really notice the difference? [life impact/practical significance]
  2. Is the total progress and acceleration being achieved – for example, across the 6-8 years students are in primary (elementary) school - large enough and fast enough to have the vast majority of these students sufficiently ‘up to speed’ in reading and writing to be able to access the rest of the curriculum as they transition into high school? [destination]
  3. How fast is the acceleration relative to what has been demonstrated/achieved in other initiatives around the world aimed at students achieving below curriculum expectations in literacy? [relative magnitude of shift]

It’s an interesting and challenging mix of ‘absolute’ and ‘relative’ considerations, and the notion of how good is good enough also depends on which subgroup of students one is referring to – all students, those who started school substantially below expectations, those who started well below, students in groups (e.g. based on ethnicity) that have historically performed well below their true potential, students with special needs, transient students, English language learners, …

It’s not cut-and-dried or straightforward by any means, but does that mean we can’t say anything? Not at all. I still believe it’s possible – and important – to at least get approximate answers to the important question of how good/valuable/worthwhile/substantial key outcomes are. AND to make transparent the logic and methodology used to draw those conclusions.

One tool I have found incredibly useful for this is evaluative rubrics, which work particularly well in participatory evaluation mode, but are just as useful for more independent evaluations (and those that are somewhere in between). More on those in a future post – and, for those of you coming to AEA in San Antonio next month, I’ll be covering those in a preconference workshop as well.


5 comments to How good is a “good” outcome?

  • Ricardo Wilson-Grau

    I do not disagree with your three key considerations but feel they are a bit too limited, at least when seen from a perspective outside of the literacy achievement examples that you give. Specifically, I believe the relative importance of the outcome as a part of a process of change must be considered.

    I work with organisations–especially international networks–that are devoted to social change. Thus, I use a very specific, generic definition of outcome (adapted from IDRC’s Outcome Mapping methodology): an outcome is a significant change in the behaviour, relationships, actions, policies or practices of individuals, groups, organisations or institutions that have been influenced by the social change initiative. The rationale is that in essence social change involves social actors doing things differently than they have been up until now; they are the subjects and the protagonists of change.

    Furthermore, social change is not the result of an accumulation of more positive than negative changes in social actors but of the articulation or synergy generated between a diversity of outcomes that enhance each other and accompanying activities and outputs, weighing in the direction of the desired social change. An outcome then, however big or small, positive or negative, is one of many results that together constitute a process of social change and the outcome can be considered good or bad depending, in part, on whether it contributes to or detracts from that process.

    From that perspective, I find missing in your three key considerations the notion of an outcome’s (positive or negative) ROLE IN A PROCESS OF SOCIAL CHANGE. That is, a fourth consideration would be the extent to which the outcome is part of a process of change that led, leads or potentially could lead to (or detract from) noticeable impact on people’s lives. Similarly, in addition to the outcome’s absolute or relative intrinsic importance for eliminating the most important sources of disadvantage or dysfunction, I believe equally important is the outcome’s role in strengthening (or undermining) other activities, outputs and outcomes in the process of social change of which it is a part.

    Said another way, an outcome that does not apparently lead to significant impact, or is not especially noteworthy in its direct effect on eliminating disadvantage or dysfunction or when compared to other outcomes, could in fact be an very important part of a process of social change, whether as a catalyst or simply another grain of sand that helps tip the balance.

  • I with Ricardo on this one. A comment in Saville Kushner’s “Personalising Evaluation” has had a lasting impression on me. Paraphrasing somewhat the comment said that most evaluators had their telescope the wrong way around. Instead of training it on the impact of a program on some stakeholder, evaluation should turn it the other way around and ask the question from the stakeholder’s point of view. How relevant has the program been to their life? So what (in this case) that the music program was wildly successful to the eight year old kid, did it prevent him getting bad food and beaten up in the playground after the music lesson …did it matter to him in the grand scheme of things.

    One of the most important lessons I’ve learned when seeking to apply systems ideas to evaluation is that the idea of a “program” is just that, an idea, a construction…. a perspective with a boundary around it. To those who’ve constructed it, that is their framing and inevitably the focal point of their interest. But to others it is just another set of activities and ideas alongside many more. By and large I think those working in the international development field (like Ricardo) understand this more clearly and have worked through the implications more extensively than evaluators working in other spheres.

  • Jane Davidson

    Ricardo and Bob, thanks so much for your thoughtful comments!

    And YES, I totally agree with you that a thorough piece of evaluative work looks at the role/contribution of a particular program impact from the perspective of the impactee and as one part in the mix of things that comes together to supposedly create social/life change. I’m going to incubate that thought for a while to try and work out how the nuts and bolts of such analysis might be translated in a way that can help us visualise how it’s done in practice …

    The reality for me in this particular project, though, is a bit different. Just as “programs” are constructs with boundaries set around them, so too do “evaluations” get subjected to the same kind of restrictive boundary-related limitations imposed by those who fund them. Those boundaries exist around funder-defined scope, to some extent, but the really crucial ones are time and budget (because they limit how much scope can be haggled).

    In this case, we are having to work off student outcome data gathered (by the provider) as part of monitoring data. Just making sense of this chews up 110% of the very limited budget available. There is no possibility of actually getting into the 60 schools involved to talk directly to students or teachers and get a sense of how the effects of this initiative contribute to any greater change (if any) in people’s lives.

    So, it’s far from ideal, and perhaps a cleverer evaluator would have turned down something like this in favor of something better funded. But I must confess I am somewhat drawn to the challenge of something that’s clearly ridiculously underfunded to see whether the small slice of what we can do might at least provide a useful evaluative interpretation of the outcome data, as opposed to the usual mind-numbing summary of t-tests, ANOVAs and post-hoc comparisons. This piece of work could barely be called “an evaluation”, but rather “an evaluative interpretation” of the outcome data gathered as part of program monitoring.

    Boundary limitations of this particular piece of work aside, I do agree that the interpretation of the value of outcomes does need some expansion. We’ll need to include that notion of not just considering a particular outcome on its own isolated merits, but what its contribution is to the mix – which, as you have both pointed out, may be more or less than first appears.

    A seemingly substantial change might simply dissipate because of the other forces and realities swirling around it – or, even if sustained over time, it might be meaningless in the context of other more pressing realities that prevent the greater social change from happening (e.g. literacy might improve, but kids might still keep dropping out of school too young to get the intended advantage of being able to better access the curriculum and gain better school qualifications).

    Similarly, a seemingly trivial improvement (e.g. small increase in literacy skills) may get them over an important ‘hump’. It might be the one small piece of the jigsaw that has been holding some kids back from a total change in perspective about how doable academic success is for them and where life might lead them.

    This is really useful stuff – thanks!

  • Actually there are two other major boundary issues that evaluation confronts. Other than “purpose” and “resources”, the there is “expertise” (and what actually constitutes knowledge) and “legitimacy” (whose views do we seek and thus legitimate, whose do we not seek and thus marginalise). All this stems from critical systems theory. It is a pity that the concepts underpinning criticial systems have largely failed to gain traction within the evaluation community since they are really a theory of valuing; something the evaluation cannon badly needs.

    Oh and for the record he’s what Saville said in his book :

    “What might program evaluation look like and involve were we to invert the conventional relationship between individual and program – that is, rather than document the program and “read” the lives of individuals in that context; to document the lives and work of people and to use that as context within which to “read” the significance and meaning of programs?”. Better and more succinct than my bumbling effort at paraphrasing.

  • Jane Davidson

    Thanks for posting that quote, Bob. It’s a hugely important insight of Saville’s and IMHO something of a Copernican revolution in its own right.

    Totally agree with your point about the boundaries around “expertise”. This happens not just within evaluation but also at the evaluation commissioning stage, where clients convey insights into what they value/seek in the way of expertise for different kinds of work, whether it’s content expertise, academic research expertise, practitioner experience, top management credibility/expertise, etc.

    Interesting stuff!