Valuing processes and distribution of costs and benefits as well as outcomes

As part of our exploration of the need for making explicit the values in evaluation, let’s be clear that this is not just about what outcomes are valued. Sometimes we are not just interested in whether we’ve reached a destination, we’re concerned about the path we’ve taken to reach it.

Pic from http://noimpactman.typepad.com

Pic from http://noimpactman.typepad.com

Notions of “what success looks like” very often include values about processes. For example, if you found that children’s vocabulary was improved by teachers hitting them with dictionaries, we might find it an effective and good program in terms of its outcomes, but unacceptable because of its processes (partly because there would be a risk of additional negative outcomes from these processes, such as injuries or normalizing physical violence). We might find a product very good in terms of its functionality and cost but still find it unacceptable because of the labor conditions under which it is produced.

The distribution of costs and benefits is also an important aspect of values that is rarely made explicit. When deciding scarce resources (such as who gets access to a service, or which services get funded) it more important to get the overall maximum benefit (in which case you might focus on cases where the resources can be leveraged with other resources, such as other funding, talent, and support), or to focus on the most needy (even if this means the overall results might not be as great), or to spread resources across different geographic areas (achieving greater equity of access)?

In Jane’s recent presentation on Visible Values: Striving for truth, beauty and justice at the Aotearoa/New Zealand Evaluation Association (Auckland branch), she drew on Ernie House’s work on values and set out three common options:

Utilitarian ethics

Maximising the average „good? across all individuals
One person’s loss is balanced by another’s gain

Intuitionist/pluralist ethics

Fundamental principles or criteria are weighed and balanced by whoever is engaged in doing the evaluation

Justice-as-fairness

Same basic rights for all, including self-respect
Social & economic inequities allowable onlyif they benefit the least advantaged in society

The distribution of costs and benefits is the most invisible value in evaluation. Much of the discussion about evidence-based policy focuses only on comparing estimates of the “mean net effect” of programs – which assumes that any negative effects on an outcome measure for some participants are worth the same as positive effects for other participants – that is, looking at the overall utility. If we take a different value position, we might prioritize not causing further harm to the most disadvantaged.

Gill Westhorp’s research into early years early intervention programs has highlighted how many programs do produce negative effects for some clients – a fact that is rarely reported or addressed when these programs are recommended in systems of evidence-based policy.

In work done by a subgroup of NONIE (the Network Of Networks for Impact Evaluation) as part of developing the Guidelines for Impact Evaluation, we put forward these as important issues:

Impact evaluation needs to assess the value of the results derived from an intervention. This is not only an empirical question but inherently about values –which impacts are judged as significant
(whether positive or negative), what types of processes are valued in themselves (either positive or
negative), and what and whose values are used to judge the distribution of costs and benefits of
interventions.

Comments are closed.