The real values behind ‘value-undiscussable’ evaluation

I posted yesterday about the importance of visible values in evaluation. This means being clear and transparent about the definitions of quality and value used when identifying criteria, evaluating performance against them based on evidence, and weighing up the pros and cons.

The perfect most of transport for the visible values evaluator - the transparent canoe! Click image to go to vendors site

The perfect mode of transport for the 'visible values' evaluator - the transparent canoe! Click image to go to vendor's site

Many evaluations, I argued, are NOT transparent about how quality and value are determined in an evaluation. Some even pretend that the entire piece of work if ‘value-free’ – which is partially true, but only part of the story.

Whose values are really embedded in ‘value-free’ evaluation?

Let’s reflect on values a little more broadly and consider what might really be going on when there is no explanation of how a particular outcome was deemed to be “potentially valuable” (and therefore worthy of inclusion) while others were not.

Who is it who walks through life without ever having to seriously explain their assumptions and reasoning? Who is it who takes exception to any questioning of how or why they made particular choices? Who is it who does a really weak job at explaining their assumptions because they don’t think they are making any? Who is it who thinks that their version of common sense is ‘the default’?

The majority. The so-called ‘mainstream’. That’s who.

Many (but not all) of us have spent some or all of our lives living as a minority in some part of the world where people just don’t make sense in the way we are used to – and people think we don’t make any sense either! I’ve done that twice for any length of time, four years in Japan and eight in the States. It’s a real eye-opener to the assumptions we carry around with us.

When you are in the majority, you seldom have to explain yourself. It’s the minorities, those who think or make sense in different ways, who are always having to justify their reasoning.

I am reminded of an earlier comment from my friend and colleague Kirimatao Paipa, who said:

Often building bridges between cultural understanding rests solely on the Maori shoulders. Translations of everything Maori is into English. Bi-lingualism aside, if this country was controlled by Maori and the boot was on the other foot, how and who would build those bridges from the Pakeha community?

Historically, it has always been the cultural and ethnic minorities who have been asked to explain themselves, while the cultural and ethnic majority’s view of the world is seen as ‘the default’ and not requiring any explanation.

The same is also true, when you think about it, when it comes to the methodological minorities of the research and evaluation universe – those who specialize in using qualitative data. This is not just because those raised in the qualitative traditions have had the importance of values and transparent reasoning drilled into them; it’s also because the ‘methodological majority’ (Team Quantoid) continually asks them to justify themselves as ‘proper scientists’.

And it’s also true that those who are disciplinary minorities in the evaluation scene (e.g. because they come out of horticulture, energy efficiency, or business rather than education or psychology) more often have to explain/justify their approaches and reasoning to those who are in the majority.

What?! You’re saying that all ‘majority’ evaluators do value-free or value-undiscussable work?

Not at all.

I’m saying that ALL good evaluation needs to be transparent about the values that are brought to bear when:

  • identifying outcomes (or dimensions of quality design, implementation, etc) to cover in an evaluation
  • saying what the evidence should/might look like if performance on those outcomes is “good” (as opposed to “mediocre” or “excellent”, for example)
  • interpreting the evidence to draw evaluative conclusions – not just about whether the entire program is worthwhile, but on ALL of the main criteria included in the evaluation.

I’m also saying that some evaluators have been more clear and explicit than others about how they do these (because they think it is important to make such things clear AND/OR because they are members of an ethnic/cultural, methodological or disciplinary minority within a particular context).

Transparency is challenging, but rewarding!

There is no denying that, when one is living as a minority, it does get tiring to have to continually explain where you’re coming from. Likewise, as an evaluator, it takes considerable effort to generate work that is explicitly and transparently value-based.

But there’s an upside.

As a minority living in a foreign culture, and even in visiting some of the lands of my ancestors (Scotland, England), I became much more clear about who I was and who I wasn’t as a cultural being on Planet Earth. And I became much more acutely aware of assumptions I never even knew I had, and taken the opportunity to refine or ditch the ones that turned out not to be well-based.

In evaluation, I have found that the discipline of explicitly explaining the evaluative parts – the basis on which particular criteria are defined as representing or having differing amounts of quality, value, or importance; how evidence is interpreted to draw evaluative conclusions; how the mix of varying outcomes is weighed; why one positive outcome compensates for another negative one (or not) – the quality of my reasoning improved dramatically.

Clarity of evaluative reasoning in a report (be it written or verbal) becomes not just a symptom of, but a means to achieving clarity of thinking. And as Michael Quinn Patton reminded us in the opening plenary at the AEA 2010 conference, ‘evaluation quality’ isn’t just about the rigor of the methods or technical quality; it’s about the quality of the thinking and the quality of the interactions that run through the evaluation itself.

Related posts and other references:

4 comments to The real values behind ‘value-undiscussable’ evaluation

  • Chad Green

    Great minds think alike, Jane. I came to the same conclusion a little while ago after studying the dynamics of interacting tridentities:

    You see, minorities already operate in a state of uncertainty, and it is this state of mind which creates the necessary conditions for creativity to flourish. Thus, minorities should be seen as one of society’s most precious resources given the pace of change in today’s conceptual age.

    But I do not see how this perspective alone will contribute to a revolution in our field and beyond.

    To attain this outcome, we need to return to the science of logic. For example, the logic model in my previous post was complete with exception to the most important process of all: justice. In hindsight, I think I now know what that process entails. But first, perhaps someone else would like to give it a go? :)

    Moein, what do you think could be the source of logical activity in the model?


  • Salaam to All,

    In today’s conceptual age, WHY NO Chad? Please continue you are most welcome.




  • Chad Green

    Moein, you will need to be more specific than that! :)

    I have an idea. Let’s create a science competition to determine who can provide the most coherent responses to the following challenge question: What is the primeval source of all logical activity? The goal of the competition is to complete the logic model described earlier by defining the missing process for justice. And by the word justice, I refer specifically to the science of logic and its most important construct as House (1980) similarly concluded in his book.

    By engaging in this exercise of transdisciplinary thinking, perhaps we can initiate a series of revolutions within ourselves that spiral inward intuitionistically to the point of creating a critical mass of values that we can then leverage to effect change within our field and beyond.


  • Salaam Chad

    Well said! But I need more time for more specific than you!