There’s a great discussion going on right now on the AEA Thought Leaders’ Forum. This week it’s being led by Jean King, who has raised the question of credentialing for evaluators.

Not all our subscribers are AEA members and following this forum, so I’m just cross-posting a revised and expanded version of a contribution – and encourage you all to check out the wider discussion!

The problem of competency ‘laundry lists’

One problem with the various lists of evaluation competencies we see around is that they cover an enormous range of the skills that evaluators have and use in our work, but FAR MORE than any one evaluator (or even one evaluation team) could or even should have.

This leads people to think that:

• “competent” = “can demonstrate every single one of the competencies”
• “missing a few” = “incompetent”

… and of course, because no-one has the full repertoire, even top-notch evaluators will be looking at the list and saying “What?! You’re calling me incompetent because I can’t [insert skill]?”

It seems to me that we need to distinguish between:

1. “the core” – the absolutely essential stuff that you really must have if you are to call yourself an evaluator
2. “specialized competencies” – the specific methodologies, content areas, and other specialties that you choose to be particularly strong in

Defining ourselves professionally

I think we need to do this at two levels:

1. defining ourselves as a profession (by defining “the core”)
2. defining ourselves as individual evaluators, evaluation teams, or evaluation units or businesses (by defining our specialized competencies and approaches – which must include the core)

Defining “the core” of our profession

I think we all agree that there are people who pedal evaluation services who basically have no idea of the difference between evaluation and, say, measurement, or descriptive research.

They are generally not aware that there are degrees or certificates in evaluation or professional associations for evaluators – and if they were aware, they probably wouldn’t opt in anyway because they don’t believe there’s anything unique about evaluation, nothing worth talking about, puzzling over, improving on.

So, what is that “core”?

In various discussions I’ve had with colleagues about this, somehow we keep coming back to one thing as being the fundamental difference, the core of what distinguishes evaluation (done right) from other work, and that is the values and ‘valuing’ piece:

• We ask questions about how good/worthwhile/valuable/important things like design, implementation, and outcomes are;
• We actually have a shot at answering those questions (not just free associating to them with whatever data seems vaguely relevant)

In the New Zealand context, we have strong agreement that cultural values are absolutely central to this – how we define what’s good/worthwhile/valuable/important (both the process of doing this and what ends up in the criteria, plus how we evidence it).

The recent NDE (#133), edited by George Julnes, is a fantastic resource for thinking really seriously about how we as evaluators judge value in evaluation. It’s a must read!

Defining “who we are” as evaluation practitioners

Every individual evaluator and every evaluation consultancy/business/contracting unit needs to be clear about “who they are” as evaluators – what is it that distinguishes their practice or approach from that of others working in this space?

It’s impossible for any individual or even any evaluation team or consultancy to be all things to all people – and it is dishonest to imply that we are.

So, who are you? What are you particularly good at? What defines your approach? And, importantly, what are you NOT strong in? What kind of work do you steer clear of?

It is up to each evaluator (and each evaluation unit/business/consultancy) to define the profile of competencies they want and need to develop in order to work effectively in the space they have carved out for themselves.

YES, that means it’s perfectly OK to position yourself as (for example) someone who does highly collaborative evaluation, works primarily with qualitative evidence, works in the United States, in English-speaking communities of color, on programs related to addiction and homelessness – so long as you are doing that core evaluative activity of asking and answering evaluative questions – like how good the program design is, how well it’s been targeted and implemented, how valuable the outcomes have been so far, and so forth.

If this were you, you’d likely turn down work that involved heavy number crunching or non-English speaking participants or a requirement for a very independent style of evaluation.

It doesn’t make you any less of an evaluator if you have specialized in a particular approach, context, or content area; it simply means you are focusing on getting really good in that space.

And nor is the generalist evaluator any less competent for choosing to practice across a range of domains, drawing on others’ expertise as required.

Credentialing – who is ‘in’? Who gets sidelined?

Credentialing (if we need it – and the answer to this varies depending on where you live and work – see Michael Scriven’s post on the Thought Leader Forum discussion) has the potential to wrongly include or exclude people.

It also has the potential to appropriately include and exclude.

Here’s my take on inclusion/exclusion:

• We will inappropriately exclude if we define the “must have” competencies more widely than what really genuinely is at the core of evaluation. [Or if we use a long list of competencies and assume they are all required to do any decent evaluation.]
• We will inappropriately include if we say there is no core, or if we define it wrongly (e.g. as measurement or monitoring or applied research or providing information for decision making or …).

It’s always important to consider carefully who wins and who loses when any particular credentialing system is initiated – and whether one is needed at all.

We’ve had this discussion in New Zealand and decided no, we don’t need or want credentialing at this point. Instead, we are opting for:

1. A list of competencies that practitioners can use to self-assess, reflect, and plan their professional development
2. Professional development aligned with the needs most lacking and desired by professional association members
3. Efforts to build the capability of clients so they become more effective evaluation scopers, purchasers, project managers, utilization advocates, and (in some cases) collaborators

Related posts and references

• AEA Thought Leaders’ Forum (April 2012 – Jean King leading discussion on credentialing)
• Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation (NDE #133)
• Lifting the quality of evaluation #2: Capable evaluators who know their ‘space’

I’ve been doing some thinking about the various ways in which evaluation can, does and should have influence, in preparation for the forthcoming Australasian Evaluation Society conference on this topic.

That useful, modern resource, The Urban Dictionary, which has prompted some of our thinking about genuine evaluation and inspired our GE badge and jingle, has alerted me to a potentially useful concept for evaluation, created by ChristinaM33 Sep 22, 2008:

heads-uppiness
	1. the quality of communication in which one gives a warning of impending action without necessarily giving all the details. “I just wanted to voice my opinion, and support for better communication/inclusion/ heads-uppiness.”

Does this have relevance for evaluation, which tends to be looking backwards, or is it more for other types of activity which are focused on looking forward? Is it relevant for the “Now What” element of evaluation only? 

You’ve probably noticed some recent changes in the blog.  Not so much planned as responding to some recurring glitches that brought the blog down several times.

We are gradually rebuilding the site, adding back in links to previous posts and comments and to other sites.

We’ve also taken the opportunity to experiment with some different visuals.  The picture currently in the header is an Australasian hybrid – a New Zealand scene (just outside Queenstown) taken by an Australian.

We will also be revising the tags and categories, drawing on the discussions we’ve had over the past 12 months about the different aspects of genuine evaluation and recurrent issues.

But not until we meet some pressing commitments, including a piece we’re jointly writing about Australasian evaluation theory and theorists.

So any comments or suggestions on structure and style are welcome – either as comments to this post or emails to us directly.

I’m in Washington DC this week, after teaching a course on Using Program Theory and Logic Models for Evaluation at The Evaluators Institute.

Fortunately I will be able to stay on for a seminar being presented this coming Wednesday by The Evaluators’ Institute and The Washington Evaluators on

“StrengtheningEvaluation Effectiveness”

Wednesday, January 19th, 6:00pm to 7:30pm
Jack Morton Auditorium, Media and Public Affairs building, 1st floor
805 21stSt NW Washington DC 20052

I wonder if the session will consider strengthening evaluation effectiveness in terms of:

• increasing the utility of evaluation for informing evidence-based policy and practice, especially in terms of assisting policymakers and practitioners to translate findings to new situations
• increasing the attention paid to heterogeneous outcomes (rather than simply the mean net effect), and in particular to who benefits and who loses from programs and policies with heterogeneous outcomes
• increasing the attention paid to unintended outcomes (positive and negative) and to the costs incurred beyond the formal resources expended for implementation, and including the resources contributed by other stakeholders, including other organizations and clients
• improving the use of credible non-experimental approaches such as process tracing in situations where experimental or quasi-experimental designs are not appropriate or feasible
• increasing the ability of managers to commission and manage external evaluators to ensure high quality work is done
• increasing the ability of managers and service deliverers to evaluate their own programs and to make use of the evaluations of other programs

Stay tuned for a report after the session – or see you there.
Speakers include:
Gary T. Henry, Duncan MacRae’09 and Rebecca Kyle MacRaeProfessorship Public Policy
Department of Public Policy, Director, Carolina Institute for Public Policy at the University of North Carolina at Chapel Hill
Harry P. Hatry, Distinguished Fellow and Director of the Public Management Program, UrbanInstitute, Washington, DC
Kathryn E. Newcomer, Director, The Trachtenberg School of Public Policy and Public Administration, George Washington University
Theodore H. Poister, Professor of Public Management and Policy, Andrew Young School of Policy Studies at Georgia State University

Moderator – Ann M. Doucette, Director, The Evaluators’ Institute and Midge Smith Center for Evaluation Effectiveness

Reception to follow.Please RSVP (reception) to tei@gwu.edu

The “no value-free” line of the Genuine Evaluation song (composed by the incomparable Kataraina Pipi, evaluator and composer/musician, with input from several other genuine evaluators!) was inspired by an earlier post where we defined genuine evaluation and drew some lines in the sand about what was in and what was out.

One of the things that is definitely out is so-called ‘value-free’ evaluation.

What are ‘value-free’ evaluations?

These are “evaluations” (not!) that avoid the all-important task of answering any questions about how good, valuable, worthwhile, or important the outcomes, the design, the implementation, and/or the entire evaluand (program, policy, product, service, etc) are.

Value-free evaluations simply present a Rorschach inkblot of evidence, complete with various descriptive analyses, such as key themes, most prevalent responses, statistical significance of differences, graphs and illustrations of changes – BUT they stop short of saying whether these findings represent worthwhile outcomes or quality programming.

Why the ‘values’ bit is part of our job description

I’ve mentioned in an earlier post (Why genuine evaluation must be value-based) that the word eVALUation doesn’t have the word ‘value’ in it for nothing!

The whole point of evaluation is to ask and answer questions about quality, value and importance – which clients and other readers/listeners can USE in some way (e.g. to take action, to inform decision making, to inform future design or programming, etc).

If we fail to do this we are, in my view, shirking our responsibilities as evaluators. The message to the client is “You work it out.”

‘Value-free’ is often a mix of ‘value-undiscussable’ and ‘value-poor’

There is an important reality that those who commission, use, or produce ‘value-free’ work should bear in mind:

Values ARE in fact being applied in that work; they are just not being made explicit (that’s the ‘value-undiscussable’ bit) – and they don’t go far enough (the evaluation is ‘value-poor’).

Here’s what I mean by value-undiscussable:

1. When we select the outcomes to be explored as part of an evaluation, we are implicitly saying that “these outcomes are potentially valuable” (the ones we decide to include) and “these outcomes are not” (the outcomes we decide not to bother exploring).
2. When we are not clear and explicit about how and why the outcomes were selected for exploration and why others weren’t, we are making those value-based decisions undiscussable and [deliberately?] not open for question or challenge.

Here’s what I mean by value-poor:

3. If we go to the bother of [implicitly or explicitly] declaring an outcome ‘potentially valuable’ (and this includes outcomes that are potentially detrimental – they have negative value), then why not take the next step and say something about just HOW valuable the actual outcome IS when the evidence is in. If we don’t bother with that part, the evaluation is value-poor; it uses only a weak application of values, excluding the most useful bit.

Does this basically mean that I have to be clear about “my personal values” as an evaluator and as a person?

That’s one useful place to start, and good practice in general, BUT I am not just talking about “personal values” here.

I am talking about how we define quality and value within a professional evaluation context.

I don’t mean “Which outcomes do I value?”

I don’t even mean “Which outcomes are valued by stakeholders?”

I mean “Which outcomes are demonstrably valuable in this context and for these recipients/impactees?”

The key here is that professional evaluation needs to make a case for which outcomes are “demonstrably valuable” in a particular context by going a lot further than just taking opinions or preferences out of their own heads or even the heads of stakeholders.

I’m not saying don’t ask stakeholders – that is almost always an extremely important source of information. But I am saying don’t stop there.

Outcomes are valuable, important, and worth achieving not just because somebody consciously desires or values them, but because they add value to people’s lives by helping them realize their potential, meeting a need, fulfilling an aspiration, making them healthier or more successful in work or business, etc.

The keys to evaluative transparency – visible values – are

1. Defining things like “valuable outcomes”, “quality programming and delivery”, and/or “good value for money/time/effort” based on concrete evidence from a range of sources, not just by plucking opinions from people’s heads and saying that’ll do.
2. This includes:

• • identifying the criteria (what we will look at) AND
  • the definitions of quality or value on those criteria (how we will interpret the evidence) AND
  • how the evidence is interpreted against those criteria AND
  • how the successes and disappointments, pros and cons are weighed in drawing overall conclusions about the evaluand

Visible values make for better quality ‘valuing’

The clearer and more transparent the ‘values’ that underlie the ‘valuing’ (or, evaluative reasoning) done in evaluation, the easier they are to criticize. This may make many evaluators extremely nervous – it’s like putting yourself up for target practice.

HOWEVER, as evaluators we are all ‘selling’ the idea that criticism and feedback are important for the improvement of programs, policies, products, services, and so forth. Surely the same applies to evaluation itself.

The upside is that, if the values and the evaluative reasoning are clear, criticisms are more likely to be aimed at the specifics of the evaluative reasoning (or the evidence used) rather than at the evaluator personally. So, the conversation shifts from “Well, that’s just YOUR opinion” to “I believe you have set the bar too high here when you define ‘acceptable’ levels on these outcomes.”

It’s those kinds of evaluative conversations that can really engage intended users AND lead to better evaluative thinking on the part of the client as well as the evaluator.

Related posts and other references:

• Visible values: Striving for truth, beaty and justice in evaluation [Originally a presentation delivered as part of the AEA 2010 conference's presidential strand - alongside Rodney Hopson, Katrina Bledsoe, and Ernie House - this handout is from a revised version presented to the anzea Auckland branch last week.]
• Why genuine evaluation must be value-based (Jane Davidson)
• How good is a ‘good’ outcome? (Jane Davidson)