There’s a great discussion going on right now on the AEA Thought Leaders’ Forum. This week it’s being led by Jean King, who has raised the question of credentialing for evaluators.

Not all our subscribers are AEA members and following this forum, so I’m just cross-posting a revised and expanded version of a contribution – and encourage you all to check out the wider discussion!

The problem of competency ‘laundry lists’

One problem with the various lists of evaluation competencies we see around is that they cover an enormous range of the skills that evaluators have and use in our work, but FAR MORE than any one evaluator (or even one evaluation team) could or even should have.

This leads people to think that:

• “competent” = “can demonstrate every single one of the competencies”
• “missing a few” = “incompetent”

… and of course, because no-one has the full repertoire, even top-notch evaluators will be looking at the list and saying “What?! You’re calling me incompetent because I can’t [insert skill]?”

It seems to me that we need to distinguish between:

1. “the core” – the absolutely essential stuff that you really must have if you are to call yourself an evaluator
2. “specialized competencies” – the specific methodologies, content areas, and other specialties that you choose to be particularly strong in

Defining ourselves professionally

I think we need to do this at two levels:

1. defining ourselves as a profession (by defining “the core”)
2. defining ourselves as individual evaluators, evaluation teams, or evaluation units or businesses (by defining our specialized competencies and approaches – which must include the core)

Defining “the core” of our profession

I think we all agree that there are people who pedal evaluation services who basically have no idea of the difference between evaluation and, say, measurement, or descriptive research.

They are generally not aware that there are degrees or certificates in evaluation or professional associations for evaluators – and if they were aware, they probably wouldn’t opt in anyway because they don’t believe there’s anything unique about evaluation, nothing worth talking about, puzzling over, improving on.

So, what is that “core”?

In various discussions I’ve had with colleagues about this, somehow we keep coming back to one thing as being the fundamental difference, the core of what distinguishes evaluation (done right) from other work, and that is the values and ‘valuing’ piece:

• We ask questions about how good/worthwhile/valuable/important things like design, implementation, and outcomes are;
• We actually have a shot at answering those questions (not just free associating to them with whatever data seems vaguely relevant)

In the New Zealand context, we have strong agreement that cultural values are absolutely central to this – how we define what’s good/worthwhile/valuable/important (both the process of doing this and what ends up in the criteria, plus how we evidence it).

The recent NDE (#133), edited by George Julnes, is a fantastic resource for thinking really seriously about how we as evaluators judge value in evaluation. It’s a must read!

Defining “who we are” as evaluation practitioners

Every individual evaluator and every evaluation consultancy/business/contracting unit needs to be clear about “who they are” as evaluators – what is it that distinguishes their practice or approach from that of others working in this space?

It’s impossible for any individual or even any evaluation team or consultancy to be all things to all people – and it is dishonest to imply that we are.

So, who are you? What are you particularly good at? What defines your approach? And, importantly, what are you NOT strong in? What kind of work do you steer clear of?

It is up to each evaluator (and each evaluation unit/business/consultancy) to define the profile of competencies they want and need to develop in order to work effectively in the space they have carved out for themselves.

YES, that means it’s perfectly OK to position yourself as (for example) someone who does highly collaborative evaluation, works primarily with qualitative evidence, works in the United States, in English-speaking communities of color, on programs related to addiction and homelessness – so long as you are doing that core evaluative activity of asking and answering evaluative questions – like how good the program design is, how well it’s been targeted and implemented, how valuable the outcomes have been so far, and so forth.

If this were you, you’d likely turn down work that involved heavy number crunching or non-English speaking participants or a requirement for a very independent style of evaluation.

It doesn’t make you any less of an evaluator if you have specialized in a particular approach, context, or content area; it simply means you are focusing on getting really good in that space.

And nor is the generalist evaluator any less competent for choosing to practice across a range of domains, drawing on others’ expertise as required.

Credentialing – who is ‘in’? Who gets sidelined?

Credentialing (if we need it – and the answer to this varies depending on where you live and work – see Michael Scriven’s post on the Thought Leader Forum discussion) has the potential to wrongly include or exclude people.

It also has the potential to appropriately include and exclude.

Here’s my take on inclusion/exclusion:

• We will inappropriately exclude if we define the “must have” competencies more widely than what really genuinely is at the core of evaluation. [Or if we use a long list of competencies and assume they are all required to do any decent evaluation.]
• We will inappropriately include if we say there is no core, or if we define it wrongly (e.g. as measurement or monitoring or applied research or providing information for decision making or …).

It’s always important to consider carefully who wins and who loses when any particular credentialing system is initiated – and whether one is needed at all.

We’ve had this discussion in New Zealand and decided no, we don’t need or want credentialing at this point. Instead, we are opting for:

1. A list of competencies that practitioners can use to self-assess, reflect, and plan their professional development
2. Professional development aligned with the needs most lacking and desired by professional association members
3. Efforts to build the capability of clients so they become more effective evaluation scopers, purchasers, project managers, utilization advocates, and (in some cases) collaborators

Related posts and references

• AEA Thought Leaders’ Forum (April 2012 – Jean King leading discussion on credentialing)
• Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation (NDE #133)
• Lifting the quality of evaluation #2: Capable evaluators who know their ‘space’

How can we improve the quality of evaluations – especially evaluations that are done by external evaluators of evaluation teams?

Improving the procurement process is an important part of this, as Jane has recently discussed (9 hot tips for commissioning, managing (and doing!) actionable evaluation). As part of the BetterEvaluation project, I’ve been bringing together resources on improving the Terms of Reference for evaluation, since this is where a lot of evaluations go badly wrong, with unreasonable timelines, vague and contradictory requirements, and unrealistic expectations about the numbers and types of questions that an evaluation can answer.

But there are other strategies that need to be used.  One of these has recently been suggested by Michael Scriven, evaluator, author and GenuineEvaluation guest blogger, whose earlier thinking has given us such core concepts as formative and summative evaluation and meta-evaluation (the evaluation of evaluation) (Scriven, M. (1967). ‘The methodology of evaluation’. In R.W. Tyler, R.W. Gagne & M. Scriven (Eds.), American Educational Research Association Monograph Series on Curriculum Evaluation, Vol .1: Perspectives of Curriculum Evaluation. Rand McNally).

On his website michaelscriven.info, he has posted a paper on ‘The Evaluation Advisor: A New Role for Evaluators?’ which sets out what the role of an evaluation advisor might involve and why it is important:

The evaluation advisor … is a person who serves as a helper or guide about evaluation, but not as an evaluator, for an individual or organization or program that is being, or is about to be, evaluated, or is considering sponsoring external or internal evaluation of themselves

He argues that this role needs to operate under a very strict written NDA (non-disclosure agreement):

The NDA means that they can discuss, with the prospective evaluees, the nature and costs and benefits of evaluations of various types, and how to do it or get it done, or survive it or benefit from it, or avoid it, or cheat on it, but cannot discuss anything about that discussion with anyone else, including in particular the agency that is funding the evaluee and wants or requires it to be done.

Some approaches to evaluation include this role as part of what the evaluator does.  For example, I’ve been reading  recently Michael Patton’s new book Essentials of Utilization-Focused Evaluation, which discusses how the evaluator can and should be involved in these discussions with the intended users of an evaluation.

Michael Scriven’s proposal is for these two roles to be separated, to avoid any possible conflict of interest:

the NDA should, usually, also preclude the EA from actually taking on the job of evaluating the organizations s/he advises. This is to prevent a possible conflict of interest in the EA between giving good advice and selling his/her services. The funding agency may or may not want to allow petitions for an exemption from this requirement in special cases.

He advocates that funding agencies should fund this role, as part of improving the quality of the data gathered in evaluations, and to improve the use made of evaluations by funded projects.

A slightly different version of this role has been used by the NGO Pact in South Africa, which has been working with Community Service Organizations funded by USAID both in group workshops and individual Technical Assistance to increase their understanding of evaluation and have input into the planning for independent external evaluations of their projects.   A presentation by Rita Sonko-Najjemba, Ana Coghlan, Addis Berhanu, and Priscilla Ngwenya at the 2011 American Evaluation Association conference on their work has been uploaded to the AEA e-library.

High quality, worthwhile, actionable evaluation doesn’t just depend on the technical competence and effective consultation skills of the evaluator.

Decisions made and actions taken (or not taken) by the client can make or break the value of evaluation for an organization.

High-value evaluation is the product of a fruitful interaction between a well-informed client and a responsive, appropriately skilled evaluation team.

What have we – both clients and evaluators – learned from both stunningly high value evaluative work (“dream projects”) and bitter disappointments (a.k.a. “Nightmares on Eval Street”)?

And, how can evaluators help clients get maximum utilization and value for their evaluation dollar by becoming informed, demanding, savvy consumers of (and partners in) evaluation?

There is much that the client can do to get evaluation on the right track at the outset, in the commissioning (e.g. RfP) phase.

The same is true when it comes to working alongside the evaluation team to ensure the client organization’s needs are met, and that the insights and learnings from evaluation are both influential and used. [This point holds whether the evaluation is participatory or independent.]

There is a LOT to say on this topic, so I’m giving just a taster of ideas here. Over the next few weeks and months I’ll be picking up on each of these – and more – and fleshing them out a bit more and expanding on some earlier material.

6 hot tips for evaluation clients:

1. Save time and energy with a two-phase process: “Expression of Interest” (EoIs) to shortlist qualified bidders followed by a full RFP to select the right team
2. Select contractors based on capabilities, not an evaluation plan they have written ‘blind’ (i.e. without speaking with stakeholders)
3. Use our list of great questions that cut to the chase and reveal the evaluation team’s approach, capabilities, and fit with the project
4. Ask for a ‘skeleton report’ before any data are collected so you can discuss what you need as an end product
5. Consider not requesting a long final report at all – they are seldom read; often, presentations and short written updates at several time points are more useful and usable
6. Have a contingency plan for ‘pulling the plug’

3 hot tips for evaluation contractors:

1. Help the client figure out whether or not you are the right fit by getting really clear about your brand – “who you are” as an evaluation practitioner or organization. What distinguishes you from the rest of the pack? What are your core values? Your “signature skill sets”? Your unique value proposition?
2. Put serious time and effort in at the front end to ensure you clearly understand the client’s needs, including any decision timelines (e.g. MQP’s point that summative decisions are often made 18 months ahead of time, but usually evaluation findings available then have only been designed with formative uses in mind)
3. Develop an overarching set of big-picture evaluation questions to guide the work – and talk through with the client what a real answer would look like, to them

What would you add to this list? Fire away in the comments section below!

It’s time for a radical rethink of the RFP process and the usual approach to evaluation project management – watch this space!

Related posts:

• 9 golden rules for commissioning a waste-of-money evaluation
• Lifting the quality of evaluation #1: Savvy clients
• Extreme Genuine Evaluation Makeovers (XGEMs) for Commissioning
• Commissioning XGEMs – the sequel
• The Friday Funny: The Government Contracting Glossary
• Who’s responsible for un-genuine evaluation?

Two things that make or break the quality and value of evaluation are:

1. Evaluation-savvy clients
2. Capable evaluators who know their ‘space’

What’s #3?

#3: Professional evaluation associations with focus!

The vast majority of professional evaluation associations around the world are relatively new, having formed only in the past decade or less.

Like New Zealand’s association, anzea, many are thinking hard about what kinds of services and activities will make the greatest contribution to evaluation’s quality and value for money.

What we most urgently need are professional associations that:

1. have a clear grip on what the key capability needs are in their country/region and – more importantly – what the most pressing priorities are, for both evaluators and clients/managers who commission evaluation
2. make professional development a priority by facilitating, brokering, and/or actively encouraging courses, workshops, mentoring, and other learning opportunities to meet those needs
3. communicate effectively with ‘the market’ (clients and other key audiences) about:
   • what evaluation is (not just ‘evaluations‘ but also other forms of ‘evaluative work’)
   • how it differs from its cousins (research, auditing, monitoring, quality assurance, etc)
   • how to spot the difference between someone who knows their stuff and someone who lists evaluation in their repertoire but shouldn’t

Evaluation associations should definitely steer clear of policing the quality of evaluation work.

The reasons are numerous and have been discussed at length elsewhere, but the Big One in my view is the risk and expense of the association being sued by an evaluator who had lost work and/or livelihood after a negative judgment.

Instead, the best avenues for professional associations to drive up the quality and value of evaluation are:

1. informing, educating, and empowering current and potential clients to invest in the right kinds of evaluation for their needs and to manage that investment effectively to get the absolute most out of it
2. providing information, tools and resources, learning opportunities and support for evaluators to:
   • understand the client perspective on where evaluation work frequently falls short
   • assess and build their own capabilities, particularly in those areas that make or break the quality and value of evaluation
   • access good professional development and meta-evaluative reviews of their work

What key elements are needed to seriously drive up how well evaluation delivers quality and value for money?

Yesterday I talked about how evaluation-savvy clients can make or break the value of evaluation. Now for the other side of the equation …

What evaluator capabilities still seem to be lacking and could really drive up the value of what we do?

I believe we urgently need evaluators to:

1. clearly understand the client’s perspective – particularly where it is that evaluation contractors are currently falling short and seriously need to step up and deliver high quality, valuable work
2. build their strengths and capabilities in order to deliver for clients (this requires hard-nosed and genuinely reflective self-assessment of their own competencies, as well as genuine use of meta-evaluative feedback from the client and other well-informed critics)
3. clearly define “who they are” as evaluators, i.e. what their unique value proposition is
4. complement their formal learning and professional development with ways of gleaning the ‘tacit knowledge’ (or evaluation know-how) we can only get by observing, watching, and working alongside those who have it.

It’s particularly worth spending a little extra time thinking carefully about point #3:

Defining who we are as [individual and teams of] evaluators

In my view, far too many evaluators pitch (and view) themselves as all-rounders or capable of taking on any piece of work.

Content specialization aside (most people ‘get’ that), what is it about one evaluation team that distinguishes it from another and makes it stronger in some spaces but – more importantly – weaker in others)?

Part of the problem resides in those who choose a particular approach as the “one true way”. These are the evaluators who, when asked, are incapable of clearly articulating the circumstances under which they would not choose their preferred approach. The narrower the approach, the more horrified we should be at the lack of a coherent answer to the “when wouldn’t you use it?” question.

But even those who view themselves as relative evaluation generalists need to understand that every individual and team is strong in some suits but not in others. Otherwise, do you even have a professional identity at all?

The reality is that the knowledge bases and skill sets that make up the rich discipline of evaluation are so wide and deep that no single individual or team can possibly peg off the full range of competencies. So, to pitch oneself as the “all things to all people” dream team amounts to either deliberate misrepresentation or an ignorance of just how huge the evaluation toolkit and knowledge base has really become.

Until we (as teams and individuals) are clear about what is distinctive about the way we work and what we are good at – and the spaces we don’t (and shouldn’t) work in, we aren’t truly clear about who we are as evaluators.

Of course, one thing we all need to be unambiguously clear about if what distinguishes our work from researchers, auditors, and those who specialize in monitoring but not evaluation. In particular, we must be crystal clear about why genuine evaluation must be value-based.

Implications for professional development of individuals and teams

Each individual evaluator, evaluation team, and evaluation consulting business should be thinking hard about how they wish to build their professional repertoire so they have a strong and [relatively] unique value proposition.

This will allows them to clearly communicate to clients (and prospective employers) what it is that distinguishes their work from the other masses vying for the same contract (or job).

When clients get good at asking about this – and when evaluators get good at answering the question – we will finally start making useful strides toward ensuring better fit between evaluation projects and evaluation teams.

Related posts:

• Lifting the quality of evaluation #1: Savvy clients
• Why genuine evaluation must be value-based
• What is Genuine Evaluation?