Two things that make or break the quality and value of evaluation are:

1. Evaluation-savvy clients
2. Capable evaluators who know their ‘space’

What’s #3?

#3: Professional evaluation associations with focus!

The vast majority of professional evaluation associations around the world are relatively new, having formed only in the past decade or less.

Like New Zealand’s association, anzea, many are thinking hard about what kinds of services and activities will make the greatest contribution to evaluation’s quality and value for money.

What we most urgently need are professional associations that:

1. have a clear grip on what the key capability needs are in their country/region and – more importantly – what the most pressing priorities are, for both evaluators and clients/managers who commission evaluation
2. make professional development a priority by facilitating, brokering, and/or actively encouraging courses, workshops, mentoring, and other learning opportunities to meet those needs
3. communicate effectively with ‘the market’ (clients and other key audiences) about:
   • what evaluation is (not just ‘evaluations‘ but also other forms of ‘evaluative work’)
   • how it differs from its cousins (research, auditing, monitoring, quality assurance, etc)
   • how to spot the difference between someone who knows their stuff and someone who lists evaluation in their repertoire but shouldn’t

Evaluation associations should definitely steer clear of policing the quality of evaluation work.

The reasons are numerous and have been discussed at length elsewhere, but the Big One in my view is the risk and expense of the association being sued by an evaluator who had lost work and/or livelihood after a negative judgment.

Instead, the best avenues for professional associations to drive up the quality and value of evaluation are:

1. informing, educating, and empowering current and potential clients to invest in the right kinds of evaluation for their needs and to manage that investment effectively to get the absolute most out of it
2. providing information, tools and resources, learning opportunities and support for evaluators to:
   • understand the client perspective on where evaluation work frequently falls short
   • assess and build their own capabilities, particularly in those areas that make or break the quality and value of evaluation
   • access good professional development and meta-evaluative reviews of their work

What key elements are needed to seriously drive up how well evaluation delivers quality and value for money?

Yesterday I talked about how evaluation-savvy clients can make or break the value of evaluation. Now for the other side of the equation …

What evaluator capabilities still seem to be lacking and could really drive up the value of what we do?

I believe we urgently need evaluators to:

1. clearly understand the client’s perspective – particularly where it is that evaluation contractors are currently falling short and seriously need to step up and deliver high quality, valuable work
2. build their strengths and capabilities in order to deliver for clients (this requires hard-nosed and genuinely reflective self-assessment of their own competencies, as well as genuine use of meta-evaluative feedback from the client and other well-informed critics)
3. clearly define “who they are” as evaluators, i.e. what their unique value proposition is
4. complement their formal learning and professional development with ways of gleaning the ‘tacit knowledge’ (or evaluation know-how) we can only get by observing, watching, and working alongside those who have it.

It’s particularly worth spending a little extra time thinking carefully about point #3:

Defining who we are as [individual and teams of] evaluators

In my view, far too many evaluators pitch (and view) themselves as all-rounders or capable of taking on any piece of work.

Content specialization aside (most people ‘get’ that), what is it about one evaluation team that distinguishes it from another and makes it stronger in some spaces but – more importantly – weaker in others)?

Part of the problem resides in those who choose a particular approach as the “one true way”. These are the evaluators who, when asked, are incapable of clearly articulating the circumstances under which they would not choose their preferred approach. The narrower the approach, the more horrified we should be at the lack of a coherent answer to the “when wouldn’t you use it?” question.

But even those who view themselves as relative evaluation generalists need to understand that every individual and team is strong in some suits but not in others. Otherwise, do you even have a professional identity at all?

The reality is that the knowledge bases and skill sets that make up the rich discipline of evaluation are so wide and deep that no single individual or team can possibly peg off the full range of competencies. So, to pitch oneself as the “all things to all people” dream team amounts to either deliberate misrepresentation or an ignorance of just how huge the evaluation toolkit and knowledge base has really become.

Until we (as teams and individuals) are clear about what is distinctive about the way we work and what we are good at – and the spaces we don’t (and shouldn’t) work in, we aren’t truly clear about who we are as evaluators.

Of course, one thing we all need to be unambiguously clear about if what distinguishes our work from researchers, auditors, and those who specialize in monitoring but not evaluation. In particular, we must be crystal clear about why genuine evaluation must be value-based.

Implications for professional development of individuals and teams

Each individual evaluator, evaluation team, and evaluation consulting business should be thinking hard about how they wish to build their professional repertoire so they have a strong and [relatively] unique value proposition.

This will allows them to clearly communicate to clients (and prospective employers) what it is that distinguishes their work from the other masses vying for the same contract (or job).

When clients get good at asking about this – and when evaluators get good at answering the question – we will finally start making useful strides toward ensuring better fit between evaluation projects and evaluation teams.

Related posts:

• Lifting the quality of evaluation #1: Savvy clients
• Why genuine evaluation must be value-based
• What is Genuine Evaluation?

What key elements are needed to seriously drive up the quality and value for money delivered by evaluation?

Earlier this year at the Aotearoa New Zealand Evaluation Association (anzea) conference, I reflected on this question and came up with three ingredients we should be thinking really hard about. Here’s one … (and watch this week for the sequels) …

 
1. Savvy clients – informed, demanding consumers of evaluation work, who can and do:

• figure out what the right piece of evaluation work is in the first place  – position it strategically, scope it clearly, and resource it right (and don’t overscope it – leave some room for discussion and negotiation!)
• hire the right people for the job (based on a good understanding of what kinds of work need what kinds of skills and capabilities – no one evaluator can be all things to all people or is the right person for all jobs) using a sound non-onerous process
• work effectively alongside evaluators to ensure information needs and expectations are clear and that the evaluation is on track to deliver those (but without micro-managing)
• build in and drive utilization from inside the organization – set the evaluation up so it’s on the radars and destined to be used by the right people, and put a sound strategy in place to follow through and make sure this keeps happening once the main evaluation project is over

It seems to me that much of the focus on good/shoody evaluation tends to be focused on evaluator competencies. We tend to forget how the evaluation savviness of clients makes or breaks evaluation.

I have a lot more to say on this topic – as you’ll see in future weeks and months!

Related posts:

• 9 golden rules for commissioning a waste-of-money evaluation
• Extreme Genuine Evaluation Makeovers (XGEMs) for Commissioning
• Commissioning XGEMs – the sequel

• Lifting the quality of evaluation #2: Capable evaluators who know their ‘space’
• Lifting the quality of evaluation #3: Evaluation associations with focus

Our guest blogger this week is Katherine Hay, a senior member of the Evaluation Unit of the International Centre for Development Research. Based in New Delhi, India, she is an expert on the role of evaluation in development in South Asia. She promotes approaches that assess how women and other marginalized groups benefit from development in the region. Katherine joined IDRC’s South Asia office in New Delhi in 2000 and has undertaken research in South Asia for more than 15 years. Her work with IDRC includes building evaluation curriculum in universities in the region, and supporting evaluation communities of practice spanning South Asia and Afghanistan. She has written on women’s empowerment, evaluation, and the policy research environment in South Asia. Katherine holds a master’s degree in international affairs from Carleton University in Ottawa. Katherine is sharing with us perspectives from her recent keynote address to the conference of the Sri Lankan Evaluation Association.

In reading the newspapers lately, I’ve noticed an increasing expectation that evidence can give us the answers that policy makers need. I practice evaluation because I believe that evaluation can help identify what is working from what is not working, and for whom.  So I should be pleased to see these calls for “the evidence.”   I am….and yet, I am also somewhat alarmed by this faith in data.

Some people seem to suggest that if we would just get enough evidence we will be able to ‘fix’ poverty.  I think that is both naïve and dangerous. In the New York Times, Nicolas Kristoff had a piece, called “Getting Smart on Humanitarian Aid,” where he said: “How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, economists provide answers that are rigorously field-tested.” That sounds good but do we really have answers, and to what?

The evidence that Kristoff was pointing to drew on the excellent work of Duflo and Banerjee on randomized controlled trials.  Kristoff, and a string of other journalists, came to the conclusion that “we now have the answers” based on 2-3 examples that included the cost effectiveness improving school attendance by deworming kids and providing them with school uniforms.  I’ve read the studies.  I’m pretty convinced that schools should deworm and that school uniforms in Africa are probably worth the money. But do education policy makers now have all the answers whereas before they just had ‘hot air?’  Not quite.

These are fairly simple interventions.  I don’t doubt that they are helpful. But idea that we have all the evidence we need or can get it through trials, is not helpful.  It dumbs down development problems by arguing that, until now, everyone working in development has been running around with no clue.  It suggests that governments, implementing agencies, funding agencies, just need to run some experiments to find out what the policy should be. It’s a simple idea.  But poverty and development are complex.

There is nothing wrong with experiments.  The right tool in any situation is the one that best answers the questions being asked.  My critique is of the idea that development is just about getting the data right, or that evidence ‘neutral’ or has nothing to do with politics.

Why is this a dangerous idea? Kristoff goes on to suggest that “For those who want to be sure, to get the most bang for your buck, there is also a “proven impact fund” that supports interventions like deworming…that have proved to be cost-effective in rigorous trials. But what would happen if we only  fund the proven, cost effective things, the sure things?  It’s hard to be sure about many things that matter.

Funding only the sure things would certainly rule out a great deal of things that many of us think are important including work to address:  climate change, violence against women, son preference, human rights, or conflict.   Much of this work takes generations to see results and is deeply contextual; in many of these areas we don’t have ‘sure things.’

I’m hopeful that the current controversy over Greg Mortenson’s book ‘Three Cups of Tea’ and the operations of the related NGO ‘Central Asia Institute’ (detailed in John Krakauer’s book Three Cups of Deceit and a 60 minutes story)  will lead to improvements in how development interventions and organizations are planned, implemented, evaluated and reported.  There are risks, however, that it could make it more difficult to raise funding in the areas of operation (girls’ schooling, Afghanistan, Pakistan).  And that it could provide further reasons for dismissing discussions about development evaluation that draw on complexity language.

In response to the criticisms raised by 60 minutes and by author,  Greg Mortenson, in a letter to supporters on the IKAT website, said:

Although we would like the world to be linear, orderly and peaceful, the reality is that our world is a dynamic, fluid place, often filled with chaos and confusion. In that space, I thrive and get the courage to help bring change and empower people. …

The “60 Minutes” program may appear to ask simple questions, but the answers are often complex, not easily encapsulated in 10-second sound bites.
Working in isolated areas, in communities that are not on any map, and often in areas of turmoil, religious extremism or natural disasters where education is still relatively rare and ancient codes of conduct and social hierarchies still dominate – all these things demand constant adjustment, accommodation and patience.

It’s interesting to analyze the case using the different lenses of simple, complicated, complex.

There appear to be simple aspects of this that should have been managed better .  Watch where the money goes, and make sure that most of it is going to the central activities (in this case, the description of the CAI on its financial accounts talks only about its international education programs, even though more money goes to the US ‘outreach’ book tours).  Check if the schools have been built, and are being used.

There appear to be complicated aspects that should have been managed better.  Schools need bulldings plus teachers plus course material.  Although these are listed as funded activities, it seems there has not been enough attention to ensuring the full ‘causal package’ is in place.

And, despite the language of complexity, there appear to be complex aspects that should have been managed better.  Since complex situations are unpredictable and emergent, it requires adaptive and responsive management.  Processes are needed to spot problems and work out how to fix them, and to spot successes and work out how to amplify them.  Instead, in Krakauer’s book, many staff and board members have recounted their experiences of finding problems and trying unsuccessfully to have them addressed.

Dealing with complexity effectively does not mean ad hoc or laissez-faire actions and reactions.  Nor does it mean ignoring the simple or complicated elements of the situation.

About a month ago, I wrote a post that wondered whether evaluation was really useful for accountability. The issues being raised about the Central Asia Institute have made me rethink this a little.  In that post I suggested that evaluation could usefully assess whether the management had displayed ‘smart accountability’ – being aware of what was going on, and adapting as necessary.  But, since most evaluators are not well trained or experienced in checking accountability for inputs, I suggested that auditors might do a better job of this.

However, the 2009 Audited Financial Statement for CAI, shows the limitations of audit processes:

Use of Estimates
Preparation of financial statements in conformity with accounting principles generally accepted in the United States of America requires the use of management’s estimates. Actual results of operations may differ from those estimates.

While evaluation needs to do more than see where the money has gone, how many activities have been undertaken, and how many outputs have been produced, it is important to have reasonable evidence of these – both for external accountability and to support informed, adaptive management that can truly respond to complexity.