Our guest blogger this week is Katherine Hay, a senior member of the Evaluation Unit of the International Centre for Development Research. Based in New Delhi, India, she is an expert on the role of evaluation in development in South Asia. She promotes approaches that assess how women and other marginalized groups benefit from development in the region. Katherine joined IDRC’s South Asia office in New Delhi in 2000 and has undertaken research in South Asia for more than 15 years. Her work with IDRC includes building evaluation curriculum in universities in the region, and supporting evaluation communities of practice spanning South Asia and Afghanistan. She has written on women’s empowerment, evaluation, and the policy research environment in South Asia. Katherine holds a master’s degree in international affairs from Carleton University in Ottawa. Katherine is sharing with us perspectives from her recent keynote address to the conference of the Sri Lankan Evaluation Association.

In reading the newspapers lately, I’ve noticed an increasing expectation that evidence can give us the answers that policy makers need. I practice evaluation because I believe that evaluation can help identify what is working from what is not working, and for whom.  So I should be pleased to see these calls for “the evidence.”   I am….and yet, I am also somewhat alarmed by this faith in data.

Some people seem to suggest that if we would just get enough evidence we will be able to ‘fix’ poverty.  I think that is both naïve and dangerous. In the New York Times, Nicolas Kristoff had a piece, called “Getting Smart on Humanitarian Aid,” where he said: “How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, economists provide answers that are rigorously field-tested.” That sounds good but do we really have answers, and to what?

The evidence that Kristoff was pointing to drew on the excellent work of Duflo and Banerjee on randomized controlled trials.  Kristoff, and a string of other journalists, came to the conclusion that “we now have the answers” based on 2-3 examples that included the cost effectiveness improving school attendance by deworming kids and providing them with school uniforms.  I’ve read the studies.  I’m pretty convinced that schools should deworm and that school uniforms in Africa are probably worth the money. But do education policy makers now have all the answers whereas before they just had ‘hot air?’  Not quite.

These are fairly simple interventions.  I don’t doubt that they are helpful. But idea that we have all the evidence we need or can get it through trials, is not helpful.  It dumbs down development problems by arguing that, until now, everyone working in development has been running around with no clue.  It suggests that governments, implementing agencies, funding agencies, just need to run some experiments to find out what the policy should be. It’s a simple idea.  But poverty and development are complex.

There is nothing wrong with experiments.  The right tool in any situation is the one that best answers the questions being asked.  My critique is of the idea that development is just about getting the data right, or that evidence ‘neutral’ or has nothing to do with politics.

Why is this a dangerous idea? Kristoff goes on to suggest that “For those who want to be sure, to get the most bang for your buck, there is also a “proven impact fund” that supports interventions like deworming…that have proved to be cost-effective in rigorous trials. But what would happen if we only  fund the proven, cost effective things, the sure things?  It’s hard to be sure about many things that matter.

Funding only the sure things would certainly rule out a great deal of things that many of us think are important including work to address:  climate change, violence against women, son preference, human rights, or conflict.   Much of this work takes generations to see results and is deeply contextual; in many of these areas we don’t have ‘sure things.’

Pic by Magnus Akselvoll

What are the big, enduring challenges in evaluation? Where are we making progress?

The ‘BetterEvaluation’ project has announced a Request for Challenges (RfC), with a particular emphasis in this round on evaluation in the areas of Capacity Development, Climate Change Adaptation and Network initiatives.

Nominations for challenges are being sought by April 30 and will help us to focus our efforts to bring together examples of good evaluation practice, and to support research and development in areas where new methods need to be developed or trialled.

The challenges might relate to any aspect of evaluation, including:

• Articulating theories of change
• Engaging stakeholders in evaluation planning, analysis and implementation
• Framing evaluation questions
• Identifying and negotiating relevant values to underpin the evaluation
• Designing evaluation plans
• Identifying meaningful and feasible indicators
• Gathering appropriate data
• Understanding direct and indirect contributions of the intervention
• Making overall judgments about the program
• Forming useful recommendations
• Facilitating the use of the evaluation results for learning;

BetterEvaluation will coordinate a process of consultation and collective problem solving addressing these challenges. In some cases, we will be able to support field testing of suggested responses.A summary of challenges and suggested responses will be posted on the BetterEvaluation website.

Please send your Challenges to bettereval@pactworld.org by April 30, 2011 with the following information:

ABOUT YOU

• Name/Position
• Organization/Affiliation
• Country
• Contact Information (phone, email)
• Program/Initiative name, duration and a brief description (5-10 sentences)

ABOUT THE CHALLENGE

• Describe your challenge and any steps you have taken to resolve this challenge so far (10-15 sentences).

BetterEvaluation is an international collaboration to improve evaluation by sharing information about how to choose and implement appropriate combinations of methods. It is being developed with the support of the Rockefeller Foundation and IFAD (International Fund for Agricultural Development) by a diverse community who share a commitment to situationally responsive evaluation. Founding partners are: RMIT University (Royal Melbourne Institute of Technology, Australia); Pact (Washington D.C, USA and Bangkok, Thailand offices); ILAC (Institutional Learning and Change initiative of the Consultative Group on International Agriculture, Italy); ODI (Overseas Development Institute, UK).

For more information about BetterEvaluation please check the current site www.betterevaluation.org. We are currently in the beta-phase of development and will open the full site in summer 2011.

After years working on this with my co-author Sue Funnell, I’m looking forward to the launch of the book Purposeful Program Theory: Effective Use of Theories of Change and Logic Models in Canberra next Thursday (Yes, on St Patrick’s Day).

Thursday 17 March, 5.30 for 5.45 pm

Venue: Central Courtyard University House, Balmain Crescent, Acton (in case of bad weather the launch will be held in the Scarth Room)

AES members and non-members are invited.

RSVP: by 5pm Monday 14 March to aes@aes.asn.au, 6262 9093 for catering purposes. Light refreshments will be served.

The book will be available for sale at the launch at a discounted rate.

I’m in Washington DC this week, after teaching a course on Using Program Theory and Logic Models for Evaluation at The Evaluators Institute.

Fortunately I will be able to stay on for a seminar being presented this coming Wednesday by The Evaluators’ Institute and The Washington Evaluators on

“StrengtheningEvaluation Effectiveness”

Wednesday, January 19th, 6:00pm to 7:30pm
Jack Morton Auditorium, Media and Public Affairs building, 1st floor
805 21stSt NW Washington DC 20052

I wonder if the session will consider strengthening evaluation effectiveness in terms of:

• increasing the utility of evaluation for informing evidence-based policy and practice, especially in terms of assisting policymakers and practitioners to translate findings to new situations
• increasing the attention paid to heterogeneous outcomes (rather than simply the mean net effect), and in particular to who benefits and who loses from programs and policies with heterogeneous outcomes
• increasing the attention paid to unintended outcomes (positive and negative) and to the costs incurred beyond the formal resources expended for implementation, and including the resources contributed by other stakeholders, including other organizations and clients
• improving the use of credible non-experimental approaches such as process tracing in situations where experimental or quasi-experimental designs are not appropriate or feasible
• increasing the ability of managers to commission and manage external evaluators to ensure high quality work is done
• increasing the ability of managers and service deliverers to evaluate their own programs and to make use of the evaluations of other programs

Stay tuned for a report after the session – or see you there.
Speakers include:
Gary T. Henry, Duncan MacRae’09 and Rebecca Kyle MacRaeProfessorship Public Policy
Department of Public Policy, Director, Carolina Institute for Public Policy at the University of North Carolina at Chapel Hill
Harry P. Hatry, Distinguished Fellow and Director of the Public Management Program, UrbanInstitute, Washington, DC
Kathryn E. Newcomer, Director, The Trachtenberg School of Public Policy and Public Administration, George Washington University
Theodore H. Poister, Professor of Public Management and Policy, Andrew Young School of Policy Studies at Georgia State University

Moderator – Ann M. Doucette, Director, The Evaluators’ Institute and Midge Smith Center for Evaluation Effectiveness

Reception to follow.Please RSVP (reception) to tei@gwu.edu

Looks like another book to add to the possible reading pile, based on the recent article in the NY Times.

What is “proofiness?”

It’sthe mathematical analog of Stephen Colbert’s “truthiness.” It’s using numbers to prove what you know in your heart is true, even when you know it’s not. Numbers have a particular ability to fool us. It’s using that ability to turn nonsense into something that is believable with numbers.

What is causuistry?

Casuistry is using bogus arguments through seemingly sound principles. Causuistry is my shorthand for wrongly implying causation.

What is randumbness?

We’re hard wired to reject the idea that there’s no reason for something happening, … our minds see these pattens we think we can exploit, and this leads us to phony beliefs.

Has anyone read it? Do you recommend it? Are there good arguments behind the newly coined words?