Katherine Hay continues her guest blogging on evidence and evaluation.

Ben Goldacre in The Guardian wrote that UK politicians “are ignorant about trials and they’re weird about evidence.” He contrasts this with international development where he talks about the “amazing work testing interventions around the world with proper, randomised trials.”  He goes on to say, policy makers in the UK, just need to “define your outcome, randomise…and you’ll have the answer by the end of next parliament.”  He notes all these trials (somehow) won’t cost money but will save unprecedented amounts of money.  He then concludes that, “politicians are …too arrogant to have their ideologies questioned, and too scared…of hard data on their interventions.’

It’s an entertaining article.  The idea of doing a trial on every single UK policy is funny.  The idea that they are free is even funnier.  Imagine how British parents would react when they brought their kids to school and were told what group their child was randomly selected to be in?  Perhaps:

In a large class but with a highly rated teacher.

• In a small class but with a less experienced teacher.
• In a small class with no hot lunch…

And so on.   The permutations to test every UK policy would be never-ending.

Goldacre is obviously being extreme to make a point.  But is he correct?  Have countries who conducted randomized trials saved huge amounts on their interventions?

I’ve seen no evidence that trials are more likely to inform policies than other evaluations or research. I expect that they are subject to the same challenges of use as other types of evidence.

If we accept that “working” can mean different things to different groups, and that views on what is ‘worth the money’ usually vary based on people’s values and their position in society, than why would we assume that studies with statistical power will lead to change on the ground? Evaluation can give us more evidence – and must give us better quality evidence – but the idea that policy making is just a computation of evidence is wrong.   Evidence is only one piece of policy making.  Evidence can, and often is, interpreted and used to reinforce dominant policies.

For example, the country where I live, India, has a system that distributes grains to the poor.  Some people think this system should be replaced with a system where cash is giving to poor families who can buy the food or grains that they choose.  Others feel that dismantling this system will mean food grain that was getting to the poor and children will be replaced by spending on things like alcohol.  Different groups have done studies on whether people want this change.  Some studies show that people do and some studies show that people don’t.

Part of the solution is about design.  You have to be confident that the evidence you have is good quality.  Do people really want it or not? But it’s not just about design.  Even with convincing findings, the policy maker has multiple elements to weigh. In New Delhi, the capital, the government decided that they wanted to experiment with the cash transfer.  But was it because they were comfortable having their ‘ideology tested’ or because their ideology lead them to prefer such a system? They were criticized for the latter; for having a position.  But they were elected on their positions; pushing for more open reality testing is not about wishing away positions.

If they get the design right they may know pretty accurately how many families want or do not want changes in this system and if they run a trial well they may also know about some outcomes.  But that data won’t tell them the ‘right choice’.  For example, how much of an increase in alcohol consumption is ok, or is trumped by increasing the poor’s control of spending choices, or increased efficiencies?  Those decisions are values based and values are often political. For example, Abhijit Sen, a noted economist and member of the Indian Planning Commission, noted “politicians will never accept a dismantling of the PDS’ but added, “Forget the politicians, what matters most is what the voters think.”

We cannot wish away politics and nor should we want to.  My point is that we need to get much more strategic on pathways to use if we want to influence policy with evaluation.

Let me give you a wonderful final little example. Two PhD candidates from Yale did an experiment in a New Delhi slum. The subjects wanted to apply for ration cards. They were randomly assigned to one of four groups. The first group applied for the ration card and did nothing more, the second attached a letter of recommendation from an NGO to their application, the third paid a bribe after putting in their application, and the fourth enquired about the status of their ration card application through a right to information (RTI) request.  The researchers found that the group that paid a bribe was the most successful, but, the group that put in an RTI request was almost as successful. Hardly anyone in the other two groups received their ration card.

Clever experiment.  They answered an interesting question and will likely get their PHD’s in the process.

But corruption in the ration card system is not fixed. Also, NGO’s and others were already using the RTI for things exactly like this.  This experiment adds more evidence to existing evidence that the RTI Act is a useful tool against corruption.  Is it THE answer?  No.  Is it helpful?  Yes.

Assuming that evidence alone will change things is wrong.  Evidence matters, and should be made to matter more, but it’s not the only thing that matters.  Recognizing this doesn’t weaken evaluation, quite the opposite, it actually creates greater opportunities for ‘genuine evaluation’

Our guest blogger this week is Katherine Hay, a senior member of the Evaluation Unit of the International Centre for Development Research. Based in New Delhi, India, she is an expert on the role of evaluation in development in South Asia. She promotes approaches that assess how women and other marginalized groups benefit from development in the region. Katherine joined IDRC’s South Asia office in New Delhi in 2000 and has undertaken research in South Asia for more than 15 years. Her work with IDRC includes building evaluation curriculum in universities in the region, and supporting evaluation communities of practice spanning South Asia and Afghanistan. She has written on women’s empowerment, evaluation, and the policy research environment in South Asia. Katherine holds a master’s degree in international affairs from Carleton University in Ottawa. Katherine is sharing with us perspectives from her recent keynote address to the conference of the Sri Lankan Evaluation Association.

In reading the newspapers lately, I’ve noticed an increasing expectation that evidence can give us the answers that policy makers need. I practice evaluation because I believe that evaluation can help identify what is working from what is not working, and for whom.  So I should be pleased to see these calls for “the evidence.”   I am….and yet, I am also somewhat alarmed by this faith in data.

Some people seem to suggest that if we would just get enough evidence we will be able to ‘fix’ poverty.  I think that is both naïve and dangerous. In the New York Times, Nicolas Kristoff had a piece, called “Getting Smart on Humanitarian Aid,” where he said: “How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, economists provide answers that are rigorously field-tested.” That sounds good but do we really have answers, and to what?

The evidence that Kristoff was pointing to drew on the excellent work of Duflo and Banerjee on randomized controlled trials.  Kristoff, and a string of other journalists, came to the conclusion that “we now have the answers” based on 2-3 examples that included the cost effectiveness improving school attendance by deworming kids and providing them with school uniforms.  I’ve read the studies.  I’m pretty convinced that schools should deworm and that school uniforms in Africa are probably worth the money. But do education policy makers now have all the answers whereas before they just had ‘hot air?’  Not quite.

These are fairly simple interventions.  I don’t doubt that they are helpful. But idea that we have all the evidence we need or can get it through trials, is not helpful.  It dumbs down development problems by arguing that, until now, everyone working in development has been running around with no clue.  It suggests that governments, implementing agencies, funding agencies, just need to run some experiments to find out what the policy should be. It’s a simple idea.  But poverty and development are complex.

There is nothing wrong with experiments.  The right tool in any situation is the one that best answers the questions being asked.  My critique is of the idea that development is just about getting the data right, or that evidence ‘neutral’ or has nothing to do with politics.

Why is this a dangerous idea? Kristoff goes on to suggest that “For those who want to be sure, to get the most bang for your buck, there is also a “proven impact fund” that supports interventions like deworming…that have proved to be cost-effective in rigorous trials. But what would happen if we only  fund the proven, cost effective things, the sure things?  It’s hard to be sure about many things that matter.

Funding only the sure things would certainly rule out a great deal of things that many of us think are important including work to address:  climate change, violence against women, son preference, human rights, or conflict.   Much of this work takes generations to see results and is deeply contextual; in many of these areas we don’t have ‘sure things.’

One of the hallmarks of genuine evaluation is thoughtful analysis and sound evaluative reasoning that the audience can follow.

Some of our readers are not only evaluators, but also teachers or students of evaluation. We thought you might enjoy this little classic that’s done the rounds on the Internet. We found this version on Pinetree.net’s Inbox Humor page.

As you study for exams, remember its not the quantity it’s the quantity. And remember there is no substitute for pure unadulterated bull

---

Dr. Schambaugh, of the University of Oklahoma School of Chemical Engineering, Final Exam question for May of 1997. Dr. Schambaugh is known for asking questions such as, “why do airplanes fly?” on his final exams. His one and only final exam question in May 1997 for his Momentum, Heat and Mass Transfer II class was: “Is hell exothermic or endothermic? Support your answer with proof.”

Most of the students wrote proofs of their beliefs using Boyle’s Law or some variant. One student, however, wrote the following:

“First, We postulate that if souls exist, then they must have some mass. If they do, then a mole of souls can also have a mass. So, at what rate are souls moving into hell and at what rate are souls leaving? I think we can safely assume that once a soul gets to hell, it will not leave.

Therefore, no souls are leaving. As for souls entering hell, let’s look at the different religions that exist in the world today. Some of these religions state that if you are not a member of their religion, then you will go to hell. Since there are more than one of these religions and people do not belong to more than one religion, we can project that all people and souls go to hell. With birth and death rates as they are, we can expect the number of souls in hell to increase exponentially.

Now, we look at the rate of change in volume in hell. Boyle’s Law states that in order for the temperature and pressure in hell to stay the same, the ratio of the mass of souls and volume needs to stay constant. Two options exist:

1. If hell is expanding at a slower rate than the rate at which souls enter hell, then the temperature and pressure in hell will increase until all hell breaks loose.
2. If hell is expanding at a rate faster than the increase of souls in hell, then the temperature and pressure will drop until hell freezes over.

So which is it? If we accept the quote given to me by Theresa Manyan during Freshman year, “that it will be a cold night in hell before I sleep with you” and take into account the fact that I still have NOT succeeded in having sexual relations with her, then Option 2 cannot be true…Thus, hell is exothermic.”

The student, Tim Graham, got the only A.

The other genuine evaluation lesson here is this:

Whether the reasoning is sound or flawed – and in the messy, nuanced real world of evaluation, it is sometimes difficult to tell – the absolutely most important thing is that it is clear and can be followed. That’s how we make evaluation accessible, easy to engage with, easy to criticize – and therefore easier to improve.

Have a great weekend!

In a week of international differences in evaluation approaches, and Australia Day (26 Jan), this week’s Friday Funny comes from Kirsty Fenton, Senior Evaluation Officer with the Department of Primary Industries, Victoria, Australia.

It’s been a busy week for Patricia, traveling from The Evaluators Institute and the seminar on ‘Strengthening the Effectiveness of Evaluation in Washington‘, DC, to the GIZ conference on ‘Systemic approaches in evaluation‘, in Frankfurt, and back to a meeting of the Board of the American Evaluation Association in Anaheim, CA. Stay tuned for some thoughts and useful links about systemic approaches, systems approaches and systematization in evaluation.

This week’s Friday Funny reminds us that conclusions are not only about evidence, but about the interpretation of that evidence, or lack of evidence.

Pic from www.guardian.co.uk

After having dug to a depth of 10 feet last year, British scientists found traces of copper wire dating back 200 years and came to the conclusion that their ancestors already had a telephone network more than 150 years ago.

Not to be outdone by the Brit’s, in the weeks that followed, an American archaeologist dug to a depth of 20 feet, and shortly after, a story published in the New York Times: “American archaeologists, finding traces of 250-year-old copper wire, have concluded that their ancestors already had an advanced high-tech communications network 50 years earlier than the British”.
One week later, the Dept of Minerals and Energy in Western Australia reported the following:
“After digging as deep as 30 feet in Western Australia’s Pilbara region, Jack Lucknow, a self-taught archaeologist, reported that he found absolutely nothing. Jack has therefore concluded that 250 years ago, Australia had already gone wireless.”

Why are evaluation and needs assessment NOT simply a matter of reporting what people said they needed, or what they said met or didn’t meet their needs?

This classic, which has done the rounds on the Internet for many years, lists actual comments apparently left in 1996 on U. S. Forest Service registration sheets and comment cards by backpackers completing wilderness camping trips:

1. “A small deer came into my camp and stole my bag of pickles. Is there a way I can get reimbursed? Please call.”
2. “Escalators would help on steep uphill sections.”
3. “Instead of a permit system or regulations, the Forest Service needs to reduce worldwide population growth to limit the number of visitors to wilderness.”
4. “Trails need to be wider so people can walk while holding hands.”
5. “Ban walking sticks in wilderness. Hikers that use walking sticks are more likely to chase animals.”
6. “All the mile markers are missing this year.”
7. “Found a smoldering cigarette left by a horse.”
8. “Trails need to be reconstructed. Please avoid building trails that go uphill.”
9. “Too many bugs and leeches and spiders and spider webs. Please spray the wilderness to rid the area of these pests.”
10. “Please pave the trails so they can be plowed of snow in the winter.”
11. “Chair lifts need to be in some places so that we can get to wonderful views without having to hike to them.”
12. “The coyotes made too much noise last night and kept me awake. Please eradicate these annoying animals.”
13. “Reflectors need to be placed on trees every 50 feet so people can hike at night with flashlights.”
14. “Need more signs to keep area pristine.”
15. “A McDonald’s would be nice at the trail head.”
16. “The places where trails do not exist are not well marked.”
17. “Too many rocks in the mountains.”

Seriously!!!

[Original author unknown; we found this version on Pinetree.net's Inbox Humor page]