Pic by JitterBuffer

Over the Antipodean summer Genuine Evaluation goes to the beach instead of blogging.  We’re back now, brushing off the sand, and planning more discussions about what it means to do genuine evaluation, plus sharing some insights from the African evaluation conference in Accra, Ghana.

To start the year, we wanted to highlight one of the more disturbing aspects of public policy discussions in recent years  -  the tendency to put forward opinions as if they were as compelling as solid evidence. We suspect that this will be the first in an ongoing series of examples.

Are working hours getting longer? Hopefully this example reflects someone being misquoted in the article in The Age in Melbourne, rather than how it appears – a researcher suggesting it’s too hard to get reasonable estimates of the extent of a problem and then pronouncing that the problem has diminished:

Some recent studies suggest this may now be a relic of history and that Australians work the longest hours in the developed world.

But Professor Mark Wooden, of the Institute of Applied Economic and Social Research, disagrees … strongly.

“The idea we work the most hours in the world is absolute crap,” he says.

“Lots of people work long hours and lots of people work short hours. We have a mix.”

He argues that workers in Japan and Korea work longer than Australians and that comparing working hours between countries was an inexact science.

People tend to overestimate how long they work as a sort of “badge of courage” and find it difficult to estimate the hours they work accurately, Professor Wooden says.

“I don’t think we can count,” he says. “It’s impossible to know. The study would need to be so invasive.”

His research shows the number of Australians working 50 hours a week or more peaked in the mid-1990s.

“Over the last 10 years, the proportion of Australians working long hours has been dropping.”

Photo: Joe Armao (The Age)

What’s worse than no evaluation? An evaluation that is wrong but you think is right.

Organizations that provide authoratitive evaluations have an obligation to meet high standards of accuracy and consistency. It is therefore hard to believe the series of events that led to Victoria’s Environmental Protection Agency listing Melbourne’s bayside beaches as “good”, and suitable for swimming when the level of bacteria was 40 times the acceptable limit.

According to The Age report,

[EPA] staff were not working on the ChristmasDay and Boxing Day public holidays, and as a result old forecast information was fed automatically into the website from Saturday until yesterday. These forecasts were made on Friday afternoon, and did not take into account the ferociousness of the Christmas Day storms.

On Sunday (Christmas Day) a massive storm hit Melbourne, sending debris, rubbish, cigarette butts and dog droppings into storm water drains. But for two days the EPA was operating on autopilot, issuing reports on the web site and tweets based on the projected water quality not the actual water quality.

And what has the EPA learned from this? Nothing, apparently. According to The Age, no EPA staff will be working on the New Year’s Day public holiday, and once again reports will be based on the weather forecast not on actual testing. A spokesman said beachgoers “should use their own judgment” in deciding to go swimming after storms like those that hit on Christmas Day.

Which does raise the question – if beachgoers should use their own judgment, because it is more likely to be accurate than the official reports, what’s the point of having the official reports?

Here’s what SHOULD happen. Either roster someone to work on these public holidays and ensure the reports are actually based on data ( it’s summer here and people are going to the beaches) OR issue a clear statement on the website, tweets and to news media that an accurate report cannot be provided due to the public holidays. No report is better than an inaccurate report.

Katherine Hay continues her guest blogging on evidence and evaluation.

Ben Goldacre in The Guardian wrote that UK politicians “are ignorant about trials and they’re weird about evidence.” He contrasts this with international development where he talks about the “amazing work testing interventions around the world with proper, randomised trials.”  He goes on to say, policy makers in the UK, just need to “define your outcome, randomise…and you’ll have the answer by the end of next parliament.”  He notes all these trials (somehow) won’t cost money but will save unprecedented amounts of money.  He then concludes that, “politicians are …too arrogant to have their ideologies questioned, and too scared…of hard data on their interventions.’

It’s an entertaining article.  The idea of doing a trial on every single UK policy is funny.  The idea that they are free is even funnier.  Imagine how British parents would react when they brought their kids to school and were told what group their child was randomly selected to be in?  Perhaps:

In a large class but with a highly rated teacher.

• In a small class but with a less experienced teacher.
• In a small class with no hot lunch…

And so on.   The permutations to test every UK policy would be never-ending.

Goldacre is obviously being extreme to make a point.  But is he correct?  Have countries who conducted randomized trials saved huge amounts on their interventions?

I’ve seen no evidence that trials are more likely to inform policies than other evaluations or research. I expect that they are subject to the same challenges of use as other types of evidence.

If we accept that “working” can mean different things to different groups, and that views on what is ‘worth the money’ usually vary based on people’s values and their position in society, than why would we assume that studies with statistical power will lead to change on the ground? Evaluation can give us more evidence – and must give us better quality evidence – but the idea that policy making is just a computation of evidence is wrong.   Evidence is only one piece of policy making.  Evidence can, and often is, interpreted and used to reinforce dominant policies.

For example, the country where I live, India, has a system that distributes grains to the poor.  Some people think this system should be replaced with a system where cash is giving to poor families who can buy the food or grains that they choose.  Others feel that dismantling this system will mean food grain that was getting to the poor and children will be replaced by spending on things like alcohol.  Different groups have done studies on whether people want this change.  Some studies show that people do and some studies show that people don’t.

Part of the solution is about design.  You have to be confident that the evidence you have is good quality.  Do people really want it or not? But it’s not just about design.  Even with convincing findings, the policy maker has multiple elements to weigh. In New Delhi, the capital, the government decided that they wanted to experiment with the cash transfer.  But was it because they were comfortable having their ‘ideology tested’ or because their ideology lead them to prefer such a system? They were criticized for the latter; for having a position.  But they were elected on their positions; pushing for more open reality testing is not about wishing away positions.

If they get the design right they may know pretty accurately how many families want or do not want changes in this system and if they run a trial well they may also know about some outcomes.  But that data won’t tell them the ‘right choice’.  For example, how much of an increase in alcohol consumption is ok, or is trumped by increasing the poor’s control of spending choices, or increased efficiencies?  Those decisions are values based and values are often political. For example, Abhijit Sen, a noted economist and member of the Indian Planning Commission, noted “politicians will never accept a dismantling of the PDS’ but added, “Forget the politicians, what matters most is what the voters think.”

We cannot wish away politics and nor should we want to.  My point is that we need to get much more strategic on pathways to use if we want to influence policy with evaluation.

Let me give you a wonderful final little example. Two PhD candidates from Yale did an experiment in a New Delhi slum. The subjects wanted to apply for ration cards. They were randomly assigned to one of four groups. The first group applied for the ration card and did nothing more, the second attached a letter of recommendation from an NGO to their application, the third paid a bribe after putting in their application, and the fourth enquired about the status of their ration card application through a right to information (RTI) request.  The researchers found that the group that paid a bribe was the most successful, but, the group that put in an RTI request was almost as successful. Hardly anyone in the other two groups received their ration card.

Clever experiment.  They answered an interesting question and will likely get their PHD’s in the process.

But corruption in the ration card system is not fixed. Also, NGO’s and others were already using the RTI for things exactly like this.  This experiment adds more evidence to existing evidence that the RTI Act is a useful tool against corruption.  Is it THE answer?  No.  Is it helpful?  Yes.

Assuming that evidence alone will change things is wrong.  Evidence matters, and should be made to matter more, but it’s not the only thing that matters.  Recognizing this doesn’t weaken evaluation, quite the opposite, it actually creates greater opportunities for ‘genuine evaluation’

We are looking forward to next week’s guest blogs from Katherine Hay, who works in New Delhi, India as part of the Evaluation Unit of Canada’s International Development Research Centre. Katherine will be blogging on the role of evidence in informing policy and what this means for the types of evidence that evaluation should aim to produce.

This week’s Friday Funny comes from Javier Ekboir, leader of the Institutional Learning And Change initiative of the Consultative Group on International Agricultural Research. It reminds us of the importance of considering:

• the change trajectories of the interventions we are evaluating (see Michael Woolcock’s blog post on this important issue),
• the timing of when we evaluate (see Michael Patton’s story in Utilization-Focused Evaluation about whether having your horse go lame might be a good thing or a bad thing),
• the reasons why what people tell you and show you might not be accurate representations of what is going on (see Jack Douglas’ classic book Investigative Social Research)

The version below is adapted from www.gotjokes.net.

While walking down the street one day a [insert appropriate local politician – senator, president, Member of Parliament) is tragically hit by a truck and dies. His soul arrives in heaven and is met by St. Peter at the entrance.

“Welcome to heaven,” says St. Peter. “Before you settle in, it seems there is a problem. We seldom see a high official around these parts, you see, so we’re not sure what to do with you.”

“No problem, just let me in,” says the politican.

“Well, I’d like to but I have orders from higher up. What we’ll do is have you spend one day in hell and one in heaven. Then you can choose where to spend eternity.”

“Really, I’ve made up my mind. I want to be in heaven,” says the politician.

“I’m sorry but we have our rules.” And with that, St. Peter escorts him to the elevator and he goes down, down, down to hell.

The doors open and he finds himself in the middle of a green golf course. In the distance is a club and standing in front of it are all his friends and other politicians who had worked with him. Everyone is very happy and in evening dress.

They run to greet him, shake his hand, and reminisce about the good times they had while getting rich at expense of the people. They play a friendly game of golf and then dine on lobster, caviar and champagne.

Also present is the devil, who really is a very friendly guy who has a good time dancing and telling jokes. They are having such a good time that, before he realizes it, it is time to go. Everyone gives him a hearty farewell and waves while the elevator rises.

The elevator goes up, up, up and the door reopens on heaven where St. Peter is waiting for him.

“Now it’s time to visit heaven.”

So, 24 hours pass with the head of state joining a group of contented souls moving from cloud to cloud, playing the harp and singing.

They have a good time and, before he realizes it, the 24 hours have gone by and St. Peter returns.

“Well then, you’ve spent a day in hell and another in heaven. Now choose your eternity.”

The politician reflects for a minute, then the senator answers: “Well, I would never have said it before, I mean heaven has been delightful, but I think I would be better off in hell.”

So St. Peter escorts him to the elevator and he goes down, down, down to hell. Now the doors of the elevator open and he’s in the middle of a barren land covered with waste and garbage.

He sees all his friends, dressed in rags, picking up the trash and putting it in black bags. The devil comes over to him and puts his arm around his shoulder.

“I don’t understand,” stammers the politician. “Yesterday I was here and there was a golf course and club, and we ate lobster and caviar,drank champagne, and danced and had a great time. Now all there is is a wasteland full of garbage and my friends look miserable. What happened?”

The devil looks at him, smiles and says, “Yesterday we were campaigning……Today you voted for us!”

Pic by Magnus Akselvoll

What are the big, enduring challenges in evaluation? Where are we making progress?

The ‘BetterEvaluation’ project has announced a Request for Challenges (RfC), with a particular emphasis in this round on evaluation in the areas of Capacity Development, Climate Change Adaptation and Network initiatives.

Nominations for challenges are being sought by April 30 and will help us to focus our efforts to bring together examples of good evaluation practice, and to support research and development in areas where new methods need to be developed or trialled.

The challenges might relate to any aspect of evaluation, including:

• Articulating theories of change
• Engaging stakeholders in evaluation planning, analysis and implementation
• Framing evaluation questions
• Identifying and negotiating relevant values to underpin the evaluation
• Designing evaluation plans
• Identifying meaningful and feasible indicators
• Gathering appropriate data
• Understanding direct and indirect contributions of the intervention
• Making overall judgments about the program
• Forming useful recommendations
• Facilitating the use of the evaluation results for learning;

BetterEvaluation will coordinate a process of consultation and collective problem solving addressing these challenges. In some cases, we will be able to support field testing of suggested responses.A summary of challenges and suggested responses will be posted on the BetterEvaluation website.

Please send your Challenges to bettereval@pactworld.org by April 30, 2011 with the following information:

ABOUT YOU

• Name/Position
• Organization/Affiliation
• Country
• Contact Information (phone, email)
• Program/Initiative name, duration and a brief description (5-10 sentences)

ABOUT THE CHALLENGE

• Describe your challenge and any steps you have taken to resolve this challenge so far (10-15 sentences).

BetterEvaluation is an international collaboration to improve evaluation by sharing information about how to choose and implement appropriate combinations of methods. It is being developed with the support of the Rockefeller Foundation and IFAD (International Fund for Agricultural Development) by a diverse community who share a commitment to situationally responsive evaluation. Founding partners are: RMIT University (Royal Melbourne Institute of Technology, Australia); Pact (Washington D.C, USA and Bangkok, Thailand offices); ILAC (Institutional Learning and Change initiative of the Consultative Group on International Agriculture, Italy); ODI (Overseas Development Institute, UK).

For more information about BetterEvaluation please check the current site www.betterevaluation.org. We are currently in the beta-phase of development and will open the full site in summer 2011.