Stuart Henderson’s mention of The Two Second Advantage (see the LinkedIn discussion referred to in the post from earlier this week) reminds me of the work of business strategist Arie De Geus, who discusses how learning organizations use scenario planning to create “memories of the future”.

It seems to me that this idea has potential not only in looking at emergent trends in evaluation (such as those Stuart is asking about on LinkedIn), but also as a lens we can apply to programs and organizations to help them consider how to ‘future-proof’ themselves.

De Geus points out that, although we can’t predict exactly what changes will happen in the future, it is possible to brainstorm a few different possible scenarios about what might happen.

Some scenarios will be more likely than others – some will seem almost certain; some will seem incredibly unlikely; some may incorporate natural disasters or substantial shifts in the world or in the community (either gradual or dramatic).

The scenarios may be developed – minimally or extensively, depending how likely the scenario seems – into contingency plans (i.e., What would we do if …).

The idea behind scenario planning is this:

• We know from psychology that the human brain is more likely to spot the early signs of change if the person has rehearsed the possibility beforehand. [I am sure some clever person has a citation for this!]

• Likewise, organizations are more likely to perceive change early – and have ideas for how to react effectively to take advantage of it – if they have gone through the process of dreaming up the various possible future scenarios.

Now, how might this help programs and organizations build strength and sustainability?

Think of the kinds of major and minor changes that could have major impacts on what the organization does or how it does it. Examples might include:

• A key funding source completely disappearing or dramatically reducing
• An immigration policy altering the demographics (and therefore the needs) of the population they serve
• A new technology enabling them to access remote areas, or making some aspect of their services obsolete
• A natural disaster such as an earthquake, volcanic eruption, or flood
• Two or three key people leaving the organization

Evaluations often have to answer questions about the sustainability, transferability, or applicability of programs, services, and activities over time, over space, over borders. One way of doing this is to help stakeholders think of possible scenarios and how well equipped the program or organization or community is to handle them.

Scenario planning is a tool that, as De Geus says, helps organizations create “memories of the future” that can help them detect the early signs of even unlikely change and to be better equipped to respond to it.

I am sure the folks involved in disaster-preparedness and emergency evaluation have a lot to say on this topic, as will those working in countries where political volatility is high and those with an interest in complexity and unpredictability – please chime in and share your thoughts!

Katherine Hay continues her guest blogging on evidence and evaluation.

Ben Goldacre in The Guardian wrote that UK politicians “are ignorant about trials and they’re weird about evidence.” He contrasts this with international development where he talks about the “amazing work testing interventions around the world with proper, randomised trials.”  He goes on to say, policy makers in the UK, just need to “define your outcome, randomise…and you’ll have the answer by the end of next parliament.”  He notes all these trials (somehow) won’t cost money but will save unprecedented amounts of money.  He then concludes that, “politicians are …too arrogant to have their ideologies questioned, and too scared…of hard data on their interventions.’

It’s an entertaining article.  The idea of doing a trial on every single UK policy is funny.  The idea that they are free is even funnier.  Imagine how British parents would react when they brought their kids to school and were told what group their child was randomly selected to be in?  Perhaps:

In a large class but with a highly rated teacher.

• In a small class but with a less experienced teacher.
• In a small class with no hot lunch…

And so on.   The permutations to test every UK policy would be never-ending.

Goldacre is obviously being extreme to make a point.  But is he correct?  Have countries who conducted randomized trials saved huge amounts on their interventions?

I’ve seen no evidence that trials are more likely to inform policies than other evaluations or research. I expect that they are subject to the same challenges of use as other types of evidence.

If we accept that “working” can mean different things to different groups, and that views on what is ‘worth the money’ usually vary based on people’s values and their position in society, than why would we assume that studies with statistical power will lead to change on the ground? Evaluation can give us more evidence – and must give us better quality evidence – but the idea that policy making is just a computation of evidence is wrong.   Evidence is only one piece of policy making.  Evidence can, and often is, interpreted and used to reinforce dominant policies.

For example, the country where I live, India, has a system that distributes grains to the poor.  Some people think this system should be replaced with a system where cash is giving to poor families who can buy the food or grains that they choose.  Others feel that dismantling this system will mean food grain that was getting to the poor and children will be replaced by spending on things like alcohol.  Different groups have done studies on whether people want this change.  Some studies show that people do and some studies show that people don’t.

Part of the solution is about design.  You have to be confident that the evidence you have is good quality.  Do people really want it or not? But it’s not just about design.  Even with convincing findings, the policy maker has multiple elements to weigh. In New Delhi, the capital, the government decided that they wanted to experiment with the cash transfer.  But was it because they were comfortable having their ‘ideology tested’ or because their ideology lead them to prefer such a system? They were criticized for the latter; for having a position.  But they were elected on their positions; pushing for more open reality testing is not about wishing away positions.

If they get the design right they may know pretty accurately how many families want or do not want changes in this system and if they run a trial well they may also know about some outcomes.  But that data won’t tell them the ‘right choice’.  For example, how much of an increase in alcohol consumption is ok, or is trumped by increasing the poor’s control of spending choices, or increased efficiencies?  Those decisions are values based and values are often political. For example, Abhijit Sen, a noted economist and member of the Indian Planning Commission, noted “politicians will never accept a dismantling of the PDS’ but added, “Forget the politicians, what matters most is what the voters think.”

We cannot wish away politics and nor should we want to.  My point is that we need to get much more strategic on pathways to use if we want to influence policy with evaluation.

Let me give you a wonderful final little example. Two PhD candidates from Yale did an experiment in a New Delhi slum. The subjects wanted to apply for ration cards. They were randomly assigned to one of four groups. The first group applied for the ration card and did nothing more, the second attached a letter of recommendation from an NGO to their application, the third paid a bribe after putting in their application, and the fourth enquired about the status of their ration card application through a right to information (RTI) request.  The researchers found that the group that paid a bribe was the most successful, but, the group that put in an RTI request was almost as successful. Hardly anyone in the other two groups received their ration card.

Clever experiment.  They answered an interesting question and will likely get their PHD’s in the process.

But corruption in the ration card system is not fixed. Also, NGO’s and others were already using the RTI for things exactly like this.  This experiment adds more evidence to existing evidence that the RTI Act is a useful tool against corruption.  Is it THE answer?  No.  Is it helpful?  Yes.

Assuming that evidence alone will change things is wrong.  Evidence matters, and should be made to matter more, but it’s not the only thing that matters.  Recognizing this doesn’t weaken evaluation, quite the opposite, it actually creates greater opportunities for ‘genuine evaluation’

Our guest blogger this week is Katherine Hay, a senior member of the Evaluation Unit of the International Centre for Development Research. Based in New Delhi, India, she is an expert on the role of evaluation in development in South Asia. She promotes approaches that assess how women and other marginalized groups benefit from development in the region. Katherine joined IDRC’s South Asia office in New Delhi in 2000 and has undertaken research in South Asia for more than 15 years. Her work with IDRC includes building evaluation curriculum in universities in the region, and supporting evaluation communities of practice spanning South Asia and Afghanistan. She has written on women’s empowerment, evaluation, and the policy research environment in South Asia. Katherine holds a master’s degree in international affairs from Carleton University in Ottawa. Katherine is sharing with us perspectives from her recent keynote address to the conference of the Sri Lankan Evaluation Association.

In reading the newspapers lately, I’ve noticed an increasing expectation that evidence can give us the answers that policy makers need. I practice evaluation because I believe that evaluation can help identify what is working from what is not working, and for whom.  So I should be pleased to see these calls for “the evidence.”   I am….and yet, I am also somewhat alarmed by this faith in data.

Some people seem to suggest that if we would just get enough evidence we will be able to ‘fix’ poverty.  I think that is both naïve and dangerous. In the New York Times, Nicolas Kristoff had a piece, called “Getting Smart on Humanitarian Aid,” where he said: “How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, economists provide answers that are rigorously field-tested.” That sounds good but do we really have answers, and to what?

The evidence that Kristoff was pointing to drew on the excellent work of Duflo and Banerjee on randomized controlled trials.  Kristoff, and a string of other journalists, came to the conclusion that “we now have the answers” based on 2-3 examples that included the cost effectiveness improving school attendance by deworming kids and providing them with school uniforms.  I’ve read the studies.  I’m pretty convinced that schools should deworm and that school uniforms in Africa are probably worth the money. But do education policy makers now have all the answers whereas before they just had ‘hot air?’  Not quite.

These are fairly simple interventions.  I don’t doubt that they are helpful. But idea that we have all the evidence we need or can get it through trials, is not helpful.  It dumbs down development problems by arguing that, until now, everyone working in development has been running around with no clue.  It suggests that governments, implementing agencies, funding agencies, just need to run some experiments to find out what the policy should be. It’s a simple idea.  But poverty and development are complex.

There is nothing wrong with experiments.  The right tool in any situation is the one that best answers the questions being asked.  My critique is of the idea that development is just about getting the data right, or that evidence ‘neutral’ or has nothing to do with politics.

Why is this a dangerous idea? Kristoff goes on to suggest that “For those who want to be sure, to get the most bang for your buck, there is also a “proven impact fund” that supports interventions like deworming…that have proved to be cost-effective in rigorous trials. But what would happen if we only  fund the proven, cost effective things, the sure things?  It’s hard to be sure about many things that matter.

Funding only the sure things would certainly rule out a great deal of things that many of us think are important including work to address:  climate change, violence against women, son preference, human rights, or conflict.   Much of this work takes generations to see results and is deeply contextual; in many of these areas we don’t have ‘sure things.’

Personnel evaluation is always a rich source of evaluation humor – and often with interesting parallel implications for program, policy and project evaluation. Here’s a classic that’s done the rounds on the Internet; we found this one at Pinetree.com’s inbox humor file.

We’ve all been interviewed for jobs. And, we’ve all spent most of those interviews thinking about what not to do. Don’t bite your nails. Don’t fidget. Don’t interrupt. Don’t belch. If we did any of the don’ts, we knew we’d disqualify ourselves instantly. But some job applicants go light years beyond this. We surveyed top personnel executives of 100 major American corporations and asked for stories of unusual behavior by job applicants.

The lowlights:

Said he was so well qualified [that] if he didn’t get the job, it would prove that the company’s management was incompetent.

Stretched out on the floor to fill out the job application.

Brought her large dog to the interview.

Chewed bubble gum and constantly blew bubbles.

Candidate kept giggling through serious interview.

She wore an ipod and said she could listen to the music and me at the same time.

Balding candidate abruptly excused himself. Returned to office a few minutes later wearing a hairpiece.

Applicant challenged interviewer to arm wrestle.

Asked to see interviewer’s resume to see if the personnel executive was qualified to judge the candidate.

Announced she hadn’t had lunch and proceeded to eat a hamburger and French fries in the interviewer’s office.

Without saying a word, candidate stood up and walked out during the middle of the interview.

Man wore jogging suit to interview for position as financial vice president.

Said if he were hired, he would demonstrate his loyalty by having the corporate logo tattooed on his forearm.

Interrupted to phone his therapist for advice on answering specific interview questions.

Wouldn’t get out of the chair until I would hire him. I had to call the police.

When I asked him about his hobbies, he stood up and started tap dancing around my office.

Had a little pinball game and challenged me to play with him.

Bounced up and down on my carpet and told me I must be highly thought of by the company because I was given such a thick carpet.

Took a brush out of my purse, brushed his hair and left.

Pulled out a Polaroid camera and snapped a flash picture of me. Said he collected photos of everyone who interviewed him.

Candidate asked me if I would put on a suit jacket to insure that the offer was formal.

Said he wasn’t interested because the position paid too much.

While I was on a long-distance phone call, the applicant took out a copy of Penthouse, and looked through the photos only, stopping longest at the centerfold.

During the interview, an alarm clock went off from the candidate’s briefcase. He took it out, shut it off, apologized and said he had to leave for another interview.

A telephone call came in for the job applicant. It was from his wife. His side of the conversation went like this: “Which company? When do I start? What’s the salary?” I said, “I assume you’re not interested in conducting the interview any further.” He promptly responded, “I am as long as you’ll pay me more.” I didn’t hire him, but later found out there was no other job offer. It was a scam to get a higher offer.

An applicant came in wearing only one shoe. She explained that the other shoe was stolen off her foot in the bus.

His attache [case] opened when he picked it up and the contents spilled, revealing ladies’ undergarments and assorted makeup and perfume.

He came to the interview with a moped and left it in the reception area. He didn’t want it to get stolen, and stated that he would require indoor parking for the moped.

He took off his right shoe and sock, opened a medicated foot powder and dusted it on the foot and in the shoe. While he was putting back the shoe and sock, he mentioned that he had to use the powder four times a day, and this was the time.

Candidate said he really didn’t want to get a job, but the unemployment office needed proof that he was looking for one.

He whistled when the interviewer was talking.

Asked who the lovely babe was, pointing to the picture on my desk. When I said it was my wife, he asked if she was home now and wanted my phone number. I called security.

She threw up on my desk, and immediately started asking questions about the job, like nothing had happened.

Pointing to a black case he carried into my office, he said that if he was not hired, the bomb would go off. Disbelieving, I began to state why he would never be hired and that I was going to call the police. He then reached down to the case, flipped a switch and ran. No one was injured, but I did need to get a new desk.

Asked if I wanted some cocaine before starting the interview.

I’m hopeful that the current controversy over Greg Mortenson’s book ‘Three Cups of Tea’ and the operations of the related NGO ‘Central Asia Institute’ (detailed in John Krakauer’s book Three Cups of Deceit and a 60 minutes story)  will lead to improvements in how development interventions and organizations are planned, implemented, evaluated and reported.  There are risks, however, that it could make it more difficult to raise funding in the areas of operation (girls’ schooling, Afghanistan, Pakistan).  And that it could provide further reasons for dismissing discussions about development evaluation that draw on complexity language.

In response to the criticisms raised by 60 minutes and by author,  Greg Mortenson, in a letter to supporters on the IKAT website, said:

Although we would like the world to be linear, orderly and peaceful, the reality is that our world is a dynamic, fluid place, often filled with chaos and confusion. In that space, I thrive and get the courage to help bring change and empower people. …

The “60 Minutes” program may appear to ask simple questions, but the answers are often complex, not easily encapsulated in 10-second sound bites.
Working in isolated areas, in communities that are not on any map, and often in areas of turmoil, religious extremism or natural disasters where education is still relatively rare and ancient codes of conduct and social hierarchies still dominate – all these things demand constant adjustment, accommodation and patience.

It’s interesting to analyze the case using the different lenses of simple, complicated, complex.

There appear to be simple aspects of this that should have been managed better .  Watch where the money goes, and make sure that most of it is going to the central activities (in this case, the description of the CAI on its financial accounts talks only about its international education programs, even though more money goes to the US ‘outreach’ book tours).  Check if the schools have been built, and are being used.

There appear to be complicated aspects that should have been managed better.  Schools need bulldings plus teachers plus course material.  Although these are listed as funded activities, it seems there has not been enough attention to ensuring the full ‘causal package’ is in place.

And, despite the language of complexity, there appear to be complex aspects that should have been managed better.  Since complex situations are unpredictable and emergent, it requires adaptive and responsive management.  Processes are needed to spot problems and work out how to fix them, and to spot successes and work out how to amplify them.  Instead, in Krakauer’s book, many staff and board members have recounted their experiences of finding problems and trying unsuccessfully to have them addressed.

Dealing with complexity effectively does not mean ad hoc or laissez-faire actions and reactions.  Nor does it mean ignoring the simple or complicated elements of the situation.

About a month ago, I wrote a post that wondered whether evaluation was really useful for accountability. The issues being raised about the Central Asia Institute have made me rethink this a little.  In that post I suggested that evaluation could usefully assess whether the management had displayed ‘smart accountability’ – being aware of what was going on, and adapting as necessary.  But, since most evaluators are not well trained or experienced in checking accountability for inputs, I suggested that auditors might do a better job of this.

However, the 2009 Audited Financial Statement for CAI, shows the limitations of audit processes:

Use of Estimates
Preparation of financial statements in conformity with accounting principles generally accepted in the United States of America requires the use of management’s estimates. Actual results of operations may differ from those estimates.

While evaluation needs to do more than see where the money has gone, how many activities have been undertaken, and how many outputs have been produced, it is important to have reasonable evidence of these – both for external accountability and to support informed, adaptive management that can truly respond to complexity.