Our guest blogger this week is Katherine Hay, a senior member of the Evaluation Unit of the International Centre for Development Research. Based in New Delhi, India, she is an expert on the role of evaluation in development in South Asia. She promotes approaches that assess how women and other marginalized groups benefit from development in the region. Katherine joined IDRC’s South Asia office in New Delhi in 2000 and has undertaken research in South Asia for more than 15 years. Her work with IDRC includes building evaluation curriculum in universities in the region, and supporting evaluation communities of practice spanning South Asia and Afghanistan. She has written on women’s empowerment, evaluation, and the policy research environment in South Asia. Katherine holds a master’s degree in international affairs from Carleton University in Ottawa. Katherine is sharing with us perspectives from her recent keynote address to the conference of the Sri Lankan Evaluation Association.

In reading the newspapers lately, I’ve noticed an increasing expectation that evidence can give us the answers that policy makers need. I practice evaluation because I believe that evaluation can help identify what is working from what is not working, and for whom.  So I should be pleased to see these calls for “the evidence.”   I am….and yet, I am also somewhat alarmed by this faith in data.

Some people seem to suggest that if we would just get enough evidence we will be able to ‘fix’ poverty.  I think that is both naïve and dangerous. In the New York Times, Nicolas Kristoff had a piece, called “Getting Smart on Humanitarian Aid,” where he said: “How can we most effectively break cycles of poverty? For decades, we had answers that were mostly anecdotal or hot air. But, increasingly, economists provide answers that are rigorously field-tested.” That sounds good but do we really have answers, and to what?

The evidence that Kristoff was pointing to drew on the excellent work of Duflo and Banerjee on randomized controlled trials.  Kristoff, and a string of other journalists, came to the conclusion that “we now have the answers” based on 2-3 examples that included the cost effectiveness improving school attendance by deworming kids and providing them with school uniforms.  I’ve read the studies.  I’m pretty convinced that schools should deworm and that school uniforms in Africa are probably worth the money. But do education policy makers now have all the answers whereas before they just had ‘hot air?’  Not quite.

These are fairly simple interventions.  I don’t doubt that they are helpful. But idea that we have all the evidence we need or can get it through trials, is not helpful.  It dumbs down development problems by arguing that, until now, everyone working in development has been running around with no clue.  It suggests that governments, implementing agencies, funding agencies, just need to run some experiments to find out what the policy should be. It’s a simple idea.  But poverty and development are complex.

There is nothing wrong with experiments.  The right tool in any situation is the one that best answers the questions being asked.  My critique is of the idea that development is just about getting the data right, or that evidence ‘neutral’ or has nothing to do with politics.

Why is this a dangerous idea? Kristoff goes on to suggest that “For those who want to be sure, to get the most bang for your buck, there is also a “proven impact fund” that supports interventions like deworming…that have proved to be cost-effective in rigorous trials. But what would happen if we only  fund the proven, cost effective things, the sure things?  It’s hard to be sure about many things that matter.

Funding only the sure things would certainly rule out a great deal of things that many of us think are important including work to address:  climate change, violence against women, son preference, human rights, or conflict.   Much of this work takes generations to see results and is deeply contextual; in many of these areas we don’t have ‘sure things.’

Debates about ways of investigating cause and effect have been a feature of many international evaluation meetings in recent years.  It’s something I’ve been wrestling with while finishing a forthcoming book on program theory with Sue Funnell (Purposeful Program Theory: Effective Use of  Theories of Change and Logic Models, to be published by Jossey-Bass).

Pic by ralphbijker http://www.flickr.com/photos/17258892@N05/2588347668/

Jane raised the issue of causal inference in a post back in February, in a recent presentation and her book Evaluation Methodology Basics Chapter 5 on Causation. By causal inference we mean both causal attribution (working out what was THE cause) and causal contribution (identifying what was one or more of the causes that together produced the outcomes and impacts).

We both agree that it is an issue that needs to be tackled in evaluation, in ways that are commensurate with the available evaluation resources, and that don’t assume it’s simply a matter of using a particular research design (such as using Randomised Controlled Trials) or data collection method (such as stakeholder interviews).

Where we have taken a different tack is in strategies for causal inference.

Jane’s take on causal inference for evaluation

Jane recently (in her book, Evaluation Methodology Basics) shared a list of 8 strategies for causal inference that can be used for a ‘patchwork’ or bricolage approach to causal inference:

1. Ask those who have observed or experienced the causation first-hand
2. Check if the content of the intervention (or, supposed cause) matches the nature of the outcome
3. Look for distinctive effect patterns (Scriven’s modus operandi method)
4. Check whether the timing of the outcomes makes sense
5. Look at the relationship between “dose” and “response”
6. Use a comparison or control (RCTs or quasi-experimental designs)
7. Control statistically for extraneous variables
8. Identify and check the causal mechanisms

My take on causal inference for program theory evaluations

Looking at causal inference for program theory evaluations, I’ve been thinking more broadly about three components of causal inference:

1. Congruence with the program theory – do the results match the program theory?
2. Counterfactual comparisons – what would have happened without the intervention?
3. Critical review – what are plausible alternative explanations for the results?

Combining the two frameworks

Combining these two framings of the issue has identified some additional techniques that should be added to the repertoire (such as techniques that compare results with those predicted by statistical models of the theory or the predictions of experts) and explains why experimental design by itself is inadequate for causal inference – this deals with the counterfactual but may not provide sufficient information on the factual (what happened?) and needs to be complemented by critical review. Some techniques (such as asking participants) can provide all three types of evidence, although none by itself will be sufficient.

A bricolage approach, covering these three different components, with triangulation of sources, seems to be what is needed for really credible causal inference.

What do you think? (And, yes, the cogs in the picture show what happens when there is too much interaction between causal forces).

from: Purposeful Program Theory: Effective Use of  Theories of Change and Logic Models, by Patricia J. Rogers and Sue C. Funnell, ISBN: 9780470478578, John Wiley/Jossey-Bass, (in press).

A recent RCT in Ghana that found Vitamin A supplementation did not lead to reduced maternal mortality ( a finding very different to a study in Nepal that found a 44% reduction) has raised some important questions for me about how well clinical trial procedures match with our understanding of using program theory in evaluation – in particular whether implementation failure or differential impacts might explain the findings rather than the ineffectiveness of the program.

http://www.flickr.com/photos/dfid/4420202860 CC BY-NC-ND 2.0

Here are the main findings, according to the summary published in The Lancet, Volume 375, Issue 9726, Pages 1640 – 1649, 8 May 2010

GhanWeb reported it as follows:

Accra, April 27, GNA – A study conducted in Kintampo by a team of experts has shown beyond doubt that taking Vitamin A does not reduce the risk of death in pregnancy or childbirth.

By contrast, Reuters correctly reported what had been tested:

Giving vitamin A to women aged between 15 and 45 in poor nations does not cut maternal death rates, scientists said Tuesday in a study which contradicts earlier research showing a dramatic drop in death rates.

There is an important debate going on about the use of ITT analysis.  Clearly it is important to know the end result of a policy change – but it is also essential to understand whether a lack of effects is due to implementation failure or theory failure.

Gerard Dallal has set out an interesting analysis of this issue which focuses on the fact that ITT analysis is appropriate for answering the question “What happens once a treatment is started or recommended?” but it does not answer the question about whether it is effective for those who follow the treatment.

For public policy issues, we need to know the latter – and if there is a big difference between the two we need to investigate ways to improve adherence.  This is what a group of researchers have advocated for studies of the effectiveness of microbicides for HIV prevention, where adherence is an important issue (and very hard to measure).

At the very least we need to be clear in the reporting of findings whether or not there was an effect for those compliant with the treatment. But even better would be to go beyond a simple summative judgement (did the policy work ?) and look at whether there is scope to improve it.

Were those who were most likely to benefit from the treatment screened out of the study?

Who would be most likely to benefit from vitamin A supplementation?  Women who had vitamin A deficiency of course.  Ideally the study would have been able to report on whether or not the results were different for those with vitamin A deficiencies as the effects might be lost in the noise from the total population.

The information about the trial recorded at http://clinicaltrials.gov had this surprising statement:

There will be no exclusions to participation, except for women who have nightblindness or other signs of VAD. These, and any women who develop VAD in the course of the study will be treated according to current IVACG recommendations (IVACG, 1997). They will continue to be followed, but will be given vitamin A and considered separately in the analysis.

These women might have been considered separately in the analysis, but their participation (or exclusion) and outcomes have not been reported in the summaries of the project.  If they are being excluded from the study on ethical grounds, to avoid them being denied a proven treatment for Vitamin A deficiency, wouldn’t that reduce the likelihood of the study producing significant effects – not because Vitamin A is ineffective, but because those for whom it would have made a difference were not included in the study?

I’d be very interested to hear more about this study, and whether the full report addresses these issues.  And of course I’ll be following  how these findings are reported.

When a large and expensive evaluation fails to produce useful results, it’s worth seeing if at least it can be useful as a cautionary tale.

Blueprint is a UK Government-funded drugs education programme consisting of five components: drug education in schools (for 11 and 12-year-olds – this was the main emphasis), drug education for parents, media coverage of Blueprint, partnerships with other key health policy stakeholders, and a set of shared principles for drug education across prevention practitioners in the community. Blueprint was developed to support the UK Government target to ‘reduce the use of Class A drugs and the frequent use of any illicit drugs among all young people under the age of 25, especially by the most vulnerable people’.

According to BBC home editor Mark Easton’s blog, the evaluation budget for this 5-year longitudinal study was £6 million. Surely it’s reasonable to expect a genuine evaluation that actually answers important questions when you’ve got that kind of budget – right?

Right … Check out this quote from the Blueprint evaluation report (p. 71):

Measuring Impact and Outcomes

It was originally intended that this report would describe the impacts and outcomes of the Blueprint programme. As described in the delivery report (Stead 2007), the six local schools were to act as a comparator to the 23 Blueprint schools, so that conclusions could be drawn on the efficacy of the programme. However, when analysis during the early stages of the evaluation concluded that to detect differences between the two samples would require a sample of at least 50 schools, it was decided that the implementation of the programme would become the main focus. Nonetheless, it was still planned that the local school data would be presented alongside the Blueprint school data, to enable some comparisons to be drawn between the two samples. However, a recent review concluded that to present the data in this way would be misleading, given that the two samples are not matched, and to make comparisons between the two samples or to draw conclusions on the efficacy of the programme, based on these findings, would be wrong. While it is disappointing that this stage of the evaluation has not met expectations, the evaluation has looked in detail at how the programme was received and provides valuable insights for future initiatives of this kind.

The lack of an adequate comparison sample is a basic and foreseeable flaw in the research that was picked up in the EARLY stages of the evaluation. Why wasn’t this rectified at the time by adding more schools to the comparison sample? How could a process evaluation (looking only at ‘implementation’) possibly be worth £6 million of taxpayers’ money as a knowledge and insight-generating exercise?

Where was the client (presumably the UK Home Office) when these discoveries were appearing? Why weren’t they insisting on a design that would actually answer what they needed to know? And if the evaluation team couldn’t deliver within such a substantial budget, what possessed them to allow that much money to continue to be spent on such a low-value evaluation?

Even if the size of the comparison sample was inadequate for the intended design to be the SOLE basis for inferring causation, what consideration was given to complementing the less-than adequate comparison data with other causal inference methods? With a bit of creative thinking, it’s possible that a less-than-adequate comparison could have been strengthened with some quantitative, qualitative, and mixed-method strategies so that there might at least have been an approximate answer to the causal inference/outcome-related questions.

The acknowledgments page presents a long list of 44 researchers (are we surprised they don’t list themselves as evaluators?) from seven universities and research organisations who worked on the Blueprint evaluation.

All this strikes me as a classic case of four themes I frequently see when evaluation has been a waste of time and money:

1. Hiring researchers to do an evaluators’ job – in my experience, those whose career focus is research usually lack the necessary expertise in evaluation to be able to scope the work adequately to get the most important questions answered and weed out the “wouldn’t it be nice to know” lines of inquiry. The fault lies on both sides – the client’s (it’s like hiring an electrician to do your plumbing, or vice versa – and saying “how was I to know there was a difference?” won’t cut it), and the researchers’ (for presenting themselves as being sufficiently skilled in something they are not).
2. Setting the evaluation plan in concrete way too early – an evaluation needs to retain a level of fluidity and flexibility in its design so that it can be changed and improved midstream if found to be lacking, or if the stakeholders’ needs or questions change. This fluidity is often in conflict with the traditional research approach, particularly when the research team is hoping to get academic publications out of the work, because of the traditional emphasis on the exact same measures being used, and the timing of measures being identical for experimental and comparison groups. Of course a pure Randomised Controlled Trial design (with random assignment to the treatment group or the control group) does require this to be done up front, but in some cases a moderate loss of design purity can be an acceptable price to pay for better questions, measures, comparison opportunities, or other causal evidence.
3. The “more the merrier” approach to evaluation team composition – too many people and organisations involved, which often means too much diffusion of responsibility and expertise, and (most importantly) the people who might have the right expertise are frequently not involved at critical stages of evaluation design, so that less qualified individuals make decisions that are flawed and expensive to reverse.
4. Failing to pull the plug when the evaluation is clearly not going to deliver sufficient valuable insights – According to the UK Telegraph newspaper, academic advisers to the Home Office told one of the ministers overseeing the evaluation that it would be a waste of public funds. An email sighted by the BBC from one of the advisers, Professor Sheila Bird, said: “I/we thought the decision-making so obvious = NOT to go ahead that we did not assiduously follow-up to ensure that the obvious decision was actually made!”

What’s particularly irritating to me (especially as a taxpayer) is that these same mistakes are made over and over again, often by different parts of the same organisation. If client organisations – those who commission evaluation – don’t insist on value-for-money genuine evaluation, they can expect more of the same in the future.

Curious post by Tim Harford in the Financial Times recently “Political ideas need proper testing”  As is often the case, his argument slid from advocating for better empirical investigation of public policy by systematic experimentation to discussing this only in terms of RCTs (Randomised Controlled Trials). I was reminded of  Michael Scriven’s critique of the appropriation of the word ‘experimental’ (in a recent issue of the Journal of MultiDisciplinary Evaluation)

… the RCT campaign also involves the less-remarked parallel effort, going back further, to redefine the concept of an experiment. In standard scientific usage, experiments are just carefully constrained explorations, and the RCT is simply a special case of these. To call the RCT the only “true experiment” is part of an attempt at redefinition that distorts the original and continuing usage, and excludes experiments designed to test many simple hypotheses about—or simple efforts to find out—what happens if we do this.
This effort at persuasive redefinition is allied with an implicit denigration of the so-called “quasi-experimental” designs, which are in fact perfectly respectable experiments, only ‘quasi’ with respect to the one respect in which they have less control over one possible way of excluding one type of alternative explanation.

What makes the recent post particularly surprising is that he is advocating for RCTs through an example which shows clearly the value of drawing on other types of evidence – the changing advice on infant sleeping positions to reduce the risk of Sudden Infant Death Syndrome.  No RCTs were used to investigate the effectiveness of these and yet adequate evidence was produced to be able to make effective changes in policy.

I recently used the same example to show the value and feasibility of using a range of credible evidence about effectiveness, including non-experimental designs.  It will be published shortly  by the Productivity Commission in proceedings from a Roundtable on Strengthening Evidence-based Policy in the Australian Federation.

Non-RCT data can provide good quality evidence of effectiveness

Good quality evidence of effectiveness can also come from quasi-experimental approaches, which compare program participants to a comparison group rather than to a randomly assigned control group, and from non-experimental approaches, when such approaches systematically and rigorously test causal conclusions and combine evidence thoughtfully.

Sudden Infant Death Syndrome (SIDS) is one of two exemplars in the National Health and Medical Research Council guide How to Put the Evidence into Practice: Implementation and Dissemination Strategies (NHMRC 2000).

It shows both the value of drawing on a diverse set of evidence and how it is possible to develop effective policy even when the evidence is not definitive. Bringing together evidence from many studies, including retrospective and prospective epidemiological studies, pathological studies and case studies, a number of possible contributing factors were identified, and other possible causes (such as vaccinations) were ruled out.

On the basis of this incomplete evidence, recommendations were developed — to put babies to sleep on their backs, avoid overheating and avoid cigarette smoke. No RCTs were used to test the effectiveness of these recommendations. The recommendations were communicated directly to parents and to health professionals working with parents, resulting in widespread change in the sleeping positions they used for infants.

By 2005, the number of SIDS deaths had been reduced to fewer than 100, a decline of 83 per cent (ABS 2007).