£6 million over 5 years – and STILL no genuine evaluation of Blueprint?

When a large and expensive evaluation fails to produce useful results, it’s worth seeing if at least it can be useful as a cautionary tale.

Blueprint is a UK Government-funded drugs education programme consisting of five components: drug education in schools (for 11 and 12-year-olds – this was the main emphasis), drug education for parents, media coverage of Blueprint, partnerships with other key health policy stakeholders, and a set of shared principles for drug education across prevention practitioners in the community. Blueprint was developed to support the UK Government target to ‘reduce the use of Class A drugs and the frequent use of any illicit drugs among all young people under the age of 25, especially by the most vulnerable people’.

According to BBC home editor Mark Easton’s blog, the evaluation budget for this 5-year longitudinal study was £6 million. Surely it’s reasonable to expect a genuine evaluation that actually answers important questions when you’ve got that kind of budget – right?

Right … Check out this quote from the Blueprint evaluation report (p. 71):

Measuring Impact and Outcomes

It was originally intended that this report would describe the impacts and outcomes of the Blueprint programme. As described in the delivery report (Stead 2007), the six local schools were to act as a comparator to the 23 Blueprint schools, so that conclusions could be drawn on the efficacy of the programme. However, when analysis during the early stages of the evaluation concluded that to detect differences between the two samples would require a sample of at least 50 schools, it was decided that the implementation of the programme would become the main focus. Nonetheless, it was still planned that the local school data would be presented alongside the Blueprint school data, to enable some comparisons to be drawn between the two samples. However, a recent review concluded that to present the data in this way would be misleading, given that the two samples are not matched, and to make comparisons between the two samples or to draw conclusions on the efficacy of the programme, based on these findings, would be wrong. While it is disappointing that this stage of the evaluation has not met expectations, the evaluation has looked in detail at how the programme was received and provides valuable insights for future initiatives of this kind.

The lack of an adequate comparison sample is a basic and foreseeable flaw in the research that was picked up in the EARLY stages of the evaluation. Why wasn’t this rectified at the time by adding more schools to the comparison sample? How could a process evaluation (looking only at ‘implementation’) possibly be worth £6 million of taxpayers’ money as a knowledge and insight-generating exercise?

Where was the client (presumably the UK Home Office) when these discoveries were appearing? Why weren’t they insisting on a design that would actually answer what they needed to know? And if the evaluation team couldn’t deliver within such a substantial budget, what possessed them to allow that much money to continue to be spent on such a low-value evaluation?

Even if the size of the comparison sample was inadequate for the intended design to be the SOLE basis for inferring causation, what consideration was given to complementing the less-than adequate comparison data with other causal inference methods? With a bit of creative thinking, it’s possible that a less-than-adequate comparison could have been strengthened with some quantitative, qualitative, and mixed-method strategies so that there might at least have been an approximate answer to the causal inference/outcome-related questions.

The acknowledgments page presents a long list of 44 researchers (are we surprised they don’t list themselves as evaluators?) from seven universities and research organisations who worked on the Blueprint evaluation.

All this strikes me as a classic case of four themes I frequently see when evaluation has been a waste of time and money:

1. Hiring researchers to do an evaluators’ job – in my experience, those whose career focus is research usually lack the necessary expertise in evaluation to be able to scope the work adequately to get the most important questions answered and weed out the “wouldn’t it be nice to know” lines of inquiry. The fault lies on both sides – the client’s (it’s like hiring an electrician to do your plumbing, or vice versa – and saying “how was I to know there was a difference?” won’t cut it), and the researchers’ (for presenting themselves as being sufficiently skilled in something they are not).
2. Setting the evaluation plan in concrete way too early – an evaluation needs to retain a level of fluidity and flexibility in its design so that it can be changed and improved midstream if found to be lacking, or if the stakeholders’ needs or questions change. This fluidity is often in conflict with the traditional research approach, particularly when the research team is hoping to get academic publications out of the work, because of the traditional emphasis on the exact same measures being used, and the timing of measures being identical for experimental and comparison groups. Of course a pure Randomised Controlled Trial design (with random assignment to the treatment group or the control group) does require this to be done up front, but in some cases a moderate loss of design purity can be an acceptable price to pay for better questions, measures, comparison opportunities, or other causal evidence.
3. The “more the merrier” approach to evaluation team composition – too many people and organisations involved, which often means too much diffusion of responsibility and expertise, and (most importantly) the people who might have the right expertise are frequently not involved at critical stages of evaluation design, so that less qualified individuals make decisions that are flawed and expensive to reverse.
4. Failing to pull the plug when the evaluation is clearly not going to deliver sufficient valuable insights – According to the UK Telegraph newspaper, academic advisers to the Home Office told one of the ministers overseeing the evaluation that it would be a waste of public funds. An email sighted by the BBC from one of the advisers, Professor Sheila Bird, said: “I/we thought the decision-making so obvious = NOT to go ahead that we did not assiduously follow-up to ensure that the obvious decision was actually made!”

What’s particularly irritating to me (especially as a taxpayer) is that these same mistakes are made over and over again, often by different parts of the same organisation. If client organisations – those who commission evaluation – don’t insist on value-for-money genuine evaluation, they can expect more of the same in the future.