Building on an earlier discussion Michael Scriven started about long-term effects (what to with them and without them), I’m interested in people’s thoughts on a related issue.

In the medical profession in particular, there are some very rigid beliefs about what constitutes good enough “evidence of effectiveness” to justify offering, recommending, allowing patients to try, or even just not vehemently opposing a particular type of treatment for a patient. [There are obviously some parallels in other sectors, such as education, social services, international development, criminal justice, etc, but let's start with some medical examples for now.]

There are some glimmers of hope in other sectors (e.g. in the Best Evidence Synthesis work here in New Zealand). But there are still three areas where there are very serious challenges in building a credible evidence base given the kinds of constraints and realities surrounding them. They are: (1) cutting-edge treatments;  (2) treatments that are by their very nature tailored/individualized rather than standardized across patients or populations; and (3) learning what works for small sub-populations.

1. Cutting-edge treatments

Advancements are being made in medical practice all the time, and many of these are initially developed by clinicians (doctors, specialists, surgeons) trying a new approach on a limited number of patients, e.g. when the standard treatments are either not working, or when there’s a plausible idea about how to improve benefits for patients.

In order for a new idea to be trialled on a larger scale, it must be picked up by individuals with a research/evaluation agenda, rather than just an ongoing medical practice. From there, there’s a very long and slow process from writing a grant, through getting it funded, conducting the evaluation, writing it  up, then submitting it to a peer-reviewed journal, going through the entire review process, before it is finally published and considered actual “evidence”. On top of this, top journals exhibit a strong preference for RCTs over other types of designs.

Harvard professor of anaesthesia, pediatrics, and medical ethics and chief of the Division of Critical Care Medicine at Boston Children’s Hospital Dr. Robert Truog, in a presentation entitled Ethical Conflicts in Randomized Controlled Trials, lists eight approaches to learning about what works in medicine, in ascending order of confidence:

1. Anecdotal Case Reports
2. Case Series without Controls
3. Case Series with Literature Controls
4. Case Series with Historical Controls
5. Databases
6. Case / Control Observational Studies
7. Randomized Controlled Trials
8. Meta-analyses

Truog argues that RCTs are not the only way to learn, even in the medical profession: “Phase I and II trials, which precede RCTs, often provide strong evidence for effectiveness.”

When should we think about alternatives to the RCT? Truog lists four conditions:

1. When therapies are potentially life-saving
2. When evaluating rapidly developing technologies (improvements in both experimental and control treatments may make the results of an RCT obsolete by the time it is published)
3. When RCTs are not the most efficient way to acquire knowledge
4. When the non-randomized data [are] compelling

Cutting-edge treatments often provide several of the above conditions, and the reality is that formal RCTs are always going to be way behind the technology. Because of the timeframes involved, the results of RCTs are often “old news” by the time they appear in print. In addition, there are often ethical dilemmas in the rigid use of RCTs. As Robert Truog asks …

“Who wants to be the last patient enrolled in the control arm of a positive randomized controlled trial?”

The same is equally true for a RCT of an educational, community health, international development, or business development intervention.

2. Tailored/individualized and adaptive treatments

In the medical and health professions, as in many other arenas, there are certain treatments (or programs/initiatives) that by their very nature must be completely tailored to the individual (or to the community, or to the organization) and/or that must be responsive to changing needs and need to be adapted over time.

One medical example of this is acupuncture and the use of Chinese herbs. Individuals with the same general Western diagnosis (e.g. depression, back pain, infertility), and even with the same basic underlying medical cause for that diagnosis (e.g. endometriosis, polycystic ovaries, diminished ovarian reserve), the Chinese medicine diagnosis of the underlying imbalances may differ substantially. A competent acupuncturist will proceed with a highly individualized treatment based on each person’s specific (Western and Eastern) diagnosis, will reassess at each session and tweak the treatment accordingly.

Clearly, this individualization and constant tweaking of treatment are at odds with the usual approach to RCTs, which is to standardize treatment and have each practitioner deliver it in exactly the same way. [There are some exceptions to this problem, e.g. some RCTs have been conducted to evaluate specific acupuncture treatments before and after IVF transfer, with statistically and practically significant effects documented. In fertility treatment, this covers just one very specific short-term application, but not the kinds of longer-term treatments that are also commonly used by couples experiencing infertility.]

An additional complication for evaluating acupuncture treatment is that diagnosis requires skilled professional judgment and (given that treatment cannot be simplistically standardized) treatment efficacy is highly dependent on the competence of the practitioner. A large-scale RCT would need to use several practitioners whose competence may vary widely, and this cause of variance could easily wash out effects.

This challenge is not limited to healthcare and medicine. Think about organizational development or community development initiatives. We have all heard countless examples of programs that really only worked amazingly well because of the passion of one or two highly committed people at key locations. Or that needed to be adapted locally to respond to changing needs and aspirations (or because they were initially not well enough understood). If the intervention couldn’t be standardized across multiple locations, it doesn’t fit the mold very well for an RCT.

3. What works for small subpopulations?

A third major challenge in working out “what works for whom” in medicine is that some patient subgroups have very specific combinations of factors that may lend themselves to particular kinds of treatments, but these populations are too small in number to even develop an RCT or any other quantitative design with sufficient statistical power to meet the usual requirements for publication. Or, the “target audience” for the findings is considered too narrow.

A good example is looking at the effectiveness of IVF treatment. It’s very easy to find a substantial sample size of women in their 30s with, say, blocked fallopian tubes or endometriosis – they often have insurance coverage for infertility or are eligible for publicly funded treatment, so there are plenty trying various IVF protocols (large N) and there is quite good knowledge about what works for them.

But suppose we wanted to understand what works for women over 40, or (even harder) over 42, who have specific diagnoses? First, the numbers are naturally lower for this group because most couples have completed their families by this age. For those still trying, the woman’s age and/or her specific diagnoses often mean that she is not eligible for insurance coverage or publicly funded treatment. So, there are far fewer trying IVF, and even fewer again for the specific diagnoses that are likely to make one ineligible for insurance or publicly funded treatment.

The reality is that some specific sub-populations will never be large enough in numbers to allow the use of RCTs to learn what works. But at the same time, certain clinicians will refuse to allow the patient to try treatment approaches that have not been supported by what they consider to be “solid” clinical trials.

At the same time, there are certain clinicians around the world who are known as top of their fields in dealing with specific types of case (such as women over 40). However, only some of them publish their findings, and often their work is sidelined by mainstream medicine as being “fringe” – and the limited sample sizes and only semi-standardized treatment protocols trigger further snorts of derision about the quality of their “evidence”.

The same is again true in education, community health, international development, business, and just about any other field one can name.

Where does this leave us – and where to next?

Right now, in medicine (and to varying degrees elsewhere), it’s only a small exaggeration to say:

• If you are seeking a “tried and true” (as supported by RCTs, or by other studies published in peer-reviewed journals) approach, you will only have access to “old” treatments and initiatives – and (in the case of RCT evidence) only those that can be completely standardized.
• If you’re after something cutting-edge or that needs to be tailored or adapted mid-stream, you have to pin your hopes on anecdotal evidence (and hope your physician or funder will support you).
• If you’re a member of a relatively large or typical subgroup, your treatment can be informed by evidence from RCTs and other published studies with a decent sample size.
• But if you’re in a very small minority sub-population, all we have is “anecdotal case studies” and the whole exercise is basically a crap-shoot.

Here in Aotearoa New Zealand, we have seen some very high quality government-funded work integrating a range of qualitative, quantitative and mixed method evidence about what works in education – the Iterative Best Evidence Synthesis (BES). A short quote from the Guidelines for Generating a Best Evidence Synthesis Iteration explains how evidence is selected for inclusion:

The [New Zealand] Ministry of Education is using the term ‘best’ within the best evidence synthesis programme to describe a body of evidence that provides credible evidence, and explanations for, influences that have made, and can make a bigger difference to desirable learner outcomes for diverse learners simultaneously. The criterion for selection of evidence for a best evidence synthesis is that the research provides evidence about impacts on learner outcomes. …

This criterion for selection of evidence means that research from a wide range of methodological designs (including for example, action research studies, case studies, microgenetic studies of classroom processes, ethnographic-outcome focused studies, quasi-experimental research, multiple regression studies, longitudinal studies and experimental research) can make valued contributions to a best evidence synthesis. The point of synthesis is that a cumulative body of research, carefully interrogated, provides more explanatory power than findings from any one research study or design type. (p. 33)

This is in stark contrast to the U.S.-based What Works Clearinghouse (WWC) evidence standards:

The WWC reviews each study that passes eligibility screens to determine whether the study provides strong evidence (Meets Evidence Standards), weaker evidence (Meets Evidence Standards with Reservations), or insufficient evidence (Does Not Meet Evidence Standards) for an intervention’s effectiveness. Currently, only well-designed and well-implemented randomized controlled trials (RCTs) are considered strong evidence, while quasi-experimental designs (QEDs) with equating may only meet standards with reservations; evidence standards for regression discontinuity and single-case designs are under development.

As a humorous side note, Michael Scriven recently (on EVALTALK) nicknamed the WWC the “WWQNC, standing for What Works for Quantitative Nerds Clearinghouse (pronounced ‘WONKS’)”.

While it’s very heartening to see some more enlightened evidence synthesis work such as NZ’s BES, I am still not sure we yet have good evidence accumulation and synthesis solutions for:

1. cutting-edge treatments where the technology and thinking is changing faster than RCTs (or even other large-scale long-term evaluation designs) can usefully inform
2. individualized, tailored, and adapt-as-you-go initiatives
3. small sub-populations that need to know what’s going to work for them

Are there ways, in medicine, to accumulate knowledge directly from clinicians and aggregate that to get approximate answers to these “what works for whom and under what conditions” questions? [I recently had a discussion with a medical academic who insisted it definitely was NOT possible!]

Are there ways in which outcome data and other learnings from localized small-scale initiatives can be meaningfully aggregated? I have been working on several projects that attempt to do just this (one in special education, one in primary school literacy, one for evaluating a nationwide strategy designed to help M?ori (NZ indigenous) students enjoy education success as M?ori) but would be interested how others have gone about the same.

For more on RCTs, see also my short JMDE (2006) editorial: The RCTs-Only Doctrine: Brakes on the Acquisition of Knowledge?

The new funding rules for the US Department of Education’s $650 million Investing in Innovation, published in Newsweek, show it is still using  an out-of-date model of evidence-based policy based on a problematic hierarchy of evidence.

The purpose of the funding is described as follows:

Program Description: The Investing in Innovation Fund, established under section 14007 of the American Recovery and Reinvestment Act of 2009 (ARRA), provides funding to support (1) local educational agencies (LEAs), and (2) nonprofit organizations in partnership with (a) one or more LEAs or (b) a consortium of schools. The purpose of this program is to provide competitive grants to applicants with a record of improving student achievement and attainment in order to expand the implementation of, and investment in, innovative practices that are demonstrated to have an impact on improving student achievement or student growth, closing achievement gaps, decreasing dropout rates, increasing high school graduation rates, or increasing college enrollment and completion rates.

These grants will (1) allow eligible entities to expand and develop innovative practices that can serve as models of best practices, (2) allow eligible entities to work in partnership with the private sector and the philanthropic community, and (3) identify and document best practices that can be shared and taken to scale based on demonstrated success.

This already sounds like the “leaky pipeline” model of evidence-based policy, where we find out “what works” and then scale it up:

The largest, or “scale up,” grants—worth up to $50 million each—will require “strong” evidence, such as program evaluations that used random assignment of students.

The second-tier, “validation” grants of up to $30 million each will go to proposals that show “moderate” evidence, such as those that use sophisticated statistical techniques to try to measure the true effects of a program.

The final-tier, “development” grants are wild cards to a degree; they are $5 million awards to proposals that are each based on a “reasonable” hypothesis or theory. The department made one change to the timing of that third tier of grants: No longer will applicants need to get prescreened before submitting their applications; their applications will be due at the same time as all the other i3 proposals.

“The overall design of the competition tries to account for the importance of evidence at each stage of innovation,” Mr. Shelton said.

For each tier, the level of evidence required is an all-or-nothing eligibility requirement; an applicant that doesn’t have the research to back up a proposal for that particular tier should not bother applying.

Let’s look at this carefully -at the top of the hierarchy is experimental design with a randomly assigned control group;  then come quasi-experimental designs that statistically create a comparison group, and then come good ideas.  There is literally no place for non-experimental causal inference.

What’s wrong with this?

Firstly, at least on the information currently available, there seems to be little attention given to the need for what Rohrbach and colleagues have called “Type II translation”, that is, building knowledge about how to move from an intervention that has been found to efficacious in a controlled setting to one which is effective when scaled up.

Secondly, the funding rules exclude serious impact evaluations of systemic interventions or highly customized practices where control groups or comparison groups are impractical.  That could exclude all interventions that are not focused on standardized individualized treatments. To put all these interventions in the third category only is to ignore any credible evidence they might have and judge them only on the basis of  “good ideas”.

(As an aside – to demonstrate that we can build credible evidence of the effectiveness of an intervention – consider the public health messages  about sleeping positions for babies.  In response to the high number of SIDS deaths, and based on some unproven theories, parents were advised to put babies to sleep on their backs, avoid overheating and cigarette smoke.  For ethical reasons, the advice was given to all parents, not to a randomly selected treatment group. The incidence of SIDS has fallen in a way that most people are confident to attribute to the changes in parenting practices.  A time series design, in conjunction with evidence of actual change in behavior, and some investigation of possible alternative explanations, was sufficient to endorse the intervention).

Thirdly, it is not clear  whether the requirements for evidence, which try to find the “real” effect of an intervention adequately address the “causal packages” that are often needed to produce impacts.  For example, if a new math innovation only works for certain types of students, or when used by a teacher with particular skills, or in a school where it is complemented by another program, will this be reported and used in informing appropriate scale up?

So what would I want to see in the funding guidelines?  I would want to RAISE the bar on the nature and type of evidence needed.

For the top tier, ‘scale up’ grants, I would require that studies have investigated not just the average effect but differential effects for different groups, and have recommendations about when and where the intervention is likely to be effective – AND when the proposals are funded they include serious follow-up and building evidence to support Type II translation.

For the second tier, ‘validation’ grants, I would want to see included in here projects that have credible non-experimental evidence of impact, and require the grants to include in their research designs attention to differential effects and the contribution of contextual factors to impacts.  This would include serious studies investigating the practices of effective teachers and seeing if these can be learned by others ( for example Doug Lemov’s taxonomies of effective teaching practices and Deborah Loewenberg Ball’s Practice-Based Theory of Mathematical Knowledge for Teaching, both described in detail in Elizabeth Green’s recent article in the New York Times.

And for the third group, “development grants’, I would want to see built into the grants the requirement that they properly evaluate innovations, drawing on the guidance provided by Burt Perrin on ‘How to – and how not to- evaluate innovation”, including paying careful attention to the successes and the failures and learning from all of them, and seeking to understand when and where and how it works or fails to work, and removing or reducing the powerful incentives to focus on the average effect or overclaim results.

Some further reading on the issues in evaluating complicated and complex interventions can be found in my paper on using program theory and Michael Patton’s forthcoming book on Developmental Evaluation by Guildford Press that will be out in July  (there’s a presentation to the Canadian Evaluation Society in 2009 currently available).

Patricia Rogers, in her recent post entitled Does the recent evaluation show that Head Start doesn’t work? asks the question:

Why do these reports  summarize the findings inaccurately? Is it deliberate misrespresentation or error?  Is it just too hard to include variations in the results in a brief summary, or are the reporters not sufficiently skilled? Or do the reporters judge that the results are not broad enough across the domains, or  not large enough?

I’d like to add to this question/discussion with a few observations about the media and its reporting of not just evaluation findings, but even just simple evaluative concepts.

Here in New Zealand, the government has moved to introduce National Standards for literacy and numeracy in primary schools. These are basically descriptions of what children should be able do/read/write/understand in order to be able to adequately access the rest of the curriculum at that level. Teachers are to use a range of appropriate assessment tools (as they presumably do already) to gauge where each child is at, and – this is the new piece – are required to clearly report to parents/families whether their child is performing at, above, below, or well below the National Standards in literacy and numeracy.

Just last week, New Zealand’s most watched news program (TVOne 6pm news) ran an item that began by telling us how terribly confused parents were about the new National Standards – implying, of course, that they were going enlighten everyone and help them understand the basics. Right …take a look at this 3-minute news item: Controversial new national standards introduced.

The National Standards are a criterion-referenced approach, i.e., performance (in literacy and numeracy) is evaluated against – as the title implies - “standards” (i.e., where  the child NEEDS to be).

How does TVOne news explain them? As a norm-referenced approach. The reporter states that the Standards will tell parents whether their child is at, above, or below the “national average” (i.e., where the child is relative to other peers). WRONG!!

The same news program has repeatedly said that children are going to be “ranked” - illustrating exactly the same total lack of understanding a of very very fundamental evaluation concept.

Most lay people can grasp the difference between grading/rating and ranking, so what’s wrong with the media?

My own observations lead me to believe that reporters and those controlling the quality of mainstream news media reports are, for the most part, woefully lacking in critical thinking ability, have a completely inadequate grasp of even the most basic evaluation concepts, and seem to have no sense of responsibility about getting a sound understanding before going to air and ‘educating’ the public.

The reporting of actual evaluation findings is a bit more complicated, as Patricia explains in her post. My hunch is that much of the blatantly inaccurate reporting we see is a combination of:

• ignorance of key evaluative concepts,
• a lack of critical thinking ability,
• laziness (e.g. just reporting what others have said instead of actually reading the original sources),
• cherry-picking the newsworthy (read: ratings-boosting) snippets at the expense of a more complete representation, and
• pressures to dumb down the news into tabloid-sized snippets that the public don’t have to work too hard to take in.

Of course, I’m mostly referring to TV here; the print media often do a much better job (and thank goodness someone does), but IMHO the bulk of it is still well short of good enough, genuine reporting. And, of course, I’m not watching media from all over the world (mostly New Zealand, but we get some Australian, U.S. and English channels here) – how do your local media outlets compare?

I suppose the big question for me is, what can we DO about this? Here are a few ideas that have crossed my mind, but I’d very much like to hear others’ views:

• Teach critical thinking in schools and in higher education.
• Teach some basic evaluative thinking too – it’s a life skill; it can save you a fortunate as you go through life making decisions about what to purchase, invest in, etc
• Help evaluators turn the nub of their findings into easy-to-grasp sound bites that the media could pick up and use as they are – maybe all evaluation reports should come with a set of press releases, some very short, some a little longer?
• As citizens and consumers of the news, keep criticizing the dumbed down news that is fed to the masses.
• Perhaps professional evaluation associations should consider commenting publicly on the misrepresentation of evaluative concepts and evaluation findings.

Another Head Start evaluation, another controversy about whether the results show it works or not.  In her comment on our post on the NY School Milk Study Susan Wolf drew our attention to some important differences between the recent evaluation report on Head Start, and how it was represented in an email from the Brookings Institute.

The January 21 2010 post on the Brookings Institute UpFront blog summarized the findings succinctly:

The study demonstrated that children’s attendance in Head Start has no demonstrable impact on their academic, socio-emotional, or health status at the end of first grade. That’s right. If you were a mother who lost the lottery, couldn’t get your child into Head Start, and had to care for her at home, she was no worse off at the end of first grade than she would have been had she gotten into Head Start.

The executive summary of the evaluation report , while acknowledging that many of the  results that were achieved during children’s participation in the program were not evident by the end of Grade 1, identified the following results:

Cognitive Outcomes: Head Start group children did significantly better on the PPVT (a vocabulary measure) for 4-year-olds and on the Woodcock-Johnson III test of Oral Comprehension for the 3-year-olds.

Social-Emotional Outcomes. By the end of 1st grade, there was some evidence that the 3-year-old cohort had closer and more positive relationships with their parents.

Health Outcomes. For the 4-year-old cohort, there was an impact on child health insurance coverage at the end of kindergarten and 1st grade, and an impact on child health status in kindergarten.

Parenting Outcomes. For the 3-year-old cohort, there were positive favorable impacts on use of time-out and authoritarian parenting at the end of 1st grade and on spanking and time out in kindergarten.

The Brookings summary has been widely picked up, including:

The Foundry, the blog of the Heritage Foundation which stated:

Unfortunately, a new (long overdue) report published by the Department of Health and Human found that the $150 billion that taxpayers have “invested” in Head Start since 1965 is yielding zero lasting benefits for participating children. According to the Head Start Impact Study: “the benefits of access to Head Start at age four are largely absent by 1st grade for the program population as a whole.” The Heritage Foundation reviews the findings of the new evaluation in a forthcoming Backgrounder report concluding: “Head Start has little to no effect on cognitive, socio-emotional, health, and parenting outcomes of children participating in the program.”

Why do these reports  summarize the findings inaccurately? Is it deliberate misrespresentation or error?  Is it just too hard to include variations in the results in a brief summary, or are the reporters not sufficiently skilled? Or do the reporters judge that the results are not broad enough across the domains, or  not large enough?  If the other gains are not sustained into Grade 1, does this reflect inadequacies of Head Start or of Grade 1?  Does Head Start need to be a ‘silver bullet’ to be successful?

Given the long history of controversial Head Start evaluations, I’m sure we’ll be hearing a lot more about this evaluation.  Including paying attention to differential results as well, since the evaluation clearly identified groups for whom Head Start worked particularly well, and those for whom it appeared to be harmful.