Advocating for RCTs – with a non-RCT example?

Curious post by Tim Harford in the Financial Times recently “Political ideas need proper testing”  As is often the case, his argument slid from advocating for better empirical investigation of public policy by systematic experimentation to discussing this only in terms of RCTs (Randomised Controlled Trials). I was reminded of  Michael Scriven’s critique of the appropriation of the word ‘experimental’ (in a recent issue of the Journal of MultiDisciplinary Evaluation)

… the RCT campaign also involves the less-remarked parallel effort, going back further, to redefine the concept of an experiment. In standard scientific usage, experiments are just carefully constrained explorations, and the RCT is simply a special case of these. To call the RCT the only “true experiment” is part of an attempt at redefinition that distorts the original and continuing usage, and excludes experiments designed to test many simple hypotheses about—or simple efforts to find out—what happens if we do this.
This effort at persuasive redefinition is allied with an implicit denigration of the so-called “quasi-experimental” designs, which are in fact perfectly respectable experiments, only ‘quasi’ with respect to the one respect in which they have less control over one possible way of excluding one type of alternative explanation.

What makes the recent post particularly surprising is that he is advocating for RCTs through an example which shows clearly the value of drawing on other types of evidence – the changing advice on infant sleeping positions to reduce the risk of Sudden Infant Death Syndrome.  No RCTs were used to investigate the effectiveness of these and yet adequate evidence was produced to be able to make effective changes in policy.

I recently used the same example to show the value and feasibility of using a range of credible evidence about effectiveness, including non-experimental designs.  It will be published shortly  by the Productivity Commission in proceedings from a Roundtable on Strengthening Evidence-based Policy in the Australian Federation.

Non-RCT data can provide good quality evidence of effectiveness

Good quality evidence of effectiveness can also come from quasi-experimental approaches, which compare program participants to a comparison group rather than to a randomly assigned control group, and from non-experimental approaches, when such approaches systematically and rigorously test causal conclusions and combine evidence thoughtfully.

Sudden Infant Death Syndrome (SIDS) is one of two exemplars in the National Health and Medical Research Council guide How to Put the Evidence into Practice: Implementation and Dissemination Strategies (NHMRC 2000).

It shows both the value of drawing on a diverse set of evidence and how it is possible to develop effective policy even when the evidence is not definitive. Bringing together evidence from many studies, including retrospective and prospective epidemiological studies, pathological studies and case studies, a number of possible contributing factors were identified, and other possible causes (such as vaccinations) were ruled out.

On the basis of this incomplete evidence, recommendations were developed — to put babies to sleep on their backs, avoid overheating and avoid cigarette smoke. No RCTs were used to test the effectiveness of these recommendations. The recommendations were communicated directly to parents and to health professionals working with parents, resulting in widespread change in the sleeping positions they used for infants.

By 2005, the number of SIDS deaths had been reduced to fewer than 100, a decline of 83 per cent (ABS 2007).