We recently stumbled across this all-time classic that Genuine Evaluation readers may well appreciate!
Smith, G. C. S. & Pell, J. P. (2003, December). Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials. BMJ, 327, 1459-1461.
Here’s the abstract:
Objectives: To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.
Design: Systematic review of randomised controlled trials.
Data sources: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.
Study selection: Studies showing the effects of using a parachute during free fall.
Main outcome measure: Death or major trauma, defined as an injury severity score > 15.
Results: We were unable to identify any randomised controlled trials of parachute intervention.
Conclusions: As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials.
Advocates of evidence based medicine have criticised the adoption of interventions evaluated by using only observational data.
We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.
And some particularly insightful snippets:
The parachute and the healthy cohort effect
One of the major weaknesses of observational data is the possibility of bias, including selection bias and reporting bias, which can be obviated largely by using randomised controlled trials.
The relevance to parachute use is that individuals jumping from aircraft without the help of a parachute are likely to have a high prevalence of pre-existing psychiatric morbidity. Individuals who use parachutes are likely to have less psychiatric morbidity and may also differ in key demographic factors, such as income and cigarette use. It follows, therefore, that the apparent protective effect of parachutes may be merely an example of the “healthy cohort” effect.
…
A call to (broken) arms
Only two options exist. The first is that we accept that, under exceptional circumstances, common sense might be applied when considering the potential risks and benefits of interventions.
The second is that we continue our quest for the holy grail of exclusively evidence based interventions and preclude parachute use outside the context of a properly conducted trial. The dependency we have created in our population may make recruitment of the unenlightened masses to such a trial difficult. If so, we feel assured that those who advocate evidence based medicine and criticise use of interventions that lack an evidence base will not hesitate to demonstrate their commitment by volunteering for a double blind, randomised, placebo controlled, crossover trial.
The full article is available online via the BMJ site (which requires a one-time registration for free access). Alternatively, we did find a directly accessible copy up at http://elucidation.free.fr/parachuteBMJ.pdf
Jane at Real Evaluation
Patricia at CIRCLE (RMIT)
An addendum to the Friday Funny. I’ve included this reference in a number of presentations on evidence-based policy and impact evaluation, and followed it up with the 2006 Campbell Collaboration systematic review of after school programs (www.campbellcollaboration.org/lib/download/58/)which ended up with only 5 studies after excluding non-experimental studies and was therefore unable to draw conclusions.
Interestingly, I have observed a number of strong advocates for RCTs referring to the BMJ paper, but dismissing it as humorous (for example Andrew Leigh’s paper advocating for RCTs at the Australian roundtable on Strenghtening Evidence-Based Policy http://www.pc.gov.au/__data/assets/pdf_file/0008/96218/13-chapter10.pdf)