Intention To Treat and checking for implementation failure and differential effects – questions about vitamin A trials in Ghana

A recent RCT in Ghana that found Vitamin A supplementation did not lead to reduced maternal mortality ( a finding very different to a study in Nepal that found a 44% reduction) has raised some important questions for me about how well clinical trial procedures match with our understanding of using program theory in evaluation – in particular whether implementation failure or differential impacts might explain the findings rather than the ineffectiveness of the program.

div xmlns:cc=http://creativecommons.org/ns# about=http://www.flickr.com/photos/dfid/4420202860/a rel=cc:attributionURL href=

http://www.flickr.com/photos/dfid/4420202860 CC BY-NC-ND 2.0

Here are the main findings, according to the summary published in The Lancet, Volume 375, Issue 9726, Pages 1640 – 1649, 8 May 2010

Effect of vitamin A supplementation in women of reproductive age on maternal survival in Ghana (ObaapaVitA): a cluster-randomised, placebo-controlled trial

Background - A previous trial in Nepal showed that supplementation with vitamin A or its precursor (betacarotene) in women of reproductive age reduced pregnancy-related mortality by 44% (95% CI 16—63). We assessed the effect of vitamin A supplementation in women in Ghana.

Methods – ObaapaVitA was a cluster-randomised, double-blind, placebo-controlled trial undertaken in seven districts in Brong Ahafo Region in Ghana. The trial area was divided into 1086 small geographical clusters of compounds with fieldwork areas consisting of four contiguous clusters. All women of reproductive age (15—45 years) who gave informed consent and who planned to remain in the area for at least 3 months were recruited. Participants were randomly assigned by cluster of residence to receive a vitamin A supplement (25 000 IU retinol equivalents) or placebo capsule orally once every week. Randomisation was blocked and based on an independent, computer-generated list of numbers, with two clusters in each fieldwork area allocated to vitamin A supplementation and two to placebo. Capsules were distributed during home visits undertaken every 4 weeks, when data were gathered on pregnancies, births, and deaths. Primary outcomes were pregnancy-related mortality and all-cause female mortality. Cause of death was established by verbal post mortems. Analysis was by intention to treat (ITT) with random-effects regression to account for the cluster-randomised design. Adverse events were synonymous with the trial outcomes. This trial is registered with ClinicalTrials.gov, number NCT00211341.

Findings 544 clusters (104 484 women) were randomly assigned to vitamin A supplementation and 542 clusters (103 297 women) were assigned to placebo. The main reason for participant drop out was migration out of the study area. In the ITT analysis, there were 39 601 pregnancies and 138 pregnancy-related deaths in the vitamin A supplementation group (348 deaths per 100 000 pregnancies) compared with 39 234 pregnancies and 148 pregnancy-related deaths in the placebo group (377 per 100 000 pregnancies); adjusted odds ratio 0·92, 95% CI 0·73—1·17; p=0·51. 1326 women died in 292 560 woman-years in the vitamin A supplementation group (453 deaths per 100 000 years) compared with 1298 deaths in 289 310 woman-years in the placebo group (449 per 100 000 years); adjusted rate ratio 1·01, 0·93—1·09; p=0·85.

Interpretation The body of evidence, although limited, does not support inclusion of vitamin A supplementation for women in either safe motherhood or child survival strategies.

Funding - UK Department for International Development, and USAID.

The first author of the study, Professor Bettey Kirkwood, was quoted by Reuters:

Kirkwood noted that the Ghana findings contradicted previous results from a trial in Nepal which showed a 44 percent decrease in maternal death among women given vitamin A. She said the discrepancy showed why it is wise not to rush to change maternal health policies on the basis on one piece of research. “Research is as important to identify potentially good ideas that do not work, as it is in establishing those that do,” she wrote. “This avoids governments wasting resources on ineffective interventions.”

Now it might be the case that the study has provided compelling evidence that the intervention is ineffective – but I have two lingering questions about it, and none of the available reports have provided enough information to answer these.  (If there is additional information available, please send me the details).

Is this a case of misinterpreting ITT – Intention To Treat – data?

The study involved an ITT analysis – that is, analyzing the results for those assigned to the groups regardless of compliance  with the treatment.  That is, the results for the treatment group includes all those who were assigned to the treatment group, but did not actually take the supplements.  So if there was low adherence, the intervention would appear to be ineffective, even if it was effective for those who took the supplements.  Was there low adherence?  This information is not provided in the available summary (my online journal subscriptions don’t include the most recent 2 months’ issues) or in any of the reports on the study.

In fact some of the reporting referred to the effect of TAKING the supplements, rather than the effect of being SUPPLIED with the supplements:

GhanWeb reported it as follows:

Accra, April 27, GNA – A study conducted in Kintampo by a team of experts has shown beyond doubt that taking Vitamin A does not reduce the risk of death in pregnancy or childbirth.

By contrast, Reuters correctly reported what had been tested:

Giving vitamin A to women aged between 15 and 45 in poor nations does not cut maternal death rates, scientists said Tuesday in a study which contradicts earlier research showing a dramatic drop in death rates.

There is an important debate going on about the use of ITT analysis.  Clearly it is important to know the end result of a policy change – but it is also essential to understand whether a lack of effects is due to implementation failure or theory failure.

Gerard Dallal has set out an interesting analysis of this issue which focuses on the fact that ITT analysis is appropriate for answering the question “What happens once a treatment is started or recommended?” but it does not answer the question about whether it is effective for those who follow the treatment.

For public policy issues, we need to know the latter – and if there is a big difference between the two we need to investigate ways to improve adherence.  This is what a group of researchers have advocated for studies of the effectiveness of microbicides for HIV prevention, where adherence is an important issue (and very hard to measure).

At the very least we need to be clear in the reporting of findings whether or not there was an effect for those compliant with the treatment. But even better would be to go beyond a simple summative judgement (did the policy work ?) and look at whether there is scope to improve it.

Were those who were most likely to benefit from the treatment screened out of the study?

Who would be most likely to benefit from vitamin A supplementation?  Women who had vitamin A deficiency of course.  Ideally the study would have been able to report on whether or not the results were different for those with vitamin A deficiencies as the effects might be lost in the noise from the total population.

The information about the trial recorded at http://clinicaltrials.gov had this surprising statement:

There will be no exclusions to participation, except for women who have nightblindness or other signs of VAD. These, and any women who develop VAD in the course of the study will be treated according to current IVACG recommendations (IVACG, 1997). They will continue to be followed, but will be given vitamin A and considered separately in the analysis.

These women might have been considered separately in the analysis, but their participation (or exclusion) and outcomes have not been reported in the summaries of the project.  If they are being excluded from the study on ethical grounds, to avoid them being denied a proven treatment for Vitamin A deficiency, wouldn’t that reduce the likelihood of the study producing significant effects – not because Vitamin A is ineffective, but because those for whom it would have made a difference were not included in the study?

I’d be very interested to hear more about this study, and whether the full report addresses these issues.  And of course I’ll be following  how these findings are reported.

6 comments to Intention To Treat and checking for implementation failure and differential effects – questions about vitamin A trials in Ghana

  • Michael Scriven

    Friday funnies must include funny-peculiars as well as funny-hahas (I hope that distinction has worldwide meaning!) This is certainly a tragic example of how bad evaluation design leads to colossal wastage of resources (the mind boggles at the cost of studies with n = ~200,000) and hence (possibly) a colossal delay in saving lives, hence a large loss of life, because of complete ambiguity of reports (so far at least). Doesn’t really matter whether the Nepal or the Ghana study or both failed to be absolutely clear about the dosage loophole or the (well-meaning) exclusionary loophole, it looks as if they are both at fault in failing to actually point out possible misinterpretations and reject them (unless they did so and the media ignored it). Big credits to Patricia for spotting this and I hope the copies she undoubtedly forwarded to the scientists involved result in clearing this up.

    Michael Scriven

  • Michael Scriven

    Maybe we need funny-tragic as a third category; or funny-pathetic; or . . . .?

    Michael Scriven

  • In a recent letter to The Lancet (1 May, http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(10)60648-4/fulltext) I criticized another recent study that suffered from many similar weaknesses and was critical of a unicef program. The Lancet included in the same edition an editorial on a rolling series in which they want more and better evaluation published. I hope that you will put your oar in on this objective through an article, perhaps looking across an even larger number of published studies – once you have your manuscript finished! Getting the critique outside the evaluation literature and into the journals that researchers in other fields read is critical.

  • Patricia Rogers

    Fred, I totally agree that the debate needs to go outside the evaluation profession and into the arenas where evidence-based policy and practice are being discussed. Kudos for raising this in the Lancet and I hope they do follow up your suggestion to include studies that can rigorously and systematically evaluate the impact of system-wide interventions where control groups or meaningful comparison groups are not possible or appropriate. This issue needs to be raised in many fora – for example, I noticed a recent post on a development blog which talked about general equilibrium analysis and THE alternative to RCTs (http://chrisblattman.com/2010/05/07/whats-the-alternative-to-randomized-control-trials-in-development-research/)

  • Patricia Rogers

    Michael, the media repeatedly report the conclusions of these studies uncritically. The full article is somewhat better, but questions still remain. The problem does not seem to have been one of adherence (which was about 70% – although there was no comparison of the effects for those with higher adherence, which would have been useful). Nor do there seem to have been many cases of vitamin A deficiency picked up and treated separately. The break in the causal chain seems to have come at the point of serum retinol. In the sample who were tested, pregnant women who were in the treatment group had LOWER serum retinol than pregnant women in the control group. No explanation was offered for this, apart from ruling out errors in randomization (mixing up the placebo and real capsules). Hard to think of a possible explanation. If the dosage was simply too low, the levels would have been similar. If it had not been a double-blind study, it might have been some compensatory behavior (for example, eating less) when taking a treatment. It’s a puzzle, and despite the huge cost of the study, there don’t seem to be any answers.

  • Dugan Fraser

    Seems to sugggest the importance of proper analysis of the underlying programme theory…

    I’d go with funny-sad as the third type of funny.