Genuine Evaluation » RCT http://genuineevaluation.com Patricia J Rogers and E Jane Davidson blog about real, genuine, authentic, practical evaluation Fri, 03 Feb 2012 19:49:48 +0000 en hourly 1 http://wordpress.org/?v=3.1 Intention To Treat and checking for implementation failure and differential effects – questions about vitamin A trials in Ghana http://genuineevaluation.com/intention-to-treat-and-checking-for-implementation-failure-and-differential-effects/ http://genuineevaluation.com/intention-to-treat-and-checking-for-implementation-failure-and-differential-effects/#comments Fri, 07 May 2010 09:30:48 +0000 Patricia Rogers http://genuineevaluation.com/?p=983 Continue reading →]]>

A recent RCT in Ghana that found Vitamin A supplementation did not lead to reduced maternal mortality ( a finding very different to a study in Nepal that found a 44% reduction) has raised some important questions for me about how well clinical trial procedures match with our understanding of using program theory in evaluation – in particular whether implementation failure or differential impacts might explain the findings rather than the ineffectiveness of the program.

div xmlns:cc=http://creativecommons.org/ns# about=http://www.flickr.com/photos/dfid/4420202860/a rel=cc:attributionURL href=

http://www.flickr.com/photos/dfid/4420202860 CC BY-NC-ND 2.0

Here are the main findings, according to the summary published in The Lancet, Volume 375, Issue 9726, Pages 1640 – 1649, 8 May 2010

Effect of vitamin A supplementation in women of reproductive age on maternal survival in Ghana (ObaapaVitA): a cluster-randomised, placebo-controlled trial

Background - A previous trial in Nepal showed that supplementation with vitamin A or its precursor (betacarotene) in women of reproductive age reduced pregnancy-related mortality by 44% (95% CI 16—63). We assessed the effect of vitamin A supplementation in women in Ghana.

Methods – ObaapaVitA was a cluster-randomised, double-blind, placebo-controlled trial undertaken in seven districts in Brong Ahafo Region in Ghana. The trial area was divided into 1086 small geographical clusters of compounds with fieldwork areas consisting of four contiguous clusters. All women of reproductive age (15—45 years) who gave informed consent and who planned to remain in the area for at least 3 months were recruited. Participants were randomly assigned by cluster of residence to receive a vitamin A supplement (25 000 IU retinol equivalents) or placebo capsule orally once every week. Randomisation was blocked and based on an independent, computer-generated list of numbers, with two clusters in each fieldwork area allocated to vitamin A supplementation and two to placebo. Capsules were distributed during home visits undertaken every 4 weeks, when data were gathered on pregnancies, births, and deaths. Primary outcomes were pregnancy-related mortality and all-cause female mortality. Cause of death was established by verbal post mortems. Analysis was by intention to treat (ITT) with random-effects regression to account for the cluster-randomised design. Adverse events were synonymous with the trial outcomes. This trial is registered with ClinicalTrials.gov, number NCT00211341.

Findings 544 clusters (104 484 women) were randomly assigned to vitamin A supplementation and 542 clusters (103 297 women) were assigned to placebo. The main reason for participant drop out was migration out of the study area. In the ITT analysis, there were 39 601 pregnancies and 138 pregnancy-related deaths in the vitamin A supplementation group (348 deaths per 100 000 pregnancies) compared with 39 234 pregnancies and 148 pregnancy-related deaths in the placebo group (377 per 100 000 pregnancies); adjusted odds ratio 0·92, 95% CI 0·73—1·17; p=0·51. 1326 women died in 292 560 woman-years in the vitamin A supplementation group (453 deaths per 100 000 years) compared with 1298 deaths in 289 310 woman-years in the placebo group (449 per 100 000 years); adjusted rate ratio 1·01, 0·93—1·09; p=0·85.

Interpretation The body of evidence, although limited, does not support inclusion of vitamin A supplementation for women in either safe motherhood or child survival strategies.

Funding - UK Department for International Development, and USAID.

The first author of the study, Professor Bettey Kirkwood, was quoted by Reuters:

Kirkwood noted that the Ghana findings contradicted previous results from a trial in Nepal which showed a 44 percent decrease in maternal death among women given vitamin A. She said the discrepancy showed why it is wise not to rush to change maternal health policies on the basis on one piece of research. “Research is as important to identify potentially good ideas that do not work, as it is in establishing those that do,” she wrote. “This avoids governments wasting resources on ineffective interventions.”

Now it might be the case that the study has provided compelling evidence that the intervention is ineffective – but I have two lingering questions about it, and none of the available reports have provided enough information to answer these.  (If there is additional information available, please send me the details).

Is this a case of misinterpreting ITT – Intention To Treat – data?

The study involved an ITT analysis – that is, analyzing the results for those assigned to the groups regardless of compliance  with the treatment.  That is, the results for the treatment group includes all those who were assigned to the treatment group, but did not actually take the supplements.  So if there was low adherence, the intervention would appear to be ineffective, even if it was effective for those who took the supplements.  Was there low adherence?  This information is not provided in the available summary (my online journal subscriptions don’t include the most recent 2 months’ issues) or in any of the reports on the study.

In fact some of the reporting referred to the effect of TAKING the supplements, rather than the effect of being SUPPLIED with the supplements:

GhanWeb reported it as follows:

Accra, April 27, GNA – A study conducted in Kintampo by a team of experts has shown beyond doubt that taking Vitamin A does not reduce the risk of death in pregnancy or childbirth.

By contrast, Reuters correctly reported what had been tested:

Giving vitamin A to women aged between 15 and 45 in poor nations does not cut maternal death rates, scientists said Tuesday in a study which contradicts earlier research showing a dramatic drop in death rates.

There is an important debate going on about the use of ITT analysis.  Clearly it is important to know the end result of a policy change – but it is also essential to understand whether a lack of effects is due to implementation failure or theory failure.

Gerard Dallal has set out an interesting analysis of this issue which focuses on the fact that ITT analysis is appropriate for answering the question “What happens once a treatment is started or recommended?” but it does not answer the question about whether it is effective for those who follow the treatment.

For public policy issues, we need to know the latter – and if there is a big difference between the two we need to investigate ways to improve adherence.  This is what a group of researchers have advocated for studies of the effectiveness of microbicides for HIV prevention, where adherence is an important issue (and very hard to measure).

At the very least we need to be clear in the reporting of findings whether or not there was an effect for those compliant with the treatment. But even better would be to go beyond a simple summative judgement (did the policy work ?) and look at whether there is scope to improve it.

Were those who were most likely to benefit from the treatment screened out of the study?

Who would be most likely to benefit from vitamin A supplementation?  Women who had vitamin A deficiency of course.  Ideally the study would have been able to report on whether or not the results were different for those with vitamin A deficiencies as the effects might be lost in the noise from the total population.

The information about the trial recorded at http://clinicaltrials.gov had this surprising statement:

There will be no exclusions to participation, except for women who have nightblindness or other signs of VAD. These, and any women who develop VAD in the course of the study will be treated according to current IVACG recommendations (IVACG, 1997). They will continue to be followed, but will be given vitamin A and considered separately in the analysis.

These women might have been considered separately in the analysis, but their participation (or exclusion) and outcomes have not been reported in the summaries of the project.  If they are being excluded from the study on ethical grounds, to avoid them being denied a proven treatment for Vitamin A deficiency, wouldn’t that reduce the likelihood of the study producing significant effects – not because Vitamin A is ineffective, but because those for whom it would have made a difference were not included in the study?

I’d be very interested to hear more about this study, and whether the full report addresses these issues.  And of course I’ll be following  how these findings are reported.

]]> http://genuineevaluation.com/intention-to-treat-and-checking-for-implementation-failure-and-differential-effects/feed/ 6