“Given the norms that govern most patriarchal societies in Africa, should the Western epistemology, ethics and concepts be the main default lens for evaluation” “Despite their blindness to social cultural context, are these evaluations valid even though they are said to be based on scientific evidence”

A, B, and C—the ways of HIV/AIDS preventing transmission

Uganda, Adaptation of the Fleet of Hope, 1995. (click image to go to source article)

Figure – Social Marketing to Prevent HIV/AIDS: Uganda, Adaptation of the Fleet of Hope, 1995. (Jhuccp.org)

I conducted a metaevaluation on a number of evaluations from some of the most prominent bilateral agencies fighting HIV/AIDS in Sub Sahara Africa. Despite insufficient evidence to support evaluation claims, the majority of the evaluations assessed include recommendations which indicate either a need for continued funding, and/or increase of monitoring staff.

Here are some of the selected examples of my findings;

Troubling Question

The critical question of concern is where is the value judgment underlying recommendations found in some evaluation reports which fail to seek for side effects

 Conclusion

Obvious the basis upon which HIV/AIDS prevention interventions maybe too narrow to shift the potential underlying social ecology (negative gender dynamics and socio-cultural norms) that gives rise to women’s HIV/AIDS vulnerability. The western lens may not be the main default lens in evaluation; social cultural and gender norms are an important part of the landscape for Sub-Saharan Africa. These dynamics and norms represent a Pandora box entrenched with hidden contents (i.e., side effects) which affect gender and HIV/AIDS prevention.

Watch for my next post, where I will be discussing the hidden contents of the Pandora’s box which are being ignored by evaluation.

In the last few days before the Australian federal election, a curious $5million advertising campaign has been launched by the Alliance of Australian Retailers against the current government’s policy to introduce plain paper packaging for cigarettes.

Their television advertisements are based on 2 arguments:

1. there is no evidence this will work, in terms of reducing smoking among either current smokers or new smokers
2. it will add to the cost of running their businesses

Their statement of “what we stand for” states:

The government proposal to mandate plain packaging for cigarettes is the last straw.
Let’s be clear – we believe that reducing smoking is good for our community. But good policies require more than good intentions.
There is no reliable evidence anywhere in the world that plain packaging will stop people from taking up smoking, or help people to quit. But we do know that it will make it harder for us to run our businesses.

“No real evidence”

One of the ads argues strongly that policy must be based on evidence not just good intentions. Despite the appeal of this line, there is something not so convincing about the argument.

Well, there is evidence from experimental studies, such as a 2008 study published in the British Medical Journal.

If they are asking for evidence in terms of a longitudinal study of the actual effects of this when introduced as policy, it’s pretty hard to get this, as it has not been introduced anywhere.

And of course there is a self-perpetuating pattern to this – if this argument is run everywhere, it never will, making it impossible to produce the evidence.

“It will make it harder to run my business”

Another ad argues that the new policy would be damaging to business, without specifying how.

The spokesperson for the AAR, Sheryle Moon, interviewed on the ABC current affairs show Lateline, tried to explain how this could be the case:

SHERYLE MOON, ALLIANCE OF AUSTRALIAN RETAILERS: If I’m a small business owner, perhaps I’m a single owner operator, so I’m in my store, I’ve got plain packaging of cigarettes, I’ve got to put them away on a shelf – it’s difficult to do that. I can’t identify necessarily which product is which product.

PETER LLOYD: Why not?

SHERYLE MOON: Because it’s harder to see them, they’re not identified, they all look very similar. If I’m serving customers, it’s difficult for me to find the right product for the customer. All those increase my transaction times and make it more difficult for me to run my business efficiently.

PETER LLOYD: But the cigarette brand name would still be printed on the bottom of the package so you could still see, for example, Marlboro Light, on the packet, so why is that so difficult?

SHERYLE MOON: So I think the issue here, Peter, is more about that plain packaging is not a proven policy.

A more convincing argument comes from the webpage of the AAR, which reports the results of a survey (no details of sample size or type…):

What we do know about plain packaging is that it will make it harder for us to run our businesses.

A May 2010 Galaxy poll found:

	80 per cent of retailers believe the plain packaging policy would hurt their business
	81 per cent of businesses surveyed consider tobacco sales important to their business
	78 per cent of retailers believe their business will suffer and they may have to lay off staff if their customers turn to the black market for tobacco
	87 per cent of retailers believe small business in Australia is faced with too much red tape and regulation.

So… tobacco sales are important and they predict a plain packaging policy would hurt their business. I guess that would be by reducing sales – that is, by working?

The AAR is a new organization comprising The Service Station Association, Australian Newsagents’ Federation and National Independent Retailers Association, which is supported by British American Tobacco Australia Limited, Philip Morris Limited and Imperial Tobacco Australia Limited.

Why can’t newspapers be more critical when they report findings from research and evaluation, and provide easy links to more details?

A new study by researchers from Australia’s major government research instution (the Commonwealth Scientific and Industrial Research Organisation – CSIRO) , reviewing the health effects of eating apples, has received the usual standard of reporting in the press. (I have a particular interest in this issue, which I have used as the opening example in my program theory book, because it shows clearly how different theories about how ‘an apple a day’ might work lead to different intervention strategies and different measures for evaluation).

The review was funded by Horticulture Australia, which, as The Melbourne Age acknowledges in its reporting (one thing they get right), “represents the apple industry”. Given this potential conflict of interest, some more detail is needed about how the review was done.

No information is available in The Age article, or on the CSIRO website and an email enquiry received an automated response that I would get a reply within 2 working days.

I finally tracked down the report ( at least the glossy summary version) thanks to the Geelong Advertiser, which at least gave the name of the report, and Emma Stirling’s blog www.scoopnutrition.com, which has uploaded a copy.

The information provided about the methodology is very meagre:

The research studies included in this review were sourced via detailed and strategic electronic searches of medical, scientific and technical literature

Published human studies selected for retrieval were assessed for methodological validity. The levels of evidence used were those followed by the National Health and Medical Research Council for the assessment and application of scientific evidence.

The scientific review was prepared by Dr Peter Roupas and Associate Professor Manny Noakes, CSIRO Food and Nutritional Sciences, on behalf of Horticulture Australia Limited, in May 2010.

No information is provided about how many references were found, and how many were excluded on methodological or other grounds. Did the researchers only report studies that found positive results for apples? There is no information about this in the report, making it impossible to judge the credibility of the conclusions.

Maybe there is more information in the full report written by the researchers but the glossy summary, prepared by Horticulture Australia Ltd and reviewed by the researchers, gives no lead to accessing the actual review report.

It also has the curious disclaimer “For health professional use only”, hidden away at the back under the references,

The findings have been reported in the press in a very uncritical way. Can you spot the credibility gap in this reporting from the Melbourne Age:

A new CSIRO report, commissioned by Horticulture Australia, which represents the apple industry, claims that eating apples daily may reduce the risk of a range of health problems including diabetes and high cholesterol – a key factor in heart disease.

The report, which reviewed 10 years of scientific research into apples, also found health benefits for asthma and allergy suffers as well as weight loss.

”We’re often told apples are good for us, but what’s emerging now is the specific reasons why they are beneficial,” Associate Professor Manny Noakes of the CSIRO’s human nutrition program said. ”One really exciting part to come out of the report is that the polyphenols in the skin of apples can lower cholesterol by 5 to 8 per cent, when eaten three times a day.”

Yes, the study is being reported as if it refers to having ONE apple a day, but some of the studies are about intake three times this amount (not impossible, but not what is being reported).

Now, don’t get me wrong. I like apples. I think it’s very likely that they are good for you. But I know that reporting of results from research and evaluation that simply repeat media releases without any critical review are not good for our health.

Not that any of this has stopped an advertising campaign for apples, which also comes via scoopnutrition.com

Building on an earlier discussion Michael Scriven started about long-term effects (what to with them and without them), I’m interested in people’s thoughts on a related issue.

In the medical profession in particular, there are some very rigid beliefs about what constitutes good enough “evidence of effectiveness” to justify offering, recommending, allowing patients to try, or even just not vehemently opposing a particular type of treatment for a patient. [There are obviously some parallels in other sectors, such as education, social services, international development, criminal justice, etc, but let's start with some medical examples for now.]

There are some glimmers of hope in other sectors (e.g. in the Best Evidence Synthesis work here in New Zealand). But there are still three areas where there are very serious challenges in building a credible evidence base given the kinds of constraints and realities surrounding them. They are: (1) cutting-edge treatments;  (2) treatments that are by their very nature tailored/individualized rather than standardized across patients or populations; and (3) learning what works for small sub-populations.

1. Cutting-edge treatments

Advancements are being made in medical practice all the time, and many of these are initially developed by clinicians (doctors, specialists, surgeons) trying a new approach on a limited number of patients, e.g. when the standard treatments are either not working, or when there’s a plausible idea about how to improve benefits for patients.

In order for a new idea to be trialled on a larger scale, it must be picked up by individuals with a research/evaluation agenda, rather than just an ongoing medical practice. From there, there’s a very long and slow process from writing a grant, through getting it funded, conducting the evaluation, writing it  up, then submitting it to a peer-reviewed journal, going through the entire review process, before it is finally published and considered actual “evidence”. On top of this, top journals exhibit a strong preference for RCTs over other types of designs.

Harvard professor of anaesthesia, pediatrics, and medical ethics and chief of the Division of Critical Care Medicine at Boston Children’s Hospital Dr. Robert Truog, in a presentation entitled Ethical Conflicts in Randomized Controlled Trials, lists eight approaches to learning about what works in medicine, in ascending order of confidence:

1. Anecdotal Case Reports
2. Case Series without Controls
3. Case Series with Literature Controls
4. Case Series with Historical Controls
5. Databases
6. Case / Control Observational Studies
7. Randomized Controlled Trials
8. Meta-analyses

Truog argues that RCTs are not the only way to learn, even in the medical profession: “Phase I and II trials, which precede RCTs, often provide strong evidence for effectiveness.”

When should we think about alternatives to the RCT? Truog lists four conditions:

1. When therapies are potentially life-saving
2. When evaluating rapidly developing technologies (improvements in both experimental and control treatments may make the results of an RCT obsolete by the time it is published)
3. When RCTs are not the most efficient way to acquire knowledge
4. When the non-randomized data [are] compelling

Cutting-edge treatments often provide several of the above conditions, and the reality is that formal RCTs are always going to be way behind the technology. Because of the timeframes involved, the results of RCTs are often “old news” by the time they appear in print. In addition, there are often ethical dilemmas in the rigid use of RCTs. As Robert Truog asks …

“Who wants to be the last patient enrolled in the control arm of a positive randomized controlled trial?”

The same is equally true for a RCT of an educational, community health, international development, or business development intervention.

2. Tailored/individualized and adaptive treatments

In the medical and health professions, as in many other arenas, there are certain treatments (or programs/initiatives) that by their very nature must be completely tailored to the individual (or to the community, or to the organization) and/or that must be responsive to changing needs and need to be adapted over time.

One medical example of this is acupuncture and the use of Chinese herbs. Individuals with the same general Western diagnosis (e.g. depression, back pain, infertility), and even with the same basic underlying medical cause for that diagnosis (e.g. endometriosis, polycystic ovaries, diminished ovarian reserve), the Chinese medicine diagnosis of the underlying imbalances may differ substantially. A competent acupuncturist will proceed with a highly individualized treatment based on each person’s specific (Western and Eastern) diagnosis, will reassess at each session and tweak the treatment accordingly.

Clearly, this individualization and constant tweaking of treatment are at odds with the usual approach to RCTs, which is to standardize treatment and have each practitioner deliver it in exactly the same way. [There are some exceptions to this problem, e.g. some RCTs have been conducted to evaluate specific acupuncture treatments before and after IVF transfer, with statistically and practically significant effects documented. In fertility treatment, this covers just one very specific short-term application, but not the kinds of longer-term treatments that are also commonly used by couples experiencing infertility.]

An additional complication for evaluating acupuncture treatment is that diagnosis requires skilled professional judgment and (given that treatment cannot be simplistically standardized) treatment efficacy is highly dependent on the competence of the practitioner. A large-scale RCT would need to use several practitioners whose competence may vary widely, and this cause of variance could easily wash out effects.

This challenge is not limited to healthcare and medicine. Think about organizational development or community development initiatives. We have all heard countless examples of programs that really only worked amazingly well because of the passion of one or two highly committed people at key locations. Or that needed to be adapted locally to respond to changing needs and aspirations (or because they were initially not well enough understood). If the intervention couldn’t be standardized across multiple locations, it doesn’t fit the mold very well for an RCT.

3. What works for small subpopulations?

A third major challenge in working out “what works for whom” in medicine is that some patient subgroups have very specific combinations of factors that may lend themselves to particular kinds of treatments, but these populations are too small in number to even develop an RCT or any other quantitative design with sufficient statistical power to meet the usual requirements for publication. Or, the “target audience” for the findings is considered too narrow.

A good example is looking at the effectiveness of IVF treatment. It’s very easy to find a substantial sample size of women in their 30s with, say, blocked fallopian tubes or endometriosis – they often have insurance coverage for infertility or are eligible for publicly funded treatment, so there are plenty trying various IVF protocols (large N) and there is quite good knowledge about what works for them.

But suppose we wanted to understand what works for women over 40, or (even harder) over 42, who have specific diagnoses? First, the numbers are naturally lower for this group because most couples have completed their families by this age. For those still trying, the woman’s age and/or her specific diagnoses often mean that she is not eligible for insurance coverage or publicly funded treatment. So, there are far fewer trying IVF, and even fewer again for the specific diagnoses that are likely to make one ineligible for insurance or publicly funded treatment.

The reality is that some specific sub-populations will never be large enough in numbers to allow the use of RCTs to learn what works. But at the same time, certain clinicians will refuse to allow the patient to try treatment approaches that have not been supported by what they consider to be “solid” clinical trials.

At the same time, there are certain clinicians around the world who are known as top of their fields in dealing with specific types of case (such as women over 40). However, only some of them publish their findings, and often their work is sidelined by mainstream medicine as being “fringe” – and the limited sample sizes and only semi-standardized treatment protocols trigger further snorts of derision about the quality of their “evidence”.

The same is again true in education, community health, international development, business, and just about any other field one can name.

Where does this leave us – and where to next?

Right now, in medicine (and to varying degrees elsewhere), it’s only a small exaggeration to say:

• If you are seeking a “tried and true” (as supported by RCTs, or by other studies published in peer-reviewed journals) approach, you will only have access to “old” treatments and initiatives – and (in the case of RCT evidence) only those that can be completely standardized.
• If you’re after something cutting-edge or that needs to be tailored or adapted mid-stream, you have to pin your hopes on anecdotal evidence (and hope your physician or funder will support you).
• If you’re a member of a relatively large or typical subgroup, your treatment can be informed by evidence from RCTs and other published studies with a decent sample size.
• But if you’re in a very small minority sub-population, all we have is “anecdotal case studies” and the whole exercise is basically a crap-shoot.

Here in Aotearoa New Zealand, we have seen some very high quality government-funded work integrating a range of qualitative, quantitative and mixed method evidence about what works in education – the Iterative Best Evidence Synthesis (BES). A short quote from the Guidelines for Generating a Best Evidence Synthesis Iteration explains how evidence is selected for inclusion:

The [New Zealand] Ministry of Education is using the term ‘best’ within the best evidence synthesis programme to describe a body of evidence that provides credible evidence, and explanations for, influences that have made, and can make a bigger difference to desirable learner outcomes for diverse learners simultaneously. The criterion for selection of evidence for a best evidence synthesis is that the research provides evidence about impacts on learner outcomes. …

This criterion for selection of evidence means that research from a wide range of methodological designs (including for example, action research studies, case studies, microgenetic studies of classroom processes, ethnographic-outcome focused studies, quasi-experimental research, multiple regression studies, longitudinal studies and experimental research) can make valued contributions to a best evidence synthesis. The point of synthesis is that a cumulative body of research, carefully interrogated, provides more explanatory power than findings from any one research study or design type. (p. 33)

This is in stark contrast to the U.S.-based What Works Clearinghouse (WWC) evidence standards:

The WWC reviews each study that passes eligibility screens to determine whether the study provides strong evidence (Meets Evidence Standards), weaker evidence (Meets Evidence Standards with Reservations), or insufficient evidence (Does Not Meet Evidence Standards) for an intervention’s effectiveness. Currently, only well-designed and well-implemented randomized controlled trials (RCTs) are considered strong evidence, while quasi-experimental designs (QEDs) with equating may only meet standards with reservations; evidence standards for regression discontinuity and single-case designs are under development.

As a humorous side note, Michael Scriven recently (on EVALTALK) nicknamed the WWC the “WWQNC, standing for What Works for Quantitative Nerds Clearinghouse (pronounced ‘WONKS’)”.

While it’s very heartening to see some more enlightened evidence synthesis work such as NZ’s BES, I am still not sure we yet have good evidence accumulation and synthesis solutions for:

1. cutting-edge treatments where the technology and thinking is changing faster than RCTs (or even other large-scale long-term evaluation designs) can usefully inform
2. individualized, tailored, and adapt-as-you-go initiatives
3. small sub-populations that need to know what’s going to work for them

Are there ways, in medicine, to accumulate knowledge directly from clinicians and aggregate that to get approximate answers to these “what works for whom and under what conditions” questions? [I recently had a discussion with a medical academic who insisted it definitely was NOT possible!]

Are there ways in which outcome data and other learnings from localized small-scale initiatives can be meaningfully aggregated? I have been working on several projects that attempt to do just this (one in special education, one in primary school literacy, one for evaluating a nationwide strategy designed to help M?ori (NZ indigenous) students enjoy education success as M?ori) but would be interested how others have gone about the same.

For more on RCTs, see also my short JMDE (2006) editorial: The RCTs-Only Doctrine: Brakes on the Acquisition of Knowledge?

This week we are revisting a classic moment in Australian comedy which seems strangely relevant to this week’s headlines.

 The Hollowmen is an comedy-drama from 2008 set in the offices of the Central Policy Unit, a special think tank personally set up by the Prime Minister to help him in the most important job of all – getting re-elected. Their brief is “long term vision”; to stop worrying about tomorrow’s headlines, and focus on next week’s. 

 In episode one, Fat Chance the Prime Minister is caught out on talk-back radio, promising to tackle the problem of childhood obesity. Now the Unit must come up with a policy that keeps everyone happy.  Their investigation begins with consultations and research review that support significant regulatory changes - and ends with a token call for self-regulation. 

This week life seemed to imitate art as the Australian government decided not to implement the recommendations of a taskforce set up to investigate these issues. The Age reports:

In contrast to the crackdown on cigarettes, the Rudd government has rejected its own experts’ recommendations to take on the powerful food and alcohol industries.

Obesity was recently found to trigger more diseases in Australia than tobacco, but the government has given the thumbs-down to the call from its preventative health taskforce for a ban on junk-food advertising before 9pm.

It has also refused the taskforce recommendation to phase-out alcohol advertising during live sport broadcasts, in a detailed response to the taskforce recommendations released with Tuesday’s budget.