An apple a day – or cherry-picking the studies?

Why can’t newspapers be more critical when they report findings from research and evaluation, and provide easy links to more details?

A new study by researchers from Australia’s major government research instution (the Commonwealth Scientific and Industrial Research Organisation – CSIRO) , reviewing the health effects of eating apples, has received the usual standard of reporting in the press. (I have a particular interest in this issue, which I have used as the opening example in my program theory book, because it shows clearly how different theories about how ‘an apple a day’ might work lead to different intervention strategies and different measures for evaluation).

The review was funded by Horticulture Australia, which, as The Melbourne Age acknowledges in its reporting (one thing they get right), “represents the apple industry”. Given this potential conflict of interest, some more detail is needed about how the review was done.

No information is available in The Age article, or on the CSIRO website and an email enquiry received an automated response that I would get a reply within 2 working days.

I finally tracked down the report ( at least the glossy summary version) thanks to the Geelong Advertiser, which at least gave the name of the report, and Emma Stirling’s blog www.scoopnutrition.com, which has uploaded a copy.

The information provided about the methodology is very meagre:

The research studies included in this review were sourced via detailed and strategic electronic searches of medical, scientific and technical literature

Published human studies selected for retrieval were assessed for methodological validity. The levels of evidence used were those followed by the National Health and Medical Research Council for the assessment and application of scientific evidence.

The scientific review was prepared by Dr Peter Roupas and Associate Professor Manny Noakes, CSIRO Food and Nutritional Sciences, on behalf of Horticulture Australia Limited, in May 2010.

No information is provided about how many references were found, and how many were excluded on methodological or other grounds. Did the researchers only report studies that found positive results for apples? There is no information about this in the report, making it impossible to judge the credibility of the conclusions.

Maybe there is more information in the full report written by the researchers but the glossy summary, prepared by Horticulture Australia Ltd and reviewed by the researchers, gives no lead to accessing the actual review report.

It also has the curious disclaimer “For health professional use only”, hidden away at the back under the references,

The findings have been reported in the press in a very uncritical way. Can you spot the credibility gap in this reporting from the Melbourne Age:

A new CSIRO report, commissioned by Horticulture Australia, which represents the apple industry, claims that eating apples daily may reduce the risk of a range of health problems including diabetes and high cholesterol – a key factor in heart disease.

The report, which reviewed 10 years of scientific research into apples, also found health benefits for asthma and allergy suffers as well as weight loss.

”We’re often told apples are good for us, but what’s emerging now is the specific reasons why they are beneficial,” Associate Professor Manny Noakes of the CSIRO’s human nutrition program said. ”One really exciting part to come out of the report is that the polyphenols in the skin of apples can lower cholesterol by 5 to 8 per cent, when eaten three times a day.

Yes, the study is being reported as if it refers to having ONE apple a day, but some of the studies are about intake three times this amount (not impossible, but not what is being reported).

Now, don’t get me wrong. I like apples. I think it’s very likely that they are good for you. But I know that reporting of results from research and evaluation that simply repeat media releases without any critical review are not good for our health.

Not that any of this has stopped an advertising campaign for apples, which also comes via scoopnutrition.com

2 comments to An apple a day – or cherry-picking the studies?

  • Gabby Fennessy

    The challenge of the media is that they like their (often poorly informed) snippets for a ‘good news’ story, why let the facts get in the way?

    The challenge for researchers when talking to the media is to distill often complex messages into easy to digest (forgive the pun) chunks of information. Do readers also relate to the possibility that it ‘may’ have an impact on health? Apples “may reduce the risk of a range of health problems” could mean anything, or are health consumers so tired of reading conflicting information about things that may work, they switch off?

    You may be interested in an article about this in BMJ about the problem ‘Why promote the findings of single research studies?’ http://www.bmj.com/cgi/content/extract/336/7646/722

  • I’d say the criticism doesn’t go nearly far enough.

    First of all, before considering any report at all, it is necessary to evaluate the credibility of the source. If the source has any reason to misinform, as one would think is obviously the case with an industry association, then there is no reason to look further.

    The test of a reliable source is (1) is the source the most knowledgeable available; (2) does the source have a well-established reputation for absolute truthfulness; and (3) does the source have a real dedication to fully informing the audience.

    Unfortunately, too few researchers meet the test, and virtually no one in the media qualifies — reporters commonly lack knowledge, frequently report misinformation, and have their primary obligation to serve their advertisers.