Misreporting evaluation findings – Example 1

Part of the concern with the report we discussed yesterday (which tracked changes in school milk purchases only without any data on calorie intake or obesity) was in how easily carefully phrased conclusions could be paraphrased as bold statements that went way beyond the data.

The original report provided by the researchers was quite careful to spell out the assumptions under their estimates:

In 2004, approximately 18.3 billion calories and 520 million grams of fat were purchased by DOE in the form of milk. In 2009, as a result of DOE’s switch to lower-fat milk, those numbers decreased to 13.7 billion calories and 98 million grams of fat, representing a 25% and 81% decline in available calories and fat from milk, respectively. Comparing 2004 with 2009, if calorie and fat savings were distributed over all enrolled students, 3,484 fewer calories and 382 fewer grams of fat were averted each school year as a result of the milk policy change. When distribution of fat and calories from milk were limited to the percentage of students who were estimated to drink milk during the school day (62% in 2004 and 63% in 2009), these savings increased to 5,960 calories and 619 fat grams per year.

The CDC added an editorial that spelled out these limitations clearly:

no data were collected on total food consumption during the school day, so the effect of the milk switch on overall diet is unknown. Students might compensate for the averted calories/fat from milk by changing their consumption patterns.

However when these findings were reported further, these limitations were ignored and the estimated reductions in calories were reported as if they had been measured not estimated in a way that was very contestable. Even the CDC used an unfortunate summary on their front page advertizing the story:

MedPageToday for example,which is peer reviewed by  University of Pennsylvania Medical  School reported: Switch to Low-Fat Milk in Schools Shows Benefit

When New York City public schools made the switch from whole milk to skim or low-fat milk, students cut their annual fat and total calorie consumption, department researchers found. Milk-drinking students consumed 5,960 fewer calories and 619 fewer grams of fat per year after they made the switch, Philip M. Alberti, PhD, of the New York Department of Health and Mental Hygiene, and colleagues reported in the Jan. 29 issue of CDC’s Morbidity & Mortality Weekly Report.

ProCor, a program of Lowan Cardiovascular Research Foundation also reported the results  as if the calorie reductions were real: US: Low-fat/fat-free milk in schools

To decrease the city’s approximately 1.1 million schoolchildren’s fat and calorie intake while maintaining calcium consumption, New York City public schools removed whole milk from all cafeterias in 2005-2006 and switched from low-fat to fat-free chocolate milk. Because of the change, in 2009 NYC public school milk-drinking students consumed about 5,960 fewer calories and 619 fewer grams of fat than in 2004.

Business Week reported NYC Schools Cut 4.6 Billion Calories Switching Milk

ABC News reported Switch to Low-Fat Milk Lowers Calories for NYC Schoolkids

The blog of Prevention Matters named the study as “the Best Prevention Idea of the Week” and copied the bloomberg report with its headline of NYC Schools Cut 4.6 Billion Calories Switching Milk

And because so many reports are simply copied for other publications, there are many, many more.

Whose responsibility is it to report findings from research and evaluation acurately?

2 comments to Misreporting evaluation findings – Example 1

  • Interesting that you posted this at this time. I was just thinking about this same issue. Last week I was startled when I received an e-mail from the Brookings Institution to the effect that Head Start is ineffective. They wrote:

    “The study demonstrated that children’s attendance in Head Start has no demonstrable impact on their academic, socio-emotional, or health status at the end of first grade.”

    I was curious so I went to the actual report’s executive summary. The actual findings were:

    “However, the advantages children gained during their Head Start and age 4 years yielded only a few statistically significant differences in outcomes at the end of 1st grade for the sample as a whole. Cognitive Outcomes. By the end of 1st grade, only a single cognitive impact was found for each cohort. Head Start group children did significantly better on the PPVT (a vocabulary measure) for 4-year-olds and on the Woodcock-Johnson III test of Oral Comprehension for the 3-year-olds.
    o Social-Emotional Outcomes. By the end of 1st grade, there was some evidence that the 3-year-old cohort had closer and more positive relationships with their parents. These impacts were preceded by other social-emotional impacts (improvements in behavior-hyperactive behavior and total problem behavior, and social skills and positive approaches to learning) in the earlier years. The findings
    Random Assignment
    Newly entering 3- and 4-year-old Head Start applicants were randomly assigned either to a Head Start group that in the initial year had access to Head Start services or to a control group that could receive any other non-Head Start services chosen by their parents.
    for the 4-year-old cohort are inconsistent with teachers reporting that children in the Head Start group are more shy and socially reticent and have more problems with student and teacher interactions than control group children while their parents are reporting that they are less withdrawn.
    o Health Outcomes. For the 4-year-old cohort, there was an impact on child health insurance coverage at the end of kindergarten and 1st grade, and an impact on child health status in kindergarten. For the 3-year-olds, there was an impact on child health insurance coverage in kindergarten only.
    o Parenting Outcomes. For the 3-year-old cohort, there were positive favorable impacts on use of time-out and authoritarian parenting at the end of 1st grade and on spanking and time out in kindergarten. These favorable impacts for authoritarian parenting and spanking were also demonstrated in the earlier years. For the 4-year-old cohort, there were no significant parenting practices impacts in kindergarten or 1st grade.”

  • […] controversy about whether the results show it works or not.  In her comment on our post on the NY School Milk Study Susan Wolf drew our attention to some important differences between the recent evaluation report on […]