Part of the concern with the report we discussed yesterday (which tracked changes in school milk purchases only without any data on calorie intake or obesity) was in how easily carefully phrased conclusions could be paraphrased as bold statements that went way beyond the data.
The original report provided by the researchers was quite careful to spell out the assumptions under their estimates:
In 2004, approximately 18.3 billion calories and 520 million grams of fat were purchased by DOE in the form of milk. In 2009, as a result of DOE’s switch to lower-fat milk, those numbers decreased to 13.7 billion calories and 98 million grams of fat, representing a 25% and 81% decline in available calories and fat from milk, respectively. Comparing 2004 with 2009, if calorie and fat savings were distributed over all enrolled students, 3,484 fewer calories and 382 fewer grams of fat were averted each school year as a result of the milk policy change. When distribution of fat and calories from milk were limited to the percentage of students who were estimated to drink milk during the school day (62% in 2004 and 63% in 2009), these savings increased to 5,960 calories and 619 fat grams per year.
The CDC added an editorial that spelled out these limitations clearly:
no data were collected on total food consumption during the school day, so the effect of the milk switch on overall diet is unknown. Students might compensate for the averted calories/fat from milk by changing their consumption patterns.
However when these findings were reported further, these limitations were ignored and the estimated reductions in calories were reported as if they had been measured not estimated in a way that was very contestable. Even the CDC used an unfortunate summary on their front page advertizing the story:
When New York City public schools made the switch from whole milk to skim or low-fat milk, students cut their annual fat and total calorie consumption, department researchers found. Milk-drinking students consumed 5,960 fewer calories and 619 fewer grams of fat per year after they made the switch, Philip M. Alberti, PhD, of the New York Department of Health and Mental Hygiene, and colleagues reported in the Jan. 29 issue of CDC’s Morbidity & Mortality Weekly Report.
ProCor, a program of Lowan Cardiovascular Research Foundation also reported the results as if the calorie reductions were real: US: Low-fat/fat-free milk in schools
To decrease the city’s approximately 1.1 million schoolchildren’s fat and calorie intake while maintaining calcium consumption, New York City public schools removed whole milk from all cafeterias in 2005-2006 and switched from low-fat to fat-free chocolate milk. Because of the change, in 2009 NYC public school milk-drinking students consumed about 5,960 fewer calories and 619 fewer grams of fat than in 2004.
Business Week reported NYC Schools Cut 4.6 Billion Calories Switching Milk
ABC News reported Switch to Low-Fat Milk Lowers Calories for NYC Schoolkids
The blog of Prevention Matters named the study as “the Best Prevention Idea of the Week” and copied the bloomberg report with its headline of NYC Schools Cut 4.6 Billion Calories Switching Milk
And because so many reports are simply copied for other publications, there are many, many more.
Whose responsibility is it to report findings from research and evaluation acurately?