Photo: Trevor Pinder, Herald Sun

We recently reported on some curious developments in evidence-based policy in the state of Victoria in Australia, where the newly-elected State Government had overturned previous policy to keep cattle out of alpine national parks, arguing there was insufficient evidence to support this.

Now the federal government (which is a different political party to the Victorian State government) has ordered the removal of cattle,using robust terms to refer the quality of research proposed. The Melbourne Age reports:

Declaring that cattle must be removed by April 8, Mr Burke said the Baillieu government had repeatedly failed to answer key questions about the trial since releasing about 400 cattle into the park in January. He said the state had failed to provide a copy of the research proposal or explain what steps it was taking to protect sensitive areas.

”For reasons I do not really comprehend, the Victorian government decided they didn’t really care if they were on the right side of the law,” he said.

”The information that has come back from the Victorian government is a joke … For something that is meant to be a university research project, they have provided information that wouldn’t pass as a high school science project.”

However, the severity of this response is somewhat limited by the timing for withdrawal of the cattle, which seems largely ceremonial:

Mr Burke’s deadline for cattle to leave the park roughly coincides with the end of the grazing season. Cattle leave the park at the end of April and return in October.

The Victorian Government has responded in very robust, and bovinely appropriate, terms:

State Environment Minister Ryan Smith last night accused Mr Burke of political grandstanding. He said the federal government knew summer grazing was about to finish and the state had answered all questions asked of it. ”To claim anything else is just bull.”

Mr Smith said he had yet to receive formal notification of the decision, and would consider his options once he had.

The Melbourne Sun-Herald asked even harder questions, and reported:

The minister refused to say if the trial was successful in helping to reduce the risk of bushfires.

He also said there is no timeline for when the final report on the first stage of the grazing trial is to be submitted to government.

 

I had thought most governments were now largely avowed supporters of evidence-based policy. Some recent examples from Australia have made me wonder what’s going on. Here’s the first of a worrying series.

1. Target 155

This program aimed to get Melbournians to reduce their residential water consumption to 155 litres per person per day. In addition to advertising, the target was included in water bills.

The newly elected State Government has terminated the program, claiming it was ‘just a political slogan’. The Melbourne Age reported last week:

Mr Walsh said the target had not played a major role in reducing water consumption.

”I think in general Melburnians did a great job with the water restrictions that were in place, and I don’t think they needed the political spin of the Target 155 campaign to know that they needed to be sensible with water use,” he said.

Mr Walsh said he didn’t have an opinion on how much water people should use each day, but said he was confident the change would not cause consumption to soar.

The Age also reported that the private water companies, which make more money when more water is used, have been campaigning for withdrawal of the program.

Now The Age has reported on research commissioned by the water companies, claiming the program had had a significant impact on reducing water usage:

But the report released by Ms Barker yesterday – which was jointly conducted by Melbourne’s three water retailers – claimed the influence of Target 155 could be identified using modelling that eliminated the impact of daily temperatures and rainfall on consumption.

The modelling found the introduction of Target 155 sent consumption below the volumes that would have been expected in the weather conditions that occurred.

Water savings of 53 billion litres were calculated for the period between the scheme’s introduction in December 2008, and August 2010.

”The T155 campaign has been effective in reducing water consumption in Melbourne,” the report said.

The classic tactic when a pesky bit of research gets in the way of a senior decision maker’s opinion is to discredit the research. Since this has been endorsed by the water companies, this might be harder to do (lthough those with a narrow view of evidence-based policy might parrot the erroneous line “without a control group, you can’t determine impact” – hey, it worked for tobacco companies for years).

Will it be rebutted, or ignored? Or will evidence overturn opinion and policy decisions? I’ll be watching with interest developments in this and other issues.

Policy that is developed in response to clearly identified needs and through careful processes of community engagement – while being feasible in an adversarial political system and short timeframes?  Time for the Hollowmen to show us how it can be done.

image from zazzle.com

Principles of Genuine Evaluation

When we set out to explore the notion of ‘Genuine Evaluation’, we identified 5 important aspects of it:

The centrality of sincerity

Are only new governments sincere about evaluation?

‘There will be no hidden agenda, no spin, no secrecy. Accountability and transparency will the principles that underpin our government,’ Mr Baillieu said on Tuesday.

Government should be transparent. Transparency promotes accountability and provides information for citizens about what their Government is doing.  Information maintained by the Federal Government is a national asset. My Administration will take appropriate action, consistent with law and policy, to disclose information rapidly in forms that the public can readily find and use. Executive departments and agencies should harness new technologies to put information about their operations and decisions online and readily available to the public. Executive departments and agencies should also solicit public feedback to identify information of greatest use to the public.

3. Indian National Government

In India, a new independent evaluation office has been established that will focus on assessing the impact of government’s flagship programs (thaks to Denis Jobin for this news):

The Union Cabinet today approved the establishment of an Independent Evaluation Office (IEO) to undertake impartial and objective assessment of the various public programmes and improve the effectiveness of public interventions. This is in pursuance of the Presidential address to the Joint Session of both Houses of Parliament in June, 2009 to establish an Independent Evaluation Office at an arms’ distance from the Government to assess the outcomes and impact of the major flagship programmes of the Government of India.

The IEO will be an independent office attached to the Planning Commission under a Governing Board chaired by the Deputy Chairman, Planning Commission. It will be funded by the Planning Commission and will have, as its head, a full-time Director General (DG) in the rank and status of Member, Planning Commission. It will have full functional autonomy to discharge its functions. The IEO will also advise the Planning Commission and the implementing agencies in developing appropriate management systems consistent with the evaluation objectives.

The IEO will strengthen the existing evaluation process by drawing the best resources available from leading research organizations. The findings of the independent evaluation will be reported to the Government of India and also be placed in the public domain.

4. UPDATE New York City’s Mayor’s Office of Operations.

( thanks to GreeneBarrett on twitter) -A searchable display of indicators about agency performance, and overall graphics.

(It does not however provide any commentary on them.  For example, the Brooklyn Public Library is way under its targets in terms of opening hours.  Is this due to lack of staff (or other factors the program should be managing) or heavy snow (or other factors the program might not be able to influence)? In the new year we will return to this and other performance reporting sites to compare them to good practices for performance reporting)

Are there other examples of goverments promising transparency – or even continuing to deliver on this several years down the track?

Jessica Miglio/NBC

Developing good performance indicators is not easy.  The history of their use is littered with examples of how they can produce a distorted picture of performance and provide dysfunctional incentives.  Burt Perrin’s report to the OECD (Organization for Economic Co-operation and Development) Implementing the vision – addressing challenges to results-focused management and budgeting summarizes a number of these and should be required reading for anyone developing performance information systems or results-based management systems.

Even if there is not formal fieldwork pilot testing performance indicators, there should be a ‘thought experiment’ to check:

1. Is it possible to score well on the performance indicator – but to actually be performing poorly?
2. Is it possibe to score badly on the performance indicator – but to actually be performing well?

A recent example from Australia has shown the perils of not adequately testing  performance indicators.  ERA (Excellence in Research for Australia ), a new system  to rate the quality of academic researchers, has rated some highly productive researchers low because it has used an average of all their published work.  A recent article in The Australian outlined an example from The Australian National University:

Andrew Cockburn, director of the[Australian National University's]  college of medicine, biology and environment, said that when compiling data for the ERA, a high-achieving geneticist’s output became a drag on the results of the college. This was because of the proportion of “building block” work published in journals that were not of the highest ranking. Professor Cockburn said the anomalous result would not have occurred had ERA focused on the academic’s best work, rather than requiring all journal articles to be submitted for the period under review.

He said the British system, in which academics submitted their best four or six papers, was preferable. “If she had been able to submit six papers published in Nature and Science, they would have delivered a higher ranking. “Although her work was within my college the most outstanding, in the ERA metric it dragged the average college performance down. It shows the way these metrics are counted can have unfortunate consequences.”

[Full disclosure: as an Australian academic, I am not just a disinterested observer of this new system.  Like everyone working in evaluation or applied research generally, I  would probably have been better off under the Research Quality Framework that the previous government had been developing,  which included assessment of the impact of research, not just publication and citation.  On the other hand, under the new system ithe AEA serial New Directions for Evaluation, which previously didn’t count at all as it did not fit the traditional form of a journal, received the only A* (top) rating for evaluation journals, suddenly making my publication record look a whole lot better – from zero to hero overnight with no effort on my part!)