More utterly uncritical media reporting of evaluation

As an evaluator, or even as an interested member of the public, what very basic fundamental information would you expect to see in a newspaper article that kicked off like this?

Chiefs get good marks even if departments don’t

Public sector leaders are mostly embracing challenges thrown to them by the Government, though at least one is “lost at sea”, a report says.

The Trans Tasman Media report, which ranks government agencies and bosses, has found that chief executives are generally performing better than their departments.

I don’t think I’m alone in having the following critically important questions leap into my mind when I see something like this:

  1. Who publishes the Trans Tasman Media report, and for what reasons? Is it an academic analysis, a report from a central government watchdog, a politically aligned propaganda organization, a fundamentally commercial venture?
  2. What are these ratings based on? What were the criteria? What was the evidence used?
  3. If they are heavily judgment-based, what are the credentials of those making the judgments?

Do we get anything close to answers to these in the full newspaper article? Not by a long shot!

  1. The publisher appears to be Trans Tasman Media, but the article tell us nothing about who they are, why they exist, or what their motivations are for writing a report like this.
  2. “Trans Tasman editor Max Bowden said all chief executives had the skills, but it was how they implemented government policy, and how easy they were to deal with, that affected rankings.” – hardly a clue here about what was rated, let alone based on what.
  3. As for those who are making the judgments? “independent panelists” – that’s all the clue we get.

What we do get is a series of titillating snippets about specific Public sector chief executives as being “engaging, sensible and thoughtful”, “continuing to step up”, or “lost at sea”. And reactions from the top- and bottom-ranked CEs, who have of course been asked to comment without having seen the report or its methodology.

Forgive me for being cynical, but, rather than decent reporting of a supposedly important piece of evaluative work, it all rather smells of an incendiary bit of free publicity for a publication that (one discovers after tracking the report down on the publishers’ website) punters are able to purchase for the bargain price of NZ$497. And, of course, rated agencies are forced to fork out this much now to see what on earth they are being rated on and why. Top marks to Trans Tasman Media for a cheap and presumably highly effective marketing ploy.

The newspaper that published this article about the report presumably thinks it should be taken seriously as an important news source. Why, then, do we continue to see such examples of lazy reporting without a skerrick of genuine critical or evaluative thinking?

The report may well be a fantastically insightful and robust piece of analysis, in which case the public surely deserves to know more. But based on the scant information given, it could just as easily be a sloppy piece of armchair theorizing from people who are far from qualified to do so. Even if the report isn’t out yet, it doesn’t hurt to ask a few basic questions to make sure the story is sound.

See also the earlier related posts:

Comments are closed.