This week’s Friday Funny (thanks to businessballs.com) reminds us about the need to check what others might call a successful outcome.
An old lady had a hearing-aid fitted, hidden underneath her hair.
A week later she returned to the doctor for her check-up.
“It’s wonderful – I can hear everything now,” she reported very happily to the doctor.
“And is your family pleased too?” asked the doctor.
“Oh I haven’t told them yet,” said the old lady, “And I’ve changed my will twice already..”
In Genuine Evaluation we focus a lot on asking the right questions, bringing an evaluative frame, and basing answers on sound evidence. But effective communication is also an important part of genuine evaluation, which is why this video caught our eye. Hat tip to Stephanie Evergreen on twitter (@evalu8r) for sharing this.
Pic by JitterBuffer
Over the Antipodean summer Genuine Evaluation goes to the beach instead of blogging. We’re back now, brushing off the sand, and planning more discussions about what it means to do genuine evaluation, plus sharing some insights from the African evaluation conference in Accra, Ghana.
To start the year, we wanted to highlight one of the more disturbing aspects of public policy discussions in recent years - the tendency to put forward opinions as if they were as compelling as solid evidence. We suspect that this will be the first in an ongoing series of examples.
Are working hours getting longer? Hopefully this example reflects someone being misquoted in the article in The Age in Melbourne, rather than how it appears – a researcher suggesting it’s too hard to get reasonable estimates of the extent of a problem and then pronouncing that the problem has diminished:
Some recent studies suggest this may now be a relic of history and that Australians work the longest hours in the developed world.
But Professor Mark Wooden, of the Institute of Applied Economic and Social Research, disagrees … strongly.
“The idea we work the most hours in the world is absolute crap,” he says.
“Lots of people work long hours and lots of people work short hours. We have a mix.”
He argues that workers in Japan and Korea work longer than Australians and that comparing working hours between countries was an inexact science.
People tend to overestimate how long they work as a sort of “badge of courage” and find it difficult to estimate the hours they work accurately, Professor Wooden says.
“I don’t think we can count,” he says. “It’s impossible to know. The study would need to be so invasive.”
His research shows the number of Australians working 50 hours a week or more peaked in the mid-1990s.
“Over the last 10 years, the proportion of Australians working long hours has been dropping.”
Photo: Joe Armao (The Age)
What’s worse than no evaluation? An evaluation that is wrong but you think is right.
Organizations that provide authoratitive evaluations have an obligation to meet high standards of accuracy and consistency. It is therefore hard to believe the series of events that led to Victoria’s Environmental Protection Agency listing Melbourne’s bayside beaches as “good”, and suitable for swimming when the level of bacteria was 40 times the acceptable limit.
According to The Age report,
[EPA] staff were not working on the ChristmasDay and Boxing Day public holidays, and as a result old forecast information was fed automatically into the website from Saturday until yesterday. These forecasts were made on Friday afternoon, and did not take into account the ferociousness of the Christmas Day storms.
On Sunday (Christmas Day) a massive storm hit Melbourne, sending debris, rubbish, cigarette butts and dog droppings into storm water drains. But for two days the EPA was operating on autopilot, issuing reports on the web site and tweets based on the projected water quality not the actual water quality.
And what has the EPA learned from this? Nothing, apparently. According to The Age, no EPA staff will be working on the New Year’s Day public holiday, and once again reports will be based on the weather forecast not on actual testing. A spokesman said beachgoers “should use their own judgment” in deciding to go swimming after storms like those that hit on Christmas Day.
Which does raise the question – if beachgoers should use their own judgment, because it is more likely to be accurate than the official reports, what’s the point of having the official reports?
Here’s what SHOULD happen. Either roster someone to work on these public holidays and ensure the reports are actually based on data ( it’s summer here and people are going to the beaches) OR issue a clear statement on the website, tweets and to news media that an accurate report cannot be provided due to the public holidays. No report is better than an inaccurate report.
Antipodean Holidays
It’s been a busy year, and we’re about to head off for our sunny, Antipodean Christmas break. Thanks to everyone for your comments and suggestions on the blog and when we have met face to face.
We’ve both had heavy project reporting commitments this year which have limited our capacity for blogging – the Friday Funnies have quite outweighed more serious blogs.
Next year we’re planning to share emerging thoughts as we work on new books – Jane on commissioning evaluation and Patricia on evidence-based policy and practice. We look forward to discussing these issues with you.
To finish the year we wanted to share this brief evaluation of Christmas Gifts, from From the Development Research Institute (DRI), New York University via Pablo Rodriguez-Bilella. We enjoyed it all except the scurrilous reference to fruit cakes, which are a greatly revered gift in the Antipodes.
Lack of Rigorous Methodology: Regrettably, this evaluation had to proceed without the required Randomized Controlled Trial on Christmas Gifts, which failed to be completed as planned. Project managers did a poor job explaining the advantages of RCT participation to the Control Group.
Lack of Targeting: The Christmas Gifts aid program was not sufficiently well-targeted to the poor. Recipients of Christmas Gifts indiscriminately included well-off regions, groups, genders, and individuals.
Lack of Efficient Modalities: The Christmas Gifts appeared to consist largely of in-kind aid. This contradicts abundant evidence of best practices emphasizing cash transfers as superior to in-kind aid. There was some evidence of #SWEDOW (“Stuff We Don’t Want”) in-kind transfers, the worst possible kind of aid, usually involving fruitcakes.
Lack of Efficient Timing: Contrary to the recommendation that aid consist of an even, predictable flow, the Christmas Gifts program is mostly concentrated on one day, with a few unpredictable lags ranging from a few days (“late deliveries”) to months (“handmade gifts”).
Lack of Net Flows: Evaluators found Christmas Gift recipients engaged in behavior that frustrated the aid program, with Recipients acting as Donors to their own Donors, reducing their own net aid intake. They explained their counterproductive behavior with non-standard concepts such as “Tis more bless’d to give than to receive.”
Jane at Real Evaluation
Patricia at CIRCLE (RMIT)