In Genuine Evaluation we focus a lot on asking the right questions, bringing an evaluative frame, and basing answers on sound evidence.  But effective communication is also an important part of genuine evaluation, which is why this video caught our eye.  Hat tip to Stephanie Evergreen on twitter (@evalu8r) for sharing this.

Pic by JitterBuffer

Over the Antipodean summer Genuine Evaluation goes to the beach instead of blogging.  We’re back now, brushing off the sand, and planning more discussions about what it means to do genuine evaluation, plus sharing some insights from the African evaluation conference in Accra, Ghana.

To start the year, we wanted to highlight one of the more disturbing aspects of public policy discussions in recent years  -  the tendency to put forward opinions as if they were as compelling as solid evidence. We suspect that this will be the first in an ongoing series of examples.

Are working hours getting longer? Hopefully this example reflects someone being misquoted in the article in The Age in Melbourne, rather than how it appears – a researcher suggesting it’s too hard to get reasonable estimates of the extent of a problem and then pronouncing that the problem has diminished:

Some recent studies suggest this may now be a relic of history and that Australians work the longest hours in the developed world.

But Professor Mark Wooden, of the Institute of Applied Economic and Social Research, disagrees … strongly.

“The idea we work the most hours in the world is absolute crap,” he says.

“Lots of people work long hours and lots of people work short hours. We have a mix.”

He argues that workers in Japan and Korea work longer than Australians and that comparing working hours between countries was an inexact science.

People tend to overestimate how long they work as a sort of “badge of courage” and find it difficult to estimate the hours they work accurately, Professor Wooden says.

“I don’t think we can count,” he says. “It’s impossible to know. The study would need to be so invasive.”

His research shows the number of Australians working 50 hours a week or more peaked in the mid-1990s.

“Over the last 10 years, the proportion of Australians working long hours has been dropping.”

Photo: Joe Armao (The Age)

What’s worse than no evaluation? An evaluation that is wrong but you think is right.

Organizations that provide authoratitive evaluations have an obligation to meet high standards of accuracy and consistency. It is therefore hard to believe the series of events that led to Victoria’s Environmental Protection Agency listing Melbourne’s bayside beaches as “good”, and suitable for swimming when the level of bacteria was 40 times the acceptable limit.

According to The Age report,

[EPA] staff were not working on the ChristmasDay and Boxing Day public holidays, and as a result old forecast information was fed automatically into the website from Saturday until yesterday. These forecasts were made on Friday afternoon, and did not take into account the ferociousness of the Christmas Day storms.

On Sunday (Christmas Day) a massive storm hit Melbourne, sending debris, rubbish, cigarette butts and dog droppings into storm water drains. But for two days the EPA was operating on autopilot, issuing reports on the web site and tweets based on the projected water quality not the actual water quality.

And what has the EPA learned from this? Nothing, apparently. According to The Age, no EPA staff will be working on the New Year’s Day public holiday, and once again reports will be based on the weather forecast not on actual testing. A spokesman said beachgoers “should use their own judgment” in deciding to go swimming after storms like those that hit on Christmas Day.

Which does raise the question – if beachgoers should use their own judgment, because it is more likely to be accurate than the official reports, what’s the point of having the official reports?

Here’s what SHOULD happen. Either roster someone to work on these public holidays and ensure the reports are actually based on data ( it’s summer here and people are going to the beaches) OR issue a clear statement on the website, tweets and to news media that an accurate report cannot be provided due to the public holidays. No report is better than an inaccurate report.

We’re looking forward to the workshops we’ll be conducting next week before  and after  the American Evaluation Association conference. It’s always fun to explore with new people the ideas we’ve discussed in our books.

Jane will be looking at ways to ‘get genuine’ with evaluation – really asking evaluative questions that really matter, really answering them, and really reporting them. Patricia will be discussing how to ‘get genuine’ with logic models and theories of change -  thinking about different ways of developing, representing and using logic models and theories of change and choosing the most appropriate ways, especially to adequately address complicated and complex aspects of interventions.

There are 53 different pre- or post- conference workshops at Anaheim this year and some great opportunities to learn more about a wide range of important evaluation topics.

Getting Actionable Answers for Real-World Decision Makers: Evaluation Nuts and Bolts that Deliver

Ever read an evaluation report and still wondered how worthwhile the outcomes really were or whether the program was a waste of money? What if evaluations actually asked evaluative questions and gave clear, direct, evaluative answers? This workshop covers 1) big-picture thinking about key stakeholders, their information needs, and the evaluative questions they need answered; 2) a hands-on introduction to evaluative rubrics as a way of directly answering those questions; 3) guidance for designing interview and survey questions that are more easily interpreted against evaluative rubrics and capture evidence of causation; and 4) a reporting structure that gets to the point, delivering direct evaluative answers that decision makers can really use.

This workshop combines mini lectures, small and large group exercises to build big picture thinking to focus the evaluation on what really matters, and the most important “nuts and bolts” concepts and tools needed to deliver actionable answers.

You will learn:

• How to write a set of big picture overarching questions to guide the evaluation;
• How to use evaluation rubrics to get direct, evaluative answers to these questions;
• How to design innovative interview and survey questions geared directly toward answering evaluative questions and that capture evidence of causation;
• Evaluation conceptualization and reporting tips that maximize the chances of a clear, to-the-point, and actionable evaluation.

Session 3: Actionable Answers
Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Beginner, no prerequisites

Purposeful Program Theory

While program theory has become increasingly popular over the past 10 to 20 years, guides for developing and using logic models sometimes sacrifice contextual difference of practice in the interests of clear guidance and consistency across organizations. This session is designed for advanced evaluators who are seeking to explore ways to develop and use program theory in ways that suit the particular characteristics of the intervention, the evaluation purpose and the organizational environment. In addition to challenges identified by participants, the workshop will use mini-lectures, exercises, and discussions to address three particularly important issues- improving the quality of the models by drawing on generic theories of change and program archetypes; balancing the tension between simple models which communicate clearly and complicated models which better represent reality; and using the model to develop and answer evaluation questions that go beyond simply meeting targets.

You will learn:

• A strategy for purposeful development and use of program theory, drawing from an expanded repertoire of options;
• Additional ways to represent logic models, including results chains, and multi-level logic models;
• How to better represent complicated and complex interventions.

Prerequisites: Previous experience using program theory/logic models in evaluation; previous experience planning and undertaking evaluations
Sunday, November 6, 9 AM to 12 PM
Level: Advanced

 

We are looking forward to next week’s guest blogs from Katherine Hay, who works in New Delhi, India as part of the Evaluation Unit of Canada’s International Development Research Centre. Katherine will be blogging on the role of evidence in informing policy and what this means for the types of evidence that evaluation should aim to produce.

This week’s Friday Funny comes from Javier Ekboir, leader of the Institutional Learning And Change initiative of the Consultative Group on International Agricultural Research. It reminds us of the importance of considering:

• the change trajectories of the interventions we are evaluating (see Michael Woolcock’s blog post on this important issue),
• the timing of when we evaluate (see Michael Patton’s story in Utilization-Focused Evaluation about whether having your horse go lame might be a good thing or a bad thing),
• the reasons why what people tell you and show you might not be accurate representations of what is going on (see Jack Douglas’ classic book Investigative Social Research)

The version below is adapted from www.gotjokes.net.

While walking down the street one day a [insert appropriate local politician – senator, president, Member of Parliament) is tragically hit by a truck and dies. His soul arrives in heaven and is met by St. Peter at the entrance.

“Welcome to heaven,” says St. Peter. “Before you settle in, it seems there is a problem. We seldom see a high official around these parts, you see, so we’re not sure what to do with you.”

“No problem, just let me in,” says the politican.

“Well, I’d like to but I have orders from higher up. What we’ll do is have you spend one day in hell and one in heaven. Then you can choose where to spend eternity.”

“Really, I’ve made up my mind. I want to be in heaven,” says the politician.

“I’m sorry but we have our rules.” And with that, St. Peter escorts him to the elevator and he goes down, down, down to hell.

The doors open and he finds himself in the middle of a green golf course. In the distance is a club and standing in front of it are all his friends and other politicians who had worked with him. Everyone is very happy and in evening dress.

They run to greet him, shake his hand, and reminisce about the good times they had while getting rich at expense of the people. They play a friendly game of golf and then dine on lobster, caviar and champagne.

Also present is the devil, who really is a very friendly guy who has a good time dancing and telling jokes. They are having such a good time that, before he realizes it, it is time to go. Everyone gives him a hearty farewell and waves while the elevator rises.

The elevator goes up, up, up and the door reopens on heaven where St. Peter is waiting for him.

“Now it’s time to visit heaven.”

So, 24 hours pass with the head of state joining a group of contented souls moving from cloud to cloud, playing the harp and singing.

They have a good time and, before he realizes it, the 24 hours have gone by and St. Peter returns.

“Well then, you’ve spent a day in hell and another in heaven. Now choose your eternity.”

The politician reflects for a minute, then the senator answers: “Well, I would never have said it before, I mean heaven has been delightful, but I think I would be better off in hell.”

So St. Peter escorts him to the elevator and he goes down, down, down to hell. Now the doors of the elevator open and he’s in the middle of a barren land covered with waste and garbage.

He sees all his friends, dressed in rags, picking up the trash and putting it in black bags. The devil comes over to him and puts his arm around his shoulder.

“I don’t understand,” stammers the politician. “Yesterday I was here and there was a golf course and club, and we ate lobster and caviar,drank champagne, and danced and had a great time. Now all there is is a wasteland full of garbage and my friends look miserable. What happened?”

The devil looks at him, smiles and says, “Yesterday we were campaigning……Today you voted for us!”