Pic by JitterBuffer

Over the Antipodean summer Genuine Evaluation goes to the beach instead of blogging.  We’re back now, brushing off the sand, and planning more discussions about what it means to do genuine evaluation, plus sharing some insights from the African evaluation conference in Accra, Ghana.

To start the year, we wanted to highlight one of the more disturbing aspects of public policy discussions in recent years  -  the tendency to put forward opinions as if they were as compelling as solid evidence. We suspect that this will be the first in an ongoing series of examples.

Are working hours getting longer? Hopefully this example reflects someone being misquoted in the article in The Age in Melbourne, rather than how it appears – a researcher suggesting it’s too hard to get reasonable estimates of the extent of a problem and then pronouncing that the problem has diminished:

Some recent studies suggest this may now be a relic of history and that Australians work the longest hours in the developed world.

But Professor Mark Wooden, of the Institute of Applied Economic and Social Research, disagrees … strongly.

“The idea we work the most hours in the world is absolute crap,” he says.

“Lots of people work long hours and lots of people work short hours. We have a mix.”

He argues that workers in Japan and Korea work longer than Australians and that comparing working hours between countries was an inexact science.

People tend to overestimate how long they work as a sort of “badge of courage” and find it difficult to estimate the hours they work accurately, Professor Wooden says.

“I don’t think we can count,” he says. “It’s impossible to know. The study would need to be so invasive.”

His research shows the number of Australians working 50 hours a week or more peaked in the mid-1990s.

“Over the last 10 years, the proportion of Australians working long hours has been dropping.”

A recent conversation with a colleague has reminded me of how traditional social science training has managed to hardwire our brains into some default thinking that needs to be questioned.

Obviously, there are a lot of places one could go with this as an opening statement, but for now, let’s look at the design of survey questions.

Are we unknowingly being caught in a Likert scale trap?

I mentioned a while ago, in a discussion following a Friday Funny post (Data as “the truth”), that I generally avoid the use of Likert scales because they are evaluatively uninterpretable. Andrew Hawkins asked me why. So, belatedly, here’s why.

Real, genuine evaluation is evaluative. In other words, it doesn’t just report descriptive evidence for others to interpret; it combines this evidence with appropriate definitions of ‘quality’ and ‘value’ and draws conclusions about such things as:

• the quality of program/policy/product/etc design and implementation
• the value and practical significance of outcomes
• whether the entire evaluand was a good (or, the best possible) use of time/money/resources or not

Now, as I’ve mentioned before (in the ‘No Value-Free’ post), many so-called evaluations are what we call ‘value-free’, a.k.a. “evaluations NOT”! They skip this whole evaluative inference step. My view: This is not acceptable. JMHO.

The usual alternative is to take descriptive evidence, often gathered using traditional social science methods, and attempt to interpret it relative to the relevant definitions of quality and value.

This sounds straightforward enough, but it’s actually quite tricky. Over the years, I have worked on some ways of making it easier.

Like, why not build evaluative elements into survey questions themselves?

Building evaluative elements into survey questions

A typical survey/questionnaire might ask questions like:

To what extent do you agree or disagree with the following:

OK, so what exactly does a mean of 3.8 (for example) mean? Is that well organized? Excellently? Mediocrely?

In some cases, the mean score (and distribution of scores) is provided alongside the mean and s.d. across a range of programs (this is very common with training programs). This does give an inkling of relative merit compared to others. In other words, you can see if you are generally doing better or worse than ‘the rest of the pack’.

But what if we want to know how good something was in some absolute sense? We might be better than average, but is that actually any good?

Most ‘evaluations’ seem to use the so-called Rorschach inkblot approach (a.k.a. the value-free, or “you work it out” approach). Basically, this means presenting descriptive data such as the above (possibly including comparison data) and letting clients and stakeholders draw their own conclusions about how good the findings are.

If we do want to take it that one step further, to say something explicit about the quality or value of something, how do we do that? Invent cut-offs, e.g. saying that 3.5-3.8 is good, 3.8-4.2 is very good, etc? What would be the basis for these?

Or, what if we ditch the agree-disagree response scale and opt for something that has evaluative terms built right in. Like, for example …

How would you rate the following:

Now, it has to be said, simply reporting summaries of participants’ ratings is not, by itself, “doing” an evaluation. The evaluator still needs to draw an overall conclusion.

However, by using evaluative terms right in the questionnaire, the participant ratings become a lot easier to interpret in terms of quality or value.

I use item designs like these primarily for process evaluation – getting a handle on the quality of content, design, and implementation. They are far more evaluatively interpretable than the traditional Likert scale-type items.

Later this week, stand by for ideas for outcome/impact evaluation that build causation right into the items.

For those new to metrics – and as the sizes of computer hard drives increase astronomically each year – it can be hard to stay on top of all that “giga-this” and “terra-that” terminology. Here’s a helpful guide that’s done the rounds on the Internet (we found it at Michelle’s Joke Vault):

1. Ratio of an igloo’s circumference to its diameter: Eskimo Pi
2. 2000 pounds of Chinese soup: Won ton
3. 1 millionth of a mouthwash: 1 microscope
4. Time between slipping on a peel and smacking the pavement: 1 bananosecond
5. Weight an evangelist carries with God: 1 billigram
6. Time it takes to sail 220 yards at 1 nautical mile per hour: Knot-furlong
7. 365.25 days of drinking low-calorie beer because it’s less filling: 1 lite year
8. Half of a large intestine: 1 semicolon
9. 1000 aches: 1 megahurtz
10. Basic unit of laryngitis: 1 hoarsepower
11. Shortest distance between two jokes: A straight line (think about it for a moment)
12. 453.6 graham crackers: 1 pound cake
13. 1 million-million microphones: 1 megaphone
14. 1 million bicycles: 2 megacycles
15. 365.25 days: 1 unicycle
16. 2000 mockingbirds: two kilomockingbirds
17. 10 cards: 1 decacards
18. 1 kilogram of falling figs: 1 Fig Newton
19. 1000 grams of wet socks: 1 literhosen
20. 1 millionth of a fish: 1 microfiche
21. 1 trillion pins: 1 terrapin
22. 10 rations: 1 decoration
23. 100 rations: 1 C-ration
24. 2 monograms: 1 diagram
25. 2.4 statute miles of intravenous surgical tubing at Yale University Hospital: 1 I.V. League

We also found some more gems to add the list on alphaDictionary.com:

26. 1 dialog = 2 monologs
27. 5 dialogs = 1 decalog
28. 8 nickels = 2 paradigms
29. 2 wharves = 1 paradox
30. 1 million billion picolos = 1 gigolo
31. 10 millipedes = 1 centipede
32. 3 1/3 tridents = 1 decadent
33. 10 cards = 1 decacards
34. One acre = 43,560 square feet = a lotsa toes
35.  4 lawyers = 2 paralegals
36. 2 untruths = 1 paralyze
37. All perjured testimony in a gov’t coverup = 1 scandalize

Got any more to add?

One of the important features of genuine evaluation is appropriate measurement, including dealing with uncertainty, as I was reminded by Chris Coryn of the Evaluation Center at Western Michigan University, in our discussions at the International Summer School on Public Policy Evaluation Research last week.

A free webinar on 16 September 10.30am – 11.30am CDT by Doug Hubbbard, “Measuring Risk – what doesn’t work and what does”, promises to address issues of appropriate measurement when dealing with intangibles and uncertainty.

Doug Hubbard, author of ‘How to Measure Anything: Finding the value of intangibles in business”, has developed the approach of “Applied Information Economics”, which promised to thoughtfully address uncertainty in developing estimates.

According to its Wikipedia entry:

AIE differs in several ways from other popular methods of decision analysis:

Unlike the accounting-style business case or cost benefit analysis, it does not rely entirely on point estimates of uncertain values. Since it uses the Monte Carlo method, uncertainty can be modeled explicitly.

Has anyone read the book? Or will attend the webinar? (As it is scheduled for 1.30am Friday Melbourne time, I don’t think I will be there).

Is it a useful new approach to measuring the hard-to-measure or an oversell on quantification? Is it only applicable to business or does it have applications in the government and not-for-profit sectors? Are the criticisms discussed in some reviews valid? Are there other texts on this issue that are better?

The webinar will focus on:

· The Problem – Why your method may be a ‘management placebo’ and why that is the biggest risk you have

· Problems that many methods ignore – and problems some methods introduce

· What Does Work – Studies reveal some methods show consistent, measurable improvements on the forecasts

· Examples of Real Improvements

· Overview of Applied Information Economics (AIE) Process

· Common Objections to quantitative methods and the misconceptions behind them

Webinar registration is at https://www1.gotomeeting.com/register/746702064

Some more developments in UK development funding, an issue we looked at in a post a few weeks ago.

Lawrence Haddad, Director of the Institute for Development Studies has  an interesting article in the Guardian in response to David Cameron’s statements on international aid.

The best ways to deliver overseas aid are often not easy to measure

David Cameron’s plans risk diverting help away from where it’s really needed

Lawrence discusses David Cameron’s concerns about waste and corruption, with some telling comparisons to UK and USA examples.  He then discusses the risks in focusing on and rewarding only results that can be readily measured.

But my real worry about the new approach is that it will slant aid towards items on which it is easiest to demonstrate delivery. While the focus on “the things that aid can best deliver and that can make a real long term difference” sounds sensible, these very things may not be the most easily assessed.

The comments from readers are also interesting, including this one from “tiojo”:

I have a lot of respect for Mr Haddad’s institution. But he’s demonstrating why he’s an academic and not a practising aid worker. He ends his article by saying,

The most important spots that aid needs to hit may be the hardest to see.

Well Mr Haddad – you and your institution have been hard at work for many years now. Undertaking all the studies, research and evaluation. So tell us please – what are those hard spots that aid needs to hit and how do we hit them?

How do we balance the need to demonstrate progress (or lack of it) in readily understood measures or indicators, and the real challenges in measuring or describing long-term or diffuse outcomes such as community capacity which are also extremely important?

Thanks to Niels Keijzer from the Pelican Initiative – Platform for Evidence-Based Learning and Communication for Social Change for the link