Photo: Phillippe Halsman

 

“Not everything that counts can be counted, and not everything that can be counted counts.” — Albert Einstein

It’s a familiar discussion in the evaluation world – the importance of getting approximate answers to important questions about what really matters, rather than being seduced into the “indicators” game where we measure the living daylights out of whatever is easiest to measure and then worship that.

Because, after all, quantification gives things a much higher truth value – right?

Right …

The business world in particular still struggles to get buy-in to a mixed method approach to evaluation.

There is a heavy emphasis on quantitative indicators such as financial/bottom line and the easy-to-measure such as web hits, etc.

A recent article in Forbes magazine by marketing specialist Susan Gunelius tries to get her marketing colleagues to think outside the box about the evaluation of marketing and include important, hard-to-measure outcomes such as the nature of engagement with the brand.

Susan suggests we should consider not just traditional Return on [Financial] Investment, but also:

Return on Impression – Eyeballs: the number of people who actually see your ad, marketing material, or other tangible marketing piece

Return on Impression – Perceptions: feelings for those brands and emotional connections to them

Return on Opportunity: the indirect marketing opportunities that it could lead to as people discuss it and share it across the social web (and offline) versus the time and monetary commitment that effort requires

Return on Engagement: the value of a positive online buzz about your brand, products, services, and business; how well your brand is performing in terms of building and sustaining relationships with both consumers and influencers

Return on Objectives: how well marketing efforts help a business move in the right direction to meet its long-term objectives (such as brand building)

With an evaluator’s lens on, perhaps we can suggest some improvements to this line-up? Please chime in with comments, and I will plan on a post on this topic as a follow-up.

The comments section of Susan Gunelius’ article shows just what an uphill battle it is in the business sector to get buy-in to any non-financial considerations. For example:

Sorry but in my opinion this smacks of a desperate attempt to cover up marketing’s tenuous or fictional link to real business metrics. Either marketing can contribute directly or indirectly to growth in a tangible way or it cannot. No amount of window dressing (fancy new handles, job titles, buzzwords) will help–in fact they only contribute to the perception that marketing is clinging to relevancy. Effective CEOs, CFOs and CMOs who “get it” have gotten past these arguments; they have made their determination on marketing’s relevance and effectiveness based on their industry and channels and have right-sized their marketing investments accordingly! — Bob London

Sound familiar?

As evaluators we often have to respond to those who believe that a mixed methods approach means we are ignoring the bottom line. [And actually, a lot of evaluations DO ignore the bottom line - but that's another story, and not an inherent flaw of genuine mixed methods evaluation.]

I did like Susan’s response to this and another “I agree” comment:

I think that’s the point though. Both you and Bob left comments on this post and Bob also tweeted it. Is the only value you’ll both derive from those comments the number of page views that this post gets (assuming every person who viewed this article also read the comments)? Or are there soft metrics that matter as well — in terms of audience perception and sentiment related to what you said in your comments and how those comments are discussed outside of this post? If you only track page views and retweets (which I’m assuming you don’t), you’re missing part of the story.

From an evaluation perspective, a few other points occur.

Financial outcomes are ‘lag’ indicators of success. They usually tell you quite a while after the fact whether there has been any change in the bottom line.

In general it’s better to also get an early read on what’s working or likely to work – the ‘lead’ indicators. These include, for example, how consumers react to or engage with the brand.

A lot of this earlier evaluation can actually happen prior to investing a huge spend in a particular marketing strategy.

This is one of the key tenets of the Balanced Scorecard approach, which is widely used in business.

The so-called ‘soft’ indicators of success are part of a causal chain that includes financial outcomes. As one of the commenters on the article put it:

Often – impressions > inquiries > leads > opps > $ … engagement can be around the loyalty constellation or opportunity around market share … we mostly use proxies anyway in linking soft to hard. — mperla

See? There’s often an important theory of change buried in all sorts of evaluative thinking!

But a very important point here is that following this causal chain is a key way to be able to attribute financial outcomes to the marketing activities (or, to estimate their contribution). Causal inference is notoriously difficult if you only track the downstream (long term) indicators.

Marketing is not just about advertising. A serious strategic approach to marketing involves truly understanding the needs and wants of consumers and building these right into the product or service development in the first place.

And if this groundwork has happened, then a logical place to go when looking at the effectiveness of advertising is to see how well it keys on solving that problem for them, or creating that value, or forging that emotional relationship with the brand.

An important message here is that what you sell (and therefore what we must evaluate) is not the “thing” (the product or the service) but the value it generates for consumers or recipients (and then, down the causal chain, to value for shareholders, or for society).

And that value may be solving a problem, making life better, or creating an emotional connection that produces enjoyment – or all three.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” Poet, Dancer, Producer, Playwright, Director, Author Maya Angelou (born 1928)

Thoughts?

Implications for evaluation in your sector?

Or, for how we market our services?

There is as much for us to learn from marketing as we have for them to learn from us.

Old mistake in today’s article on European responses to austerity measures – here, as reported by Karen Kissane in The Age in Melbourne:

Meanwhile, in Greece, a country spiralling into poverty with more than half of its young people unemployed,

This seems to be a common misinterpretation. A Reuters report led with the headline:

Over half of Greek youth unemployed

A BBC report stated:

More than half of young people in Greece are unemployed …

Well. no, actually.  Unemployment rates are the percentage of those available for and looking for work who are unemployed.  Since a large proportion of young people are engaged in full time study, the proportion of the population who are unemployed is much less.

According to the European Commission’s Eurostat page “Statistics explained” on unemployment statistiscs, Greece’s youth unemployment rate has gone up from 25.7% in 2009 to 44.4 in 2011 and 49.3 in the last quarter of 2011 – but the ratio – that is, youth unemployed as a percentage of the youth population has only increased from 8% in 2009 to 13% in 2011.  Still a tragedy for those affected, but very different to the picture painted.

Discussions about public policy need to be based in accurate representation of the situation. This is a basic mistake that should not be made.

Pic by JitterBuffer

Over the Antipodean summer Genuine Evaluation goes to the beach instead of blogging.  We’re back now, brushing off the sand, and planning more discussions about what it means to do genuine evaluation, plus sharing some insights from the African evaluation conference in Accra, Ghana.

To start the year, we wanted to highlight one of the more disturbing aspects of public policy discussions in recent years  -  the tendency to put forward opinions as if they were as compelling as solid evidence. We suspect that this will be the first in an ongoing series of examples.

Are working hours getting longer? Hopefully this example reflects someone being misquoted in the article in The Age in Melbourne, rather than how it appears – a researcher suggesting it’s too hard to get reasonable estimates of the extent of a problem and then pronouncing that the problem has diminished:

Some recent studies suggest this may now be a relic of history and that Australians work the longest hours in the developed world.

But Professor Mark Wooden, of the Institute of Applied Economic and Social Research, disagrees … strongly.

“The idea we work the most hours in the world is absolute crap,” he says.

“Lots of people work long hours and lots of people work short hours. We have a mix.”

He argues that workers in Japan and Korea work longer than Australians and that comparing working hours between countries was an inexact science.

People tend to overestimate how long they work as a sort of “badge of courage” and find it difficult to estimate the hours they work accurately, Professor Wooden says.

“I don’t think we can count,” he says. “It’s impossible to know. The study would need to be so invasive.”

His research shows the number of Australians working 50 hours a week or more peaked in the mid-1990s.

“Over the last 10 years, the proportion of Australians working long hours has been dropping.”

Photo: Joe Armao (The Age)

What’s worse than no evaluation? An evaluation that is wrong but you think is right.

Organizations that provide authoratitive evaluations have an obligation to meet high standards of accuracy and consistency. It is therefore hard to believe the series of events that led to Victoria’s Environmental Protection Agency listing Melbourne’s bayside beaches as “good”, and suitable for swimming when the level of bacteria was 40 times the acceptable limit.

According to The Age report,

[EPA] staff were not working on the ChristmasDay and Boxing Day public holidays, and as a result old forecast information was fed automatically into the website from Saturday until yesterday. These forecasts were made on Friday afternoon, and did not take into account the ferociousness of the Christmas Day storms.

On Sunday (Christmas Day) a massive storm hit Melbourne, sending debris, rubbish, cigarette butts and dog droppings into storm water drains. But for two days the EPA was operating on autopilot, issuing reports on the web site and tweets based on the projected water quality not the actual water quality.

And what has the EPA learned from this? Nothing, apparently. According to The Age, no EPA staff will be working on the New Year’s Day public holiday, and once again reports will be based on the weather forecast not on actual testing. A spokesman said beachgoers “should use their own judgment” in deciding to go swimming after storms like those that hit on Christmas Day.

Which does raise the question – if beachgoers should use their own judgment, because it is more likely to be accurate than the official reports, what’s the point of having the official reports?

Here’s what SHOULD happen. Either roster someone to work on these public holidays and ensure the reports are actually based on data ( it’s summer here and people are going to the beaches) OR issue a clear statement on the website, tweets and to news media that an accurate report cannot be provided due to the public holidays. No report is better than an inaccurate report.

Katherine Hay continues her guest blogging on evidence and evaluation.

Ben Goldacre in The Guardian wrote that UK politicians “are ignorant about trials and they’re weird about evidence.” He contrasts this with international development where he talks about the “amazing work testing interventions around the world with proper, randomised trials.”  He goes on to say, policy makers in the UK, just need to “define your outcome, randomise…and you’ll have the answer by the end of next parliament.”  He notes all these trials (somehow) won’t cost money but will save unprecedented amounts of money.  He then concludes that, “politicians are …too arrogant to have their ideologies questioned, and too scared…of hard data on their interventions.’

It’s an entertaining article.  The idea of doing a trial on every single UK policy is funny.  The idea that they are free is even funnier.  Imagine how British parents would react when they brought their kids to school and were told what group their child was randomly selected to be in?  Perhaps:

In a large class but with a highly rated teacher.

• In a small class but with a less experienced teacher.
• In a small class with no hot lunch…

And so on.   The permutations to test every UK policy would be never-ending.

Goldacre is obviously being extreme to make a point.  But is he correct?  Have countries who conducted randomized trials saved huge amounts on their interventions?

I’ve seen no evidence that trials are more likely to inform policies than other evaluations or research. I expect that they are subject to the same challenges of use as other types of evidence.

If we accept that “working” can mean different things to different groups, and that views on what is ‘worth the money’ usually vary based on people’s values and their position in society, than why would we assume that studies with statistical power will lead to change on the ground? Evaluation can give us more evidence – and must give us better quality evidence – but the idea that policy making is just a computation of evidence is wrong.   Evidence is only one piece of policy making.  Evidence can, and often is, interpreted and used to reinforce dominant policies.

For example, the country where I live, India, has a system that distributes grains to the poor.  Some people think this system should be replaced with a system where cash is giving to poor families who can buy the food or grains that they choose.  Others feel that dismantling this system will mean food grain that was getting to the poor and children will be replaced by spending on things like alcohol.  Different groups have done studies on whether people want this change.  Some studies show that people do and some studies show that people don’t.

Part of the solution is about design.  You have to be confident that the evidence you have is good quality.  Do people really want it or not? But it’s not just about design.  Even with convincing findings, the policy maker has multiple elements to weigh. In New Delhi, the capital, the government decided that they wanted to experiment with the cash transfer.  But was it because they were comfortable having their ‘ideology tested’ or because their ideology lead them to prefer such a system? They were criticized for the latter; for having a position.  But they were elected on their positions; pushing for more open reality testing is not about wishing away positions.

If they get the design right they may know pretty accurately how many families want or do not want changes in this system and if they run a trial well they may also know about some outcomes.  But that data won’t tell them the ‘right choice’.  For example, how much of an increase in alcohol consumption is ok, or is trumped by increasing the poor’s control of spending choices, or increased efficiencies?  Those decisions are values based and values are often political. For example, Abhijit Sen, a noted economist and member of the Indian Planning Commission, noted “politicians will never accept a dismantling of the PDS’ but added, “Forget the politicians, what matters most is what the voters think.”

We cannot wish away politics and nor should we want to.  My point is that we need to get much more strategic on pathways to use if we want to influence policy with evaluation.

Let me give you a wonderful final little example. Two PhD candidates from Yale did an experiment in a New Delhi slum. The subjects wanted to apply for ration cards. They were randomly assigned to one of four groups. The first group applied for the ration card and did nothing more, the second attached a letter of recommendation from an NGO to their application, the third paid a bribe after putting in their application, and the fourth enquired about the status of their ration card application through a right to information (RTI) request.  The researchers found that the group that paid a bribe was the most successful, but, the group that put in an RTI request was almost as successful. Hardly anyone in the other two groups received their ration card.

Clever experiment.  They answered an interesting question and will likely get their PHD’s in the process.

But corruption in the ration card system is not fixed. Also, NGO’s and others were already using the RTI for things exactly like this.  This experiment adds more evidence to existing evidence that the RTI Act is a useful tool against corruption.  Is it THE answer?  No.  Is it helpful?  Yes.

Assuming that evidence alone will change things is wrong.  Evidence matters, and should be made to matter more, but it’s not the only thing that matters.  Recognizing this doesn’t weaken evaluation, quite the opposite, it actually creates greater opportunities for ‘genuine evaluation’