Photo: Phillippe Halsman

 

“Not everything that counts can be counted, and not everything that can be counted counts.” — Albert Einstein

It’s a familiar discussion in the evaluation world – the importance of getting approximate answers to important questions about what really matters, rather than being seduced into the “indicators” game where we measure the living daylights out of whatever is easiest to measure and then worship that.

Because, after all, quantification gives things a much higher truth value – right?

Right …

The business world in particular still struggles to get buy-in to a mixed method approach to evaluation.

There is a heavy emphasis on quantitative indicators such as financial/bottom line and the easy-to-measure such as web hits, etc.

A recent article in Forbes magazine by marketing specialist Susan Gunelius tries to get her marketing colleagues to think outside the box about the evaluation of marketing and include important, hard-to-measure outcomes such as the nature of engagement with the brand.

Susan suggests we should consider not just traditional Return on [Financial] Investment, but also:

Return on Impression – Eyeballs: the number of people who actually see your ad, marketing material, or other tangible marketing piece

Return on Impression – Perceptions: feelings for those brands and emotional connections to them

Return on Opportunity: the indirect marketing opportunities that it could lead to as people discuss it and share it across the social web (and offline) versus the time and monetary commitment that effort requires

Return on Engagement: the value of a positive online buzz about your brand, products, services, and business; how well your brand is performing in terms of building and sustaining relationships with both consumers and influencers

Return on Objectives: how well marketing efforts help a business move in the right direction to meet its long-term objectives (such as brand building)

With an evaluator’s lens on, perhaps we can suggest some improvements to this line-up? Please chime in with comments, and I will plan on a post on this topic as a follow-up.

The comments section of Susan Gunelius’ article shows just what an uphill battle it is in the business sector to get buy-in to any non-financial considerations. For example:

Sorry but in my opinion this smacks of a desperate attempt to cover up marketing’s tenuous or fictional link to real business metrics. Either marketing can contribute directly or indirectly to growth in a tangible way or it cannot. No amount of window dressing (fancy new handles, job titles, buzzwords) will help–in fact they only contribute to the perception that marketing is clinging to relevancy. Effective CEOs, CFOs and CMOs who “get it” have gotten past these arguments; they have made their determination on marketing’s relevance and effectiveness based on their industry and channels and have right-sized their marketing investments accordingly! — Bob London

Sound familiar?

As evaluators we often have to respond to those who believe that a mixed methods approach means we are ignoring the bottom line. [And actually, a lot of evaluations DO ignore the bottom line - but that's another story, and not an inherent flaw of genuine mixed methods evaluation.]

I did like Susan’s response to this and another “I agree” comment:

I think that’s the point though. Both you and Bob left comments on this post and Bob also tweeted it. Is the only value you’ll both derive from those comments the number of page views that this post gets (assuming every person who viewed this article also read the comments)? Or are there soft metrics that matter as well — in terms of audience perception and sentiment related to what you said in your comments and how those comments are discussed outside of this post? If you only track page views and retweets (which I’m assuming you don’t), you’re missing part of the story.

From an evaluation perspective, a few other points occur.

Financial outcomes are ‘lag’ indicators of success. They usually tell you quite a while after the fact whether there has been any change in the bottom line.

In general it’s better to also get an early read on what’s working or likely to work – the ‘lead’ indicators. These include, for example, how consumers react to or engage with the brand.

A lot of this earlier evaluation can actually happen prior to investing a huge spend in a particular marketing strategy.

This is one of the key tenets of the Balanced Scorecard approach, which is widely used in business.

The so-called ‘soft’ indicators of success are part of a causal chain that includes financial outcomes. As one of the commenters on the article put it:

Often – impressions > inquiries > leads > opps > $ … engagement can be around the loyalty constellation or opportunity around market share … we mostly use proxies anyway in linking soft to hard. — mperla

See? There’s often an important theory of change buried in all sorts of evaluative thinking!

But a very important point here is that following this causal chain is a key way to be able to attribute financial outcomes to the marketing activities (or, to estimate their contribution). Causal inference is notoriously difficult if you only track the downstream (long term) indicators.

Marketing is not just about advertising. A serious strategic approach to marketing involves truly understanding the needs and wants of consumers and building these right into the product or service development in the first place.

And if this groundwork has happened, then a logical place to go when looking at the effectiveness of advertising is to see how well it keys on solving that problem for them, or creating that value, or forging that emotional relationship with the brand.

An important message here is that what you sell (and therefore what we must evaluate) is not the “thing” (the product or the service) but the value it generates for consumers or recipients (and then, down the causal chain, to value for shareholders, or for society).

And that value may be solving a problem, making life better, or creating an emotional connection that produces enjoyment – or all three.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” Poet, Dancer, Producer, Playwright, Director, Author Maya Angelou (born 1928)

Thoughts?

Implications for evaluation in your sector?

Or, for how we market our services?

There is as much for us to learn from marketing as we have for them to learn from us.

Business Cartoon by Andertoons

It’s a puzzling fact that one of the most costly and stressful organizational change strategies – restructuring – is virtually NEVER subjected to any serious evaluation.

The reshuffling of lines and boxes; the layoffs; the unwanted ‘domino effect’ turnover of valuable employees; the ripple effects on families; whether the intended improvements in focus, efficiency, productivity, and competitive advantage emerge; or whether they outweigh the costs and negative side effects … all subjected to little more than some armchair analysis by management.

Public and private, for-profit, non-profit, and government organizations spend hundreds of thousands of dollars (sometimes more) on working out the best restructure and implementing the change.

But for some unfathomable reason, no-one ever says “I wonder if we should see whether all that expenditure and upheaval was worth it” or “We should try and learn from this huge initiative so we can get it right as it unfolds and do it better next time.”

Or do they?

In 2005, four intrepid health researchers from Norway and Canada published a review of all the articles that evaluated reorganizations.

The following is their abstract, but it is well worth clicking through to the full article!

A surrealistic mega-analysis of redisorganization theories

SUMMARY

Background We are sick and tired of being
redisorganized.

Objective To systematically review the empirical
evidence for organizational theories and repeated
reorganizations.

Methods We did not find anything worth reading,
other than Dilbert, so we fantasized. Unfortunately, our
fantasies may well resemble many people’s realities.
We are sorry about this, but it is not our fault.

Results We discovered many reasons for repeated
reorganizations, the most common being ‘no good
reason’. We estimated that trillions of dollars are being
spent on strategic and organizational planning
activities each year, thus providing lots of good reasons
for hundreds of thousands of people, including us,
to get into the business. New leaders who are
intoxicated with the prospect of change further fuel
perpetual cycles of redisorganization. We identified
eight indicators of successful redisorganizations,
including large consultancy fees paid to friends and
relatives.

Conclusions We propose the establishment of ethics
committees to review all future redisorganization
proposals in order to put a stop to uncontrolled,
unplanned experimentation inflicted on providers and
users of the health services.

It was Michael Scriven‘s birthday this week, which is a fine time to introduce our Friday Funny with a short quote from the Evaluation Thesaurus, which lists the following entry. As evaluators, we are all familiar with this phenomenon in our work or everyday lives, in one form or another:

KILL THE MESSENGER (phenomenon)  The tendency to punish the bearer of bad tidings, who is often the evaluator. A phenomenon related to valuephobia – mindless striking back at the proximate cause of pain, even when it’s clear this is neither just punishment nor likely to be curative of the problem. Much of the current attack on testing – for example, minimum competence testing for graduation or teaching licensure – is pure KTM, like many of the elaborate and rationalized earlier attacks on course grades. The presence of the rationalizations identify these specimens of the subspecies, AFTOC: Kill The Messenger – After a Fair Trial, Of Course.

But there are some settings we do in fact seem to see very gracious acceptance of less-than-hoped-for evaluative conclusions. One such setting is in the movie industry awards.

But how often do they really speak their minds?

This hilarious sketch from British comedian Rowan Atkinson shows an non-awardee accepting an award on behalf of his colleague, and finishing off the acceptance speech with some feedback for the judging panel. Enjoy!

[If you can't see the video (e.g. because you are reading this on the email feed), you will need to click through to the site.]

Stuart Henderson’s mention of The Two Second Advantage (see the LinkedIn discussion referred to in the post from earlier this week) reminds me of the work of business strategist Arie De Geus, who discusses how learning organizations use scenario planning to create “memories of the future”.

It seems to me that this idea has potential not only in looking at emergent trends in evaluation (such as those Stuart is asking about on LinkedIn), but also as a lens we can apply to programs and organizations to help them consider how to ‘future-proof’ themselves.

De Geus points out that, although we can’t predict exactly what changes will happen in the future, it is possible to brainstorm a few different possible scenarios about what might happen.

Some scenarios will be more likely than others – some will seem almost certain; some will seem incredibly unlikely; some may incorporate natural disasters or substantial shifts in the world or in the community (either gradual or dramatic).

The scenarios may be developed – minimally or extensively, depending how likely the scenario seems – into contingency plans (i.e., What would we do if …).

The idea behind scenario planning is this:

• We know from psychology that the human brain is more likely to spot the early signs of change if the person has rehearsed the possibility beforehand. [I am sure some clever person has a citation for this!]

• Likewise, organizations are more likely to perceive change early – and have ideas for how to react effectively to take advantage of it – if they have gone through the process of dreaming up the various possible future scenarios.

Now, how might this help programs and organizations build strength and sustainability?

Think of the kinds of major and minor changes that could have major impacts on what the organization does or how it does it. Examples might include:

• A key funding source completely disappearing or dramatically reducing
• An immigration policy altering the demographics (and therefore the needs) of the population they serve
• A new technology enabling them to access remote areas, or making some aspect of their services obsolete
• A natural disaster such as an earthquake, volcanic eruption, or flood
• Two or three key people leaving the organization

Evaluations often have to answer questions about the sustainability, transferability, or applicability of programs, services, and activities over time, over space, over borders. One way of doing this is to help stakeholders think of possible scenarios and how well equipped the program or organization or community is to handle them.

Scenario planning is a tool that, as De Geus says, helps organizations create “memories of the future” that can help them detect the early signs of even unlikely change and to be better equipped to respond to it.

I am sure the folks involved in disaster-preparedness and emergency evaluation have a lot to say on this topic, as will those working in countries where political volatility is high and those with an interest in complexity and unpredictability – please chime in and share your thoughts!

Katherine Hay continues her guest blogging on evidence and evaluation.

Ben Goldacre in The Guardian wrote that UK politicians “are ignorant about trials and they’re weird about evidence.” He contrasts this with international development where he talks about the “amazing work testing interventions around the world with proper, randomised trials.”  He goes on to say, policy makers in the UK, just need to “define your outcome, randomise…and you’ll have the answer by the end of next parliament.”  He notes all these trials (somehow) won’t cost money but will save unprecedented amounts of money.  He then concludes that, “politicians are …too arrogant to have their ideologies questioned, and too scared…of hard data on their interventions.’

It’s an entertaining article.  The idea of doing a trial on every single UK policy is funny.  The idea that they are free is even funnier.  Imagine how British parents would react when they brought their kids to school and were told what group their child was randomly selected to be in?  Perhaps:

In a large class but with a highly rated teacher.

• In a small class but with a less experienced teacher.
• In a small class with no hot lunch…

And so on.   The permutations to test every UK policy would be never-ending.

Goldacre is obviously being extreme to make a point.  But is he correct?  Have countries who conducted randomized trials saved huge amounts on their interventions?

I’ve seen no evidence that trials are more likely to inform policies than other evaluations or research. I expect that they are subject to the same challenges of use as other types of evidence.

If we accept that “working” can mean different things to different groups, and that views on what is ‘worth the money’ usually vary based on people’s values and their position in society, than why would we assume that studies with statistical power will lead to change on the ground? Evaluation can give us more evidence – and must give us better quality evidence – but the idea that policy making is just a computation of evidence is wrong.   Evidence is only one piece of policy making.  Evidence can, and often is, interpreted and used to reinforce dominant policies.

For example, the country where I live, India, has a system that distributes grains to the poor.  Some people think this system should be replaced with a system where cash is giving to poor families who can buy the food or grains that they choose.  Others feel that dismantling this system will mean food grain that was getting to the poor and children will be replaced by spending on things like alcohol.  Different groups have done studies on whether people want this change.  Some studies show that people do and some studies show that people don’t.

Part of the solution is about design.  You have to be confident that the evidence you have is good quality.  Do people really want it or not? But it’s not just about design.  Even with convincing findings, the policy maker has multiple elements to weigh. In New Delhi, the capital, the government decided that they wanted to experiment with the cash transfer.  But was it because they were comfortable having their ‘ideology tested’ or because their ideology lead them to prefer such a system? They were criticized for the latter; for having a position.  But they were elected on their positions; pushing for more open reality testing is not about wishing away positions.

If they get the design right they may know pretty accurately how many families want or do not want changes in this system and if they run a trial well they may also know about some outcomes.  But that data won’t tell them the ‘right choice’.  For example, how much of an increase in alcohol consumption is ok, or is trumped by increasing the poor’s control of spending choices, or increased efficiencies?  Those decisions are values based and values are often political. For example, Abhijit Sen, a noted economist and member of the Indian Planning Commission, noted “politicians will never accept a dismantling of the PDS’ but added, “Forget the politicians, what matters most is what the voters think.”

We cannot wish away politics and nor should we want to.  My point is that we need to get much more strategic on pathways to use if we want to influence policy with evaluation.

Let me give you a wonderful final little example. Two PhD candidates from Yale did an experiment in a New Delhi slum. The subjects wanted to apply for ration cards. They were randomly assigned to one of four groups. The first group applied for the ration card and did nothing more, the second attached a letter of recommendation from an NGO to their application, the third paid a bribe after putting in their application, and the fourth enquired about the status of their ration card application through a right to information (RTI) request.  The researchers found that the group that paid a bribe was the most successful, but, the group that put in an RTI request was almost as successful. Hardly anyone in the other two groups received their ration card.

Clever experiment.  They answered an interesting question and will likely get their PHD’s in the process.

But corruption in the ration card system is not fixed. Also, NGO’s and others were already using the RTI for things exactly like this.  This experiment adds more evidence to existing evidence that the RTI Act is a useful tool against corruption.  Is it THE answer?  No.  Is it helpful?  Yes.

Assuming that evidence alone will change things is wrong.  Evidence matters, and should be made to matter more, but it’s not the only thing that matters.  Recognizing this doesn’t weaken evaluation, quite the opposite, it actually creates greater opportunities for ‘genuine evaluation’