Evaluation of marketing – grappling with the important but hard to measure outcomes

Photo: Phillippe Halsman


“Not everything that counts can be counted, and not everything that can be counted counts.” — Albert Einstein

It’s a familiar discussion in the evaluation world – the importance of getting approximate answers to important questions about what really matters, rather than being seduced into the “indicators” game where we measure the living daylights out of whatever is easiest to measure and then worship that.

Because, after all, quantification gives things a much higher truth value – right?

Right …

The business world in particular still struggles to get buy-in to a mixed method approach to evaluation.

There is a heavy emphasis on quantitative indicators such as financial/bottom line and the easy-to-measure such as web hits, etc.

A recent article in Forbes magazine by marketing specialist Susan Gunelius tries to get her marketing colleagues to think outside the box about the evaluation of marketing and include important, hard-to-measure outcomes such as the nature of engagement with the brand.

Susan suggests we should consider not just traditional Return on [Financial] Investment, but also:

Return on Impression – Eyeballs: the number of people who actually see your ad, marketing material, or other tangible marketing piece

Return on Impression – Perceptions: feelings for those brands and emotional connections to them

Return on Opportunity: the indirect marketing opportunities that it could lead to as people discuss it and share it across the social web (and offline) versus the time and monetary commitment that effort requires

Return on Engagement: the value of a positive online buzz about your brand, products, services, and business; how well your brand is performing in terms of building and sustaining relationships with both consumers and influencers

Return on Objectives: how well marketing efforts help a business move in the right direction to meet its long-term objectives (such as brand building)

With an evaluator’s lens on, perhaps we can suggest some improvements to this line-up? Please chime in with comments, and I will plan on a post on this topic as a follow-up.

The comments section of Susan Gunelius’ article shows just what an uphill battle it is in the business sector to get buy-in to any non-financial considerations. For example:

Sorry but in my opinion this smacks of a desperate attempt to cover up marketing’s tenuous or fictional link to real business metrics. Either marketing can contribute directly or indirectly to growth in a tangible way or it cannot. No amount of window dressing (fancy new handles, job titles, buzzwords) will help–in fact they only contribute to the perception that marketing is clinging to relevancy. Effective CEOs, CFOs and CMOs who “get it” have gotten past these arguments; they have made their determination on marketing’s relevance and effectiveness based on their industry and channels and have right-sized their marketing investments accordingly! — Bob London

Sound familiar?

As evaluators we often have to respond to those who believe that a mixed methods approach means we are ignoring the bottom line. [And actually, a lot of evaluations DO ignore the bottom line – but that’s another story, and not an inherent flaw of genuine mixed methods evaluation.]

I did like Susan’s response to this and another “I agree” comment:

I think that’s the point though. Both you and Bob left comments on this post and Bob also tweeted it. Is the only value you’ll both derive from those comments the number of page views that this post gets (assuming every person who viewed this article also read the comments)? Or are there soft metrics that matter as well — in terms of audience perception and sentiment related to what you said in your comments and how those comments are discussed outside of this post? If you only track page views and retweets (which I’m assuming you don’t), you’re missing part of the story.

From an evaluation perspective, a few other points occur.

Financial outcomes are ‘lag’ indicators of success. They usually tell you quite a while after the fact whether there has been any change in the bottom line.

In general it’s better to also get an early read on what’s working or likely to work – the ‘lead’ indicators. These include, for example, how consumers react to or engage with the brand.

A lot of this earlier evaluation can actually happen prior to investing a huge spend in a particular marketing strategy.

This is one of the key tenets of the Balanced Scorecard approach, which is widely used in business.

The so-called ‘soft’ indicators of success are part of a causal chain that includes financial outcomes. As one of the commenters on the article put it:

Often – impressions > inquiries > leads > opps > $ … engagement can be around the loyalty constellation or opportunity around market share … we mostly use proxies anyway in linking soft to hard. — mperla

See? There’s often an important theory of change buried in all sorts of evaluative thinking!

But a very important point here is that following this causal chain is a key way to be able to attribute financial outcomes to the marketing activities (or, to estimate their contribution). Causal inference is notoriously difficult if you only track the downstream (long term) indicators.

Marketing is not just about advertising. A serious strategic approach to marketing involves truly understanding the needs and wants of consumers and building these right into the product or service development in the first place.

And if this groundwork has happened, then a logical place to go when looking at the effectiveness of advertising is to see how well it keys on solving that problem for them, or creating that value, or forging that emotional relationship with the brand.

An important message here is that what you sell (and therefore what we must evaluate) is not the “thing” (the product or the service) but the value it generates for consumers or recipients (and then, down the causal chain, to value for shareholders, or for society).

And that value may be solving a problem, making life better, or creating an emotional connection that produces enjoyment – or all three.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” Poet, Dancer, Producer, Playwright, Director, Author Maya Angelou (born 1928)


Implications for evaluation in your sector?

Or, for how we market our services?

There is as much for us to learn from marketing as we have for them to learn from us.

1 comment to Evaluation of marketing – grappling with the important but hard to measure outcomes

  • Couple of thoughts along the line of ‘marketing folk are not keeping in touch with changes in the world.’

    1. Many of us fill out feedback surveys from really big outfits (this includes philanthropies and service orgs, not just product sellers) that show incredibly crude approaches to the nuances of impact, nuances that clearly are linked to future buying and other tangible changes in behavior. And how many of those surveys bother to ask for suggestions about improving the survey (not just the product or service)? Precious few, that’s how much systematic self-improvement is going on. So we know that for all the hard-nosed ROI adherents, there’s plenty of slippage in getting a grip on even hard-core ROI, i.e., plenty of bad marketing practice, despite the rhetoric.

    2. Perhaps even more importantly, there is plenty of evidence that the hard nose ROI’rs don’t even bother to use the data that is already there without doing any more market research. It’s buried in the time-stamped negative feedback publicly available on the websites of the big retailers like Amazon. Focus on the detailed feedback provided by unhappy campers when their numbers get up into the scores (i.e. enough to reduce chances of feedback faked by competitors’ hirelings), and span several years of reports of simple failures, rude online ‘help,’ and failures to improve bad design, and you’re seeing a disconnect between marketing and production/design that means there’s a serious problem in the organization’s management of marketing data.

    In other words, for 1% to 5% of the cost of a new marketing effort, a few days work by a smart evaluator would show how to get big results on basic ROI, and then the client might start in on the nuances from the Forbes article that will surely yield still further gains. Dissing them would be worth listening to if the critics showed signs of basic competence.

    Michael Scriven