Photo: Phillippe Halsman

 

“Not everything that counts can be counted, and not everything that can be counted counts.” — Albert Einstein

It’s a familiar discussion in the evaluation world – the importance of getting approximate answers to important questions about what really matters, rather than being seduced into the “indicators” game where we measure the living daylights out of whatever is easiest to measure and then worship that.

Because, after all, quantification gives things a much higher truth value – right?

Right …

The business world in particular still struggles to get buy-in to a mixed method approach to evaluation.

There is a heavy emphasis on quantitative indicators such as financial/bottom line and the easy-to-measure such as web hits, etc.

A recent article in Forbes magazine by marketing specialist Susan Gunelius tries to get her marketing colleagues to think outside the box about the evaluation of marketing and include important, hard-to-measure outcomes such as the nature of engagement with the brand.

Susan suggests we should consider not just traditional Return on [Financial] Investment, but also:

Return on Impression – Eyeballs: the number of people who actually see your ad, marketing material, or other tangible marketing piece

Return on Impression – Perceptions: feelings for those brands and emotional connections to them

Return on Opportunity: the indirect marketing opportunities that it could lead to as people discuss it and share it across the social web (and offline) versus the time and monetary commitment that effort requires

Return on Engagement: the value of a positive online buzz about your brand, products, services, and business; how well your brand is performing in terms of building and sustaining relationships with both consumers and influencers

Return on Objectives: how well marketing efforts help a business move in the right direction to meet its long-term objectives (such as brand building)

With an evaluator’s lens on, perhaps we can suggest some improvements to this line-up? Please chime in with comments, and I will plan on a post on this topic as a follow-up.

The comments section of Susan Gunelius’ article shows just what an uphill battle it is in the business sector to get buy-in to any non-financial considerations. For example:

Sorry but in my opinion this smacks of a desperate attempt to cover up marketing’s tenuous or fictional link to real business metrics. Either marketing can contribute directly or indirectly to growth in a tangible way or it cannot. No amount of window dressing (fancy new handles, job titles, buzzwords) will help–in fact they only contribute to the perception that marketing is clinging to relevancy. Effective CEOs, CFOs and CMOs who “get it” have gotten past these arguments; they have made their determination on marketing’s relevance and effectiveness based on their industry and channels and have right-sized their marketing investments accordingly! — Bob London

Sound familiar?

As evaluators we often have to respond to those who believe that a mixed methods approach means we are ignoring the bottom line. [And actually, a lot of evaluations DO ignore the bottom line - but that's another story, and not an inherent flaw of genuine mixed methods evaluation.]

I did like Susan’s response to this and another “I agree” comment:

I think that’s the point though. Both you and Bob left comments on this post and Bob also tweeted it. Is the only value you’ll both derive from those comments the number of page views that this post gets (assuming every person who viewed this article also read the comments)? Or are there soft metrics that matter as well — in terms of audience perception and sentiment related to what you said in your comments and how those comments are discussed outside of this post? If you only track page views and retweets (which I’m assuming you don’t), you’re missing part of the story.

From an evaluation perspective, a few other points occur.

Financial outcomes are ‘lag’ indicators of success. They usually tell you quite a while after the fact whether there has been any change in the bottom line.

In general it’s better to also get an early read on what’s working or likely to work – the ‘lead’ indicators. These include, for example, how consumers react to or engage with the brand.

A lot of this earlier evaluation can actually happen prior to investing a huge spend in a particular marketing strategy.

This is one of the key tenets of the Balanced Scorecard approach, which is widely used in business.

The so-called ‘soft’ indicators of success are part of a causal chain that includes financial outcomes. As one of the commenters on the article put it:

Often – impressions > inquiries > leads > opps > $ … engagement can be around the loyalty constellation or opportunity around market share … we mostly use proxies anyway in linking soft to hard. — mperla

See? There’s often an important theory of change buried in all sorts of evaluative thinking!

But a very important point here is that following this causal chain is a key way to be able to attribute financial outcomes to the marketing activities (or, to estimate their contribution). Causal inference is notoriously difficult if you only track the downstream (long term) indicators.

Marketing is not just about advertising. A serious strategic approach to marketing involves truly understanding the needs and wants of consumers and building these right into the product or service development in the first place.

And if this groundwork has happened, then a logical place to go when looking at the effectiveness of advertising is to see how well it keys on solving that problem for them, or creating that value, or forging that emotional relationship with the brand.

An important message here is that what you sell (and therefore what we must evaluate) is not the “thing” (the product or the service) but the value it generates for consumers or recipients (and then, down the causal chain, to value for shareholders, or for society).

And that value may be solving a problem, making life better, or creating an emotional connection that produces enjoyment – or all three.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” Poet, Dancer, Producer, Playwright, Director, Author Maya Angelou (born 1928)

Thoughts?

Implications for evaluation in your sector?

Or, for how we market our services?

There is as much for us to learn from marketing as we have for them to learn from us.

Old mistake in today’s article on European responses to austerity measures – here, as reported by Karen Kissane in The Age in Melbourne:

Meanwhile, in Greece, a country spiralling into poverty with more than half of its young people unemployed,

This seems to be a common misinterpretation. A Reuters report led with the headline:

Over half of Greek youth unemployed

A BBC report stated:

More than half of young people in Greece are unemployed …

Well. no, actually.  Unemployment rates are the percentage of those available for and looking for work who are unemployed.  Since a large proportion of young people are engaged in full time study, the proportion of the population who are unemployed is much less.

According to the European Commission’s Eurostat page “Statistics explained” on unemployment statistiscs, Greece’s youth unemployment rate has gone up from 25.7% in 2009 to 44.4 in 2011 and 49.3 in the last quarter of 2011 – but the ratio – that is, youth unemployed as a percentage of the youth population has only increased from 8% in 2009 to 13% in 2011.  Still a tragedy for those affected, but very different to the picture painted.

Discussions about public policy need to be based in accurate representation of the situation. This is a basic mistake that should not be made.

There’s a great discussion going on right now on the AEA Thought Leaders’ Forum. This week it’s being led by Jean King, who has raised the question of credentialing for evaluators.

Not all our subscribers are AEA members and following this forum, so I’m just cross-posting a revised and expanded version of a contribution – and encourage you all to check out the wider discussion!

The problem of competency ‘laundry lists’

One problem with the various lists of evaluation competencies we see around is that they cover an enormous range of the skills that evaluators have and use in our work, but FAR MORE than any one evaluator (or even one evaluation team) could or even should have.

This leads people to think that:

• “competent” = “can demonstrate every single one of the competencies”
• “missing a few” = “incompetent”

… and of course, because no-one has the full repertoire, even top-notch evaluators will be looking at the list and saying “What?! You’re calling me incompetent because I can’t [insert skill]?”

It seems to me that we need to distinguish between:

1. “the core” – the absolutely essential stuff that you really must have if you are to call yourself an evaluator
2. “specialized competencies” – the specific methodologies, content areas, and other specialties that you choose to be particularly strong in

Defining ourselves professionally

I think we need to do this at two levels:

1. defining ourselves as a profession (by defining “the core”)
2. defining ourselves as individual evaluators, evaluation teams, or evaluation units or businesses (by defining our specialized competencies and approaches – which must include the core)

Defining “the core” of our profession

I think we all agree that there are people who pedal evaluation services who basically have no idea of the difference between evaluation and, say, measurement, or descriptive research.

They are generally not aware that there are degrees or certificates in evaluation or professional associations for evaluators – and if they were aware, they probably wouldn’t opt in anyway because they don’t believe there’s anything unique about evaluation, nothing worth talking about, puzzling over, improving on.

So, what is that “core”?

In various discussions I’ve had with colleagues about this, somehow we keep coming back to one thing as being the fundamental difference, the core of what distinguishes evaluation (done right) from other work, and that is the values and ‘valuing’ piece:

• We ask questions about how good/worthwhile/valuable/important things like design, implementation, and outcomes are;
• We actually have a shot at answering those questions (not just free associating to them with whatever data seems vaguely relevant)

In the New Zealand context, we have strong agreement that cultural values are absolutely central to this – how we define what’s good/worthwhile/valuable/important (both the process of doing this and what ends up in the criteria, plus how we evidence it).

The recent NDE (#133), edited by George Julnes, is a fantastic resource for thinking really seriously about how we as evaluators judge value in evaluation. It’s a must read!

Defining “who we are” as evaluation practitioners

Every individual evaluator and every evaluation consultancy/business/contracting unit needs to be clear about “who they are” as evaluators – what is it that distinguishes their practice or approach from that of others working in this space?

It’s impossible for any individual or even any evaluation team or consultancy to be all things to all people – and it is dishonest to imply that we are.

So, who are you? What are you particularly good at? What defines your approach? And, importantly, what are you NOT strong in? What kind of work do you steer clear of?

It is up to each evaluator (and each evaluation unit/business/consultancy) to define the profile of competencies they want and need to develop in order to work effectively in the space they have carved out for themselves.

YES, that means it’s perfectly OK to position yourself as (for example) someone who does highly collaborative evaluation, works primarily with qualitative evidence, works in the United States, in English-speaking communities of color, on programs related to addiction and homelessness – so long as you are doing that core evaluative activity of asking and answering evaluative questions – like how good the program design is, how well it’s been targeted and implemented, how valuable the outcomes have been so far, and so forth.

If this were you, you’d likely turn down work that involved heavy number crunching or non-English speaking participants or a requirement for a very independent style of evaluation.

It doesn’t make you any less of an evaluator if you have specialized in a particular approach, context, or content area; it simply means you are focusing on getting really good in that space.

And nor is the generalist evaluator any less competent for choosing to practice across a range of domains, drawing on others’ expertise as required.

Credentialing – who is ‘in’? Who gets sidelined?

Credentialing (if we need it – and the answer to this varies depending on where you live and work – see Michael Scriven’s post on the Thought Leader Forum discussion) has the potential to wrongly include or exclude people.

It also has the potential to appropriately include and exclude.

Here’s my take on inclusion/exclusion:

• We will inappropriately exclude if we define the “must have” competencies more widely than what really genuinely is at the core of evaluation. [Or if we use a long list of competencies and assume they are all required to do any decent evaluation.]
• We will inappropriately include if we say there is no core, or if we define it wrongly (e.g. as measurement or monitoring or applied research or providing information for decision making or …).

It’s always important to consider carefully who wins and who loses when any particular credentialing system is initiated – and whether one is needed at all.

We’ve had this discussion in New Zealand and decided no, we don’t need or want credentialing at this point. Instead, we are opting for:

1. A list of competencies that practitioners can use to self-assess, reflect, and plan their professional development
2. Professional development aligned with the needs most lacking and desired by professional association members
3. Efforts to build the capability of clients so they become more effective evaluation scopers, purchasers, project managers, utilization advocates, and (in some cases) collaborators

Related posts and references

• AEA Thought Leaders’ Forum (April 2012 – Jean King leading discussion on credentialing)
• Promoting Valuation in the Public Interest: Informing Policies for Judging Value in Evaluation (NDE #133)
• Lifting the quality of evaluation #2: Capable evaluators who know their ‘space’

A couple of years ago we quoted a paper on psychological research called “Keeping it simple” (Peterson & Park, 2010) that observed:

… the evidence of history is clear that the research studies with the greatest impact in psychology are breathtakingly simple in terms of the questions posed, the methods and designs used, the statistics brought to bear on the data, and the take-home messages.

In the post, Simplicity and Genuine Utilization, Jane lamented our tendency in evaluation to overcomplicate things thanks to our training in the [social] sciences.

Nevertheless, we do realize that some of our colleagues are also in the business of communicating not to normal people (clients) but to editors and reviewers of academic journals who have the power to publish them or let them academically perish.

For the benefit of our evaluation colleagues in academia, here is a selection of snippets from Science Magazine columnist Adam Ruben, author of Surviving Your Stupid, Stupid Decision to Go to Grad School … Enjoy!

How to Write Like a Scientist

(click the title above to read Adam Ruben’s full post on the Science Magazine website – we’re posting just a few tantalizing snippets here)

2. Using the first person in your writing humanizes your work. If possible, therefore, you should avoid using the first person in your writing. Science succeeds in spite of human beings, not because of us, so you want to make it look like your results magically discovered themselves.

4. The more references you include, the more scholarly your reader will assume you are. Thus, if you write a sentence like, “Much work has been done in this field,” you should plan to spend the next 9 hours tracking down papers so that your article ultimately reads, “Much work has been done in this field^{1,3,6-27,29-50,58,61,62-65,78-315,952-Avogadro’s Number}.” If you ever write a review article, EndNote might explode.

9. Starting sentences with “obviously” or “as everyone knows” demonstrates your intellectual superiority. If possible, start sentences with, “As super-intelligent beings like myself know,” or “Screw your stupidity; here’s a fact-bomb for you.”

10. Your paper will be peer reviewed, so include flattering descriptions of all of your peers. Scientists call these “shout-outs” or “mad props.”

12. If you’re co-authoring a paper, most of your notoriety will derive from the order of authors and not from the content of your paper — so make sure to have vehement and petty debates about whose name goes first. Here are the general rules for authorship:

FIRST AUTHOR: Weary graduate student who spent hours doing the work.

SECOND AUTHOR: Resentful graduate student who thinks he or she spent hours doing the work.

THIRD AUTHOR: Undergraduate just happy to be named.

FOURTH AUTHOR: Collaborator no one has ever met whose name is only included for political reasons.

FIFTH AUTHOR: Postdoctoral fellow who once made a chance remark on the subject.

SIXTH AUTHOR: For some reason, Vladimir Putin.

LAST AUTHOR: Principal investigator whose grant funded the project but who hasn’t stood at a lab bench in decades, except for that one weird photo shoot for some kind of pamphlet, and even then it was obvious that he or she didn’t know where to find basic things.

Related posts:

• Simplicity and Genuine Utilization
• Managing genuine evaluation paradoxes: Genuine reporting

References/further reading:

Keeping it simple: Christopher Peterson and Nansook Park on the lasting impact of minimally sufficient research. The Psychologist, Vol. 23(5), 2010, pp. 398-401.

Unlearning some of our social scientist habits: Jane Davidson on how academic training in the social sciences can impede genuine evaluation. Journal of Multidisciplinary Evaluation, 4(8), 2007, pp. iii-vi.

It was Michael Scriven‘s birthday this week, which is a fine time to introduce our Friday Funny with a short quote from the Evaluation Thesaurus, which lists the following entry. As evaluators, we are all familiar with this phenomenon in our work or everyday lives, in one form or another:

KILL THE MESSENGER (phenomenon)  The tendency to punish the bearer of bad tidings, who is often the evaluator. A phenomenon related to valuephobia – mindless striking back at the proximate cause of pain, even when it’s clear this is neither just punishment nor likely to be curative of the problem. Much of the current attack on testing – for example, minimum competence testing for graduation or teaching licensure – is pure KTM, like many of the elaborate and rationalized earlier attacks on course grades. The presence of the rationalizations identify these specimens of the subspecies, AFTOC: Kill The Messenger – After a Fair Trial, Of Course.

But there are some settings we do in fact seem to see very gracious acceptance of less-than-hoped-for evaluative conclusions. One such setting is in the movie industry awards.

But how often do they really speak their minds?

This hilarious sketch from British comedian Rowan Atkinson shows an non-awardee accepting an award on behalf of his colleague, and finishing off the acceptance speech with some feedback for the judging panel. Enjoy!

[If you can't see the video (e.g. because you are reading this on the email feed), you will need to click through to the site.]