One of the hallmarks of genuine evaluation is thoughtful analysis and sound evaluative reasoning that the audience can follow.

Some of our readers are not only evaluators, but also teachers or students of evaluation. We thought you might enjoy this little classic that’s done the rounds on the Internet. We found this version on Pinetree.net’s Inbox Humor page.

As you study for exams, remember its not the quantity it’s the quantity. And remember there is no substitute for pure unadulterated bull

---

Dr. Schambaugh, of the University of Oklahoma School of Chemical Engineering, Final Exam question for May of 1997. Dr. Schambaugh is known for asking questions such as, “why do airplanes fly?” on his final exams. His one and only final exam question in May 1997 for his Momentum, Heat and Mass Transfer II class was: “Is hell exothermic or endothermic? Support your answer with proof.”

Most of the students wrote proofs of their beliefs using Boyle’s Law or some variant. One student, however, wrote the following:

“First, We postulate that if souls exist, then they must have some mass. If they do, then a mole of souls can also have a mass. So, at what rate are souls moving into hell and at what rate are souls leaving? I think we can safely assume that once a soul gets to hell, it will not leave.

Therefore, no souls are leaving. As for souls entering hell, let’s look at the different religions that exist in the world today. Some of these religions state that if you are not a member of their religion, then you will go to hell. Since there are more than one of these religions and people do not belong to more than one religion, we can project that all people and souls go to hell. With birth and death rates as they are, we can expect the number of souls in hell to increase exponentially.

Now, we look at the rate of change in volume in hell. Boyle’s Law states that in order for the temperature and pressure in hell to stay the same, the ratio of the mass of souls and volume needs to stay constant. Two options exist:

1. If hell is expanding at a slower rate than the rate at which souls enter hell, then the temperature and pressure in hell will increase until all hell breaks loose.
2. If hell is expanding at a rate faster than the increase of souls in hell, then the temperature and pressure will drop until hell freezes over.

So which is it? If we accept the quote given to me by Theresa Manyan during Freshman year, “that it will be a cold night in hell before I sleep with you” and take into account the fact that I still have NOT succeeded in having sexual relations with her, then Option 2 cannot be true…Thus, hell is exothermic.”

The student, Tim Graham, got the only A.

The other genuine evaluation lesson here is this:

Whether the reasoning is sound or flawed – and in the messy, nuanced real world of evaluation, it is sometimes difficult to tell – the absolutely most important thing is that it is clear and can be followed. That’s how we make evaluation accessible, easy to engage with, easy to criticize – and therefore easier to improve.

Have a great weekend!

A salutary reminder that just because things are measured precisely (such as money) doesn’t mean that the measurements are valid or useful. As reported by Louise Story, Landon Thomas Jr and Nelson D. Schwartz, in the New York Times on 13 Feb 2010 :

As in the American subprime crisis and the implosion of the American International Group, financial derivatives played a role in the run-up of Greek debt. Instruments developed by Goldman Sachs, JPMorgan Chase and a wide range of other banks enabled politicians to mask additional borrowing in Greece, Italy and possibly elsewhere.

In dozens of deals across the Continent, banks provided cash upfront in return for government payments in the future, with those liabilities then left off the books. Greece, for example, traded away the rights to airport fees and lottery proceeds in years to come.

Critics say that such deals, because they are not recorded as loans, mislead investors and regulators about the depth of a country’s liabilities.

The second-funniest joke in the world, according to Richard Wiseman’s LaughLab, was the joke submitted by Geoff Anandappa of Blackpool, which is clearly about the importance of focusing on key messages in evaluation summaries.

Sherlock Holmes and Dr Watson were going camping. They pitched their tent under the stars and went to sleep. Sometime in the middle of the night Holmes woke Watson up and said: “Watson, look up at the stars, and tell me what you see.”

Watson replied: “I see millions and millions of stars.”

Holmes said: “And what do you deduce from that?”

Watson replied: “Well, if there are millions of stars, and if even a few of those have planets, it’s quite likely there are some planets like earth out there. And if there are a few planets like earth out there, there might also be life.”

And Holmes said: “Watson, you idiot, it means that somebody stole our tent.”

The version on Wikipedia is even more appropriate:

Sherlock Holmes and Dr. Watson go on a camping trip, and after finishing their dinner they retire for the night, and go to sleep. Some hours later, Holmes wakes up and nudges his faithful friend.”Watson, look up at the sky and tell me what you see.”

“I see millions and millions of stars, Holmes” exclaims Watson.

“And what do you deduce from that?”

Watson ponders for a minute. “Well, astronomically, it tells me that there are millions of galaxies and potentially billions of planets. Astrologically, I observe that Saturn is in Leo. Horologically, I deduce that the time is approximately a quarter past three. Meteorologically, I suspect that we will have a beautiful day tomorrow. Theologically, I can see that God is all powerful, and that we are a small and insignificant part of the universe. What does it tell you, Holmes?”

“Watson, you idiot!” He exclaims, “Somebody’s stolen our tent!”

Recently I was asked to review an evaluation report that the client expressed disappointment in because the report did not adequately address the key evaluation questions. (See http://genuineevaluation.com/whos-responsible-for-un-genuine-evaluation/comment-page-1/#comment-23 ).

Whilst it was possible to make some recommendations to improve the overall presentation of findings, any suggestions were limited because the evaluation approach, methodology and data collection had been completed – and these core components could not be revisited.

A simple logic might be –> disappointing evaluation report –> implement report review suggestions –> improved evaluation report –> improved evaluation.  Of course an evident flaw in this logic is that an improved evaluation report does not necessarily equate to an improved evaluation and ‘tinkering’ with the output of the evaluation effort – the evaluation report – will in many cases be too little, too late.

This  raised for me the bigger question of ‘what does it take to get a ‘good’ evaluation?’ Focusing on the discrete components of an evaluation is of course valuable, but in reality there is  need for each of the core evaluation components, tasks or activities all to be done well.

So what does it take to get a ‘good’ evaluation?

This is not an easy question to answer and there are many textbooks and evaluation checklists dedicated to achieving this very goal.  They are often written for evaluation practitioners, and therein lies part of the problem.

Whilst not wanting to trivialize the diversity and complexity of the practice of evaluation, I got to thinking: What four or five key messages or advice would we offer to evaluation managers and commissioners – who are not evaluation specialists – about what it takes to get a good evaluation?  (Five is an arbitrary number intended to provide focus; there may well be more than five.)

Okay, so here’s a first stab at a core five evaluation components:

1. Key evaluation questions that are explicitly evaluative.
2. An appropriate match of methods and methodology to address the key evaluation questions.
3. An evaluation framework that provides a clear and transparent method and process for drawing evidence based evaluative conclusions.
4. The alignment of data collection to the key evaluation questions
5. Clear communication of evaluation findings

All of the planning, design, implementation etc tasks and activities are important to achieving a ‘good’ evaluation.  The attempt here is to identify some core components which we can suggest that commissioners focus on to improve the likelihood of getting a good evaluation.

I’ve suggested five core components (a fuller discussion below follows), and the invitation now is to comment on:

1. What are the five core messages for commissioners of evaluation to get to a good evaluation? Are there more or less than five core messages?
2. What’s the value, if any, of this type of approach to getting a ‘good’ evaluation.

1. Key evaluation questions that are explicitly evaluative.

That is questions, which specifically ask about the quality, value or importance of the evaluand or some aspect of it (Davidson, 2005).  I briefly touch on the limitations of research questions as a poor substitute for explicitly evaluation questions in the earlier post comment.  Some commissioners of evaluation will need some help in developing or refining their key evaluation questions (see Davidson 2009 AEA presentation).

2. Appropriate match of methods and methodology to address the key evaluation questions.

The aim here is to ensure that the proposed methods and methodology address/connect to the key evaluation questions. For example commissioners might require: (1) evaluation tenders or bids to discuss the strengths and limitations of their proposed methods/approach in addressing the key evaluation questions; (2) an evaluation plan as the first deliverable of any contract that clearly demonstrates how the methods and methodology address/connect to the key evaluation questions.

3. An evaluation framework (including an analysis and synthesis methodology) which provides a clear and transparent method and process for drawing evidence-based evaluative conclusions.

I know from my own practice whilst making data-driven judgments against evaluative criteria hasn’t posed a problem,  being clear and transparent about the basis for these determinations – including the values, assumptions and preferences prioritized in this process – has been an emergent practice.  So, we might suggest to commissioners that they seek specific feedback in tender documents and evaluation plans about how evaluators will  develop a framework for making judgments and drawing evaluative conclusions. I find Jane’s work (Davidson, 2005) particularly useful in this respect.

4. The alignment of data collection to the key evaluation questions

It is at the data collection stage the reality of what was envisaged in the evaluation plan and what is actually feasible and affordable ‘in the field’ surfaces.  It can be a particular point of vulnerability as time and budget pressures typically loom large.  In particular, commissioners need to (1) keep a ‘close eye’ on the data collection tools for their alignment to the key evaluation questions; and (2) if methods are to be scaled back, changed or discarded those changes are made knowing full well the implication on the overall evaluation and the ability to draw evaluative conclusions.  In my experience, it is at this stage that commissioners are particularly vulnerable to ‘dropping the ball’ – not understanding the importance of these activities, nor the potential impact on the evaluation.

5. Clear communication of evaluation findings

Written evaluation reports continue to be the common method of reporting evaluation findings (despite the increased range of communication options available).  The normal conventions of report writing apply to evaluation reports; for example, the report is written in a clear and easily readable style and there is a logical sequence and presentation of findings. Other  reporting elements specifically applicable to an evaluation report might include: (1) clearly evidences the data and findings to conclusions drawn; (2) takes account of and critically explores various likely alternatives; and (3) draws explicitly evaluative conclusions and makes transparent the basis on which judgments have been made.

The evaluation report is the sum of the evaluation effort rendered to a report, and it goes without saying that it needs to be done well.

We might remember ‘regression to the mean‘ from those lists of threats to validity (in terms of causal analysis). But when is it actually likely to be a problem for genuine evaluation? In a recent post by Rebecca Goldin on the stats.org blog, “Why any ol’ diet will work (if your BMI is high enough): A case study in regression toward the mean” statements were made about its relevance for studies of BMI that I have found very confusing. This is an issue I’ve puzzled over during a particular evaluation and I wonder if the argument in the post stacks up.

My understanding is that regression to the mean is an artefact of the unreliability of measurement. So, for example, a student’s test score might be affected by whether they had a good or bad night’s sleep, a good or bad breakfast, and a good or bad social interaction just before the test, in addition to their actual skills and knowledge in the topic (and in taking tests on the topic). So if we select students who have scored low, implement an intervention, and then test them again, we might find that a number who scored low initially (but whose test scores were lower than their actual level of knowledge) achieve higher scores due to more accurate measurement the second time.

The post discusses the example of blood pressure, which notoriously fluctuates (so a high reading on one day might not indicate much) and coin flipping (which is a random process, so you could get an odd high measure) – and then generalizes to BMI, which doesn’t seem to me to be relevant. Wearing heavier shoes for the first weighing, or having just eaten, might make you a little heavier, but it’s unlikely to be significant if the measurements are being done well (and we have pretty good measures of mass and height). As many of us can testify, BMI doesn’t tend to fluctuate much, and is in fact quite slow to change.

The title of the blog also seems to be contradicted by the conclusion to the post which acknowledges:

What diets do not advertise is that people with the highest BMI people are (on average) losing weight even without a diet – though they may well hover at a very high BMI. For full disclosure: the random variance of weight measurements may in fact be very small.

For people with higher-than-average BMI trying to shed some pounds, regression to the mean is an encouraging thought — but of course, it’s only reduced calorie consumption and exercise that will shed pounds instead of ounces

So is regression to the mean only likely to explain tiny reductions (which would be useful for diets that focused on the number who had lost any weight) but is not likely to be the explanation for large reductions? So does it come down to what we mean by “it works” – any reduction, or a clinically significant reduction? Or is the title misleading?

Any helpful statistical advice on this one?