Evaluation-Specific Methodology: The methodologies that are distinctive to evaluation

I raised a few eyebrows last week when I mentioned the idea of Evaluation-Specific Methodology (ESM) as being an essential part of what defines us as a discipline.

Of course, a large proportion of people who identify as evaluators consider that evaluation is merely the application of social science research methods to support decision making – or something along those lines.

So, not surprisingly, someone asked (in the LinkedIn discussion that started me on this series of blog posts) what everyone else was probably wondering:

“What methodologies are those?”

The methodologies distinctive to evaluation are the ones that go directly after values.

Examples of evaluation-specific methodologies include:

  • needs and values assessment
  • merit determination methodologies (blending values with evidence about performance, e.g. with evaluative rubrics)
  • importance weighting methodologies (both qualitative and quantitative)
  • evaluative synthesis methodologies (combining evaluative ratings on multiple dimensions or components to come to overall conclusions)
  • value-for-money analysis (not just standard cost-effective analysis or SROI, but also strategies for handling VfM analysis that involves a lot of intangibles, for example

I wouldn’t count the following as evaluation-specific: statistics or any of the standard research methods (interviews, observations, surveys, content analysis, or even causal inference methodologies). We clearly draw on these and use them a lot, but they are not distinctive to evaluation because they are not specifically about the “values” piece.

In other words, you could use these (non-evaluative qualitative and quantitative research methods) and still NOT be doing evaluation.

But if you are using ESM (evaluation-specific methodology), you sure ARE evaluating, i.e. drawing conclusions about quality, value, or importance.

And in fact, if you don’t use any ESM, you basically aren’t doing real, genuine evaluation. Either you are skipping the whole evaluative conclusions piece, or you are getting to it by logical leap (e.g. “I looked upon it and saw that it was good”). ESM is what allows us to get systematically and transparently from evidence about performance to evaluative conclusion, by weaving in the values (“how good is good”) piece.

It’s true that several disciplines use evaluation-specific methodologies (e.g. industrial & organizational psychology uses cost-effectiveness analysis). That doesn’t make them “not evaluation-specific” any more than statistics becomes psychology just because psychologists use it.

As I mentioned in the previous post, Michael Scriven and I have proposed a pre-conference workshop on Evaluation-Specific Methodology for AEA (in Washington DC) and a mini-workshop in the main program for AES (in Brisbane, Australia) this year, so if accepted we look forward to clarifying these concepts further!

My 2-day AEA workshop, Actionable Evaluation, has already been approved, and will cover (among other things) how to use evaluative rubrics to draw explicitly evaluative conclusions.

And, of course, see also the following books from Michael and myself!
[Already read these? Who would each one be particularly useful for? Please add your thoughtful evaluative reviews on Amazon – click the book to post one now!]

Related posts:

4 comments to Evaluation-Specific Methodology: The methodologies that are distinctive to evaluation

  • Cath

    Thank you for this brief but understandable explanation. I am currently a “want to be” evaluator undertaking the 3rd paper of a Post Grad Dip in Social Sector Evaluation and Research at Massey University. This paper is known as the “methods paper”. I have a grasp on the first modules of the paper – quantitative, qualitative and mixed methods (no surprises there) – but have been struggling to define what is meant by the following terms used subsequently in the paper: innovative methods and evaluation specific methods. I have found it even harder to find examples of evaluations using either of the later two methods.

    On trawling various government websites for evaluation reports there appear to be a plethora of ways to combine methods but the methods themselves seemed to be nothing new or innovative, or specific to evaluation. Having also read your blog on the likert scale I am now going to look at things a little differently. It may not be the tool that it is unique to evaluation but the way in which it is used. The way in which rating scales are worded in the example of likert scales.

    I think I will be ringing student loans to convince them that loaning me the money to buy your book would add value to my learning experience and be worth the investment. I also very much hope that you can make your notes from the AEA workshop available.

  • Cath, thanks so much for your comment!

    You are right that it is very hard to find much on evaluation-specific methods, which is one reason I’ve put quite a bit of energy into trying to contribute in that space.

    If you want to see examples of ESM in action, check out:
    * Judy Oakden’s example on BetterEvaluation http://betterevaluation.org/blog/guest-rubrics
    * Mathea Roorda & colleagues’ evaluation of a seasonal employer policy http://dol.govt.nz/publications/research/rse-evaluation-final-report/final-18.asp (this evaluation won the AES award a couple of years ago!)
    * the example in my Actionable Evaluation Basics minibook (the one that’s cheap enough that no student loan is required!) – and in fact, this covers most of what I cover in the AEA workshop
    * the Heifer evaluation conducted by Michael Scriven, Thomaz Chianca and colleagues from WMU http://www.heifer.org/ourwork/measure-of-success (and see also Thomaz’s explanation of the Heifer Hoofprint Model for impact evaluation http://vimeo.com/8918447)

    All the best with the PGDipSSER!

  • Cath

    Many thanks for the information. I have copied the link to this discussion to my classmates too as I think it will be more than helpful to all.

  • Thanks for the very helpful site and posts, looking forward to reading your books. I have worked in mental health evaluation and writing on qualitative methods, ethnography and mental health service users’ views, keen to hear of any developments in evaluation and service user experience, thanks again, Roberta