What is Genuine Evaluation?

When we think about the type of evaluation we want to do and to support, and the types we want to hold up for critique and as cautionary tales, five elements stand out:

  1. VALUE-BASED -transparent and defensible values (criteria of merit and worth and standards of performance)
  2. EMPIRICAL – credible evidence about what has happened and what has caused this,
  3. USABLE – reported in such a way that it can be understood and used by those who can and should use it (which doesn’t necessarily mean it’s used or used well, of course)
  4. SINCERE – a commitment by those commissioning evaluation to respond to information about both success and failure (those doing evaluation can influence this but not control it)
  5. HUMBLE – acknowledges its limitations

This does not restrict ‘genuine evaluation’ to any particular type of method or research design – experimental, quasi-experimental and non-experimental designs can all be appropriate in particular circumstances. But it does exclude:

  • evaluations that don’t get to the point of making a judgment about something being good or bad, better or worse
  • evaluations that uncritically accept stated objectives as the only evaluative criteria, leaving out any unintended impacts
  • evaluations that focus only on the average effect, ignoring or being unaware of differential effects where a program might be effective on average but actually harmful in some circumstances
  • evaluations that only draw on summarized participant responses as evidence of what has happened and how it has happened
  • evaluations that are ignored, buried or censored.

What do you think?

11 comments to What is Genuine Evaluation?

  • Michael Patton

    Genuine Evaluation sounds like a great book. And we’ll be seeing it when?

  • It’s about time for something like this. Let’s hope that it generates “genuine” interest. Good work, both of you.

  • Ricardo Wilson-Grau

    I feel (and think) I would like to know more, especially about the role of the evaluand in Genuine Evaluation.

  • Hi professors

    I think Genuine Evaluation is an ideal type that has not same specific characteristics in Contexts , cultures & …. Program or organization evaluation capacity, the level of evaluation quality could take place and many other factors are important and mediate.
    I learn from both many imagine and evaluative knowledge. You are most welcome and many thanks for new humanistic, social and scientific efforts.

    Best

    Moein

  • Abdou Ndoye

    This is a great idea. Good job to both of you. I look forward to learning more about Genuine evaluation and particularly the steps one needs to take to implement it and guarantee its success.

  • Patricia

    Thanks for the encouragement – and I hope you will all join us in exploring these ideas and sharing examples.
    I’m sure that context will have a big influence on the scope for doing genuine evaluation and what are effective strategies.
    There are issues here both for those doing evaluation and for those commissioning or using evaluation. I think it requires, as a mimimum, a lot of discussion, clarification and negotiation, as well as sincere intent, and very little staff turnover.

  • Mateusz Zych

    Dear Madams,

    The five principles you have pointed out, are well fixed rules of doing research in general. How does it differs from rigour of social sciences methodology? What is genuine in your approach?

    Yours faithfully
    Mateusz Zych

  • Patricia Rogers & Jane Davidson

    Thanks for the question, Mateusz. You’re right that some of these are attributes of genuine research as well, and some of the examples we have been discussing have focused on issues of empirical quality.

    Like all good consultants we’ll take the liberty of rewriting the question slightly to clarify the point:
    What’s genuinely evaluative about our approach?

    Genuine evaluation carries with it virtually all the hallmarks of genuine applied research of the type applied research consultants would carry out.

    The one thing that is fundamentally different about genuine evaluation is that it asks and very explicitly answers questions about the quality, value and/or importance of things (“things” being programs, policies, services, etc – and also performances on criteria used to evaluate those programs, policies, services, etc). The following post goes directly after this issue: http://genuineevaluation.com/why-genuine-evaluation-must-be-value-based/ – we welcome your comments.

    This is not to say there’s an absolutely clear distinction between the two. Much applied research is quite evaluative in nature, and many evaluation projects use a lot of applied research methods. But that doesn’t make the distinction unimportant, in our view.

    Many thanks!
    Jane & Patricia

  • I really like these five principles as criteria standards for doing good evaluation work. I also like they operationalize fairly easily into explicit, testable questions. They also appear to be internally consistent, embodying the principles they espouse.

    However, I am not convinced that these principles provide a unique enough application of evaluation thinking and practice to warrant the articulation of a new brand of evaluation, especially one with a title “Genuine Evaluation” that might be seen to imply a major distinction between itself and the general practice of high quality social science.

  • Patricia Rogers

    Hi Michael,
    Thanks for your comments. I don’t think of Genuine Evaluation as a ‘brand’ of evaluation, even though we do have a jingle and lapel badges!

    For me it’s more of a set of lenses that I pop on to look at an evaluation, either planned or actual. In the same way, I pop on utilization-focused lenses, key evaluation checklist lenses, and appreciative inquiry lenses to look at what we are planning to do from different perspectives.

    Jane’s previous comments on this post discuss the differences and similarities between evaluation and research.

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>