Strategies for improving the quality of evaluation – the independent evaluation advisor

How can we improve the quality of evaluations – especially evaluations that are done by external evaluators of evaluation teams?

Improving the procurement process is an important part of this, as Jane has recently discussed (9 hot tips for commissioning, managing (and doing!) actionable evaluation). As part of the BetterEvaluation project, I’ve been bringing together resources on improving the Terms of Reference for evaluation, since this is where a lot of evaluations go badly wrong, with unreasonable timelines, vague and contradictory requirements, and unrealistic expectations about the numbers and types of questions that an evaluation can answer.

But there are other strategies that need to be used.  One of these has recently been suggested by Michael Scriven, evaluator, author and GenuineEvaluation guest blogger, whose earlier thinking has given us such core concepts as formative and summative evaluation and meta-evaluation (the evaluation of evaluation) (Scriven, M. (1967). ‘The methodology of evaluation’. In R.W. Tyler, R.W. Gagne & M. Scriven (Eds.), American Educational Research Association Monograph Series on Curriculum Evaluation, Vol .1: Perspectives of Curriculum Evaluation. Rand McNally).

On his website, he has posted a paper on ‘The Evaluation Advisor: A New Role for Evaluators?’ which sets out what the role of an evaluation advisor might involve and why it is important:

The evaluation advisor … is a person who serves as a helper or guide about evaluation, but not as an evaluator, for an individual or organization or program that is being, or is about to be, evaluated, or is considering sponsoring external or internal evaluation of themselves

He argues that this role needs to operate under a very strict written NDA (non-disclosure agreement):

The NDA means that they can discuss, with the prospective evaluees, the nature and costs and benefits of evaluations of various types, and how to do it or get it done, or survive it or benefit from it, or avoid it, or cheat on it, but cannot discuss anything about that discussion with anyone else, including in particular the agency that is funding the evaluee and wants or requires it to be done.

Some approaches to evaluation include this role as part of what the evaluator does.  For example, I’ve been reading  recently Michael Patton’s new book Essentials of Utilization-Focused Evaluation, which discusses how the evaluator can and should be involved in these discussions with the intended users of an evaluation.

Michael Scriven’s proposal is for these two roles to be separated, to avoid any possible conflict of interest:

the NDA should, usually, also preclude the EA from actually taking on the job of evaluating the organizations s/he advises. This is to prevent a possible conflict of interest in the EA between giving good advice and selling his/her services. The funding agency may or may not want to allow petitions for an exemption from this requirement in special cases.

He advocates that funding agencies should fund this role, as part of improving the quality of the data gathered in evaluations, and to improve the use made of evaluations by funded projects.

A slightly different version of this role has been used by the NGO Pact in South Africa, which has been working with Community Service Organizations funded by USAID both in group workshops and individual Technical Assistance to increase their understanding of evaluation and have input into the planning for independent external evaluations of their projects.   A presentation by Rita Sonko-Najjemba, Ana Coghlan, Addis Berhanu, and Priscilla Ngwenya at the 2011 American Evaluation Association conference on their work has been uploaded to the AEA e-library.



1 comment to Strategies for improving the quality of evaluation – the independent evaluation advisor