When we think about the type of evaluation we want to do and to support, and the types we want to hold up for critique and as cautionary tales, five elements stand out:
- VALUE-BASED -transparent and defensible values (criteria of merit and worth and standards of performance)
- EMPIRICAL – credible evidence about what has happened and what has caused this,
- USABLE – reported in such a way that it can be understood and used by those who can and should use it (which doesn’t necessarily mean it’s used or used well, of course)
- SINCERE – a commitment by those commissioning evaluation to respond to information about both success and failure (those doing evaluation can influence this but not control it)
- HUMBLE – acknowledges its limitations
This does not restrict ‘genuine evaluation’ to any particular type of method or research design – experimental, quasi-experimental and non-experimental designs can all be appropriate in particular circumstances. But it does exclude:
- evaluations that don’t get to the point of making a judgment about something being good or bad, better or worse
- evaluations that uncritically accept stated objectives as the only evaluative criteria, leaving out any unintended impacts
- evaluations that focus only on the average effect, ignoring or being unaware of differential effects where a program might be effective on average but actually harmful in some circumstances
- evaluations that only draw on summarized participant responses as evidence of what has happened and how it has happened
- evaluations that are ignored, buried or censored.
What do you think?