Jane Davidson here. [You can tell on the GE site which of us is posting, but unfortunately not on the email feed. Please hit reply if you know how to do this on Feedburner!]
Have you ever been struck by the irony of clearly invalid methods used to evaluate proposals for evaluation work?
I find myself shaking my head at one such RfP as we speak. I won’t mention the commissioning organization, but I find it just staggering. And this is not at all uncommon.
Here are the criteria, which perhaps seem sensible enough to the evaluatively untrained eye …
- Description of methods (10%)
- Performance measures and targets (5%)
- Professional expertise (10%)
- Previous experience in evaluation (10%)
- Quality assurance (5%)
- Service priorities (can complete within timeframe) (10%)
- Price (50%)
As is often typical, there is no budget range given, so it’s anyone’s guess whether the organization is after a Rolls Royce/Cadillac or a bicycle/moped-scale evaluation. It’s not clear how price will be evaluated, but one has to assume that “cheaper is better”.
Perhaps the part that frightens me the most (on behalf of the taxpayer who will be funding this and the recipients whose lives depend on the quality of the program) are the low weights given to things that should (a) be weighted more heavily because they are surely the main point of the work and (b) have ‘bars’ (minima) in addition to weights. A ‘bar’ is a minimum level you must pass in order to be considered at all.
When only numerical weights are used, this means that very poor performance on one or more low-weighted criteria could, in theory, be compensated for by good performance on a highly weighted one.
Let me rephrase the above criteria as plain language questions to help illustrate the problem here:
- How sound are your ideas for doing this entire piece of work? (10%)
- How good are the measures you plan to use? (5%)
- Do you have the expertise to do a decent job? (10%)
- Have you ever actually done an evaluation before? (10%)
- How do you make sure you are doing a good job? (5%)
- Have you actually got time to do this work? (10%)
- Are you cheap? (50%)
In theory, a very cheap proposal from a bidder with no actual evaluation experience and a flimsy evaluation plan will score way ahead of an experienced evaluation team with a sound plan that, carefully costed, will require a higher budget.
Note that adjusting the weighting will only solve part of the problem; ‘bars’ (minima) are critical here. If the proposed approach is clearly inadequate; if the team simply doesn’t have the expertise to do the job; or if they have never actually done a single evaluation before, then these proposals should be excluded outright.
So, how do you synthesize performances on multiple criteria?
This is a fairly substantial topic, but here are the hottest leads:
- Check out the BetterEvaluation site’s page on how to synthesize data.
- Of particular interest here is the Qualitative Weight and Sum (QWS) option (the link here takes you to a 2000 presentation from Michael Scriven and Jane Davidson on this topic).
For more ideas on commissioning genuine evaluation:
- 9 golden rules for commissioning a waste-of-money evaluation
- Extreme Genuine Evaluation Makeovers (XGEMs) for Commissioning
- Commissioning XGEMs – the sequel
- Lifting the quality of evaluation #1: Savvy clients
- Hot tips for commissioning and managing actionable evaluation
- 6 questions that cut to the chase when choosing the right evaluation team