Extreme Genuine Evaluation Makeovers (XGEMs) for Commissioning

There may well be some managers out there who have yet to experience the thrill of commissioning a total waste-of-money evaluation. Well, thankfully, your colleagues around the world have been working diligently on building a knowledge base just for you …

We had a great discussion a while back about the 9 golden rules for commissioning a waste-of-money evaluation. We got a record 22 comments on that one – from all continents except Antarctica – so it obviously struck a chord!

In a keynote for the recent Australasian Evaluation Society conference, I talked about the incredibly ineffective commissioning processes that are often used and that frequently result in evaluations that are, as we say in this part of the planet, “about as much use as an ashtray on a motorbike”. Some of the most important problems are:

  1. Detailed plans and costings are often required to be written virtually ‘blind’ – with only the cryptic clues in the RFP, little idea of the context or politics currently in play, and rules that prevent one asking anyone who would actually know what’s going on.
  2. There’s an assumption that, even for long-term evaluations, everything is knowable in advance – no part of the evaluation will need to be figured out mid-flight. Seriously?
  3. There are often incredibly onerous RFP processes and high financial barriers to entry – this has the effect of screening out the smaller operators and limiting the pool to larger evaluation shops who have minions to copy and paste boilerplate and collate all the documents. And how incisively evaluable is a proposal that is 90% boilerplate anyway?
  4. Criteria for evaluating proposals often totally miss the mark, getting lost in the details just as badly as the evaluations that end up being commissioned!

Let me offer a few suggestions for consideration and discussion:

Step 1: Start with an Expression of Interest process rather than requiring a full proposal. A good one is brief, simple, and to the point. For example you might ask:

  • What or who is the entity/person/team planning to bid on the evaluation? (max 1 page)
  • Why are you interested in this evaluation? (max half a page)
  • What makes you think you can do it? What expertise, experience, and capacity do you have to pull it off? (max 1 page)
  • Who are you as an evaluation firm, an evaluator, or an evaluation team? What values, practices, and areas of expertise and specialization distinguish you from your competitors/colleagues? (max half a page)
  • What are the daily rates of the proposed team members, and what other overheads or incidentals would apply?
  • Please provide 2-3 executive summaries from recent evaluation reports the lead evaluator(s) have conducted/led.

For heaven’s sake do NOT under any circumstances slip in a question that asks “and tell us how you would do the evaluation” – that instantly turns it from an EOI into an RFP! Don’t ask for CVs or any extras either.

The executive summaries will be like gold. They can provide answers to such all-important questions as:

  • Are these people capable of being concise? [Any exec summary longer than 2 pages is a red flag – ten pages should put them instantly in the bin.]
  • Do they actually ask and answer evaluative questions? Are we any the wiser by the end of the exec summary how substantial or valuable the outcomes were (not just “what” they were), and whether the entire program was a waste of money or not, or is it all data and stories with no real conclusions?
  • Can they cut through the ‘noise’ and retain a sense of the big picture, or do they get lost in the details?
  • Can they write in a way that gets to the point, that makes sense?

Stand by for the sequel – a rethink of the entire concept of ‘proposals’ for commissioning evaluations.

4 comments to Extreme Genuine Evaluation Makeovers (XGEMs) for Commissioning

  • Jane, this is my favorite post to date. I can’t wait for the sequel!

    One comment though – it can be difficult to read those executive summaries out of context. Too often, we see reports that answer key questions, only to find in the body of the report that the answers aren’t linked tightly to the data. I’m afraid that I want the whole report for an accuracy and process check – but with that great executive summary front and center.

  • Jane Davidson

    Thanks, Susan!

    It’s a bit mean posting just Part 1 initially, I know, but this is the prescreening piece where we are weeding out rather than making a final call on choosing an evaluation team.

    It’s true that a good-looking executive summary doesn’t mean the conclusions are necessarily based on sound evidence and evaluative interpretation. But it’s also true that, if someone can’t make good, incisive sense in the executive summary, you can be absolutely certain the body of the report will be like wading through mud.

    So, screen the latter out, and advance the promising-looking ones onto the shortlist for further consideration.

    The alternative would be to read full reports for all those who express an interest, which may be dozens and dozens of firms/teams each with two to three 100-page reports. Not a cost-effective use of the commissioner’s time!

  • Salaam

    When I read this post, suddenly I think I lived and live in a developed country and not developing country! Because if in my country exist an evaluation opportunity, that opportunity is more similar to Jane description a bout most important problems in her keynote! And I knew what Jane suggested for doing good evaluation and more writing and speaking about it in my country but not hearing! And may be Jane has same problem more than me and more frequently!

    Yes! We live in a global village and all roads lead to Rome!

    Best

    Moein

  • Agreed Jane, and very good point. It is a matter of timing. I see what you are saying – we’re screening for writing competence in Round 1 with the ES review. I get it now. I’ll wait for Round 2 for the opportunity to ensure strong evidence for conclusions. You rock!