9 hot tips for commissioning, managing (and doing!) actionable evaluation

High quality, worthwhile, actionable evaluation doesn’t just depend on the technical competence and effective consultation skills of the evaluator.

Decisions made and actions taken (or not taken) by the client can make or break the value of evaluation for an organization.

High-value evaluation is the product of a fruitful interaction between a well-informed client and a responsive, appropriately skilled evaluation team.

What have we – both clients and evaluators – learned from both stunningly high value evaluative work (“dream projects”) and bitter disappointments (a.k.a. “Nightmares on Eval Street”)?

And, how can evaluators help clients get maximum utilization and value for their evaluation dollar by becoming informed, demanding, savvy consumers of (and partners in) evaluation?

There is much that the client can do to get evaluation on the right track at the outset, in the commissioning (e.g. RfP) phase.

The same is true when it comes to working alongside the evaluation team to ensure the client organization’s needs are met, and that the insights and learnings from evaluation are both influential and used. [This point holds whether the evaluation is participatory or independent.]

There is a LOT to say on this topic, so I’m giving just a taster of ideas here. Over the next few weeks and months I’ll be picking up on each of these – and more – and fleshing them out a bit more and expanding on some earlier material.

6 hot tips for evaluation clients:

  1. Save time and energy with a two-phase process: “Expression of Interest” (EoIs) to shortlist qualified bidders followed by a full RFP to select the right team
  2. Select contractors based on capabilities, not an evaluation plan they have written ‘blind’ (i.e. without speaking with stakeholders)
  3. Use our list of great questions (under development!) that cut to the chase and reveal the evaluation team’s approach, capabilities, and fit with the project
  4. Ask for a ‘skeleton report’ before any data are collected so you can discuss what you need as an end product
  5. Consider not requesting a long final report at all – they are seldom read; often, presentations and short written updates at several time points are more useful and usable
  6. Have a contingency plan for ‘pulling the plug’

3 hot tips for evaluation contractors:

  1. Help the client figure out whether or not you are the right fit by getting really clear about your brand – “who you are” as an evaluation practitioner or organization. What distinguishes you from the rest of the pack? What are your core values? Your “signature skill sets”? Your unique value proposition?
  2. Put serious time and effort in at the front end to ensure you clearly understand the client’s needs, including any decision timelines (e.g. MQP’s point that summative decisions are often made 18 months ahead of time, but usually evaluation findings available then have only been designed with formative uses in mind)
  3. Develop an overarching set of big-picture evaluation questions to guide the work – and talk through with the client what a real answer would look like, to them

What would you add to this list? Fire away in the comments section below!

It’s time for a radical rethink of the RFP process and the usual approach to evaluation project management – watch this space!

Related posts:

 

7 comments to 9 hot tips for commissioning, managing (and doing!) actionable evaluation

  • Jane, great post! Do you have a link to the referenced “our list of great questions”? Would love to learn and share.

  • Thanks, Susan – the “list of great questions” will be in an upcoming post! I’ve been putting my head together with one or two clients to get these right; we are almost there and would love some comments from people. So, watch (and continue to chime in on) this space!

  • Nan Wehipeihana

    Hi Jane.

    For Commissioners: I think its important to get clear about the skills, competencies, expertise and disposition/interpersonal skills that are critical for a specific evaluation. No two evaluations are exactly the same, and no two evaluators/evaluation teams are exactly, so getting clear on what matters for a specific context is important. And when selection decisions appear close, look for strengths in the essential criteria as opposed to working you way down the ‘laundry list’ of desirable and nice to have skills and competencies.

    For evaluators: ‘stick to your knitting’. Know what you’re good at, where you’re strengths are and focus on work in this area. Partner up with acknowledged experts in other fields of evaluation/subject areas to grow your own skills and knowledge and where an evaluation calls for expertise beyond your own core competencies.

  • We trialled some of these suggestions in our agency with great success. We were able to eliminate the ‘boiler plate’ ones, who were just interested in getting a bit because of their reputation in other fields.

    We interviewed the evaluators face to face and asked them some tricky questions, like ‘How suited is this evaluation to a quasi-experimental design and how might you go about it?’

    We found this really helped to identify their capability and gave us a sense of how the evaluators operated. The evaluators we contracted through this process have been very satisfactory.

    It is hard for public sector commissioners to go outside the standard procurement processes, and your suggestions helped us have the confidence to tackle things differently. Thanks Jane

  • Jane Davidson

    Great to hear this has worked well for you, Bronte!

    Thanks so much for sharing your experiences.

  • Thanks for the tips Jane. My current work involves commissioning and managing several external evaluations for partner organisations, and I find your first two tips for evaluation clients critical. I have also learnt that as part of the solicitation process, face to face interviews followed by reference checks are essential in confirming evaluator’s skills and fit for a given evaluation.
    We have also found that using a participatory process to engage stakeholders in discussing and finalizing the evaluation questions, as well as discussing the proposed approach once the evaluator is hired goes a long way in creating buy-in and support for the evaluation. Lastly, commissioners/managers requiring various brief updates as part of the contract deliverables helps ensure close follow-up and feedback to the evaluators throughout the evaluation process and thereby limiting potential bad surprises when the end product is submitted.

  • marlene laeubli

    Hello Jane

    Congrats to you and Patricia for a really great blog with many, many interesting ideas and messages! I am starting out on a project to develop a list of competencies – maybe at different levels – for managers of inhouse evaluation services. Apart from the IDEAS list of competencies, do you know of anything else or where else to look? Most interested in your help.

    Best wishes

    Marlène

Leave a Reply

  

  

  

You can use these HTML tags

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>