6 questions that cut to the chase when choosing the right evaluation team

Have you ever felt like you were nearly going cross-eyed, wading through evaluation proposals that all present screeds of information but don’t really tell you what you need to know? Or, writing one of these yourself?

The whole RFP (Request for Proposal) and evaluation contractor selection process frustrates both evaluators and clients (or commissioners) alike.

In a previous post, entitled 9 hot tips for commissioning, managing (and doing!) actionable evaluation, I outlined some alternatives to the traditional “let’s have everyone write a 50-page phone book” approach.

Central to getting the evaluation team selection process right is knowing the right questions to ask.

Great questions to ask prospective evaluators need to:

help you clearly distinguish among the candidates …
… on those strengths and weaknesses that will make or break the evaluation.

After lengthy discussions with both evaluator colleagues and clients, here are my …

Top 6 questions that cut to the chase when choosing the right evaluation team

  1. This is an evaluation of [insert evaluand] as opposed to a research project about it. How would you define the difference? What knowledge and skills will you need to apply to this because it is an evaluation rather than a research project?
  2. One of our key evaluation questions is, “How valuable are the outcomes for [insert specific recipient group]?” How would you go about answering this question? [- as opposed to a descriptive question such as, “What were the outcomes for this group?”]
  3. The various different stakeholders with an interest in this program [or evaluand] have some quite different and possibly conflicting perspectives on, for example, which outcomes should be considered most valuable and what “quality service delivery” looks like in this context. How would you manage these differing perspectives and apply them appropriately to drawing your conclusions?
  4. We anticipate another of the key challenges in this project to be [describe*]. How would you handle this challenge? Please provide examples of how you have done in the past.
  5. How would you describe yourselves, professionally, as evaluators? What is your ‘brand’? What is your ‘signature approach’, the one you are best known for? What kinds of projects particularly play to your strengths? What kinds of projects do you avoid because they are outside your areas of strength? [Note: Probe specifically about evaluation expertise and approach; don’t let them away with an answer about content expertise.]
  6. Please provide examples of three recent executive summaries for evaluations you have completed. [These speak volumes about how well the team can write, get to the point, be succinct, actually answer important questions evaluatively (not just free associate to them with data), and how well they truly understand intended users and what will make sense for them.]

* “Challenges” to ask about might include the limited budget or timeframe, a lack of baseline data, difficult-to-measure (but important) outcomes, pressure from influential stakeholders to adopt a particular design or approach, anticipated resistance, changing political or economic context, and so on.

Thoughts? Comments? Please click through to the post on the Genuine Evaluation site to chime in with your additions!

Acknowledgements, and more on Hot Tips for Commissioning and Managing Actionable Evaluation:

Mark the dates!

3 comments to 6 questions that cut to the chase when choosing the right evaluation team

  • I love using this type of an interview instead of the typical RFP for every project.

    In addition to the questions you ask, I might suggest that organizations request examples of different ways evaluators have communicated results to different audiences. For example, how does your presentation differ when it’s for the board rather than program staff?

  • Jane Davidson

    Great idea, Maria! Particularly if you use a face-to-face approach to auditioning your shortlist, you could ask for a short sample presentation for the audience that usually finds evaluation the least engaging.

    This kind of live demo is so much more telling than some sales pitch in a proposal saying “We are highly experienced and effective in presenting findings for a wide range of audiences, from Boards to program staff to communities.”

  • Daniel Ticehurst

    Dear Jane (and Maria)

    Great questions to provoke a revealing discussion. Looking at
    Cvs is also often as a way of inviting them for interview. Think of many academic researchers masquerading as evaluators!! A catalogue of publications and descriptions of activity. So many of your questions would also be really useful as a basis for
    Developing a focussed cv. Too many consultants and academics who work for public sector development agencies are encouraged to describe what they have done with the only quality standards asked for being a phd and published articles. What does this tell you? Good At writing but what about inter-personal and verbal communication skills?

    I always found a question like “of the last three evaluations you have done which one
    made the biggest difference, why and for who?” take it from there…..

    Thanks again for the great website. Daniel