Approximate answers to the ‘overall value’ question – 5 more survey and interview item ideas

Well, we had a mini theme last week of tips, tricks, and ideas for designing survey (questionnaire) items that cut to the evaluative chase. All these can be used in structured and semi-structured interviews too, of course.

As a follow-up, I wanted to pursue a little more the notion of whether – and what kind of – survey questions about overall value might be useful and informative.

An important evaluation question about a set of leadership development programs I evaluated a while back was whether each of them was ‘worth it’ in some overall sense. This was a relatively low budget evaluation relative to the size of the programs themselves, so the opportunity to do a really in-depth cost-benefit analysis didn’t exist. Besides, many of the outcomes did not easily reduce themselves to some monetary benefit.

Nevertheless, the “was it worth it?” question remained, and it needed at least an approximate answer. So, what could be done?

This question could be asked at multiple different levels.

  • First, for an individual participant, was it worth going through the selection/assessment process, spending time away from work and family to participate, putting in the extra effort to engage with the program?
  • From the perspective of the organization (which paid the fees, etc), were benefits reaped that were large enough to make it worth the financial investment (course fees, plus backfilling the manager’s job while he or she attended)?
  • In some cases (as was true here) there is also a sector-level question, where there are long-term benefits, e.g., in terms of strengthening interorganizational networks and the senior management talent pool across an entire sector

Overall value at the individual level

In the last post I gave some examples of how we could gauge the individual level overall value question, both with a ‘global’ rating of the extent to which it was worth doing (AFTER first asking about specific outcomes of value – see the earlier post), as below …

How would you rate the program overall as a worthwhile use of your time? (circle one letter grade)

A+
Excellent Good Adequate Poor

and …

To what extent do you believe the benefits of program outweigh the financial, time, opportunity and other costs of attending?

Costs clearly
outweigh benefits
Benefits of similar value to costs Benefits clearly outweigh costs Unable to
estimate
(a) From your own perspective, taking three weeks out from work plus time away from family vs. benefits to you and your career

Please explain, briefly, what led you to the above conclusions:

_____________________________________________________________

_____________________________________________________________

… and supplementing these questions with some more specific information about key outcomes of value at the individual level, such as career advancement, ability to take on even more senior roles, managerial effectiveness, and so forth (see the earlier post for examples of these items). Note that, when asking about whether the program was “worth it” for the individual, that is really about their judgement, so there is little need to triangulate by asking others.

Overall value at the organizational level

But how might we go about gauging the overall value question at the organizational level, again without the evaluation resources to be able to do a serious ROI-type cost/benefit analysis?

Well, the approach I took was to ask some questions (in surveys and interviews) from a range of key informants. Starting with the participants, I asked an open-ended question first to get them to think about this (and, to give me some concrete evidence) …

What differences (if any) has the program made for your organization? How has it benefitted by sending you on the program? How do you know?

_____________________________________________________________

_____________________________________________________________

and …

To what extent do you believe the benefits of the program outweigh the financial, time, opportunity and other costs of attending?

Costs clearly
outweigh benefits
Benefits of similar value to costs Benefits clearly outweigh costs Unable to
estimate
(b) From your organization’s perspective, paying your fees and covering your position for three weeks vs. benefits to the organization

Please explain, briefly, what led you to the above conclusions:

_____________________________________________________________

_____________________________________________________________

Now, when asking and answering this question from the organization’s perspective, we do need to go beyond simply asking the individual participants. So, I triangulated by asking chief executives (who, after all, make the financial decision of whether or not to invest in the program) the following question:

How would you rate the program as an investment compared to the other options you could choose to achieve similar benefits? Do the benefits to your agency outweigh the costs (time, money, opportunity)? Why or why not? What evidence convinces you of this?

Comparative value

It is also possible to add a more explicit comparative element to the value-for-money question by asking participants to compare the program with alternatives in a broad-brush way. I’ll save that example for another post next week.

Thoughts? Or, any neat ideas you would like to share?

Comments are closed.