In a post last week, I shared some ideas about how to get approximate answers to the ‘overall value’ question by building these concepts right into survey and interview questions themselves. Some ‘overall value’ questions pertained to cost-effectiveness, whether the costs outweighed the benefits or not.
But, there is another question that is often important in evaluation, and it’s a comparative one … was this the best investment we could have made with the available resources?
One interview question I mentioned last week tapped into this comparative cost-effectiveness question. I asked CEs who had invested in an executive leadership program:
How would you rate the program as an investment compared to the other options you could choose to achieve similar benefits? … What evidence convinces you of this?
Those who invest in programs have quite often thought about this question and can give some very insightful answers.
But there are times when some more explicit comparisons are helpful. Here’s a question I asked participants in a 3-week leadership development program (referred to here as the EFP) for senior managers. Note that, for them individually, financial costs are not the most important consideration, but rather the time and effort investment.
4 (a) Apart from the EFP, what is the best short (1 to 6 week) leadership course you have attended, if any (e.g. Leadership in Practice, Henley, Harvard, London Business School, NZIM)?
Name of program:
much less valuable
|EFP was somewhat less valuable||EFP was about as valuable||EFP was somewhat more valuable||EFP was much more valuable||N/A|
(b) How does the EFP compare with the program you named above?
Now, responses to the above give a gauge of how participants saw the relative value of the program given the time and effort they invested in attending. But, the financial costs of attending (not just course fees, but also travel and accommodation, and the cost of backfilling the manager’s position while he/she attends) can be a useful overlay here.
Here’s a slightly unconventional representation of just some of the data from this question, for those interested … a ‘costline’ showing a comparison of the evaluand (EFP) with the similar length programs participants had considered either more or less valuable. The position on the costline represents total money costs, including fees, travel, accommodation, and time costs (estimated using a typical salary for the managers who attend such courses). [Click the image to see a larger version.]
Although this analysis doesn’t give a definitive answer to the comparative value-for-money question, it does help put the program more clearly into a cost and quality/value context, which is potentially useful for stakeholders.
First, we can see that the EFP compares favorably with some of the ‘big name’ programs around the world. Second, it’s been more valuable to these participants than some of the well-known international programs, including some that are almost as expensive. Third, it’s relatively expensive compared to the other programs people found more valuable.
As a side note, comments revealed that the program has high value at a particular career stage, whereas some of the other programs considered very high value were valuable at the particular stage the participant accessed them, but wouldn’t have been a better substitute now. So, it’s not as simple as saying that some of the cheaper ones would have been better – but, CEOs and other senior managers are familiar with the comparison programs mentioned, and are in a good position to consider which would be appropriate alternatives for someone specific they are seeking to develop.
- Approximate answers to the ‘overall value’ question – 5 more survey and interview item ideas
- Building causation into survey items about outcomes
- Breaking out of the Likert scale trap