Earlier this week I posted about the uninterpretability of the standard Likert scale that asks people to agree or disagree with a statement. I suggested a more evaluative scale that is more interpretable, particularly for survey items that go after process evaluation.
Now, let’s look at survey items for outcome evaluation and a few ideas for overhauling those.
Over the years, having started from the usual applied social science approach to research design, and finding it frustratingly hard to interpret and use, I have evolved my methods to incorporate some other ideas. Two in particular …
- have the response scale reflect something about the value and practical significance of the outcomes
- build causal language into the stem
So, to illustrate, let’s take a typical item we might see fishing for outcomes from a training and development program, for example.
To what extent do you agree or disagree with the following:
|strongly disagree||disagree||neither agree nor disagree||agree||strongly agree|
|I have applied the knowledge and skills I learned in my job.||1||2||3||4||5|
OK, what’s wrong with this survey item? Well, it does tap into transfer of training (application on the job), but isn’t the main point whether the knowledge and skills actually added value on the job? For example, enhancing performance, or some other valuable outcome?
Second, the agree/disagree format is, to me, not that interpretable. I’d rather know something about how substantial the impact was – something that asks more directly about the program’s value added.
So, here are a few ideas from a questionnaire I developed a couple of years ago to evaluate outcomes from a leadership development program. Bear in mind the participant surveys were NOT the only source of evidence here. And, I am not listing ALL the questions asked, just a sample.
How much impact so far has the program had on:
|Your access to useful networks across the sector||1||2||3||4|
|Your visibility as a candidate for higher level positions||1||2||3||4|
|Your ability to take on more complex work and/or an even more senior role||1||2||3||4|
|Your career potential or career success||1||2||3||4|
[* Please note that your responses will be analysed taking into account the time you have been in the program, therefore lower levels of impact are expected on several of these outcomes for managers who have entered the program more recently. Such findings are not necessarily a poor reflection on the program.]
To supplement just some of the above, the following open-ended items were also used …
Since completing the program, have you changed job roles or responsibilities at all? If so, would you consider this change a career advancement? To what extent did the program contribute to your getting the new role or responsibilities? How do you know?
What differences (if any) has the program made for your organization? How has it benefitted by sending you on the program?
How would you rate the program overall as a worthwhile use of your time? (circle one letter grade)