Michael Scriven on the Evaluation Train you boarded

Very excited to see that Kylie Hutchinson and James Coyle have posted their long-anticipated interview with Michael Scriven!

Taken by Jane Davidson, October 2013, AEA

Welcome to a tour of the cosmology of evaluation. This time we’re going by train, and Michael Scriven has some news for us!

“Guys, you all got on a train and you didn’t check the destination. You just checked the question of what it was first going to get to, which was program eval, and then there were little hamlets along the way, like personnel eval and so on.

But actually, this train is going to follow what the definition of evaluation is in the dictionary because that’s what it chose to call itself.

And I have news for you. It’s got some pretty remarkable places where it’s going to stop and you are going to have to show your pass. And we expect you to be doing a little work on the voyage, on the trip, towards making yourself able to get yourself through the pass control system.”

What are those destinations? Here are two you should know about:

  1. Intradisciplinary Evaluation. No area of study can even call itself a discipline without sound evaluation of everything it does. No, this doesn’t just mean evaluation of policies and programs relevant to the discipline. This is about how the discipline decides what makes a significant vs. a trivial advance in its discipline, a good vs. a flawed theory, a high vs. low quality piece of research, a worthy faculty member, a good dissertation, and so forth. As evaluators, we have a duty to contribute to this, to help other disciplines make sure they get it right.
  2. Ethical analysis. No, not just “research ethics”, but the more difficult task of determining whether the policy/program/project/product itself and its outcomes are ethical. At the trickiest end of the spectrum, it might involve working out whether a “good” women’s health clinic should include pregnancy termination services.

What else are you going to need to get through the “pass control system” Michael mentions?

Evaluation-specific methodology. These are the methodologies that are both distinctive to and essential for evaluation – but that most evaluators do not yet have in their toolkits.

Listen to the full interview with Michael Scriven, or download the podcast.

And don’t forget to sign up for the podcast on iTunes as well!  add to iTunes

While you’re there, be a good evaluator and rate the podcast series.

3 comments to Michael Scriven on the Evaluation Train you boarded

  • Bob Williams

    Interesting comments on ethics. For quite a few years now I’ve been promoting evaluators adopting methods drawn from the critical systems tradition. Critical systems methodologies essentially assess (ie evaluate) the ethical dimensions of interventions. I have to say that on the whole there’s been quite a bit of resistance to the idea, largely it seems because it implies challenging the “clients” view of the world. My somewhat unkind response is to ask whether evaluation aspires to be a profession or a trade. Michael is spot on here – we can either walk into the ethical space willingly or be dragged into it. Because of evaluation’s history of largely ignoring (indeed often resisting) the ethical basis of what it is evaluating, neither option is going to be comfortable

  • Tom Grayson

    For me, Michael Scriven is simply amazing. The first thought that came to my mind while listening to the podcast was, “wow! What excitment and passion in his voice — in his choice of words. He is the voice of evaluation.”

    I was struck with his concluding comment that evaluation is the “key” to so much in the philosophy of science. This comment was linked to his beginning comments that evaluation is a special discipline, a tool for all other applied disciplines.

    I also was reinforced in my thinking about the importance of being balanced in the fact-value dichotomy, which is clearly discussed by House and Howe. The secret (in my mind) for an ethhical analysis is in democratic evaluation, i.e., inclusive, dialogical and deliberative.

  • Leslie Ayre-Jaschke

    I just listened to the podcast yesterday and was just totally engaged in the conversation. I love these podcasts–there is always something for me to mull over, agree with, disagree with, or go read about more.

    Like Tom, I was impressed with Michael’s characterization of evaluation as the sort of uber-discipline and will look forward to more on this. In the meantime, I’ve dragged out his “Evaluation Thesaurus” and am going to read my way through that before tackling the new book of readings in honour of Michael.

    We are having conversations in Alberta (Canada) about evaluation and ethics and have a provincial program that is pushing this (ARECCI), although I’d say it’s less about the bigger ethical questions that Michael is encouraging us to consider. But it’s a start.