Pushing sand uphill with a pointy stick? ‘No value-free’ in higher ed evaluation

There’s a unique and extremely challenging barrier to singing the ‘no value-free’ parts of the genuine evaluation song in a higher education (a.k.a. tertiary education) setting.

And that’s what Michael Scriven calls the value-free doctrine.

Last week I delivered the opening keynote at the Self Assessment for Quality conference for tertiary (=higher) education organizations working to implement New Zealand’s new evaluative approach to quality assurance.

This means conducting self-assessment that asks and answers questions about the quality of their offerings and services and the value of their outcomes for learners and other key stakeholders (such as employers, communities and iwi).

The theme of the conference was Self-Assessment for Quality: How do you know good when you see it?

One of the great challenges of implementing this in academic organizations is that the very idea of doing truly evaluative self-assessment runs counter to the fundamental beliefs and values that form part of the culture of those institutions.

It’s not just part of the way that most academics think about evaluating programs, courses, and other services or offerings; it’s a way of thinking and a belief/value system that is embedded deep in their very own research portfolios, across almost all disciplines.

Last month in San Antonio, a participant in my AEA pre-conference workshop said to me, “WOW, this is completely different from what the professors in my master’s program told me; they said we should never use words like ‘effective’, ‘good’, or valuable’.”

Yep, the value-free doctrine is still alive and well, people!

There is a pervasive belief in traditional Western academia that ‘values’ have no place in the analysis of data or evidence. High quality research should be objective, dispassionate, and just present the data – not interpret it using ‘value-laden’ terms like ‘excellent’, ‘high quality’, ‘effective’, or ‘ineffective’.

Anyone who asks a value-free ‘evaluator’ whether a particular outcome was actually good/valuable/worthwhile is likely to be greeted with snorts of derision. Obviously the person asking has no idea what ‘real science’ is all about.

Values should be avoided like the plague – even with rubber gloves and a mask on!

We present the [descriptive] facts.

You (the decision makers) work out what’s good and what’s not – and we’ll steer well clear while you do. Don’t expect any help from us with “that stuff”!

As Scriven points out, this ‘valuephobia’ is actually pretty rich coming from these individuals. These exact same valuephobes then trot back to their desks, where they apply clear definitions of ‘quality’ and ‘value’ to the evaluation of student work; of faculty teaching, research, and service; of papers submitted for publication …

Evaluative conclusions are by no means alien to academics, but for some reason, as soon as there’s anything resembling formal data collection, that inclination goes out the window.

Just what job placement rate, speed of obtaining positions, and quality of positions constitute really excellent employment outcomes for a Bachelor of Nursing degree delivered in a specific economic climate and job market? What would mediocre look like? Or clearly not good enough?

Just why are these questions not worth answering?

Even the decision not to go there is a value claim about the question itself!

Related posts and references:

9 comments to Pushing sand uphill with a pointy stick? ‘No value-free’ in higher ed evaluation

  • Irene Guijt

    Hi Jane. In some senses, I’m really quite amazed this is an issue. In my evaluation work, I have always been required to ‘pass judgement’ on what I saw -was something, good, bad, useful, not useful, relevant, not relevant, effective etc. Perhaps it is because I work in international development and there the DAC criteria are very common. Evaluations are usually asked to ‘pronounce’ on five issues: relevance, effectiveness, efficiency, impact and sustainability.
    Having said that, can one really be so generic for ‘evaluation’ – as there is much diversity within evaluation practice. For example, what does this all mean for ‘developmental evaluation’ – Patton’s name for a form of evaluation that works alongside implementation as part of an ongoing relationship of asking meaningful questions to improve impact or innovation or whatever the purpose is. This is my description of DE by the way. I’m in this kind of a relationship with a large program in Latin America and I am not asked to pronounce definitively on anything. I’m asked to question, doubt, probe, dig, and then put questionable issues/practices on the table that are then debated and decided upon by ‘them’. Based on this blog, I might conclude that I am a poor quality evaluator. Yet I know that I am absolutely fulfilling my mandate and meeting the evaluation need of this program.
    So perhaps ‘valuephobia’ is a good thing for some forms of evaluative practice and a problem for others… What do you think?

  • Jane Davidson

    Irene, thanks so much for asking this question because it reminds me I often forget to be clear about this – and I’ve done it again!!

    ‘Value-free’ evaluation (in my mind, anyway) doesn’t mean that the evaluator has personally failed to do any evaluative interpretation, but that he or she (or, more often, the team) hasn’t bothered facilitating or guiding the client through a process of making evaluative sense of the data either.

    Some evaluations are heavily participatory; others are totally not. But many have parts where the client and/or other key stakeholders are quite heavily involved.

    In a lot of my work, for example, I have heavy client involvement in defining “how good is good” but they may not join my team on the actual data gathering or analysis. However, they may be involved again in the evaluative interpretation of findings (or at least the more controversial or sensitive ones), or in giving me feedback on my/our evaluative interpretations, and also in the development of action plans for what to do next (if it’s part of my brief to lead them through that – a great alternative to writing recommendations due to the increased buy-in and ownership of action).

    I’m not sure whether you’re at the table when the clients make sense of the findings, but even if you’re not, there are ways of feeding the findings and questions/issues to them in ways that push them toward having some serious evaluative conversations and getting to the “is this result any good?” part.

    The kind of ‘value-free’ work I am mostly referring to is stuff that purports to be formative and/or summative, but that delivers nothing more than a sea of data – statistical significance tests, descriptive statistics, narratives, summaries of themes or opinions – but that never actually gets to the evaluative point.

    AND that doesn’t do anything to facilitate the client toward getting to the evaluative point either.

    It’s certainly possible (and desirable!) to do value-based participatory, collaborative, empowerment, and developmental evaluations. It’s not about whether the evaluator personally draws the conclusions; it’s about whether they take responsibility for ensuring they are drawn – and drawn validly, not just “whatever everybody in the room thinks”. We can help clients get to valid evaluative conclusions by really good questioning, facilitation, evidence presentation, and many other methods.

    Thoughts?

  • Chad Green

    Now that we’ve defined the situation, how can we resolve it in the quickest and most efficient way possible? Here’s a suggested recipe using the ideal gas law:

    Add two cups of behavioralism and one cup of cognitivism to a pressure cooker filled with three quarts of hot classical logic. Close the lid tightly, making sure that the safety valve of intuitionistic logic is open to prevent an explosion. Ignite the logical revolution to a temperature higher than classical logic’s normal boiling point. Boil ingredients for 10 years to soften up any remaining dualities. Serves a noble cause if you ask me. :)

    Chad

    PS I’m assuming that it will take less than half the amount of time to re-establish logic (classical and non-classical) as a dominant field in public education in comparison to the cognitive revolution.

  • Salaam, this is my two sent not as a minority but as a critical friend!

    I see! an invisible value in Patricia statements in last post “No dodgy stats” – highlights and lowlights of the year! She wrote “The first standard of evaluation, is labelled “accuracy”.”
    Invisible value is personal, poor and undiscussable value. It makes ambiguity & not evaluative transparency!
    In evaluation standards we have not the first, second, tertiary & fourth standard! Only we have four complexes of standards. Only one or two items of one category of Evaluation Standards (or finally category not first! in early Stufflebeam view) are about statistical accuracy and all accuracy matters are not statistical.
    Scientists and practitioner experiences in varied dimensions and levels of evaluation and its outcomes are indicate evaluation methods dynamics, and disapproval in world surface. Thereupon of this fickleness, evaluation is not shape or bad shape. Such opponent insight doesn’t permit evaluation knowledge profile. Evaluation is an awareness that eat itself!
    I think we counter to some paradox in applying evaluation standards! For rising some reasons in this claim, I citation to Stufflebeam from Patton!

    “The most comprehensive effort at developing standards was hammered out over 5 years by a 17-member committee appointed by 12 professional organizations, with input from hundreds of practicing evaluation professionals. The standards published by the Joint Committee on Standards in 1981 dramatically reflected the ways in which the practice of evaluation had matured. Just prior to publication, Dan Stufflebeam, Chair of the Committee, summarized the committee’s work as follows:

    The standards that will be published essentially call for evaluations that have four features. These are utility, feasibility, propriety and accuracy. And I think it is interesting that the Joint Committee decided on that particular order.
    (First)Their rationale is that an evaluation should not be done at all if there is no prospect for its being useful to some audience. Second, it should not be done if it is not feasible to conduct it in political terms, or practicality terms, or cost effectiveness terms. Third, they do not think it should be done if we cannot demonstrate that it will be conducted fairly and ethically. Finally, if we can demonstrate that an evaluation will have utility, will be feasible and will be proper in its conduct, then they said we could turn to the difficult matters of the technical adequacy of the evaluation.(Stufflebeam 1980:90)”

    In today’s RCTs imperialism, the order of evaluation standards in early Stufflebeam view is not important, and one or two items of one category are king of others and gold standards. I think if we pursuit Research based methods in evaluation work only we received to Research! Not to Evaluation. At this time many evaluators and evaluation approaches select and pursuit this fallow land! And this is the simultaneity problem of sad influence of political ideology; & methodological ideology in our field and profession.

    Mamnoon
    Moein

  • Chad Green

    Let’s define a systems concept in accordance with the situation model in Kintsch’s Construction-Integration Model of text comprehension. The systems concept, as I interpret through Kintsch’s model, consists of two dimensions: coherence and salience. The coherence of a systems concept comprises the interconnectedness of the components such that if any component is removed, the coherence of the model as a whole is destroyed. The second dimension, salience, means that some components will be deemed more important or relevant to understanding than the others, based on prior knowledge, values, contextual complexity, etc.

    So perhaps what Moein is trying to tell us is that the salience of our principles needs to be reviewed in order to counteract the value-free doctrine which, if unchallenged, may very well lead to the planned obsolescence of professional evaluation as we know it.

    Chad

  • Mamnoon Chad

    This fuel is more delicious than the above cookie!

    Thought?

    Moein

  • Chad Green

    OK, let’s fast-forward one giant leap in the history of thermodynamics to the Saturn V multistage rocket.

    Imagine that each stage of the rocket involves a process of salience and coherence seeking in the direction of firstness, which in this case is the Sea of Tranquility. The safety valve of intuitionistic logic has been replaced by the Instrument Unit, which calculates the precise position and velocity of the rocket and corrects for any deviations. In this race for technological and intellectual supremacy in space, accuracy of instrumentation is mission critical….

    But as the third and final stage rocket grows quiet, you take a look out the ship’s window to gaze upon the Earth at the beginning of an orbital sunrise: http://www.youtube.com/watch?v=ZxwN4PQe51g

    And as the Earth disappears from view from the light of the Sun, you realize for a moment that firstness was with you the entire time. :)

    Chad

  • Salaam Chad

    Your last fuel was more energetic so you take off to space by rocket please speak genuine with me and others by some simple word! Oh we have any others?

    Mamnoon

    Moein

  • Chad Green

    So who’s afraid of the big, bad question? I will repeat it below for the sake of clarity:

    What is the primeval source of all logical activity (i.e., justice)?

    Chad