There’s a unique and extremely challenging barrier to singing the ‘no value-free’ parts of the genuine evaluation song in a higher education (a.k.a. tertiary education) setting.

And that’s what Michael Scriven calls the value-free doctrine.

Last week I delivered the opening keynote at the Self Assessment for Quality conference for tertiary (=higher) education organizations working to implement New Zealand’s new evaluative approach to quality assurance.

This means conducting self-assessment that asks and answers questions about the quality of their offerings and services and the value of their outcomes for learners and other key stakeholders (such as employers, communities and iwi).

The theme of the conference was Self-Assessment for Quality: How do you know good when you see it?

One of the great challenges of implementing this in academic organizations is that the very idea of doing truly evaluative self-assessment runs counter to the fundamental beliefs and values that form part of the culture of those institutions.

It’s not just part of the way that most academics think about evaluating programs, courses, and other services or offerings; it’s a way of thinking and a belief/value system that is embedded deep in their very own research portfolios, across almost all disciplines.

Last month in San Antonio, a participant in my AEA pre-conference workshop said to me, “WOW, this is completely different from what the professors in my master’s program told me; they said we should never use words like ‘effective’, ‘good’, or valuable’.”

Yep, the value-free doctrine is still alive and well, people!

There is a pervasive belief in traditional Western academia that ‘values’ have no place in the analysis of data or evidence. High quality research should be objective, dispassionate, and just present the data – not interpret it using ‘value-laden’ terms like ‘excellent’, ‘high quality’, ‘effective’, or ‘ineffective’.

Anyone who asks a value-free ‘evaluator’ whether a particular outcome was actually good/valuable/worthwhile is likely to be greeted with snorts of derision. Obviously the person asking has no idea what ‘real science’ is all about.

Values should be avoided like the plague – even with rubber gloves and a mask on!

We present the [descriptive] facts.

You (the decision makers) work out what’s good and what’s not – and we’ll steer well clear while you do. Don’t expect any help from us with “that stuff”!

As Scriven points out, this ‘valuephobia’ is actually pretty rich coming from these individuals. These exact same valuephobes then trot back to their desks, where they apply clear definitions of ‘quality’ and ‘value’ to the evaluation of student work; of faculty teaching, research, and service; of papers submitted for publication …

Evaluative conclusions are by no means alien to academics, but for some reason, as soon as there’s anything resembling formal data collection, that inclination goes out the window.

Just what job placement rate, speed of obtaining positions, and quality of positions constitute really excellent employment outcomes for a Bachelor of Nursing degree delivered in a specific economic climate and job market? What would mediocre look like? Or clearly not good enough?

Just why are these questions not worth answering?

Even the decision not to go there is a value claim about the question itself!

Related posts and references:

• Actionable self-assessment and evaluation (powerpoint handout of Jane’s presentation to the Self-Assessment for Quality conference)
• Scriven’s Evaluation Thesaurus – a must-have classic for every evaluator’s bookshelf! See our Shelfari bookshelf on the left for what we thought of it

Jessica Miglio/NBC

Developing good performance indicators is not easy.  The history of their use is littered with examples of how they can produce a distorted picture of performance and provide dysfunctional incentives.  Burt Perrin’s report to the OECD (Organization for Economic Co-operation and Development) Implementing the vision – addressing challenges to results-focused management and budgeting summarizes a number of these and should be required reading for anyone developing performance information systems or results-based management systems.

Even if there is not formal fieldwork pilot testing performance indicators, there should be a ‘thought experiment’ to check:

1. Is it possible to score well on the performance indicator – but to actually be performing poorly?
2. Is it possibe to score badly on the performance indicator – but to actually be performing well?

A recent example from Australia has shown the perils of not adequately testing  performance indicators.  ERA (Excellence in Research for Australia ), a new system  to rate the quality of academic researchers, has rated some highly productive researchers low because it has used an average of all their published work.  A recent article in The Australian outlined an example from The Australian National University:

Andrew Cockburn, director of the[Australian National University's]  college of medicine, biology and environment, said that when compiling data for the ERA, a high-achieving geneticist’s output became a drag on the results of the college. This was because of the proportion of “building block” work published in journals that were not of the highest ranking. Professor Cockburn said the anomalous result would not have occurred had ERA focused on the academic’s best work, rather than requiring all journal articles to be submitted for the period under review.

He said the British system, in which academics submitted their best four or six papers, was preferable. “If she had been able to submit six papers published in Nature and Science, they would have delivered a higher ranking. “Although her work was within my college the most outstanding, in the ERA metric it dragged the average college performance down. It shows the way these metrics are counted can have unfortunate consequences.”

[Full disclosure: as an Australian academic, I am not just a disinterested observer of this new system.  Like everyone working in evaluation or applied research generally, I  would probably have been better off under the Research Quality Framework that the previous government had been developing,  which included assessment of the impact of research, not just publication and citation.  On the other hand, under the new system ithe AEA serial New Directions for Evaluation, which previously didn’t count at all as it did not fit the traditional form of a journal, received the only A* (top) rating for evaluation journals, suddenly making my publication record look a whole lot better – from zero to hero overnight with no effort on my part!)

Many thanks to Michael Quinn Patton for sending us through this gem (from the New York Times) about a rather interesting essay exam for selecting graduate students into All Souls College in Oxford.

Oxford Tradition Comes to This: ‘Death’ (Expound)

OXFORD, England — The exam was simple yet devilish, consisting of a single noun (“water,” for instance, or “bias”) that applicants had three hours somehow to spin into a coherent essay. An admissions requirement for All Souls College here, it was meant to test intellectual agility, but sometimes seemed to test only the ability to sound brilliant while saying not much of anything.

“An exercise in showmanship to avoid answering the question,” is the way the historian Robin Briggs describes his essay on “innocence” in 1964 …

“Brilliant fun,” a past applicant named Matthew Edward Harris wrote in The Daily Telegraph recently, recalling his 2007 essay, on “harmony.”

But did it work? Apparently not – and that’s why the one-word essay test has now been scrapped after being used annually since 1932.

“For a number of years, the one-word essay question had not proved to be a very valuable way of providing insight into the merits of the candidates,” said Sir John Vickers, the warden, or head, of the college. …

“Many candidates, including some of the best, seemed at a loss when confronted with this exercise,” said Mr. Briggs, a longtime teacher of modern history at Oxford.

Click to view the full NY Times article

There are numerous instances across the world where ineffective personnel evaluation and student assessment practices have persisted despite being poor predictors of future performance. Another that came across my desk just this morning was a survey of UK personnel selection practices showing that it is still the selection tools with the lowest predictive validity – such as informal, unstructured interviews and CVs – that are the most widely used. [Zibarras, L.D., & Woods, S.A. (2010) A survey of UK selection practices across different organization sizes and industry sectors Journal of Occupational and Organizational Psychology, 83, 2, 499-511.]

Most people working in program and policy evaluation will agree that there are several intangible aptitudes and competencies that a good evaluator needs but that aren’t easily captured in the usual selection processes for evaluation graduate programs or for professional positions. So, perhaps we should have a light-hearted test run for the one-word essay exam as applied to our own profession …

Today’s essay question: ‘Genuine’ (Expound)

You may reply in the ‘comments’ section, below! Let’s see if we can get one entry from each continent or geographic region around the world (please state where you are beaming in from).

Building on an earlier discussion Michael Scriven started about long-term effects (what to with them and without them), I’m interested in people’s thoughts on a related issue.

In the medical profession in particular, there are some very rigid beliefs about what constitutes good enough “evidence of effectiveness” to justify offering, recommending, allowing patients to try, or even just not vehemently opposing a particular type of treatment for a patient. [There are obviously some parallels in other sectors, such as education, social services, international development, criminal justice, etc, but let's start with some medical examples for now.]

There are some glimmers of hope in other sectors (e.g. in the Best Evidence Synthesis work here in New Zealand). But there are still three areas where there are very serious challenges in building a credible evidence base given the kinds of constraints and realities surrounding them. They are: (1) cutting-edge treatments;  (2) treatments that are by their very nature tailored/individualized rather than standardized across patients or populations; and (3) learning what works for small sub-populations.

1. Cutting-edge treatments

Advancements are being made in medical practice all the time, and many of these are initially developed by clinicians (doctors, specialists, surgeons) trying a new approach on a limited number of patients, e.g. when the standard treatments are either not working, or when there’s a plausible idea about how to improve benefits for patients.

In order for a new idea to be trialled on a larger scale, it must be picked up by individuals with a research/evaluation agenda, rather than just an ongoing medical practice. From there, there’s a very long and slow process from writing a grant, through getting it funded, conducting the evaluation, writing it  up, then submitting it to a peer-reviewed journal, going through the entire review process, before it is finally published and considered actual “evidence”. On top of this, top journals exhibit a strong preference for RCTs over other types of designs.

Harvard professor of anaesthesia, pediatrics, and medical ethics and chief of the Division of Critical Care Medicine at Boston Children’s Hospital Dr. Robert Truog, in a presentation entitled Ethical Conflicts in Randomized Controlled Trials, lists eight approaches to learning about what works in medicine, in ascending order of confidence:

1. Anecdotal Case Reports
2. Case Series without Controls
3. Case Series with Literature Controls
4. Case Series with Historical Controls
5. Databases
6. Case / Control Observational Studies
7. Randomized Controlled Trials
8. Meta-analyses

Truog argues that RCTs are not the only way to learn, even in the medical profession: “Phase I and II trials, which precede RCTs, often provide strong evidence for effectiveness.”

When should we think about alternatives to the RCT? Truog lists four conditions:

1. When therapies are potentially life-saving
2. When evaluating rapidly developing technologies (improvements in both experimental and control treatments may make the results of an RCT obsolete by the time it is published)
3. When RCTs are not the most efficient way to acquire knowledge
4. When the non-randomized data [are] compelling

Cutting-edge treatments often provide several of the above conditions, and the reality is that formal RCTs are always going to be way behind the technology. Because of the timeframes involved, the results of RCTs are often “old news” by the time they appear in print. In addition, there are often ethical dilemmas in the rigid use of RCTs. As Robert Truog asks …

“Who wants to be the last patient enrolled in the control arm of a positive randomized controlled trial?”

The same is equally true for a RCT of an educational, community health, international development, or business development intervention.

2. Tailored/individualized and adaptive treatments

In the medical and health professions, as in many other arenas, there are certain treatments (or programs/initiatives) that by their very nature must be completely tailored to the individual (or to the community, or to the organization) and/or that must be responsive to changing needs and need to be adapted over time.

One medical example of this is acupuncture and the use of Chinese herbs. Individuals with the same general Western diagnosis (e.g. depression, back pain, infertility), and even with the same basic underlying medical cause for that diagnosis (e.g. endometriosis, polycystic ovaries, diminished ovarian reserve), the Chinese medicine diagnosis of the underlying imbalances may differ substantially. A competent acupuncturist will proceed with a highly individualized treatment based on each person’s specific (Western and Eastern) diagnosis, will reassess at each session and tweak the treatment accordingly.

Clearly, this individualization and constant tweaking of treatment are at odds with the usual approach to RCTs, which is to standardize treatment and have each practitioner deliver it in exactly the same way. [There are some exceptions to this problem, e.g. some RCTs have been conducted to evaluate specific acupuncture treatments before and after IVF transfer, with statistically and practically significant effects documented. In fertility treatment, this covers just one very specific short-term application, but not the kinds of longer-term treatments that are also commonly used by couples experiencing infertility.]

An additional complication for evaluating acupuncture treatment is that diagnosis requires skilled professional judgment and (given that treatment cannot be simplistically standardized) treatment efficacy is highly dependent on the competence of the practitioner. A large-scale RCT would need to use several practitioners whose competence may vary widely, and this cause of variance could easily wash out effects.

This challenge is not limited to healthcare and medicine. Think about organizational development or community development initiatives. We have all heard countless examples of programs that really only worked amazingly well because of the passion of one or two highly committed people at key locations. Or that needed to be adapted locally to respond to changing needs and aspirations (or because they were initially not well enough understood). If the intervention couldn’t be standardized across multiple locations, it doesn’t fit the mold very well for an RCT.

3. What works for small subpopulations?

A third major challenge in working out “what works for whom” in medicine is that some patient subgroups have very specific combinations of factors that may lend themselves to particular kinds of treatments, but these populations are too small in number to even develop an RCT or any other quantitative design with sufficient statistical power to meet the usual requirements for publication. Or, the “target audience” for the findings is considered too narrow.

A good example is looking at the effectiveness of IVF treatment. It’s very easy to find a substantial sample size of women in their 30s with, say, blocked fallopian tubes or endometriosis – they often have insurance coverage for infertility or are eligible for publicly funded treatment, so there are plenty trying various IVF protocols (large N) and there is quite good knowledge about what works for them.

But suppose we wanted to understand what works for women over 40, or (even harder) over 42, who have specific diagnoses? First, the numbers are naturally lower for this group because most couples have completed their families by this age. For those still trying, the woman’s age and/or her specific diagnoses often mean that she is not eligible for insurance coverage or publicly funded treatment. So, there are far fewer trying IVF, and even fewer again for the specific diagnoses that are likely to make one ineligible for insurance or publicly funded treatment.

The reality is that some specific sub-populations will never be large enough in numbers to allow the use of RCTs to learn what works. But at the same time, certain clinicians will refuse to allow the patient to try treatment approaches that have not been supported by what they consider to be “solid” clinical trials.

At the same time, there are certain clinicians around the world who are known as top of their fields in dealing with specific types of case (such as women over 40). However, only some of them publish their findings, and often their work is sidelined by mainstream medicine as being “fringe” – and the limited sample sizes and only semi-standardized treatment protocols trigger further snorts of derision about the quality of their “evidence”.

The same is again true in education, community health, international development, business, and just about any other field one can name.

Where does this leave us – and where to next?

Right now, in medicine (and to varying degrees elsewhere), it’s only a small exaggeration to say:

• If you are seeking a “tried and true” (as supported by RCTs, or by other studies published in peer-reviewed journals) approach, you will only have access to “old” treatments and initiatives – and (in the case of RCT evidence) only those that can be completely standardized.
• If you’re after something cutting-edge or that needs to be tailored or adapted mid-stream, you have to pin your hopes on anecdotal evidence (and hope your physician or funder will support you).
• If you’re a member of a relatively large or typical subgroup, your treatment can be informed by evidence from RCTs and other published studies with a decent sample size.
• But if you’re in a very small minority sub-population, all we have is “anecdotal case studies” and the whole exercise is basically a crap-shoot.

Here in Aotearoa New Zealand, we have seen some very high quality government-funded work integrating a range of qualitative, quantitative and mixed method evidence about what works in education – the Iterative Best Evidence Synthesis (BES). A short quote from the Guidelines for Generating a Best Evidence Synthesis Iteration explains how evidence is selected for inclusion:

The [New Zealand] Ministry of Education is using the term ‘best’ within the best evidence synthesis programme to describe a body of evidence that provides credible evidence, and explanations for, influences that have made, and can make a bigger difference to desirable learner outcomes for diverse learners simultaneously. The criterion for selection of evidence for a best evidence synthesis is that the research provides evidence about impacts on learner outcomes. …

This criterion for selection of evidence means that research from a wide range of methodological designs (including for example, action research studies, case studies, microgenetic studies of classroom processes, ethnographic-outcome focused studies, quasi-experimental research, multiple regression studies, longitudinal studies and experimental research) can make valued contributions to a best evidence synthesis. The point of synthesis is that a cumulative body of research, carefully interrogated, provides more explanatory power than findings from any one research study or design type. (p. 33)

This is in stark contrast to the U.S.-based What Works Clearinghouse (WWC) evidence standards:

The WWC reviews each study that passes eligibility screens to determine whether the study provides strong evidence (Meets Evidence Standards), weaker evidence (Meets Evidence Standards with Reservations), or insufficient evidence (Does Not Meet Evidence Standards) for an intervention’s effectiveness. Currently, only well-designed and well-implemented randomized controlled trials (RCTs) are considered strong evidence, while quasi-experimental designs (QEDs) with equating may only meet standards with reservations; evidence standards for regression discontinuity and single-case designs are under development.

As a humorous side note, Michael Scriven recently (on EVALTALK) nicknamed the WWC the “WWQNC, standing for What Works for Quantitative Nerds Clearinghouse (pronounced ‘WONKS’)”.

While it’s very heartening to see some more enlightened evidence synthesis work such as NZ’s BES, I am still not sure we yet have good evidence accumulation and synthesis solutions for:

1. cutting-edge treatments where the technology and thinking is changing faster than RCTs (or even other large-scale long-term evaluation designs) can usefully inform
2. individualized, tailored, and adapt-as-you-go initiatives
3. small sub-populations that need to know what’s going to work for them

Are there ways, in medicine, to accumulate knowledge directly from clinicians and aggregate that to get approximate answers to these “what works for whom and under what conditions” questions? [I recently had a discussion with a medical academic who insisted it definitely was NOT possible!]

Are there ways in which outcome data and other learnings from localized small-scale initiatives can be meaningfully aggregated? I have been working on several projects that attempt to do just this (one in special education, one in primary school literacy, one for evaluating a nationwide strategy designed to help M?ori (NZ indigenous) students enjoy education success as M?ori) but would be interested how others have gone about the same.

For more on RCTs, see also my short JMDE (2006) editorial: The RCTs-Only Doctrine: Brakes on the Acquisition of Knowledge?

Disconcerting reportsof a study was undertaken for the Australian Auditor-General’s inquiry into the $A16.2 billion school building program (a major element in the economic stimulus strategy in response to the Global Financial Crisis).

A survey of all school principals was undertaken and then largely discarded – and a smaller sample survey subsequently undertaken which oversampled private schools that were more supportive of the scheme.  Although the survey company states that the reported results reweighted the responses to reflect the population (no evidence is provided of this, and some commentators, such as Andrew Bolt in the Herald Sun newspaper seem to have not understood this or chosen to ignore it), there remains the issue of why anyone, particularly a government agency, would ask busy officials to reply to a questionnaire and then ignore their responses except for some listing of “themes” in their qualitative responses.

An odd sampling strategy

The Melbourne Age newspaper reported:

The Auditor-General originally asked 7951 primary school principals receiving money for new halls, libraries, gymnasiums or other buildings to volunteer their thoughts about the program last September. His office designed the survey and methodology, but a private company, Orima Research, conducted the research.  … As many as 3100 primary school principals responded to the survey request about the federal program, 75 per cent of which were public schools. Many of the comments were far from flattering.

But instead of using this raw data alone, Orima contacted more principals over a two to three-week period in October last year to ensure a ”broader representation” of participating schools rather than the ”passionate outbursts”.

A select sample of 620 principals was chosen from the augmented list to ensure a statistically robust response rate and representative result. Government schools made up a smaller percentage in this select sample – just 40 per cent, even though they comprised 71 per cent of all program recipients.

A stratified random sample of 922 schools was selected for intensive followup and only these replies were reported numerically.

Table A 1

School principal survey response rates by sector, ANAO audit sample versus schools not actively followed up

Sector	Number of schools	Schools in audit sample	Completed audit survey	Response rate	Schools not in audit sample	Completed full survey	Response rate
Government	5659	415	258	62%	5244	2361	45%
Catholic	1332	255	167	65%	1077	457	42%
independent	892	255	158	62%	637	338	53%
Joint	68	60	39	65%	8	3	38%
Total	7951	985	622	63%	6966	3159

These figures get even more interesting when used to calculate the proportion of schools in each sector included in the audit sample:

Sector	Number of schools	Schools in audit sample	Schools not in audit sample	% of schools in audit sample
Government	5659	415	5244	7%
Catholic	1332	255	1077	19%
independent	892	255	637	29%
Joint	68	60	8	88%
Total	7951	985	6966	12%

Over sampling of small, heterogeneous sub-populations is a valid approach in many circumstances.  In this case the proportions sampled vary considerably, and it is hard to argue that government schools can be considered heterogenous.

This weighting of the responses wasparticularly problematic in view of the differential pattern of responses:

Elsewhere, the report had noted that a majority of independent schools that had taken the design, tendering and implementation into their own hands were more satisfied on questions of value for money. Public schools, the majority of which accepted a ”cookie cutter” choice of building, were far from happy. In fact, 65 per cent of the mainly government school principals said they could not agree that they got value for money with the building templates.

So the effect of this additional sampling to add more independent schools could have been to increase the proportion who were satisfied – a serious skewing of the actual situation, although The Age reported that:

In an email response, Orima Research said the sample size of government schools had been reweighted to be statistically proportional.

Ignoring responses to a survey

A bigger problem, however, is the proces of collecting data from literally thousands of schools and then not reporting it, although the reported referred to sending the survey to all schools perhaps for cathartic purposes:

The survey was sent to all 7951 primary school principals in Australia. This provided an opportunity for every principal to give their views.

While it was indeed sent to all 7,951 school principals, and 3,159 replied – these replies were barely used. Replies to the closed ended questions were not included in the analysis.  The report makes this clear:

The responses from these schools to the open?ended questions in the ANAO survey were used to identify common themes.

ABC News reported the comments of a statistical expert:

Ray Chambers, a professor at the Centre for Statistical and Survey Methodology, examined the data from the report for ABC Online and said the method was “unusual”.

“It’s uncommon to ignore data from that many,” Professor Chambers said. “You wouldn’t go to the effort of collecting the information and not use it.  To particularly ignore the information, that is something they might have decided to do if they had been very worried that it showed a bias. Personally I would have thought they should have published the results with caveats – that would have prevented a lot of these questions about the process.”

The auditor-general’s office would not officially comment on the survey, saying the report speaks for itself.

Anne Connolly has now done a follow up story, after seeking explanations from the Audit Office about the relative merits of stratified sampling compared to poor response rate surveys of the whole population.  While acknowledging that the census survey might have had more responses from dissatisfied principals, she asks the pointed question “Why weren’t they heard?”

Asking busy people to complete a survey and then not using their responses is a poor approach to evaluation.  For a government agency to do this is unacceptable.