Policy that is developed in response to clearly identified needs and through careful processes of community engagement – while being feasible in an adversarial political system and short timeframes?  Time for the Hollowmen to show us how it can be done.

The comments shared in response to the earlier post, Culturally Competent Needs Assessment By An “Outsider” raise issues that are critical to the discipline of evaluation. Two things come to mind; a) reflections on how we define evaluation theory, and practice within the context of culture; b) the role of values and valuing in evaluation.

K. Fisher raised some critical questions about how we define evaluative culture in evaluation. It made me realize how we tend to think about culture, values, norms when we are about to evaluate a program, rather than ensure these things become an integral part of our ‘evaluation roots’; i.e., it needs to become part of the academic curriculum/syllabus. Without cultural competence, evaluators find themselves with quick ‘band aid’ solutions “to make the evaluation seem proficient to address cultural contexts issues”. There is a great need for scholarship in this area, particularly by evaluation institutions.

Addressing the issue of culture at the back end is not only unfair to the client, but also demeaning to the communities impacted by the program. Thanks David Earle, Jane and Patricia for putting a spin on this discussion. While diverse cultures exhibit ‘their own’ evaluative thinking, it is important to recognize these differences and learn from them.

The spin weaved here also touched on an important aspect of evaluation; i.e., the values and valuing. What is it that the community values? Is it intrinsically valuable and or desirable? What are the local norms, and the related dynamics that give rise to positive and or negative impacts to the evaluand? Whose reflections are we hearing? Which voices are not being heard? What is it that is not being said, and yet, it seems to matter?

If Betty LaDuke had not asked the insiders to help her ‘listen to their values, ‘speak their values language’ and ‘explore the deeper meanings behind the values’ that shape the day to day discussions, the artist would have walked away with unsubstantiated evaluative conclusions. Inclusion of values and how they were identified in the actual evaluation process has a bearing on the relevance and validity of the evaluation results.

Integration of ‘values’ and the relevant data collected during the process provided the basis on which the needs assessment (and its related components/ dimensions) had been judged as good or bad (merit), worthwhile or worthless (worth), and (significance).

Thanks for the suggestions My next post will discuss the dangers of evaluation that fails to ground itself in local realities.

Standard ‘mainstream’ belief is that one element of credibility as an evaluator comes from one’s independence and the perceived objectivity (lack of bias) that derives from that.

Here in Aotearoa New Zealand, we often find the opposite is the case: one’s credibility with the community and the provider – and with funder and external audiences who understand the implications for validity -  is enhanced where there is a long-term history of trust and competent evaluation work that fairly represents what is happening, that “speaks truth to power” when necessary, and that doesn’t (as Nan Wehipeihana puts it) “trample on the mana of the people”.

Independence (having no connection or history with the community) is actually bad for credibility and therefore for validity. If we don’t know you and trust you, if you don’t have any connection with us, then why would we share insights with you?

The following interview snippet from leading M?ori (indigenous) researcher and evaluator Dr. Fiona Cram explains how, in Kaupapa M?ori research, relationships aren’t just the means to get one-off data collection done; productive relationships are long-term, ongoing, and the very foundation on which good research and evaluation is built. [Subscribers on the email feed will need to view the video on the Genuine Evaluation website.]

[kml_flashembed publishmethod="static" fversion="8.0.0" movie="http://www.rangahau.co.nz//assets/video/fiona/methodology_fiona.swf" width="400" height="300" targetclass="flashmovie" play="false" loop="false"] [/kml_flashembed] Glossary: “Kaupapa M?ori research” – research based on a M?ori worldview, conducted by, for and with M?ori (click link for more detailed explanation of principles) “the kaupapa of the center” = the way in which the [research] center works (includes values, approach, philosophy) “tino rangatiratanga” = sovereignty, autonomy, control, self-determination and independence; M?ori being in control of their own culture, aspirations and destiny [For more great interview snippets and insights from some of our top M?ori evaluators and researchers (both academics and practitioners), check out the excellent Rangahau website.]

It’s important to note here that, in an evaluation context, we are talking primarily (as I underrstand it) about connectedness to the community and people affected by the program (impactees), rather than necessarily to the providers of programs and services. However, in some cases the providers are  a community-based organization, in which case they may well be friends or relatives of the impactees.

One of the ways to build credibility and trust over time is to deliver any feedback in mana-enhancing ways, i.e. in ways that respect and enhance the status/influence/standing of both program/service recipients (and their communities) and the individuals managing and running the program or service.

What I take this to mean for genuine evaluation is:

• criticize the program or the practices if necessary, not the people personally
• criticism of the community has absolutely no place in evaluation or research – we need to understand strengths and needs that the program or service can and should be building on or addressing, not use evaluation to take deficit model ‘pot shots’ at communities that have often had the economic and social rug pulled out from under them for generation after generation
• acknowledge the strengths, enablers, and pockets of success or promising practices – in the community as well as in the program
• avoid causing people or communities public embarrassment or loss of face – speak truth to power; don’t trample on the relatively powerless
• provide (or, if possible and appropriate, facilitate the creation of) insights and ideas that people can really use to change the program and the community for the better

Although many of these insights come out of the indigenous (and more specifically Kaupapa M?ori) evaluation space, I am always struck by how these ideas also have legs in general population evaluation work in this country. Perhaps this is because we have a small population. If you trample on the mana of the people in one evaluation project, you can be absolutely sure this news will have reached the next potential client or community well before you bid for or begin your next piece of work! We call this the “bush telegraph” here; friends in Hawai`i tell me the “coconut wireless” is the correct term there. With social and professional networking mushrooming all around the planet, it seems only a matter of time before the entire planet will seem like one very small village, and relationships (direct or through others) will be critically important for credibility wherever you are …

See also the following related posts:

• How to spot a ‘lip service’ approach to culturally responsive evaluation (a checklist for evaluation commissioners)
• Working across the cultural divide in evaluation: roles, challenges and benefits
• Culture – insiders’ and outsiders’ insights – and genuine evaluation

So you’ve put out an RFP for an evaluation of a policy, program or initiative intended to serve and effect positive change in a “minority” community. All the proposals look terribly impressive, and they all include “cultural experts” on the evaluation team. How can you distinguish the proposals that show a clear understanding of what it takes to do effective and culturally responsive evaluations from those that merely pay ‘lip service’ to cultural competence?

How to spot an evaluation proposal that is likely to miss the point:

• The only brown faces on the evaluation team are the $15/hr research assistants
• “Cultural expertise” is budgeted for data collection and translation services, NOT for the initial conceptualization, development of evaluation questions to guide the evaluation, or evaluative interpretation of evidence
• There is a leap to outcomes measurement without any attention to questioning the fundamental assumptions. The evaluation proposal takes as given that the policy, program or initiative is the right one for this community, and that the intended outcomes (goals) as defined by funder and/or provider are the criteria to be used.

How to spot an evaluation proposal that “gets” the relevance of cultural expertise and cultural values:

* I’ll have some more to say in another post about how we should be valuing cultural expertise in $$ terms.

Reflections from discussions with some of Aotearoa’s leading Pasifika and Maori evaluators

Last week I attended a really invigorating regional symposium in Auckland run by anzea (Aotearoa New Zealand Evaluation Association) where we had a ‘critical mass’ of some of the top Maori and Pasifika minds in the profession.

The final session for the conference was a plenary discussion led by top Samoan evaluator Pale Sauni, who facilitated a  powerful and constructive discussion about where evaluations in Pasifika communities can go wrong and what we as an evaluation community can do to support and promote the best possible evaluation practices that respect, support, and include the needs, values, and aspirations of those communities. A lot of the points listed above arose in the discussion, and I’d like to acknowledge all those involved for helping me clarify my thinking on this.

I particularly liked Pale’s reference to how ‘lip service’ approaches can seriously undermine the validity of the data itself. Paraphrasing (but hopefully not misrepresenting) what he said …

When the evaluation process, the questions, and the design don’t reflect a clear understanding of community values and aspirations, what you’ll get is “McDonalds” evidence:

“Would you like lies with that?”

Pay our people $20 gift vouchers, come in for your 10-minute interview, and that’s exactly what you’ll get: Lies.

As someone behind me in the audience muttered, “Yep, Margaret Mead all over again …”

More to come …

I have a few more thoughts on this topic that I hope to get to soon in future posts, including:

• how we should be valuing cultural expertise in $$ terms
• mana-enhancing ways of learning from evaluations that ran into problems on the cultural front

Building on an earlier discussion Michael Scriven started about long-term effects (what to with them and without them), I’m interested in people’s thoughts on a related issue.

In the medical profession in particular, there are some very rigid beliefs about what constitutes good enough “evidence of effectiveness” to justify offering, recommending, allowing patients to try, or even just not vehemently opposing a particular type of treatment for a patient. [There are obviously some parallels in other sectors, such as education, social services, international development, criminal justice, etc, but let's start with some medical examples for now.]

There are some glimmers of hope in other sectors (e.g. in the Best Evidence Synthesis work here in New Zealand). But there are still three areas where there are very serious challenges in building a credible evidence base given the kinds of constraints and realities surrounding them. They are: (1) cutting-edge treatments;  (2) treatments that are by their very nature tailored/individualized rather than standardized across patients or populations; and (3) learning what works for small sub-populations.

1. Cutting-edge treatments

Advancements are being made in medical practice all the time, and many of these are initially developed by clinicians (doctors, specialists, surgeons) trying a new approach on a limited number of patients, e.g. when the standard treatments are either not working, or when there’s a plausible idea about how to improve benefits for patients.

In order for a new idea to be trialled on a larger scale, it must be picked up by individuals with a research/evaluation agenda, rather than just an ongoing medical practice. From there, there’s a very long and slow process from writing a grant, through getting it funded, conducting the evaluation, writing it  up, then submitting it to a peer-reviewed journal, going through the entire review process, before it is finally published and considered actual “evidence”. On top of this, top journals exhibit a strong preference for RCTs over other types of designs.

Harvard professor of anaesthesia, pediatrics, and medical ethics and chief of the Division of Critical Care Medicine at Boston Children’s Hospital Dr. Robert Truog, in a presentation entitled Ethical Conflicts in Randomized Controlled Trials, lists eight approaches to learning about what works in medicine, in ascending order of confidence:

1. Anecdotal Case Reports
2. Case Series without Controls
3. Case Series with Literature Controls
4. Case Series with Historical Controls
5. Databases
6. Case / Control Observational Studies
7. Randomized Controlled Trials
8. Meta-analyses

Truog argues that RCTs are not the only way to learn, even in the medical profession: “Phase I and II trials, which precede RCTs, often provide strong evidence for effectiveness.”

When should we think about alternatives to the RCT? Truog lists four conditions:

1. When therapies are potentially life-saving
2. When evaluating rapidly developing technologies (improvements in both experimental and control treatments may make the results of an RCT obsolete by the time it is published)
3. When RCTs are not the most efficient way to acquire knowledge
4. When the non-randomized data [are] compelling

Cutting-edge treatments often provide several of the above conditions, and the reality is that formal RCTs are always going to be way behind the technology. Because of the timeframes involved, the results of RCTs are often “old news” by the time they appear in print. In addition, there are often ethical dilemmas in the rigid use of RCTs. As Robert Truog asks …

“Who wants to be the last patient enrolled in the control arm of a positive randomized controlled trial?”

The same is equally true for a RCT of an educational, community health, international development, or business development intervention.

2. Tailored/individualized and adaptive treatments

In the medical and health professions, as in many other arenas, there are certain treatments (or programs/initiatives) that by their very nature must be completely tailored to the individual (or to the community, or to the organization) and/or that must be responsive to changing needs and need to be adapted over time.

One medical example of this is acupuncture and the use of Chinese herbs. Individuals with the same general Western diagnosis (e.g. depression, back pain, infertility), and even with the same basic underlying medical cause for that diagnosis (e.g. endometriosis, polycystic ovaries, diminished ovarian reserve), the Chinese medicine diagnosis of the underlying imbalances may differ substantially. A competent acupuncturist will proceed with a highly individualized treatment based on each person’s specific (Western and Eastern) diagnosis, will reassess at each session and tweak the treatment accordingly.

Clearly, this individualization and constant tweaking of treatment are at odds with the usual approach to RCTs, which is to standardize treatment and have each practitioner deliver it in exactly the same way. [There are some exceptions to this problem, e.g. some RCTs have been conducted to evaluate specific acupuncture treatments before and after IVF transfer, with statistically and practically significant effects documented. In fertility treatment, this covers just one very specific short-term application, but not the kinds of longer-term treatments that are also commonly used by couples experiencing infertility.]

An additional complication for evaluating acupuncture treatment is that diagnosis requires skilled professional judgment and (given that treatment cannot be simplistically standardized) treatment efficacy is highly dependent on the competence of the practitioner. A large-scale RCT would need to use several practitioners whose competence may vary widely, and this cause of variance could easily wash out effects.

This challenge is not limited to healthcare and medicine. Think about organizational development or community development initiatives. We have all heard countless examples of programs that really only worked amazingly well because of the passion of one or two highly committed people at key locations. Or that needed to be adapted locally to respond to changing needs and aspirations (or because they were initially not well enough understood). If the intervention couldn’t be standardized across multiple locations, it doesn’t fit the mold very well for an RCT.

3. What works for small subpopulations?

A third major challenge in working out “what works for whom” in medicine is that some patient subgroups have very specific combinations of factors that may lend themselves to particular kinds of treatments, but these populations are too small in number to even develop an RCT or any other quantitative design with sufficient statistical power to meet the usual requirements for publication. Or, the “target audience” for the findings is considered too narrow.

A good example is looking at the effectiveness of IVF treatment. It’s very easy to find a substantial sample size of women in their 30s with, say, blocked fallopian tubes or endometriosis – they often have insurance coverage for infertility or are eligible for publicly funded treatment, so there are plenty trying various IVF protocols (large N) and there is quite good knowledge about what works for them.

But suppose we wanted to understand what works for women over 40, or (even harder) over 42, who have specific diagnoses? First, the numbers are naturally lower for this group because most couples have completed their families by this age. For those still trying, the woman’s age and/or her specific diagnoses often mean that she is not eligible for insurance coverage or publicly funded treatment. So, there are far fewer trying IVF, and even fewer again for the specific diagnoses that are likely to make one ineligible for insurance or publicly funded treatment.

The reality is that some specific sub-populations will never be large enough in numbers to allow the use of RCTs to learn what works. But at the same time, certain clinicians will refuse to allow the patient to try treatment approaches that have not been supported by what they consider to be “solid” clinical trials.

At the same time, there are certain clinicians around the world who are known as top of their fields in dealing with specific types of case (such as women over 40). However, only some of them publish their findings, and often their work is sidelined by mainstream medicine as being “fringe” – and the limited sample sizes and only semi-standardized treatment protocols trigger further snorts of derision about the quality of their “evidence”.

The same is again true in education, community health, international development, business, and just about any other field one can name.

Where does this leave us – and where to next?

Right now, in medicine (and to varying degrees elsewhere), it’s only a small exaggeration to say:

• If you are seeking a “tried and true” (as supported by RCTs, or by other studies published in peer-reviewed journals) approach, you will only have access to “old” treatments and initiatives – and (in the case of RCT evidence) only those that can be completely standardized.
• If you’re after something cutting-edge or that needs to be tailored or adapted mid-stream, you have to pin your hopes on anecdotal evidence (and hope your physician or funder will support you).
• If you’re a member of a relatively large or typical subgroup, your treatment can be informed by evidence from RCTs and other published studies with a decent sample size.
• But if you’re in a very small minority sub-population, all we have is “anecdotal case studies” and the whole exercise is basically a crap-shoot.

Here in Aotearoa New Zealand, we have seen some very high quality government-funded work integrating a range of qualitative, quantitative and mixed method evidence about what works in education – the Iterative Best Evidence Synthesis (BES). A short quote from the Guidelines for Generating a Best Evidence Synthesis Iteration explains how evidence is selected for inclusion:

The [New Zealand] Ministry of Education is using the term ‘best’ within the best evidence synthesis programme to describe a body of evidence that provides credible evidence, and explanations for, influences that have made, and can make a bigger difference to desirable learner outcomes for diverse learners simultaneously. The criterion for selection of evidence for a best evidence synthesis is that the research provides evidence about impacts on learner outcomes. …

This criterion for selection of evidence means that research from a wide range of methodological designs (including for example, action research studies, case studies, microgenetic studies of classroom processes, ethnographic-outcome focused studies, quasi-experimental research, multiple regression studies, longitudinal studies and experimental research) can make valued contributions to a best evidence synthesis. The point of synthesis is that a cumulative body of research, carefully interrogated, provides more explanatory power than findings from any one research study or design type. (p. 33)

This is in stark contrast to the U.S.-based What Works Clearinghouse (WWC) evidence standards:

The WWC reviews each study that passes eligibility screens to determine whether the study provides strong evidence (Meets Evidence Standards), weaker evidence (Meets Evidence Standards with Reservations), or insufficient evidence (Does Not Meet Evidence Standards) for an intervention’s effectiveness. Currently, only well-designed and well-implemented randomized controlled trials (RCTs) are considered strong evidence, while quasi-experimental designs (QEDs) with equating may only meet standards with reservations; evidence standards for regression discontinuity and single-case designs are under development.

As a humorous side note, Michael Scriven recently (on EVALTALK) nicknamed the WWC the “WWQNC, standing for What Works for Quantitative Nerds Clearinghouse (pronounced ‘WONKS’)”.

While it’s very heartening to see some more enlightened evidence synthesis work such as NZ’s BES, I am still not sure we yet have good evidence accumulation and synthesis solutions for:

1. cutting-edge treatments where the technology and thinking is changing faster than RCTs (or even other large-scale long-term evaluation designs) can usefully inform
2. individualized, tailored, and adapt-as-you-go initiatives
3. small sub-populations that need to know what’s going to work for them

Are there ways, in medicine, to accumulate knowledge directly from clinicians and aggregate that to get approximate answers to these “what works for whom and under what conditions” questions? [I recently had a discussion with a medical academic who insisted it definitely was NOT possible!]

Are there ways in which outcome data and other learnings from localized small-scale initiatives can be meaningfully aggregated? I have been working on several projects that attempt to do just this (one in special education, one in primary school literacy, one for evaluating a nationwide strategy designed to help M?ori (NZ indigenous) students enjoy education success as M?ori) but would be interested how others have gone about the same.

For more on RCTs, see also my short JMDE (2006) editorial: The RCTs-Only Doctrine: Brakes on the Acquisition of Knowledge?