Michael Scriven has had us working our gray matter harder than usual this week trying to come up with a new ‘Copernican’ revolution for evaluation. The ensuing discussion has covered, among other things, the point that certain [‘Northern’ and especially ‘North American’] views of the world (and their accompanying assumptions, methodologies) have, historically, been treated as ‘the default’. Michael responded:
Right, the view that people down under have an inverted view of what’s ‘really’ true is an excellent example of a loaded perception. That opens the mental door for replacing the Northern epistemology/ethics/concept of family etc. with a less biased one. BUT, apart from redoing maps so that NZ is on top, what exactly IS the new concept going to be, i.e., the new world view in the specific fields of epistemology, ethics, methodology, and what useful new results does it produce? Copernican revolutions are not JUST assertions that an alternative view is right, they are fully argued proofs that it’s better e.g., because it explains or predicts some phenomena that the original view did not. OR, not quite as good but very important, the revolutionary view provides an equally good general account.
In other aspects of the discussion, we have confirmed that a Copernican revolution doesn’t have to be one that there is broad consensus on, just one that’s based on sound logic and a different way of thinking about evaluation.
Well, let me try for a very southern hemisphere-flavored candidate for rethinking evaluation globally – the realization that the various different evaluation theories, approaches, models, and methodologies are not in fact ideologies to which one swears lifelong allegiance. Rather, some of the best genuine evaluations are the ones that ‘sample across the silos’ and combine approaches that were heretofore thought to be incompatible.
Let me explain where I’m coming from here. I did my early evaluation training in the United States (as an international student) and attended the AEA conference every year I was there. As with any situation where one is the cultural outsider, there are things that seem puzzling, things that the profession seems to take for granted, things that take a while to decipher.
The big one that struck me was how evaluation theories, models, approaches, and methodologies were viewed far less as tools or perspectives to draw on as and where appropriate, but more as aspects of one’s identity as an evaluator. It’s considered quite normal in the States to identify as ‘theory-driven evaluator’ or an ‘empowerment evaluator’ or a ‘qualitative evaluator’ – and to use that chosen approach in every evaluation done. As an outsider, this struck me as bizarre.
Now, as Bob Williams has very correctly pointed out to me, one of the major reasons for this is that the U.S. evaluation market is huge. It’s big enough to allow evaluators to position themselves as specialists, take calls only from clients who require that type of evaluation, and basically make a career out of that niche. This is one advantage of a large economy in that people can afford to bury themselves deeply in one area and devote enough time to it to develop it fully.
Where I (and other kiwis, and many other evaluators from around the world) live and work, the evaluation market – and the economy in general – is much smaller. Although every evaluator or evaluation consulting firm has a range of services they promote themselves as being particularly skilled at, we generally have to have many more options up our sleeves. If we were very specialized, we’d be seen as useless. Put another way, New Zealand is a ‘generalist’ culture (we need and value breadth) whereas the U.S. is a ‘specialist’ culture (where depth in very specific knowledge is much more respected).
The small size of our economy means that evaluators here can’t afford to position ourselves as ‘wedded’ to one particular theory, approach or methodology. The pieces of work are often quite small too, so it’s often impossible (within budget constraints) to put together a decent-sized team to cover a wide range of expertise. As a result, individual evaluators need a diverse toolkit in this context in order to survive. We need to be able to blend approaches to fit the situation and change midstream if necessary.
So, blends of evaluation approaches that are often viewed as heretic in the States (and definitely raise a few eyebrows at the AEA conference – “What are those crazy kiwis and Aussies up to now?”) are considered normal and expected here. Some of the work that I’ve done includes a goal-free theory-based evaluation, and a participatory/collaborative, explicitly evaluative, utilization-focused evaluation. There are many more, but I haven’t stopped to invent names for them.
I’d like to think the power of blending approaches will filter north across the equator and that we’ll start seeing more and more of it in the future. Perhaps Marv Alkin’s next edition of Evaluation Roots won’t have an evaluation theory tree splitting into ever smaller and more specialized twigs (populated mostly by U.S.-based specialists), but a wild, hybrid, sprawling bush with grafts from one branch sprouting out of another and vines intertwined with mingling sap, all laden with interesting and delicious fruits of new ideas.