Michael Scriven, during his time as our guest blogger, suggested it was time for a Copernican revolution in evaluation. Now that I have returned from three weeks on the road, I’d like to suggest two different revolutions that might be needed. In both cases I am taking literally the notion of what is considered the center around which the other elements revolve.
Evaluation as a component of implementation and management
What is at the center of our imagined universe – the evaluation or the evaluand? I think we have too often made it the evaluation, and one of the Copernican revolutions needed is to put the evaluand in the center.
It’s helpful to hang about with other evaluators. We learn a lot from each other. But there is a risk that when we tell stories of evaluation, in formal presentations at conferences, in informal chats over coffee or drinks, or in papers and books, we make the evaluation central to the story, rather than the intervention itself. And in our efforts to improve the quality of evaluation, we risk it becoming a specialist activity, undertaken only by specialists, and increasingly divorced from the real decision making of programs and policies.
I strongly support efforts to improve the quality of evaluation and to improve professional development of evaluators BUT I think real improvements will come when evaluation is seen as an integral part of program management and implementation – something that is a professional imperative for all staff and managers. This doesn’t mean that all evaluation should be self-evaluation – there can be value in having an external perspective and additional expertise brought to bear. But evaluation needs to be a game that everyone plays.
Evaluation by and for intended beneficiaries and civil society
The other revolution is in terms of WHO is in the center. All too often the powerful, formal decision makers – managers, policymakers, politicians- are the only ones whose needs the evaluation is intended to meet.
Citizen evaluations, often using technological developments to easily collect, analyze and share data, need to be on the map. And in a later post I want to explore some recent examples of these.
What do you think? Are these desirable revolutions for evaluation?
Jane at Real Evaluation
Patricia at CIRCLE (RMIT)
Real Evaluation on Facebook
Hi All
I think evaluation take on evaluand & both are at the center of our imagined universe! BUT professor Rogers’s polish on professor Scriven’s rubble, extend the debate to important details and show to some later entrance to this debate(Bob): this is an important evaluation evolution matter and we must appreciate and forward it.
Best
Moein
Patricia says… What is at the center of our imagined universe – the evaluation or the evaluand? Great question but I’d add another element – the recipient of the intervention. It’s what Saville Kushner was on about in Personalising Evaluation – about which Jane reminded us yesterday at the ANZEA meeting. Just where does the intervention stand in relation to the life experience of the individual recipient. Yes, the intervention may have had a beneficial impact on someone – but how significant was it to them in relation to their overall social, cultural, economic, intellectual or political well being? Or to use Glenda Eoyang’s useful phrase, was it a difference that made a difference?
Evaluand vs. Evaluation? A salesperson once told me that if you’re good at sales, it doesn’t matter what you sell – cars, vegetables, insurance, etc. What I took that to mean is that context – what you sell – matters little. Seems to me that when we evaluators focus too much on the technique (the evaluation) rather than the context (evaluand) we become like that salesperson. I advise those who are looking for an evaluator, to look for someone who has some passion and experience (oops! there goes objectivity) about the content to be evaluated.
These are good nudges to rethink our (evaluation) universe. After all, it turns out that the sun, no more than the earth, is not the real center of the universe around which everything else is revolving. That didn’t mean Copernicus was wrong; he had the next step right. Perhaps the real insight is relativity — without relativism. Each of the Copernican revolutions is right to say that we were wrong in being X-centric, but we mustn’t conclude that there’s a Y such that being Y-centric is the final answer, just that there’s nearly always a Y that’s better than any X as a central point of reference, for many important purposes.
On the importance of ‘differences that make a difference’— a phrase that was commonplace in post-positivist philosophy of science — I take it that another way of putting that point is that evaluation is obliged to look not only at merit and worth but also significance. And significance is not the same as ‘statistical significance’ which is neither a necessary nor a sufficient condition for real-world significance, although it’s close to a necessary condition in some special circumstances (just as RCTs are close to being NCs for causation in special circumstances).
Hi Michael et. al. Perhaps we can help you better, Michael, if we first understand what you mean
by the phrase “relativity without relativism.” For example, what is this new standard upon which the new “Copernican revolution” shall and will be based?
Chad
“If a man will begin with certainties, he shall end in doubts, but if he will be content to begin with doubts, he shall end in certainties.” – Francis Bacon
As aluded to above, the Copernican revolution was both political and postitional. It was part of a development of understanding that the world is bigger than the ‘village’ or the planet. It also challenged the notion of the centre itself – as Michael S points out.
For evaluation, one lesson is that we are only the centres of our own perception. In the wider world, the intense arguments we have about method and methodology are but an irritating sidebar for most of our intended audience.
A problem we have in common with Copernicus is that our models of the world often get too complex and unweildy to be useful. One that can be solved by stepping outside our perceived universe to re-centre our view.
But fundamentally, it challenges the very notion of the centre. A concept that concept that continues to be enshrined in government, and national and urban planning. It is humbling as a public servant to present to an audience, and realise that so-called central government is a peripheral irritation in the lives of the people you are talking to.
Hi David. We are definitely heading in the right direction. What do you mean by the phrase: “the centre itself – as Michael S points out.” For example, if humanity were to possess the key to infinite knowledge, how would we/i act responsibly?
Chad
“As a single footstep will not make a path on the earth, so a single thought will not make a pathway in the mind. To make a deep physical path, we walk again and again. To make a deep mental path, we must think over and over the kind of thoughts we wish to dominate our lives.” – Henry David Thoreau
“challenging the notion of the centre itself” is short hand for saying that post-Einstein physics [and several ancient philosophies] reveals that there is no single point which the entire universe revolves around. Any central point that is chosen is entirely abitrary and dependent on the observers’ frame of reference.
Having said that, the choice of central point is important to sense making. So Copernicus’s contribution was that understanding the paths of the planets is much easier if you concieve of the sun as the center rather than the earth.
Hence my last point – that if our models get too complex, it may be that we are trying to explain a phenomina from the wrong point of view.
The other thought I have been reflected on is the political challenge that Copernicus made the centrality of Rome. The Romans were the first to really got centralised government working – and the notion of Rome as the centre of the world persisted at the time of Copernicus (and still does today in some debates). The idea that nations, cities and organisations must be organised around a strong center is extremely pervasive in modern society and yet may be as limited as the pre-Copernian notions of the world and the universe.
Which leads on to thinking that we need to be cautious whenever we come up with (or up against) an centre-periphery model – and ask whether other ways of describing the world may work better (such as networked models).
Yes, I think we should be examining phenomena from the point of view that simplicity (order) arises from pressures within a dynamic, chaotic flux, similar in many respects to the formations that emerge from daily weather patterns.
To understand how these patterns function, as well as how we play a role as change agents, requires a certain level of second-order cybernetic thinking:
http://en.wikipedia.org/wiki/Second-order_cybernetics
With respect to research on social networks, I’d suggest looking into promising applications of Comte’s sociocratic model if they exist: http://en.wikipedia.org/wiki/Sociocracy
Chad