We’re looking forward to the workshops we’ll be conducting next week before  and after  the American Evaluation Association conference. It’s always fun to explore with new people the ideas we’ve discussed in our books.

Jane will be looking at ways to ‘get genuine’ with evaluation – really asking evaluative questions that really matter, really answering them, and really reporting them. Patricia will be discussing how to ‘get genuine’ with logic models and theories of change -  thinking about different ways of developing, representing and using logic models and theories of change and choosing the most appropriate ways, especially to adequately address complicated and complex aspects of interventions.

There are 53 different pre- or post- conference workshops at Anaheim this year and some great opportunities to learn more about a wide range of important evaluation topics.

Getting Actionable Answers for Real-World Decision Makers: Evaluation Nuts and Bolts that Deliver

Ever read an evaluation report and still wondered how worthwhile the outcomes really were or whether the program was a waste of money? What if evaluations actually asked evaluative questions and gave clear, direct, evaluative answers? This workshop covers 1) big-picture thinking about key stakeholders, their information needs, and the evaluative questions they need answered; 2) a hands-on introduction to evaluative rubrics as a way of directly answering those questions; 3) guidance for designing interview and survey questions that are more easily interpreted against evaluative rubrics and capture evidence of causation; and 4) a reporting structure that gets to the point, delivering direct evaluative answers that decision makers can really use.

This workshop combines mini lectures, small and large group exercises to build big picture thinking to focus the evaluation on what really matters, and the most important “nuts and bolts” concepts and tools needed to deliver actionable answers.

You will learn:

• How to write a set of big picture overarching questions to guide the evaluation;
• How to use evaluation rubrics to get direct, evaluative answers to these questions;
• How to design innovative interview and survey questions geared directly toward answering evaluative questions and that capture evidence of causation;
• Evaluation conceptualization and reporting tips that maximize the chances of a clear, to-the-point, and actionable evaluation.

Session 3: Actionable Answers
Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Beginner, no prerequisites

Purposeful Program Theory

While program theory has become increasingly popular over the past 10 to 20 years, guides for developing and using logic models sometimes sacrifice contextual difference of practice in the interests of clear guidance and consistency across organizations. This session is designed for advanced evaluators who are seeking to explore ways to develop and use program theory in ways that suit the particular characteristics of the intervention, the evaluation purpose and the organizational environment. In addition to challenges identified by participants, the workshop will use mini-lectures, exercises, and discussions to address three particularly important issues- improving the quality of the models by drawing on generic theories of change and program archetypes; balancing the tension between simple models which communicate clearly and complicated models which better represent reality; and using the model to develop and answer evaluation questions that go beyond simply meeting targets.

You will learn:

• A strategy for purposeful development and use of program theory, drawing from an expanded repertoire of options;
• Additional ways to represent logic models, including results chains, and multi-level logic models;
• How to better represent complicated and complex interventions.

Prerequisites: Previous experience using program theory/logic models in evaluation; previous experience planning and undertaking evaluations
Sunday, November 6, 9 AM to 12 PM
Level: Advanced

 

Multiple deadlines and travel have been getting in the way of either Jane or me posting on the many evaluation issues in the news at the moment. But I hope to see some Genuine Evaluation readers at tonight’s panel discussion in San Francisco, co-hosted by the San Francisco Bay Area Evaluators and The Evaluators Institute.

Mel Mark, Michael Patton and I will be discussing “Current Trends in Evaluation Use, Theory and Practice”. I want to focus on the dynamic tension between over-simplifying and over-complicating how we think about interventions and about evaluation. How do we avoid making interventions and evaluation all too hard to think about – and yet not overly simplify what is needed?

Thursday, April 7, 2011 6-8pm Embarcadero 1 (first floor) Sheraton Fisherman’s Wharf, 2500 Mason Street, San Francisco.

For more information please contact:
Uda Olabarria Walker (SFBAE): 510-291-4226 or uda@gcoinc.com
Michelle Baron (TEI): 202-994-4095 or barontei@gwu.edu

Comparison of grazed and ungrazed alpine meadow. From 2005 DSE Fact Sheet

While Japan has ‘scientific whaling’, Australia might be beginning a phase of ‘scientific alpine grazing’, reversing a policy of removing cattle from summer grazing in alpine national park in the name of research. (Thanks to a number of GenuineEvaluation readers for suggesting we feature this issue – we’ll be interested in your responses).

The evidence we used to have

There used to be sufficient evidence for government to ban cattle from alpine national parks on the grounds that:

1. they caused enormous environmental damage
2. they did not reduce the risk of uncontrolled wildfire

In 2004, an inquiry into alpine grazing received several scientific submissions, including a submission by the Australian Academy of Science, which summarized the evidence about the damage caused by cattle grazing:

Over five decades of research has shown that grazing and nature conservation in alpine areas are essentially incompatible land uses. The continuation of grazing within any of the Victorian alpine and sub-alpine national parks is at variance with established concepts and values of nature conservation.

and a position paper from the Ecological Society of Australia , the peak group of ecologists in Australia, which also argued for a ban on the basis of the damage caused:

The results of long-term studies since the 1940s have shown unequivocally that alpine grazing is an inappropriate land management practice, particularly in the Alps National Parks. The Ecological Society of Australia urges all levels of government to ensure that the ban on livestock grazing in the parks is maintained.

A study by Australia’s CSIRO (Commonwealth Scientific and Industrial Research Organisation) into the 2003 alpine fires found that cattle grazing had not reduced the risk of fire (a claim often made by those advocating alpine grazing “Alpine grazing reduces blazing”):

Widespread fires burned tens of thousands of hectares of alpine country in both Victoria and New South Wales in 2003. Not all alpine areas burned, but significant areas of all of the major alpine vegetation types were.

Preliminary surveys of burning patterns were collected on Victoria’s Bogong High Plains soon after the fires, using a variety of indices of fire extent and severity. About 100 km of transect lines were walked, across all the major regions. Measurements on whether country had burned were taken every 50–500 metres, and environmental data on vegetation types, slope and aspect were collected. A measure of fire severity – minimum twig diameter – was recorded on a sample of dominant shrubs in patches that were burned.

Statistical analysis of the data showed that there was no significant difference between grazed and ungrazed country in the proportion of the landscape that burned, in both grassland and heathland. There was no significant effect of grazing on the severity of burning in heathland.

The Taskforce report concluded that there was sufficient evidence to conclude that cattle grazing caused environmental damage and did not reduce fire risk:

Findings on the environmental benefits and impacts of cattle grazing in the Alpine National Park (excluding the issue of fire) (Chapter 3)
… On the evidence before it, the Taskforce concurs with the conclusions of the 1998 Groves report, that the scientific research is adequate and consistently reveals that grazing has a deleterious effect on biodiversity.
…
The Taskforce finds significant damaging impacts and no overall benefits for the environment from cattle grazing in the Alpine National Park.

Findings on the benefits and impacts of cattle grazing in the Alpine National Park in relation to fire (Chapter 4)
Both grazed and ungrazed areas were burnt and unburnt in the 2003 fires, with fire severity predominantly determined by the prevailing weather conditions, topography, fuel loads and fuel flammability types, not whether an area has been grazed. The Taskforce concludes that cattle grazing does not make an effective contribution to fuel reduction and wildfire behaviour in the Alpine National Park.

Now there is insufficient evidence

Early this year, in line with its election promise, the new Victorian State Government claimed there was insufficient evidence about the impact of cattle on fire risk (no mention of their damage to the environment) and moved to re-introduce cattle grazing in alpine national parks – in the form of ”strategic cattle grazing’:

To meet its obligations, the Victorian Government is investigating ways to effectively manage fuel and bushfire risk across the public land estate. This includes investigating fuel and bushfire risk management in Victoria’s high country using strategic cattle grazing.

The Secretary to the Department of Sustainability and Environment (DSE) is responsible for the management of land under the National Parks Act 1975, and must “ensure that appropriate and sufficient measures are taken to protect each national park and state park from injury by fire“.

The management of fuel and bushfire risk management using grazing has been an issue of debate for many years with a general lack of peer-reviewed science to support the differing opinions.

Before cattle are introduced into the Alpine National Park for fuel and bushfire risk management purposes, the Secretary must be satisfied with the effectiveness of fuel and bushfire risk management using strategic cattle grazing, and of the non-fuel reduction impacts.

Following a review of the available scientific evidence, the Secretary decided that there was not enough evidence to form an opinion on fuel and bushfire risk management using strategic cattle grazing.

In accordance with the National Parks Act 1975, the Secretary has commissioned a research trial to help form an opinion on the effectiveness of fuel and bushfire risk management in Victoria’s high country using strategic cattle grazing.

What about the previous research?

The announcement of the trial argued there were gaps in the available evidence:

Isn’t it proven that ‘grazing doesn’t reduce blazing’?
No. The effective management of fuel and fire using grazing has been an issue of debate for many years with a general lack of peer-reviewed science to support the differing opinions.

In its 2005 report of the investigation into the future of cattle grazing in the Alpine National Park, the Alpine Grazing Taskforce stated that “There has been only one broad-scale, systematic and statistically based investigation of patterns of burning across treeless areas of the Bogong High Plains following the 2003 fires”.

While this study found that there was no statistically significant lowering of fire incidence or severity at a landscape level because of grazing, it only focused on a small part of the Alpine National Park – grasslands, heathlands and wetlands on the Bogong High Plains region – due to practical limitations. There were also important gaps in this study including the following key variables that are known to influence fire behaviour – fuel load and structure, topography and weather.

This scientific research trial will focus on a range of ecosystems in the Alpine National Park, including forested areas, and will build upon the existing body of research by concentrating on areas where scientific evidence is lacking.

And an interesting twist

A leaked email reported in The Melbourne Age, “Government blackmailed university“, suggests that the Victorian Government has been applying some pressure to the University of Melbourne to take on the research project and to engage a particular researcher, who is supported by advocated of alpine grazing, to conduct it.

In response, the Vice-Chancellor of the University of Melbourne, in a statement to staff quoted in The Australian, declared:

The University of Melbourne does not tailor its research work to respond to political pressure. Nor in this matter has the Baillieu government used research grant powers to force particular research outcomes.

Questions

How will the study investigate the impact of cattle on fire incidence?

Were there flaws in the CSIRO study, in addition to the limitations discussed above, as claimed by Philip Maguire, a Mountain cattleman whose property was burned in the 2003 fires, and who claims

In fact there is not a lot of research on grazing and fuel reduction in the Victorian Alps. There is the notorious and thoroughly discredited CSIRO travesty by Dick Williams and that’s about it.

We will follow developments with great interest….

Well, we had a mini theme last week of tips, tricks, and ideas for designing survey (questionnaire) items that cut to the evaluative chase. All these can be used in structured and semi-structured interviews too, of course.

As a follow-up, I wanted to pursue a little more the notion of whether – and what kind of – survey questions about overall value might be useful and informative.

An important evaluation question about a set of leadership development programs I evaluated a while back was whether each of them was ‘worth it’ in some overall sense. This was a relatively low budget evaluation relative to the size of the programs themselves, so the opportunity to do a really in-depth cost-benefit analysis didn’t exist. Besides, many of the outcomes did not easily reduce themselves to some monetary benefit.

Nevertheless, the “was it worth it?” question remained, and it needed at least an approximate answer. So, what could be done?

This question could be asked at multiple different levels.

• First, for an individual participant, was it worth going through the selection/assessment process, spending time away from work and family to participate, putting in the extra effort to engage with the program?
• From the perspective of the organization (which paid the fees, etc), were benefits reaped that were large enough to make it worth the financial investment (course fees, plus backfilling the manager’s job while he or she attended)?
• In some cases (as was true here) there is also a sector-level question, where there are long-term benefits, e.g., in terms of strengthening interorganizational networks and the senior management talent pool across an entire sector

Overall value at the individual level

In the last post I gave some examples of how we could gauge the individual level overall value question, both with a ‘global’ rating of the extent to which it was worth doing (AFTER first asking about specific outcomes of value – see the earlier post), as below …

How would you rate the program overall as a worthwhile use of your time? (circle one letter grade)

and …

To what extent do you believe the benefits of program outweigh the financial, time, opportunity and other costs of attending?

Please explain, briefly, what led you to the above conclusions:

_____________________________________________________________

_____________________________________________________________

… and supplementing these questions with some more specific information about key outcomes of value at the individual level, such as career advancement, ability to take on even more senior roles, managerial effectiveness, and so forth (see the earlier post for examples of these items). Note that, when asking about whether the program was “worth it” for the individual, that is really about their judgement, so there is little need to triangulate by asking others.

Overall value at the organizational level

But how might we go about gauging the overall value question at the organizational level, again without the evaluation resources to be able to do a serious ROI-type cost/benefit analysis?

Well, the approach I took was to ask some questions (in surveys and interviews) from a range of key informants. Starting with the participants, I asked an open-ended question first to get them to think about this (and, to give me some concrete evidence) …

What differences (if any) has the program made for your organization? How has it benefitted by sending you on the program? How do you know?

_____________________________________________________________

_____________________________________________________________

and …

To what extent do you believe the benefits of the program outweigh the financial, time, opportunity and other costs of attending?

Please explain, briefly, what led you to the above conclusions:

_____________________________________________________________

_____________________________________________________________

Now, when asking and answering this question from the organization’s perspective, we do need to go beyond simply asking the individual participants. So, I triangulated by asking chief executives (who, after all, make the financial decision of whether or not to invest in the program) the following question:

How would you rate the program as an investment compared to the other options you could choose to achieve similar benefits? Do the benefits to your agency outweigh the costs (time, money, opportunity)? Why or why not? What evidence convinces you of this?

Comparative value …

It is also possible to add a more explicit comparative element to the value-for-money question by asking participants to compare the program with alternatives in a broad-brush way. I’ll save that example for another post next week.

Thoughts? Or, any neat ideas you would like to share?

I posted yesterday about the importance of visible values in evaluation. This means being clear and transparent about the definitions of quality and value used when identifying criteria, evaluating performance against them based on evidence, and weighing up the pros and cons.

The perfect mode of transport for the 'visible values' evaluator - the transparent canoe! Click image to go to vendor's site

Many evaluations, I argued, are NOT transparent about how quality and value are determined in an evaluation. Some even pretend that the entire piece of work if ‘value-free’ – which is partially true, but only part of the story.

Whose values are really embedded in ‘value-free’ evaluation?

Let’s reflect on values a little more broadly and consider what might really be going on when there is no explanation of how a particular outcome was deemed to be “potentially valuable” (and therefore worthy of inclusion) while others were not.

Who is it who walks through life without ever having to seriously explain their assumptions and reasoning? Who is it who takes exception to any questioning of how or why they made particular choices? Who is it who does a really weak job at explaining their assumptions because they don’t think they are making any? Who is it who thinks that their version of common sense is ‘the default’?

The majority. The so-called ‘mainstream’. That’s who.

Many (but not all) of us have spent some or all of our lives living as a minority in some part of the world where people just don’t make sense in the way we are used to – and people think we don’t make any sense either! I’ve done that twice for any length of time, four years in Japan and eight in the States. It’s a real eye-opener to the assumptions we carry around with us.

When you are in the majority, you seldom have to explain yourself. It’s the minorities, those who think or make sense in different ways, who are always having to justify their reasoning.

I am reminded of an earlier comment from my friend and colleague Kirimatao Paipa, who said:

Often building bridges between cultural understanding rests solely on the Maori shoulders. Translations of everything Maori is into English. Bi-lingualism aside, if this country was controlled by Maori and the boot was on the other foot, how and who would build those bridges from the Pakeha community?

Historically, it has always been the cultural and ethnic minorities who have been asked to explain themselves, while the cultural and ethnic majority’s view of the world is seen as ‘the default’ and not requiring any explanation.

The same is also true, when you think about it, when it comes to the methodological minorities of the research and evaluation universe – those who specialize in using qualitative data. This is not just because those raised in the qualitative traditions have had the importance of values and transparent reasoning drilled into them; it’s also because the ‘methodological majority’ (Team Quantoid) continually asks them to justify themselves as ‘proper scientists’.

And it’s also true that those who are disciplinary minorities in the evaluation scene (e.g. because they come out of horticulture, energy efficiency, or business rather than education or psychology) more often have to explain/justify their approaches and reasoning to those who are in the majority.

What?! You’re saying that all ‘majority’ evaluators do value-free or value-undiscussable work?

Not at all.

I’m saying that ALL good evaluation needs to be transparent about the values that are brought to bear when:

• identifying outcomes (or dimensions of quality design, implementation, etc) to cover in an evaluation
• saying what the evidence should/might look like if performance on those outcomes is “good” (as opposed to “mediocre” or “excellent”, for example)
• interpreting the evidence to draw evaluative conclusions – not just about whether the entire program is worthwhile, but on ALL of the main criteria included in the evaluation.

I’m also saying that some evaluators have been more clear and explicit than others about how they do these (because they think it is important to make such things clear AND/OR because they are members of an ethnic/cultural, methodological or disciplinary minority within a particular context).

Transparency is challenging, but rewarding!

There is no denying that, when one is living as a minority, it does get tiring to have to continually explain where you’re coming from. Likewise, as an evaluator, it takes considerable effort to generate work that is explicitly and transparently value-based.

But there’s an upside.

As a minority living in a foreign culture, and even in visiting some of the lands of my ancestors (Scotland, England), I became much more clear about who I was and who I wasn’t as a cultural being on Planet Earth. And I became much more acutely aware of assumptions I never even knew I had, and taken the opportunity to refine or ditch the ones that turned out not to be well-based.

In evaluation, I have found that the discipline of explicitly explaining the evaluative parts – the basis on which particular criteria are defined as representing or having differing amounts of quality, value, or importance; how evidence is interpreted to draw evaluative conclusions; how the mix of varying outcomes is weighed; why one positive outcome compensates for another negative one (or not) – the quality of my reasoning improved dramatically.

Clarity of evaluative reasoning in a report (be it written or verbal) becomes not just a symptom of, but a means to achieving clarity of thinking. And as Michael Quinn Patton reminded us in the opening plenary at the AEA 2010 conference, ‘evaluation quality’ isn’t just about the rigor of the methods or technical quality; it’s about the quality of the thinking and the quality of the interactions that run through the evaluation itself.

Related posts and other references:

• “No value-free”: The importance of visible values in evaluation (Jane Davidson)
• Visible values: Striving for truth, beaty and justice in evaluation [Originally a presentation delivered as part of the AEA 2010 conference's presidential strand - alongside Rodney Hopson, Katrina Bledsoe, and Ernie House - this handout is from a revised version presented to the anzea Auckland branch last week.]
• Why genuine evaluation must be value-based (Jane Davidson)
• How good is a ‘good’ outcome? (Jane Davidson)