We’re looking forward to the workshops we’ll be conducting next week before  and after  the American Evaluation Association conference. It’s always fun to explore with new people the ideas we’ve discussed in our books.

Jane will be looking at ways to ‘get genuine’ with evaluation – really asking evaluative questions that really matter, really answering them, and really reporting them. Patricia will be discussing how to ‘get genuine’ with logic models and theories of change -  thinking about different ways of developing, representing and using logic models and theories of change and choosing the most appropriate ways, especially to adequately address complicated and complex aspects of interventions.

There are 53 different pre- or post- conference workshops at Anaheim this year and some great opportunities to learn more about a wide range of important evaluation topics.

Getting Actionable Answers for Real-World Decision Makers: Evaluation Nuts and Bolts that Deliver

Ever read an evaluation report and still wondered how worthwhile the outcomes really were or whether the program was a waste of money? What if evaluations actually asked evaluative questions and gave clear, direct, evaluative answers? This workshop covers 1) big-picture thinking about key stakeholders, their information needs, and the evaluative questions they need answered; 2) a hands-on introduction to evaluative rubrics as a way of directly answering those questions; 3) guidance for designing interview and survey questions that are more easily interpreted against evaluative rubrics and capture evidence of causation; and 4) a reporting structure that gets to the point, delivering direct evaluative answers that decision makers can really use.

This workshop combines mini lectures, small and large group exercises to build big picture thinking to focus the evaluation on what really matters, and the most important “nuts and bolts” concepts and tools needed to deliver actionable answers.

You will learn:

• How to write a set of big picture overarching questions to guide the evaluation;
• How to use evaluation rubrics to get direct, evaluative answers to these questions;
• How to design innovative interview and survey questions geared directly toward answering evaluative questions and that capture evidence of causation;
• Evaluation conceptualization and reporting tips that maximize the chances of a clear, to-the-point, and actionable evaluation.

Session 3: Actionable Answers
Monday and Tuesday, October 31 and November 1, 9 AM to 4 PM
Level: Beginner, no prerequisites

Purposeful Program Theory

While program theory has become increasingly popular over the past 10 to 20 years, guides for developing and using logic models sometimes sacrifice contextual difference of practice in the interests of clear guidance and consistency across organizations. This session is designed for advanced evaluators who are seeking to explore ways to develop and use program theory in ways that suit the particular characteristics of the intervention, the evaluation purpose and the organizational environment. In addition to challenges identified by participants, the workshop will use mini-lectures, exercises, and discussions to address three particularly important issues- improving the quality of the models by drawing on generic theories of change and program archetypes; balancing the tension between simple models which communicate clearly and complicated models which better represent reality; and using the model to develop and answer evaluation questions that go beyond simply meeting targets.

You will learn:

• A strategy for purposeful development and use of program theory, drawing from an expanded repertoire of options;
• Additional ways to represent logic models, including results chains, and multi-level logic models;
• How to better represent complicated and complex interventions.

Prerequisites: Previous experience using program theory/logic models in evaluation; previous experience planning and undertaking evaluations
Sunday, November 6, 9 AM to 12 PM
Level: Advanced

 

I’m looking forward to returning to South Africa in September for the SAMEA conference. Before the conference I will be running a workshop on ‘BetterEvaluation‘ – the international collaborative website for sharing information about evaluation methods.

The workshop will focus on:

• Choosing evaluation methods that are appropriate for a particular situation, taking into account the purposes of the evaluation, the resources available, and the nature of the program or policy being evaluated. Making sense of the wide range of possible evaluation methods by focusing on the different tasks involved in an evaluation
• Learning to access material from a new free internet resource, BetterEvaluation, which aims to improve evaluation practice by sharing information about evaluation methods.

Participants will learn:

• A structured approach to choosing evaluation methods and identify resources for helping them to implement them well.
• About a range of methods for negotiating values, combining qualitative and quantitative data, understanding cause and effect in evaluation, and reporting findings, explore when these might be most appropriate, and identify resources to support further investigation of these.
• Different ways of combining qualitative and quantitative data.
• How to access information about evaluation methods from the new evaluation resource “BetterEvaluation”

Costs : R1350. Participant numbers have been capped and no bookings will be accepted once the limit has been reached. Bookings via the conference website.

It’s nearly time for the annual Summer Evaluation Institute of CDC (Centers for Disease Control) and AEA (American Evaluation Association), to be held in Atlanta, USA June 12-15. I’ll be escaping the chills of a Melbourne winter and enjoying some sunshine and hot topics in evaluation, including exploring ways of matching impact evaluation designs to the particular features of evaluations and evaluands (the thing being evaluated).

The Institute has a combination of keynote speakers and breakout sessions.  This year there are two great keynotes:

Monday, June 13, Keynote 1: 10 Steps to Data Vizardry – Making People Fall in Love with your Data

People think visually. However, most of the tools that evaluators, data analysts, and statisticians use actually confuse our natural visual interpretation skills. Today, we will share practical guidance that makes data presentation more effective, engaging, and valuable. Some of the topics we’ll be discussing include research-based information about improving layout and workflow, information visualization, chart selection, and styling. We’ll show real-life examples to demonstrate how applying these best practices enable charting laymen and experts alike to work like visualization ninjas.

Zach Gemignani is co-founder and CEO of Juice Analytics.

Wednesday, June 15, Keynote 2: Wiggling in or Sliding by: Quality and the “Good Enough” Rule with Leslie Cooksy

Rossi and Freeman (1989) advocated using the “good enough” rule in evaluation design, choosing “the best possible design, given the potential importance of the program and considerations of practicality and feasibility” (p. 232). But Rossi later said that he feared that the good enough rule may be misused to excuse sloppy work. In this talk, Cooksy will ask how we know what “good enough” is, and suggest ways that even the just-barely-good enough evaluation can make a meaningful difference.

Lesley Cooksy is the Past-President of AEA.

And an amazing list of breakout sessions:

Conducting and Using Success Stories for Capacity Building (Lavinghouze & Price)

Strengthening Evaluation Through Cultural Relevance (Hopson & Kirkhart)

4: Focus Group Research: Understanding, Designing, and Implementing (Revels)

Enhanced Group Facilitation: Techniques and Process (Dewey)

Building Evaluation Capacity: Tips and Strategies (Taylor-Powell)

Every Picture Tells a Story (Chapel)

Project Management Fundamentals (Deane)

Improving Survey Quality(Germuth)

Theory Driven Evaluation for Assessing Planning, Implementation, Effectiveness (Chen)

Case Study Methods for Evaluators (O’Sullivan)

An Introduction to Economic Evaluation(Corso)

Evaluability Assessments: Achieving Better Evaluations (Dawkins)

Handling Data: From Logic Model to Final Report (Barrington)

Dissemination and Implementation Research (Emshoff, Noonan, & Puddy)

Systems Level Evaluation of Communities of Practice (Hegedus & Jernigan)

Transformative Mixed Methods Evaluations (Mertens & Bledsoe)

Essential Competencies for Program Evaluators (King)

Evaluating and Improving Organizational Collaboration (Woodland)

Logic Models as a Platform for Program Evaluation (Chapel)

NEW! Impact Evaluation Design (Rogers)

Ensuring Evaluation Use(Christie)

NEW! Using Theory to Improve Evaluation Practice (Donaldson)

Qualitative Interviewing: Asking the Right Questions in the Right Way (Maietta)

NEW! Developmental Evaluation(Patton)

NEW! Performance Management, Quality Improvement, and Evaluation (Davis)

Bottom-Up Approach for Assessing Viability, Effectuality, & Transferability (Chen)

Evaluating Community Engagement in Translation Health (Akintobi & Evans)

NEW! 25+ Low-cost/No-cost Tools for Evaluators (Kistler & Bolton

NEW! Developing an Evaluation Plan (Lavinghouze & Jernigan)

Using the Guiding Principles to Improve Your Evaluation Practice (Goodyear)

Facing Evaluation Challenges in the Real World: A Case-Based Approach (Smith)

Using GIS in Evaluation(Meyers)

NEW! You + Graphic Design = Great Evaluation Presentations (Evergreen)

Analyzing Qualitative Data: Using Qualitative Data Analysis Software (Maietta)

Collaborative Evaluations: A Step-by-Step Model for the Evaluator (Rodriguez)

NEW! Unraveling the Participant Observer Interaction (Hall)

NEW! : Engaging Stakeholders in Data Issues: Creating a Conversation (Davis)

NEW! Design Principles of Dashboards and Interactive Applications (Hilburn & Gemignani)

 

The AES is providing grants for emerging Indigenous evaluators to meet the partial or full costs of attendance at the 2011 conference where the financial constraints may prevent participation.  The ultimate goal and vision of these grants is to increase the number of Indigenous evaluators in the Australasian area and to strengthen their evaluation capacities.

Testimonials from previous grant recipients about the significant benefits and opportunities that the grants have given them can be found at http://www.aes2011.com.au/rego-grants.asp

How you can help

The AES is asking if evaluators, evaluation commissioners, and program funders can support this initiative. Please see if you can:

• · Promote the conference to Indigenous evaluators.
• · Encourage eligible emerging Indigenous evaluators to apply for a conference support grant.
• · Provide some mentoring and support to grant applicants (the AES will also provide some support with preparing applications).
• · Sponsor or part-sponsor an Indigenous evaluator to attend – sponsors will be acknowledged in the official program, on the AES conference website and in other promotional materials.

Eligibility

The grants are open to emerging evaluators in the following groups:

• · Aboriginal and Torres Strait Islander peoples living in Australia
• · Maori living in Aotearoa / New Zealand
• · Pacific Island peoples (including those living in Aotearoa / New Zealand and PNG)
• · Native East Timorese people.

At least one grant will be reserved for an Aboriginal or Torres Strait Islander who meets the support grant criteria. Conference places will be offered to Indigenous evaluators with some experience in evaluation who:

• · have demonstrated their interest in and commitment to evaluation;
• · aim to strengthen their own evaluation capacity; and
• · can potentially contribute to the wider development of Indigenous evaluation capacity.

Key dates:

Requests for assistance with preparing the application must be received by 13 May 2011 Applications close and must be received by 27 May 2011 Applicants will be notified by 24 June 2011 Further information, as well as the application form and the selection criteria, is available at http://www.aes2011.com.au/rego-grants.asp

Multiple deadlines and travel have been getting in the way of either Jane or me posting on the many evaluation issues in the news at the moment. But I hope to see some Genuine Evaluation readers at tonight’s panel discussion in San Francisco, co-hosted by the San Francisco Bay Area Evaluators and The Evaluators Institute.

Mel Mark, Michael Patton and I will be discussing “Current Trends in Evaluation Use, Theory and Practice”. I want to focus on the dynamic tension between over-simplifying and over-complicating how we think about interventions and about evaluation. How do we avoid making interventions and evaluation all too hard to think about – and yet not overly simplify what is needed?

Thursday, April 7, 2011 6-8pm Embarcadero 1 (first floor) Sheraton Fisherman’s Wharf, 2500 Mason Street, San Francisco.

For more information please contact:
Uda Olabarria Walker (SFBAE): 510-291-4226 or uda@gcoinc.com
Michelle Baron (TEI): 202-994-4095 or barontei@gwu.edu