[UPDATE: Proposals closed at midnight EST Friday 16 March for the American Evaluation Association conference in Minneapolis. As a community service we provided a countdown clock to the deadline.]

Check the conference information pages for information about proposal types, including the new brown bag and Ignite formats, and how to submit

Ever tried to explain to someone what you do as an evaluator? Check out this version of What People Think I Do/ What I Really Do.

Technical difficulties  prevented us from sharing this with you on Friday as planned.  Thanks to the evaluator who sent this to us with a request for anonymity.

This is likely to be one of a continuing series highlighting the weaknesses of policy debates which talk about particular interventions or policy options in terms of a ‘silver bullet’ which can deliver the intended impacts without any help from other interventions or supportive context.

The release of the latest PISA results has led to handwringing in Australia as our test scores have declined – and, more to the point, the gap between good and bad schools has widened. The Age reported on a Grattan Institute report “Catching up: Learning from the best school systems in East Asia“:

SCHOOL funding levels are no guarantee of good student results, a report to be released today reveals.

The Grattan Institute report turns much accepted wisdom about education in Australia on its head.

It shows there is no clear link between school funding levels and good test results, with Korea spending less than Australia on students and performing better than them in a range of subjects.

Nor do smaller class sizes guarantee good educational outcomes; teachers in Shanghai spent 10 to 12 hours directly teaching students each week, working with an average class size of 40 students.

It’s very unhelpful to discuss funding levels and class sizes as if they could ever “guarantee” better outcomes.

For a more thoughtful discussion of these issues, check out Tim Hacsi’s book “Children As Pawns: The Politics of Education Reform” , which reviews the research evidence on this and clearly comes out with a conclusion that “additional funding, used wisely” is what is needed.  You can read some of this on google books.

And for an interesting analysis, see Tino Sanandaji’s post on his blog “Super-economy” which shows how the regression line between countries’ education expenditure and results changes when a simple revision is made to address different proportions of immigrant children (who in many cases have lower levels of language skills in the local language).

More broadly, I see this as part of a fundamental flaw in the way research and evaluation are conceptualised and used: the fundamental assumption that it is useful to think about interventions as being both necessary and sufficient to produce the intended outcomes and impacts – the ‘silver bullet’.

It’s something I’m going to be doing some more thinking about this year, given the increasing recognition of the crowded policy space in which interventions are launched, and the increasing focus on partnership approaches.

It is timely to consider ‘Occupy Valentine’s Day‘, which calls on us to reject the increasingly stereotyped rituals of Valentine’s Day and instead celebrate all forms of love in ways that are affirming, inclusive and uplifting:

Celebrating love is wonderful and romance can be great too. But we don’t need corporations to dictate how we should do it, a mainstream media chastising us for not doing it right or traditional ideas touted over and over by our friends and family.

What does this have to do with Genuine Evaluation?

• It is based on an evaluation that looks at the overall impact of an intervention, not just whether or not it reached its stated objectives (Red roses delivered? Tick!).
• It is based on an evaluation that considers the impact of the intervention on different individuals and groups, not just the average impact, or the impact on primary targets – with particular attention to the impacts on those who are marginalized.
• It encourages everyone to consider what would be locally appropriate theories of action (relevant to the discussion of scaleability and transferability that Fiona Cram is currently leading in the AEA Thought Leaders series).

(The picture was taken at a Minnesota florist recently – the advertisement reads “University research shows flower givers are considered likable, friendly and successful.” Let’s see – vaguely referenced research, self-interest, average effect, no information on possible external validity from sample… Any more?)

Enjoy your seasonal celebrations of love.

Related post: The Friday Funny: Valentine’s Day Memo

As the end of the year rushes up relentlessly, we thought this week we would share a simple story about the gap between words and deeds, and between receiving information and acting on it.

The question is “Which one of these is the evaluator?”

Charlie bit my finger – again