In the previous discussion about evaluation’s intellectual silos, Jara Dean-Coffey suggested that there were generational differences within the U.S. evaluation community with respect to the “specialist culture” that has dominated the evaluation scene to date, and wondered whether this might also be true elsewhere:
… within the US there is also a generational difference in how evaluation, regardless of what moniker you place before it, is perceived and practiced.
As someone in their early 40’s who came to evaluation or rather evaluative work as a means to strengthen the efforts of organizations engaged in social justice and equity work, for me it has always had a value based bent including both mine and that of our clients. It has always been a tool in service of something greater and thus the full range of theories and methods have always been part of the toolkit.
There may be more similarities with other non western approaches to evaluation if you move down the age continuum. What do others think?
Jara’s reflections reminded me of something else I have observed and puzzled over – and would be interested in comments and reflections on. Are there forces working against the recognition of the newer, silo-spanning generation of evaluators?
When we think about who is promoted as a significant contributor or a theorist worth listening to (e.g. in high profile AEA panels, and in the evaluation books that supposedly cover the main approaches or theories), and the formats of those, I’m seeing some patterns and wondering what patterns others see …
First, there seems to be a preference for theorists who have pegged off a distinct piece of territory and given it a catchy name – as opposed to those who skillfully blend ideas across the silos and may not have a “brand name” for what they do (partly because it varies so much that it doesn’t lend itself to distinctive “branding”).
Second, there is a preference for debate formats, “target practice”, and defending one’s own “position”. I think this tends to exclude those who can see synergies and opportunities for blending – and it discourages more constructive dialogue about what combinations of approaches to use, when, and under what conditions. Somewhere in here there’s an assumption that, unless we all violently disagree with each other, the conference session or the book is going to be boring.
Third, there is a preference for including the original inventors rather than necessarily the people who could make the best contributions to the discussion. In some cases these are one and the same, which is fine, but I do often see some missed opportunities for showcasing some of the really outstanding ‘next generation’ of thought leaders.
I am also reminded of a piece of interesting research coming out in JPSP that highlights the downside of the “take a position and defend it” approach – and then a cultural lens that can be placed over this finding. The following summary is an edited snippet from BPS Research Digest …
In a forthcoming paper, Andreas Mojzisch and Stefan Schulz-Hardt have uncovered a new reason groups so often make sub-optimal decisions. The researchers show that when a group of people begin a discussion by sharing their initial preferences, they subsequently devote less attention to the information brought to the table by each member, thus leading the group to fail to reach the optimal decision. The practical implications are clear – if you can, avoid beginning group decision-making sessions with the exchange of members’ initial preferences.
The original paper is: Andreas Mojzisch, & Stefan Schulz-Hardt (2010). Knowing others’ preferences degrades the quality of group decisions. Journal of Personality and Social Psychology. [In Press].
“The Japanese never acquired the Western habit of argument. At a Western-style meeting the participants sit there with their points of view and in many cases the conclusion they wish to see agreed upon. The meeting consists of arguing through these different points of view to see which one survives the criticism and which one attracts the most adherents.
It is hard for Westerners to understand that Japanese participants sit down at a meeting without any pre-formed ideas in their heads. The purpose of the meeting is to listen. Each participant gives neutral information. Gradually the map grows more complete. When the map is finished the route becomes obvious to everyone.
It seems odd to us that they do not argue.
It seems odd to them that we cherish argument.”
What are the lessons here for rethinking who we listen to in the evaluation community and how we listen? Assuming we are making a push toward better ways of thinking about and doing genuine evaluation, I can see a few to ponder:
- The traditional “take a position and defend it” approach to framing conference sessions and books is actually reinforcing a less effective way of creating advances in our discipline.
- This argumentative approach is also – let’s face it - a very “white Western male” set of norms and values about what constitutes “something worth listening to”. In my view this is one reason why we have seen fewer women and non-Western thought leaders in the evaluation limelight. And, building on Jara’s point, this may be a generational thing too – maybe the upcoming white Western male evaluators are less like this than the previous generation …
- We’re not giving enough airtime to the new generation of silo spanners – and we need to think about how to deliberately do this in a variety of media. Apart from anything else, it’s just good succession planning!
- The silo spanners need to step up, find each other, and showcase some highly engaging conference panels and books. The debates have (for the most part) been highly entertaining, and if they are going to be displaced, that needs to be with something that has a good energy level and is exciting to watch and participate in.