Rethinking “who’s worth listening to” – and how we listen to each other – in evaluation

In the previous discussion about evaluation’s intellectual silos, Jara Dean-Coffey suggested that there were generational differences within the U.S. evaluation community with respect to the “specialist culture” that has dominated the evaluation scene to date, and wondered whether this might also be true elsewhere:

… within the US there is also a generational difference in how evaluation, regardless of what moniker you place before it, is perceived and practiced.

As someone in their early 40’s who came to evaluation or rather evaluative work as a means to strengthen the efforts of organizations engaged in social justice and equity work, for me it has always had a value based bent including both mine and that of our clients. It has always been a tool in service of something greater and thus the full range of theories and methods have always been part of the toolkit.

There may be more similarities with other non western approaches to evaluation if you move down the age continuum. What do others think?

Jara’s reflections reminded me of something else I have observed and puzzled over – and would be interested in comments and reflections on. Are there forces working against the recognition of the newer, silo-spanning generation of evaluators?

When we think about who is promoted as a significant contributor or a theorist worth listening to (e.g. in high profile AEA panels, and in the evaluation books that supposedly cover the main approaches or theories), and the formats of those, I’m seeing some patterns and wondering what patterns others see …

First, there seems to be a preference for theorists who have pegged off a distinct piece of territory and given it a catchy name – as opposed to those who skillfully blend ideas across the silos and may not have a “brand name” for what they do (partly because it varies so much that it doesn’t lend itself to distinctive “branding”).

Second, there is a preference for debate formats, “target practice”, and defending one’s own “position”. I think this tends to exclude those who can see synergies and opportunities for blending – and it discourages more constructive dialogue about what combinations of approaches to use, when, and under what conditions. Somewhere in here there’s an assumption that, unless we all violently disagree with each other, the conference session or the book is going to be boring.

Third, there is a preference for including the original inventors rather than necessarily the people who could make the best contributions to the discussion. In some cases these are one and the same, which is fine, but I do often see some missed opportunities for showcasing some of the really outstanding ‘next generation’ of thought leaders.

I am also reminded of a piece of interesting research coming out in JPSP that highlights the downside of the “take a position and defend it” approach – and then a cultural lens that can be placed over this finding. The following summary is an edited snippet from BPS Research Digest

In a forthcoming paper, Andreas Mojzisch and Stefan Schulz-Hardt have uncovered a new reason groups so often make sub-optimal decisions. The researchers show that when a group of people begin a discussion by sharing their initial preferences, they subsequently devote less attention to the information brought to the table by each member, thus leading the group to fail to reach the optimal decision. The practical implications are clear – if you can, avoid beginning group decision-making sessions with the exchange of members’ initial preferences.

The original paper is: Andreas Mojzisch, & Stefan Schulz-Hardt (2010). Knowing others’ preferences degrades the quality of group decisions. Journal of Personality and Social Psychology. [In Press].

So, what’s the cultural lens on this? Keith McGregor, a New Zealand personnel psychology specialist, made an interesting link to a snippet from Edward de Bono’s Six Thinking Hats:

“The Japanese never acquired the Western habit of argument.  At a Western-style meeting the participants sit there with their points of view and in many cases the conclusion they wish to see agreed upon.  The meeting consists of arguing through these different points of view to see which one survives the criticism and which one attracts the most adherents.

It is hard for Westerners to understand that Japanese participants sit down at a meeting without any pre-formed ideas in their heads.  The purpose of the meeting is to listen.  Each participant gives neutral information.  Gradually the map grows more complete.  When the map is finished the route becomes obvious to everyone.

It seems odd to us that they do not argue.

It seems odd to them that we cherish argument.”

What are the lessons here for rethinking who we listen to in the evaluation community and how we listen? Assuming we are making a push toward better ways of thinking about and doing genuine evaluation, I can see a few to ponder:

  1. The traditional “take a position and defend it” approach to framing conference sessions and books is actually reinforcing a less effective way of creating advances in our discipline.
  2. This argumentative approach is also – let’s face it –  a very “white Western male” set of norms and values about what constitutes “something worth listening to”. In my view this is one reason why we have seen fewer women and non-Western thought leaders in the evaluation limelight. And, building on Jara’s point, this may be a generational thing too – maybe the upcoming white Western male evaluators are less like this than the previous generation …
  3. We’re not giving enough airtime to the new generation of silo spanners – and we need to think about how to deliberately do this in a variety of media. Apart from anything else, it’s just good succession planning!
  4. The silo spanners need to step up, find each other, and showcase some highly engaging conference panels and books. The debates have (for the most part) been highly entertaining, and if they are going to be displaced, that needs to be with something that has a good energy level and is exciting to watch and participate in.


5 comments to Rethinking “who’s worth listening to” – and how we listen to each other – in evaluation

  • David Turner

    The premium on catchy names has always bothered me (perhaps because I’m not very good at coining them?).

    The issue about generating real dialogue that involves listening and moving past defending your own positions is something I’ve struggled with a lot. I used to run sessions at the AES conference on ethics, in which I presented scenarios for people to engage with and talk through. I found that people were generally willing to grapple honestly with the issues and share their views about how to respond. These sessions allowed participants to develop their own views, but weren’t really intended to have results for a wider audience.

    People at the MIT organisational learning group have worked on the issue of how to get people to work past their entrenched positions and come to new and shared views. They talk about “Theory U”, although I forget how to explain that at the moment. Perhaps we should look beyond the evaluation field to get ideas on how to deal with this issue.

  • Jane Davidson

    David – great point not limiting ourselves to intradisciplinary silo spanning, but also finding the interdisciplinary gems from others who have been developing useful ideas. And, LOL, I am hopeless at catchy names too.

    I wanted to add, too, that I do think it’s important to have opinionated people in the evaluation scene who have interesting and cage-rattling things to say, but who don’t have their fingers in their ears. [In fact, these are important criteria for choosing our celebrity guest bloggers!] Obviously, Patricia and I have quite a few things to say, otherwise we wouldn’t have started this blog. We are very keen on silo-spanning (of both the intradisciplinary and interdisciplinary varieties); we are also keen to deliver on our original promise of a distinctly Southern Hemisphere perspective; and we value the right to change our minds when we stumble across improved ways of thinking about or doing things!

  • Hi Jane,

    I am glad and honored that my reflections led to an expanded post that continued to push the dialogue on how we, as a practice, engage with new ideas and approaches which defy a single label and who get’s to be the messenger.

    Your points about the conference/session format resonated strongly. This whole notion of argument as being core to how learning seesions are designed when actually they prevent the audience from holding multiple frames is quite interesting. Your reference to a Japanese meeting format absent of pre-determined positions and outcomes reminded me how much “we” as Westerners assume. I had in fact been doing some follow-up research of my own to a post I did on “how we learn” ( and was starting to better understand how rhteroric and debate became our default.

    As you might be aware, April in the US brought with it an interesting convergence of conferences; Council on Foundation, Emerging Practioners in Philanthropy, Young Non Profit Professionals, Association of Black Foundation Executives and I think a few more. As various bloggers reflected on their experience, a thread started to emerge sparked by a question posed by Teri Behrens on the Tactical Philanthropy Blog,”How did the 90 minute conference session become enshrined as THE format for these events?” Although the question engaged several people, not one of us mentioned the cultural frame.

    I am always fascinated when similar conversations are happening in seemingly different spheres. It suggest a readiness and thus an opportunity to move forward within and across…etc. Perhaps as you suggest, I will as a silo spanner step forward and make the connection. More to follow.

  • Hi Jane,

    I took your lead and went off on my own tangent in a related post on

    The “who is worth listening to” is still rolling around. I think it is a question that is being asked in variety of places.

  • Gabriel Della-Piana

    J. Thomas Hastings was worth listening to. Tom Hastings’ style provided prompts for focusing on information with high leverage, not only for reducing uncertainty concerning claims, but also for informing program success and encompassing concerns for all stakeholders in an enterprise. If this claim is justified, a major contribution to the field would be to keep on our minds the image of Hastings, and professionals like Hastings, in the back or side of the room or across the table saying “yah-but” or “now wait a minute” (to make sure that every voice was heard or perspective considered), or “what was the question”? (when things got in a muddle or needed better focus), or “would you describe the situation we are talking about” (when context was missing or dropped out of consideration), or even making an intentionally exaggerated comment like, “whenever you hear the word standard you know someone is planning to do something that will be harmful to children”. It is this style in Tom, this repetition of form with worthwhile function, that may help to fill the need for what Lagemann (1997) called, “understanding and surmounting the constraints of professionalization in order to develop more truly equal, genuinely respectful, and effectively collaborative relationships among the groups most directly involved in the study and practice of education”. And such a presence, posture, style and function might even satisfy Ernie House’s wish, after reading an earlier statement on Hastings, that one should try to capture the ineffable in Tom Hastings. The ineffable might be, with Tom’s presence, that Annie Dillard sense of “Now-you-don’t-see-it, now-you-do” (Dillard, 1974). And reflection on lives such as Tom’s might lead to recognition of the significance of the ephemeral but influential voices around us, not for praise or historical record, but for management of our professional affairs and reminders of our values.