Culture – insiders’ and outsiders’ insights – and genuine evaluation

This fantastic discussion that our guest blogger, Nan Wehipeihana, has been leading for us this week has got me thinking. REALLY thinking!

As part of my doctoral training in organizational psychology and management, one great area of interest for me was the study of organizational culture and organizational culture change. I’d like to present a few ideas from that area and see how they apply to the concept of culture more generally (as it relates to ethnicity and all the many other dimensions of diversity we see around us). And, how these insights might be able to drive us closer to genuine evaluation.

We can think of culture as having three levels (and I always find the iceberg metaphor helpful for visualizing this):

  • Formal policies, systems, and practices (artifacts; the highly visual stuff that is obvious to both insiders and outsiders; the official “talk” that may or may not match the “walk”)
  • Informal practices and symbolic actions – “the way we do things around here” (the norms that insiders intuitively follow and outsiders only discover when they accidentally break them; the “walk”/actions that symbolize what is really valued – may differ wildly from the official “talk”)
  • Beliefs, values, and attitudes – deep-seated understandings and assumptions shared by insiders that are buried well below consciousness

Now, one thing we must consider as evaluators is whether we take the culture we are working in as a “given” – never question it, treat all of its beliefs and values as valid, leave the undiscussables well alone … or whether some useful part of our job might be to surface some of the beliefs, values, traditions, rituals – the undiscussables – and ask some probing questions about them. [Many thanks to Bob Williams, who stimulated my thinking on this topic in a conversation the other day.]

Let’s put aside for a moment the very emotive idea of doing this in another ethnic culture and consider applying this idea to organizational culture.

Every organization has norms, practices, values, and beliefs that have become so embedded  in organizational life that no-one dares question them. In fact, it is often decidedly unsafe to question them. One of the reasons outside consultants (sometimes evaluators) are brought in is because they provide a fresh perspective of one who is not immersed in the culture. They can provide insights, they can raise questions like “Why do you do things this way? Who does it help? Who does it hurt? Is it a barrier or an enabler for what you are trying to achieve? Why does your ‘walk’ not match your ‘talk’ on this issue?” An outsider may be the only person who can safely ask such questions. This is what Argyris and Schon called surfacing the undiscussables. It’s uncomfortable, it’s difficult, but sometimes it’s precisely what needs to be done to start a conversation about what really matters.

Every organization develops a culture (ways of doing things, ways of thinking) over time, and often for very good reasons. The ways of doing things and the ways of thinking have worked in the past to ensure organizational survival, or harmony, … and sometimes they have worked very well to perpetuate power and status distinctions in the favor of one subgroup within the organization.

Over time, the context changes, society changes, we learn new things, we develop new understandings. So, after a while, it can be healthy to question underlying beliefs and assumptions and values and ask if they still work in the current context. This kind of questioning has happened at a societal level all over the world about big issues such as gender discrimination, apartheid, slavery, racial discrimination, and many more. It has happened in a smaller scale in organizations as well – questioning how the “old boys’ networks” and other organizational ‘traditions’ had created a glass ceiling that prevented women and minorities from holding positions of influence. We have questioned the underlying assumption – namely, “That doesn’t matter because they won’t add any value anyway” – and there is now a powerful body of evidence to show that this belief was simply false. We have questioned the power, the politics, the organizational structures and practices that help perpetuate the status quo.

I’d like to reiterate a point I made in a comment about one of Nan’s posts and the ensuing discussion:

One thing I have yet to see – and would like to see – is predominantly Maori and/or diversely multicultural evaluation teams evaluating high-profile general population evaluations. Seriously – why not? I am not at all convinced that all the Western/Pakeha evaluations to date have adequately served the needs of clients, the general community, the taxpayer, and other stakeholders.

There’s a need for more genuine evaluation and outside-the-[Western-]box thinking on many policies and programs in this country and around the globe. Commissioners of evaluations would do well to add this consideration to their thinking, instead of pigeon-holing Maori evaluators as only really having value in ‘their’ space.

Picking up also on Kirimatao’s excellent point

Often building bridges between cultural understanding rests solely on the M?ori shoulders. Translations of everything M?ori is into English. Bi-lingualism aside, if this country was controlled by M?ori and the boot was on the other foot, how and who would build those bridges from the Pakeha community?

Yes! Don’t policies and programs have a responsibility to align with the needs, strengths, and aspirations of Maori? Shouldn’t the bridge building be going in the other direction anyway? Why is it still acceptable to deliver a purely Pakeha perspective on the quality and value of a program or policy that is aimed at a general population that includes significant numbers of Maori?

What if organizations and agencies and governments made a deliberate decision to have some of their KEY mainstream policies, programs, and practices evaluated by a team with a very purposefully Maori (or indigenous, or other significant minority) worldview? What if they commissioned an evaluation with a distinctly feminist perspective? Or, with an immigrant (“new New Zealanders”) perspective? Or, from the perspective of one of our major trading partner economies?

This is a variation on the seldom-used advocate-adversary model of evaluation (Worthen, Sanders, & Fitzpatrick, 1997), where two evaluations are conducted and findings presented from deliberately opposing points of view – as is done by the prosecution and defense in court cases.

I’m not suggesting this should be the model for all evaluations, but I do think there are some long-existing policies, programs, even entire government agencies, that have only ever been subjected to Western “reviews” and evaluations that, not surprisingly, support the perpetuation of current structures and funding models. No-one has ever seriously questioned them – and it’s about time.

Coming back to our context here in Aotearoa New Zealand, my sense is that there is something here for all of us in considering the value of the external eye (and multicultural evaluation teams) BUT some programs and communities need it more than others.

Maori, Pasifika and all the various minority communities in this country have had their beliefs, values, and ways of doing things questioned for years and years already. There’s already an awareness of the Western/Pakeha/’mainstream’ perspective. When conducting evaluations in these communities, there’s less (but perhaps not zero) need for more of that, and more need for an ‘insider’ perspective to address the current imbalance.

But the dominant (Western) culture has had years and years of being the ‘default’, of only being evaluated by their own, of never having their undiscussables discussed. So, when it comes to evaluation, there’s a need for more of an  ‘outsider’ perspective, more outside-the-box thinking, better insights and more new ideas. Food for thought for those who commission evaluations of ‘mainstream’ and ‘general population’ programs and policies.

Some general questions for evaluators (including myself) to ponder:

  • Under what circumstances is the ‘insider-only’ perspective the right choice when putting together an evaluation team?
  • Under what circumstances is it important to deliberately include an ‘outsider’?
  • When should you push harder and ensure there’s a ‘critical mass’ of outsiders?
  • How do you know you’ve got the right mix?

4 comments to Culture – insiders’ and outsiders’ insights – and genuine evaluation

  • Nan Wehipeihana

    Hi Jane
    I’d like to respond to the question you pose: One thing you have yet to see – and would like to see – is predominantly Maori and/or diversely multicultural evaluation teams evaluating high-profile general population evaluations.

    There are a couple of reasons for this: Firstly the number of Maori evaluators is small and we are kept busy just trying to do evaluations which are Maori focused.

    Secondly I believe there is a capability and credibility perception in relation to Maori evaluators – and the minds of commissioners – being able to work in non-Maori contexts and having an extensive methodological tool kit.

    When your evaluation track record has a strong Maori focus I sense that commissioners believe/feel you can’t do evaluation with other populations, in other cultural contexts or settings. (Of course it doesn’t apply in reverse).

    This is partially because many of the methods used in Maori and indigenous communities are primarily qualitative and there is the belief, rightly or wrongly so, that because you’re Maori you won’t have a strong interest or skill level in quantitative methods.

    It is somewhat ironic, that my own entry into evaluation was in large social policy/program evaluations, highly quantitative in nature, as well as utilizing large administrative databases; and, where Maori were not the focus of the evaluation (although there were one of the analysis variables).

    In contrast, my Pakeha (NZer of European ancestry) friend and colleague Kate McKegg talks about her first evaluation work being on Maori focused programmes.

    Go figure!

  • Jane Davidson

    Thanks, Nan, for your insights on this – very interesting.

    It’s been my observation, too, that good talent is getting sidelined in this country and around the world. This is partly (as I mentioned in my original post) because of limited thinking on the part of those who commission evaluation, and partly because there is so much to do in the indigenous/minority space. But it’s also partly because many evaluators make ‘comfort zone’ project choices that IMHO unnecessarily narrow their repertoire and get them pigeon-holed.

    It’s very clear to me that there is so much worthwhile evaluation that can be done in the Maori space and only limited numbers of experienced Maori evaluators to cover it all, so that the choice to do other lines of work could mean that Maori providers and communities end up being subjected to lower quality evaluations. So, I can completely understand that feeling of obligation to step up and do that work. I suppose the only real solution is to encourage more Maori into the profession.

    But there’s also a real downside to taking on projects that use a relatively narrow range of methods and occur in a narrow range of contexts/communities – you don’t build skills, experience, or a credible track record in those other methods and contexts.

    This reminds me of a conversation I had last year with a couple of younger Maori evaluators. I advised them (and would be interested in your view on my advice – they may be listening in too!) to make absolutely sure that the mix of projects they signed up for over the next few years included a decent proportion of ‘mainstream’ projects. Last year we deliberately sought out a mainstream project to work on together – and it went well (as a project, and as a learning experience for them).

    One of the young Maori evaluators was asking me about doing some graduate study at Claremont, preferably online. I suggested the Advanced Certificate program, but also said that (since he was single with no kids and able to travel) he should definitely take 3 months’ leave from work and do a full semester on campus, particularly to go through the evaluation theory and Dale Berger’s fantastic stats courses.

    “Ewww, stats, not sure I want to do that,” was the response. Well, I said, that’s exactly why should you should do it – fill those gaps! Don’t position yourself as yet another Maori evaluator who uses qualitative methods. Be one of the few who has the full repertoire at his fingertips. The opportunities as an evaluator will be far greater. Lesser minds than you have made it through those stats courses, and Dale is absolutely world-class as a teacher and thoughtful methodologist. It’s an opportunity of a lifetime!

    All this brings me back to what we, as a profession, can do to build the capacity that will eventually have predominantly Maori evaluation teams evaluating mainstream policies and programs.

    What I have been doing already is putting thoughts like the above into the minds of anyone who will listen. [That’s not many people, perhaps – LOL!]

    I’d also like to see our national association look at offering (or persuading some provider or other to offer) some serious quantitative and mixed methods professional development opportunities that really are relevant for a wide range of settings including Maori-focused evaluations.

    I’d like to find ways to help more of the up-and-coming Maori evaluators find a really diverse range of evaluation experiences that cover a far wider range of methods and settings than they might otherwise be exposed to.

    And, get a few experienced Maori and Pasifika evaluators into some first and second year university courses as a guest speaker to talk about what a fantastic career choice this is! :)

    Kia ora!

  • Nan Wehipeihana

    Thanks Jane

    A focus on the insider – building Maori evaluation capacity is a topic near and dear to my heart

    Firstly for purely personal (and selfish) reasons. I just want more Maori – so there is more of us, so I have others to work with, and others to contribute to and challenge my thinking…. And of course it would be fantastic to have more Maori from a diverse range fields and discipline in the field evaluation such as economics, environmental scientists etc

    Building Maori evaluation capacity and capability is also about having cultural peers, who can critique your work and thinking from a cultural perspective.

    I was working with two colleagues the other day one Maori and one Pakeha. We were all engaged in some lively discussion and I was challenged by each of them on two different points – to clarify the basis for my statement, to be more explicit about what I was basing my judgment on, and then asked the extent to which that judgment reflected and or aligned to the Maori principles and values that the program was based on. On these two instances, I was held to account, as I should have been. (I of course was doing the same in return).

    So a critical mass of Maori evaluators is needed to achieve these kinds of interrogative cultural conversations.

    What can we, as a profession, do to build the capacity that will eventually have predominantly Maori evaluation teams evaluating mainstream policies and programs.

    All of the ideas you suggested are fantastic.

    Promoting evaluation at university is a good option. But we need to be promoting it when we are doing our work in the community, with clients.

    I’m often asked, what do you have to do to get a job like this? I tell them:

    it helps if you like people, like talking to people and are a really good listener. If you like to observe things and wonder how they work, how things might be improved, then that’s what I do. I talk about going to university and so on and so forth

    So I think we need to:

    1. Build the visibility and promote the attractiveness of evaluation as a career to all around us and especially its value and potential contribution to society.

    2. Broaden the skill base to include quantitative methods.

    I have to say I disliked intently the first stats course I did at university. The teaching (if you could call it that) was so abstract that it lacked a connection to the use, purpose and real life situations. So it all seemed a complete and utter waste of time. I persevered as it was core subject in my undergraduate degree.

    It wasn’t until I was working in a research company alongside people who passionate about statistics, who talked animatedly about numbers and statistics and the stories you could tell through stats, in the same way some people talk about qualitative research, that I got to appreciate the power and potential of the method.

    3. Identify the great teachers. My colleagues in the research company were great teachers (one of them happened to also be a part-time senior lecturer at university, its a pity I hadn’t run across her in my undergraduate years).

    (The emphasis here being on the great teachers and not necessarily the course.)

    My other colleague specialized in complex survey design (yip rcts, and the gamete of experimental designs) and statistical modeling. I thought some of the modeling stuff was very ‘sexy’ in a brain fodder sense.

    When I went on to my next evaluation position, in a large government agency, they had whole divisions of data analysts, and quantities – and again privileged to work with and be mentored by some highly skilled quantitative data analysts and statisticians.

    These learning contexts were incredibly powerful and influential in emphasizing a need to get a handle on quantitative methods.

    4. Seek out/develop opportunities for study and applied learning. Working around people like that was an awesome learning experience and made you realize the complimentary/mixed methods provides a richer and more nuanced story. From my limited knowledge of the US educational system, the notion of internships, seems to be one way they manage post graduate study and training.

    I think my message here is we sell the technical skill (statistics) as opposed to promoting the value, utility and power of the skill. Mastering the former is important for the latter.

    5. Semester based courses. I like the idea of a short intensive focus on any topic – although it suits people differently depending on their life stage. But in principle, I think its great.

    My concern is about the notion of sending someone off as an individual. My sense is that a going off as an individual would work for some – its the typical approach to education. However if we could build capacity using a small or large group process where we bring Maori (and others) together to study then there are some natural as Maori processes that emerge to support learning and achievement.

    The Maori Trade Training scheme is the 60’s is probably by far the single most successful education and training initiative for Maori IMHO. Historically, the Maori boarding schools and the Maori Battalion were other successful ‘educational. So thinking about collective approach to ECB is worth exploring.

    And I must acknowledge Associate Professor Robin Peace from Massey University for her responsiveness to the learning needs of Maori students on the Postgraduate Diploma in Social Sector Evaluation Research – working with this ‘delightful’ group of Maori students to meet their learning needs whilst still meeting the academic requirements of the university. Kia ora Robin.


  • Judy Oakden

    Nan and Jane you are raising really important points about ‘where we come from’ in terms of our training as evaluators. Most children don’t say “when I grow up I want to be an evaluator”. And as our entry to the field of evaluation often occurs later in our careers, we have often come to this space via a circuitous route, and hence have varying ‘toolkits’ of skills. Indeed a wide range of skills are required to work in the field of evaluation, as Skolits, Morrow & Burr (2009) identified in their recent AJE article “Reconceptualizing Evaluator Roles”. One of the benefits of working in teams is that it enables the provision of the appropriate range of skills. Further, many of us come to this space from a social science background, which has a particular culture of its own.

    Let’s focus for a moment on research skills needed in evaluation.
    Thinking about the toolbox of research skills evaluators bring to their work, traditionally in New Zealand, the market research companies have been great training grounds where new graduates learned to apply both qualitative and quantitative research skills across a range of topics. As a result they became well rounded ‘quali-quants’ who had lived through the practical experience and knew the strengths and weaknesses of both approaches (though these approaches may be more separate now). They were also places where staff got to know each other well, and formed strong working relationships. Both M?ori and non-M?ori researchers within market research organisations worked on a wide range of general population studies. And when they moved on to other careers, they took that quali-quant understanding with them.

    Some of our really strong M?ori and Pacifica evaluators included a stint in market research organisations as part of their career development. Having cut their teeth on the general population work, it is my observation that M?ori and Pacific researchers often moved on to focus on their areas of passion, which were generally their own communities.

    Over the last few years, there appear to me to have been fewer graduate positions in the large market research organisations. And so this begs the question, where is the training ground for new graduates to learn to apply both qualitative and quantitative skills in New Zealand these days?

    I’d be interested to know what others think.