Working across the cultural divide in evaluation: roles, challenges and benefits

In Aotearoa New Zealand, the role of non-Maori evaluators in evaluation with and in M?ori communities, organizations and tribes has been, and continues to be, a contested space. (Maori are the indigenous people of New Zealand, and approximately 15% of the New Zealand population.)

In more recent times there has been an increased emphasis in the commissioning of evaluation of Maori evaluators leading or playing significant and meaningful role in studies that have a primary focus on Maori. This is largely due to the socio-political history of New Zealand, the status given to the Treaty of Waitangi (the founding document of New Zealand) and Maori aspirations for self-determination.

Making space for my friends is how I now come to recognize and value the contribution that non-Maori can make to the Maori evaluation context, within clearly understood parameters. I am more clear about what my bottom line is, as Maori, in terms of non-Maori participation and their roles in this space (see the recent JMDE article I coauthored with Jane Davidson, Kate McKegg, and Vidhya Shanker, entitled: What Does it Take to do Evaluation in Communities and Cultural Contexts Other Than Our Own?)

Given the contested nature of this space for non-M?ori evaluators and for evaluators working across diverse cultural contexts and settings begs the following questions:

  • How do they come to be in this space?
  • What are the roles they play in this space?
  • What are the challenges of working in this space?
  • What are the benefits of working in this space?

8 comments to Working across the cultural divide in evaluation: roles, challenges and benefits

  • Judy Oakden

    Guest or gatecrasher – P?keh? working in M?ori focused evaluation

    Nan has started an interesting discussion on making space for our friends in M?ori focused evaluation. I have taken up the challenge to write in this space, as I am one of those friends. People often ask me how I started working on M?ori evaluation projects as there are no obvious pointers as to why I might be in this space as a P?keh? (a New Zealander of European descent).

    How do I come to be in this space?

    After several years contracting M?ori colleagues to work on my projects with the general population, a friendship emerged and trust was built on both sides based on appropriately holding the M?ori space on a wide range of projects. Then quite simply – a M?ori evaluator invited me to contribute to some projects – baby steps first, writing parts of reports here or there, or working on a proposal together. Then I was introduced to a wider group of M?ori colleagues and over time, they accepted me as I ‘earned my stripes’. But having worked in this space several years, I know I am and will always be a guest in the M?ori evaluation space.

    What are the challenges of working in this space?

    I’d like to introduce the concept of “guesthood” because it seems to provide a useful way for P?keh? to think of working in the M?ori space. What is guesthood? Harvey (2003) talks of how M?ori move strangers “by careful stages into guests” p.126). My understanding of guests generally is that they have good manners; they don’t barge in, take over, impose their views or flaunt their knowledge, they are respectful, and they contribute as appropriate. Guests differ from visitors who I see as being more passive, almost sight-seers.

    Translating this to a M?ori focused evaluation space; guests are part of a wider M?ori-lead team carrying out a range of roles in a manner that are appropriate as a guest. In being a guest you are serious about the responsibilities you take on when you are invited into the space. It’s about building good relationships, listening carefully, suspending your own world view and really trying to better understand what your hosts are telling or showing you – and offering your skills as appropriate. It’s also important there is genuine reciprocity in the relationship. As a group, we talked about this way of working at the recent Anzea conference (Wehipeihana, Oakden, Pipi, & McKegg, 2009) .

    Another way of thinking about guesthood is looking at what it is not. It’s certainly not being a gatecrasher! Jane Davidson and I recently reflected that in the New Zealand evaluation space the two prevailing attitudes that contribute to gatecrashing are; those non-M?ori who come to ‘save M?ori from themselves’, and/or those non-M?ori who believe there are so few M?ori working in the evaluation space that it is ‘their obligation to do the work for M?ori even without M?ori involvement or with very token Maori involvement’. We would contend that both attitudes do not genuinely meet the needs of M?ori, and can lead to projects which are at best unsafe and at worst very damaging to communities (and also to the evaluators).

    What is the place of P?keh? evaluators in M?ori focussed evaluations?

    So what is the role of a P?keh? guest if they are invited into the M?ori focused evaluation space? Firstly, it is not a given that P?keh? will be invited into the team. But on those occasions where P?keh? are invited, at times it’s about adding capacity to a team, at other times it’s providing a counter perspective, or helping to translate M?ori ideas/concepts so they are understandable to P?keh?. Often it’s about contributing to others’ projects. At times though, it’s also about leading projects in the population in general, ensuring a M?ori voice is genuinely and appropriately present throughout the project. Clients find the additional insight this provides really valuable.

    What are the benefits of operating in this space for me personally?

    Guesthood is a challenging relational space, and not always an easy space – feeling uncomfortable and uncertain go with the territory. A high degree of trust in those you work with is required in this space and that trust builds gradually over time. But there are benefits of operating in this space. The work is very diverse and challenging and I like that. You go places you would never go alone and you end up with a broader perspective and better understanding of our wider community than could otherwise be achieved.

    In addition, some of the lessons from the M?ori space about guesthood also apply in other diverse cultural contexts and settings such as the disability space for example. I have learned a lot from my M?ori colleagues about ways of working collectively (rather than as individuals). In addition, they are skilled evaluators who have shared their knowledge generously and it has been a privilege and a pleasure to get to know them better.

    Thanks for raising this topic Nan, I will be most interested to hear what others have to say about the idea of non-M?ori working in M?ori focused evaluations.

    And finally, in the Aoteaora tradition, by way of introduction, kia ora, I’m Judy Oakden. I am a fifth generation P?keh? of Scottish, English and French descent. My family immigrated to New Zealand in the mid 1800’s. On my mother’s side, I’m from the Canterbury region, and on my father’s side from the Rangitikei region from a farming family. I’ve been consulting in research (and more recently evaluation) for over 20 years both in NZ and the UK. For the last 3 years I have run my own evaluation and research consultancy. I live in Wellington with my husband and three teenage children.
    Harvey, G. (2003). Guesthood as ethical decolonising research method. Numen, 50 (2), 125-146.

    Wehipeihana, N., Oakden, J., McKegg, K, and Pipi, K. (2009). ‘M?ori working with non-M?ori’ presented at the Anzea Conference, 13-16 July 2009, Rotorua.

  • Robyn Bailey

    Kia ora korua Nan and Judy

    Thank you for your postings and contributions re working across the cultural divide in evaluation, both in terms of M?ori / non-M?ori and, as you noted in other postings Nan, other insider / outsider contexts. The concepts you have used to frame this discussion – making space for my friends (Nan), invited, guest and guesthood (Judy, Kate and Jane in this and a range of other postings and conversations) – resonant. Implicit in these words (for me) is ‘genuine’ stuff about evolving relationships, respect and trust (as you both described).

    My working space to date:

    I am posting from a different space from Judy. I have yet to be ‘invited’ by M?ori evaluators to work in the M?ori evaluation context. My experience is mostly with ‘general population’ evaluations (that is, evaluations of projects and programmes intending to benefit the whole population). I have undertaken an evaluation 3-5 years ago, of a programme specifically for M?ori and Pasifika peoples in which I was invited by a (non-M?ori) government organisation and worked in a co-leader role with another P?keh? evaluator. So I’ll be commenting from these experience bases.

    What are the roles I have, and would play (and not play), in this space?

    With regard to the second scenario above – evaluating a programme specifically for M?ori and Pasifika peoples, my understanding is we were invited by the government organisation given that the small number of M?ori and Pasifkia evaluators available at that time were already committed, and we were on-the-spot with an existing relationship and some understanding of the organisation and related policies and programmes. We invited and worked with senior, experienced M?ori and Pasifika evaluation advisors who provided review of the planned evaluation approach, participated in the data analysis workshops, peer reviewed the report and provided contacts for M?ori interviewers. We also worked in paired interview teams – one of the project leaders working with a M?ori or Pasifika interviewer, some of whom participated in workshopping the data analysis and reviewing the report.

    However, while we as P?keh? evaluators were conscious of and attempted to be respectful and mindful (and how this impacted) of not being M?ori or Pasifika, my experience is we still fundamentally carried out a ‘P?keh? evaluation’, that is the approach, methodology, methods, power relations, understanding – interpretation – analysis of the information, etc ultimately reflected and replicated a P?keh? (western) world view.

    My decision to take on this evaluation project and leadership role at that time was pragmatic. This experience led my co-leader, and other P?keh? evaluators for other reasons, to reflect on and engage in discussing our role in evaluations of other-than-own cultures / groups. I’m also currently engaged in exciting discussions with a M?ori-Pasifika-P?keh? group about evaluator competencies in Aotearoa NZ. These conversations have sparked long-held but dormant values and beliefs which, when applied to the above scenario, means I would not take such a leadership role in evaluations of other-than-my own culture / group, unless specifically invited by people of that culture / group and had the support / endorsement of evaluators from that culture / group.

    I am also currently rethinking about how I can work on general population evaluations in a way that genuinely shifts the power balance and imposition of western worldviews / framing which permeates all aspects of the evaluation (and often the policy / programme).

    I would also like to contribute to Nan’s other questions but have run out of time today. Another question I’m currently reflecting on, posed to me by a M?ori colleague Kataraina Pipi, is: Who is my accountability too? Where does my accountability lie (as a P?keh? evaluator)?

    Similarly to Judy … Tena koutou katoa (Hello to all). I’m Robyn Bailey, a fifth generation P?keh? of primarily English descent. My families immigrated to Aotearoa NZ around 1840. My mother’s side is from the Kaipara Harbour (farming) and my father’s from Dunedin (teaching). I’ve been working in evaluation since 1986, and as independent since 1998. I live on the lovely Kapiti Coast. Hei konei r? (Goodbye).

  • Laurie Porima

    Kia ora e te tuahine Nan, nau nei i werohia i a t?tou, etahi o ngai taua, e mahi ana i tenei tumomo mahi ko te arotakenga, hei whakautu ai. Nau hoki i tohutohungia i a t?tou, kia mahi tahi, kia mahi pai hoki ki o t?tou whanaunga o waho ra, heoi ano e mihi atu. I agree with Judy this is an interesting discussion and thanks also Judy for sharing your experiences.

    How do they come to be in this space?
    The evaluation world here in New Zealand relative to other countries is very small and when our government agencies’, who are in the main, the primary commissioners of evaluations in NZ, ‘put out’ tenders for evaluation projects involving Maori, this then provides an attractive opportunity (professionally, academically and financially) in which to respond to the tender.

    In addition, some of the projects are challenging and ground-breaking and therefore requires innovative approaches, further enhancing the attraction element for working in this space! Getting into the space is the easy part, it’s what you do, once you are in there that provides the challenges and counts the most!

    A question that comes to hand is why would an indigenous evaluator or anybody for that matter want to even get into this space? The answer was alluded to in earlier pieces and that is because one chooses either to join or to invite. We may at a later stage explore some of the motivators for making these choices.

    What are the roles they play in this space?
    I do concur with the article cited above with regards to people’s roles as it provides some good examples of what some of those roles can involve. But I think if we expand slightly on what Judy has mentioned in terms of the concept of guest and take it into the M?ori realm, one becomes what is typically known within M?ori culture as the manuhiri or visitor. These positions come with much responsibility both for the guest and the host. For some people, even M?ori, they have to be taught what the roles of a host (tangata whenua) and guest (manuhiri) entail. Again, perhaps in another discussion we can expand on this thread however the important point to remember is that these roles must be made explicit within the evaluation team and not assumed – this is critical to the good functioning of the team.

    What are the challenges of working in this space?
    I think the challenges of working in this space relate to both individual and team dynamics. For the individual it could mean their level of experience working in the field of evaluation and how they have come to be a part of the team. Who takes the lead roles and what does lead actually mean!? What has led them to being a part of this team?

    I am sure, all of us who have read and contributed to blogs such as this, have started out as the junior in the team and it is not until one has increased their level of knowledge and confidence that they feel they are able and even “qualified” to contribute something meaningful to a discussion. So the challenge for the individual is what level of confidence do they feel they have that they as part of this team will, as the medical profession espouses “do no harm,” (we won’t mention the ‘Unfortunate Experiment’) and that their presence within the team will enhance the outcome.

    For the team and particularly the leaders of the team, what responsibilities do they have for looking after their team members whether they are juniors or seniors, indigenous or not? How do they provide a safe working environment not only for the communities in which they will work in but also for their own team members? To not make this an explicit and deliberate part of the team preparation for the evaluation provides much indication of whether there is a genuine presence of mind to work respectfully within cross cultural settings.

    Like Nan, I too am clear about the parameters I have for working with non-M?ori, and these have been developed as my experience and confidence within the field has grown – as I am sure they have for working with me! Making this explicit at the outset and operating within the principles of what we as M?ori call “tikanga” does lead to some very good and long lasting relationships with colleagues

    I’ll end here for now, so in summary, it is about the individual and who they choose to work with, the environment they want to work in, and having that level of understanding and confidence to be able to contribute effectively. Whilst for the leaders in the team, to provide an environment that allows others to contribute meaningfully and purposefully.

    It can be really easy and enjoyable however some people do make it awfully hard.

  • Kia ora koutou Nan, Judy, Robyn, Laurie, and the many others following along. Great discussion!

    I must say I’m fairly new to collaborating with others on evaluation projects, either Maori or Pakeha. When I came back home to NZ in 2004 (after 12 years overseas) I knew hardly anyone in the evaluation scene here and was just starting a young family. So, I stuck to small-scale manageable projects that I could do solo and juggle to spend time with my little ones.

    I had always steered clear of projects that were clearly in the Maori or Pasifika ‘space’ – primarily because I was working solo and I didn’t feel I had the capabilities to deliver effectively on such projects (I try not to set myself up for failure!), but also because I was aware there were already several excellent Maori and Pasifika evaluators around, who I assumed were taking good care of that line of work. Until relatively recently I didn’t really know any of them well enough that either side would suggest working together (I would have felt cheeky for even suggesting it). That’s slowly changing now that I’ve made it through the first 2 years with twins (parents of multiples will understand this white knuckle ride!) and am starting to find time to connect with the profession again.

    So, this discussion is timely for me as I consider how bicultural and multicultural evaluation teams can work effectively and deliver genuine value for clients and communities. Some of the roles/projects I’ve ended up in so far are …

    1. A couple of younger Maori/Pasifika evaluators approached me to ask whether they could work with me on something that would help broaden their experience and serve as a professional development experience with them. We looked around for a project, and eventually won a bid for a general population project that served significant numbers of Maori and Pasifika. I involved them in the client engagement and conceptualization phases, one was heavily involved in the fieldwork and the other more in the write-up. I took the lead on the project and was the person responsible for the final deliverables. They both got a lot out of participating and observing how the project evolved from start to finish, and one has now gone out on her own as an independent evaluation consultant; the other is planning to make the transition in the future. Our next project together is likely to be with them taking the lead and me advising/coaching.

    2. A Maori evaluator leading her first small-scale evaluation project asked me for support/coaching on the evaluative sensemaking and write-up part of the project. She had already completed one of my evaluation methodology workshops, so knew the fundamentals (i.e. knew what my approach would be, and was on the same page about it); I made some suggestions about the report structure and content and helped provide some ideas for making sense of the data.

    3. I’m just about to launch into a project with one Maori and one Pasifika evaluator. It’s one of those projects that’s not clearly in one particular cultural space, but that obviously needs some significant input from people with cultural expertise/insights, evaluative minds, and the perspective that comes from real life experience. A very large chunk of the project requires the evaluative interpretation of descriptive data that have already been collected, which is an area of specialization for me. It’s a VERY small budget. Ideally, we’d all be working on all of it (and I wish we could), but budget won’t allow. So, the plan is that I’ll do most of the work, but will draw on the expertise of my colleagues at the conceptualization and sensemaking stages, where it’s most crucial.

    Project #3 is the one that’s really uncharted territory for me, and a challenge to ensure that the relevant cultural expertise gets its rightful [prominent] place and isn’t sidelined, or an add-on. This is incredibly hard to achieve on a small budget that limits the involvement of those collaborating on the project. There’s a lot of trust involved, and a lot of trust at stake. Given that we are “satisficing” on the roles here, my primary strategy was to ensure that I worked with competent, experienced people (not $15/hr research assistants!) who I could trust to call me on any assumptions, errors, and omissions. My job is to create an environment that enables that, and to do some serious reflection together about how it’s working.

    One thing I have yet to see – and would like to see – is predominantly Maori and/or diversely multicultural evaluation teams evaluating high-profile general population evaluations. Why not? I am not at all convinced that all the Western/Pakeha evaluations to date have adequately served the needs of clients, the general community, the taxpayer, and other stakeholders.

    There’s a need for more genuine evaluation and outside-the-[Western-]box thinking on many policies and programs in this country and around the globe. Commissioners would do well to add this consideration to their thinking, instead of pigeon-holing Maori evaluators as only really having value in ‘their’ space. JMHO …

    And finally … Kia ora! I’m Jane Davidson, a 6th generation P?keh? (NZer of European descent) descended on my father’s side from Scottish railway workers (from Muirkirk, near Glasgow) who settled in Oamaru (South Canterbury) and on my mother’s side from English farmers (from Sussex) who settled in the Akatarawa Valley (Upper Hutt). I live in Tamaki Makaurau (Auckland), the lands of the Ng?ti Wh?tua, with my husband, Susumu, and our three bilingual/bicultural poppets – Kiri (5), Ema and Mariko (2-year-old twins). I spent 4 years living and working in T?ky? (and speak reasonable Japanese), and 8 years in the States (Southern California and Michigan).

  • Laurie Porima

    Kia ora Jane,
    I think you have captured the essence of what it may be like for some people wanting to enter into the field of evaluation and being able to work in an environment where skill sets have been acknowledged yet there is a genuine desire to grow capacity. It is that willingness to share that makes working in this”space” really enjoyable. The onus is on giving back and it is fact that the more you give the more you will receive – this is the true essence of koha (reciprocity). Building evaluation capacity here in NZ I think is critical to the ensured growth of the field. With that growth will come the abilities to critically think and theorise about our profession.

  • Jane Davidson

    Kia ora Laurie,

    Yes, I do think there are interesting parallels between those sitting on the edge of the evaluation universe wondering whether to dip their toes in the water, and those in evaluation wondering when and how to dip their toes in the bicultural and multicultural evaluation team space. Will I get burnt? Will I get bitten? Will I mess it up for someone else? If my old swimming skills aren’t likely to be much use, where can I get some floaties, and who will help me if I find myself out of my depth? Should I only go with very strong swimmers? Is it worth the risk of changing pools if I am doing fine where I am?

    I, too, have found the evaluation profession generally to be incredibly generous about supporting newcomers and helping them find what they need to grow their skills and build a rewarding practice. Even at the big AEA conferences, where you’ve got several thousand people in one space, I was just amazed that the Big Wheels really are interested in talking to the newcomers, encouraging and supporting them. Conferences in other professions have been a stark contrast – you would never get close to any of the Big Wigs, who thought they knew it all already. Perhaps it’s an advantage of being in a young discipline where there still is much to learn no matter how experienced you are. I’ve certainly been very fortunate to find myself here. :)


  • Patricia Rogers

    Thanks for these reflections on “working across the cultural divide” in evaluation. Discussions about the appropriateness and value of insiders to a community or a culture, or outsiders, or a combined team, undertaking an evaluation are very important. The discussion to date certainly has made me think about processes for development projects and the extent to which they address these issues.

    I wonder if we might explore the implications of this in terms of:
    * the appropriateness of the evaluative criteria used
    * the quality of the information gathered and its interpretation
    * the uptake and use of the information and
    * the process impacts of being involved in the evaluation.

    My reading of some of the discussions about this issue is that these different rationales lead to quite different practices. And I am sure there are other rationales that would be useful to make explicit. I’d be interested to hear what others think about this.

  • Kia ora koutou,

    Yes a valuable contribution to a debate that has been around a long time. My first paper to an evaluation conference was about this issue – 1995 I think.

    I come at this from a rather different angle. I’m primarily interested in how people learn from a cognitive and social point of view rather than purely a cultural point of view. Yes of course the three are closely linked, but my understanding of the research rather than the rhetoric is that culture mediates cognition and social learning processes rather than determines them. Within this cognitive and social learning space I’m interested in the role of dialectic; basically how we handle ideas and views that challenge our existing norms and world views. It’s disputed territory but many claim that “deep” learning emanates from dialectic not data or even discussion.

    It is here where cultural mediation becomes interesting. In a Maori context the notion of manuhiri as “guest” is for me an intriguing one, since on the one hand manuhiri are expected to be relatively respectful of local cultural contexts …. but how many of us have sat at a powhiri listening to manuhiri and tangata whenua rip into each other in very dialectical ways. Same within my English context there are cultural contexts where dialectic is acceptable (in the pub, in a “debate”) and some where they are not.

    Given that evaluation is by its very nature to do with values, and values are frequently disputed territory, I’d argue that if evaluation is to be more than a superficial learning process we are going to have to develop ways of handling dialectic whatever the cultural context.