The insider/outsider mix: A ‘program logic’ for evaluation team composition

We both want to gratefully acknowledge the time and energy our first celebrity guest blogger, Nan Wehipeihana, has put into leading a truly stimulating week and a half of discussion on various themes related to genuine evaluation. Kia ora, Nan! And many thanks also to the readers who contributed their perspectives and enriched the discussion even further. It’s been an envigorating and thought-provoking experience!

We wanted to share some ideas from an offline dialogue we’ve had, reflecting on what ‘insiders’ can do for an evaluation and how different implicit ‘theories’ about this can have implications for practice … see what you think!

Patricia: I would really like to explore the different rationales for cultural insider evaluation – I think it has important implications for practice.

Jane: I would like to hear some more about what you had in mind on that earlier comment.

Patricia: Thinking about some parallel discussions in international development,  I have noticed some quite different rationales about including local community members ((as a particular type of insider) in an evaluation process .

  1. Involving locals makes sure their values are included in the evaluation” (not usually done)
  2. It makes it easier to get credible data (or any data)  if you have a local involved in field work” (quite a common  rationale – can still be part of quite an extractive approach to data collection or might be part of a broader participatory rationale)
  3. The evaluation  report will have more credibility or impact if we involve local people in the evaluation”
  4. As part of a truly developmental approach, we must share the governance of the evaluation (sometimes expressed by people working in development)

I think these lead to quite different approaches and quite different types of discussion and disagreement.  But at the moment the different rationales are often not clearly distinguished, so one person is trying to talking about governance and respectful engagement and someone else is talking about information gatekeepers.

Jane: Interesting – like a program logic  for the evaluation approach/team composition itself!

For example, any mix of the following rationales could be in play, with possible/likely practice implications as follows …

Insider Inclusion Rationale Likely Practice Implication
(How ‘Insiders’ Are Involved)
helping ensure the evaluation process is appropriate for the context (“process values”)
  • insiders may lead the evaluation
  • insiders take highly visible evaluation process design, facilitator, and community/client contact roles
  • may also be asked to be a ‘critical friend’, advising on the way in which the evaluation is conducted
including insiders’ values in the evaluation content itself (“deep values”)
  • insiders take front-end conceptual roles (determining what ‘good quality programming’ and ‘high value outcomes’ mean in this context)
  • insiders also take back-end evaluative interpretation and sense-making roles
  • insiders may lead the evaluation, but if not, will be senior and present in ‘critical mass’ numbers
making data collection easier
  • insiders hired for fieldwork, translator, and interpreter roles
  • unlikely to hold senior/influential/leadership roles on the project unless other rationales are also in play
enhancing the credibility and/or impact of the evaluation Depending on credibility “in whose eyes” …

  • relatively senior insiders will take key roles in engaging with the community and/or the client/funder;
  • when engaging with the funder, a different kind of ‘insider’ may be used, i.e. may be an ‘insider’ relative to the board/senior management/funder rather than an insider to the recipient community
developmental evaluation, capacity building, and learning
  • insiders heavily involved in setting the evaluation agenda (e.g., questions, focus), interpretation of major findings, and development of recommendations
  • insiders may be involved in data collection, if “seeing things with their own eyes” is an important part of the learning (and/or unlearning) process

Looking at this list, it strikes me that each rationale brings with it a different set of assumptions about what “the problem” or “the challenges” are with the evaluation that lead you to formulate the rationale in the first place. If you see evaluation in a cultural context other than your own largely as a language/cultural communication barrier challenge, you think about how to make data collection and translation work. If you see it as a credibility issue (“they won’t listen to us”), you think about who might be a credible ambassador to smooth the way. If you think of it as a validity challenge, you consider how to get “deep [cultural] values” right into the evaluative criteria and the evaluative interpretation itself. And so on.

Some of these considerations go right to the heart of genuine evaluation; others skirt around the edge of the important issues and engage in politically correct ‘window dressing’ to make the evaluation look right.

There are also some more general issues here about having ‘insiders’ and ‘outsiders’ on an evaluation team – not just cultural insiders/outsiders in the sense of ethnicity, but for example disciplinary insiders, methodological insiders, etc. And of course, an individual can be an insider on some dimensions but an outsider on others.

Patricia: Yes, I wonder if these issues are relevant across different types of insiders – relating to indigeneity, ethnicity, community membership, organizational membership, or disciplinary background.  And we could tease out arguments for having exclusively “outsiders” doing the evaluation in a similar way.

3 comments to The insider/outsider mix: A ‘program logic’ for evaluation team composition

  • Robyn Bailey

    Kia ora k?rua Jane and Patricia

    Great chart! I can see this being a good tool to help facilitate discussion and potentially surfacing what’s going on, sometimes at an unconscious or unspoken level.

    I’d love to see if you could incorporate the problem or challenges assumptions on the left, and then add a further column to the right which makes it clear what ‘type’ of evaluative product each rationale and set of practices would produce. I think this would help spell out the implications even more starkly. However, this maybe a challenge as the reality and possibly ideal is probably a mix, if not all of the practice suggestions (revealing my bias).

    In my experience, as illustrated in an earlier posting from me, unless particularly the top two and bottom one happens, then we end up replicating the worldviews, approaches, values etc of the policy/programme/evaluation commissioning organisation.

    This may not be a problem if the policy/programme/evaluation commissioning organisation is an ‘insider’ to the community that the programme/policy is intended to benefit but (in my view) will always be a problem if an ‘outsider’ .

    Kia ora k?rua for continuing to progress this k?rero!

    Kind regards … Robyn

  • Paula White

    Kia ora koutou katoa
    Ka pai to Jane and Patricia for intitiating this, and to others for contributing too.
    Great blog fullstop!

    I would have found such a table invaluable when I first considered ‘insider-outsider’ debates as a social research student 12 years ago. As I recall, at the time we were offered as a paradigm of practice something like this: if the research question/participants primarily involve topics/people of your own culture, then go for it. If not, best not to go there at all. We felt shortchanged by this at the time – while we ‘got’ the history and implications of inappropriate research practice, we wanted more unpacking of the rationale, and practical implications. So thanks for providing substance on the issue in an evaluation context!

    Reading the table now, I want to ask this – is it also useful to consider within this framework the (counterfactual) rationale for including the ‘outsider’ in evaluation? Or is the role of the ‘outsider’ intended as implicit in the table?

    The reason I ask is because we know there are also benefits, in some cases, to involving outsiders in evaluation research (such as providing a different perspective on an issue from outside looking in, the ability to frame different kinds of questions or consider other possibilities that might not be obvious to an insider from within). And logic aside, should we also allow for the preferences of the client or evaluand community?

    I remember at an early ANZEA hui in Taupo a few years ago, there was a discussion lead by a group of Maori/Pacific/Pakeha evaluators. When describing their process for deciding which team member(s) would work on which project(s), they explained that a key consideration was the preference of the evaluator, and the client community (primarily Maori communities), in terms of their cultural comfort (considering eg. iwi, hapu and whanau whakapapa) depending on the evaluation focus. The result being that sometimes the apparent ‘insider’ was right for an evaluation, while in other cases the outsider perspective was valued more. I realise this example relates to a sub-cultural dimension within a Maori context. Yet I took from this discussion a key message, namely that insider/outsider benefits are something to consider on an evaluation case by case basis.

    This example also reminds me of the distinction between culture and ethnicity. The concepts sometimes get used interchangeably, and this risks us making simplistic assumptions in (genuine) efforts to ensure cultural safety. So who determines who is an insider? How? Do we just assume? Is this good and ‘safe’ practice?

    In summary, to ensure this framework communicates the logic of cultural insider roles in evaluation in a progressive way, perhaps it might include both a nod to the logic of the role of outsider, and, to the very real challenge in practice of establishing ‘who is an insider’?

    Thanks again for the opportunity to comment, Paula

  • Patricia Rogers

    Hi Paula,

    Thanks for your comments. I agree that it would be useful to do a table of the different logics of involving ‘outsiders’ in an evaluation and think that this will be a useful issue to explore next week.
    Patricia