A ‘program logic’ for including ‘outsiders’ in evaluation teams

In a couple of earlier posts, we talked about some of the reasons why and the roles in which cultural ‘insiders’ are included in evaluation teams. The posts we are referring to are:

[Just a quick disclaimer: the whole insiders/outsiders conversation really makes no sense if one’s talking about general population studies in a particular country – we’re all ‘insiders’ then (assuming everyone lives in the country in question). Nevertheless, there are some lessons in these discussions for such projects, we think.]

Jane: I would like to pick up the alternative perspective that Paula White raised in her comment, i.e. considering insiders as the ‘default’ and asking when and why outsiders might be included in various roles.

Note that the ‘cultural insiders’ I am talking about here are external evaluators or evaluation team members who are not program staff or part of the specific target recipient population but who come from the same broad cultural and linguistic roots.

Suppose you are an evaluator looking to put together a team of colleagues to bid on an evaluation of a program that primarily or exclusively targets members of your own culture. We can think of culture quite broadly here in a variety of dimensions including ethnicity, gender, sexual orientation, life/health/social history (e.g. immigrants, refugees, former alcoholics, former drug users, experience of being homeless, survivors of abuse), profession or disciplinary roots (e.g. engineers, medical professionals, educators, lawyers, business people, social workers, athletes), etc.

What are the various reasons for including outsiders (people from outside that culture) on your evaluation team? What is the implicit “problem” or “challenge” you would be responding to with that rationale? In what roles would outsiders be involved? How would that influence your evaluation ‘product’ (the services and report delivered)?

Implicit “Problem” or “Challenge” Addressed Outsider Inclusion Rationale Likely Practice Implication
(How ‘Outsiders’ Are Involved)
Likely Evaluation ‘Product’
1. credibility or communication issue – “the client (and other stakeholders) won’t believe the findings if they come from a team of insiders only” OR “we don’t really have the language or approach to presenting findings that is going to really resonate with some of our important [outsider] audiences” to increase credibility, reduce the perception of ‘bias’, and/or make the communication of key concepts more effective
  • outsider(s) placed in visible role(s) to front (or, help front) the evaluation – usually at client and/or media interface rather than the community interface;
  • outsiders may also play a more behind-the-scenes role helping the evaluation team hone the reports so they are credible and in the right ‘language’ for outsiders to understand and believe;
  • alternatively (or as well), a different outsider may be asked to conduct a summative meta-evaluation to provide some assurance that the evaluation and its findings are valid, defensible, and robust.
depending on the role of the outsider and the power/influence he or she has in the evaluation team, the final product may be essentially an insider’s take on the program with an outsider simply delivering or helping hone and deliver the key messages OR a blended perspectives piece

the ‘messenger’ is credible to some audiences but possibly not to others; whether the message itself deserves credibility is a separate issue (see #2)

2. validity challenge – “we won’t get the questions, the criteria, or the evaluative conclusions right if the evaluation team is insiders only” or “we may be missing something important if we are missing the outsider perspective in the work” a fresh eye on the underlying issues, the design, the data, the interpretations; ensuring nothing important is missed
  • outsiders used in conceptualization stage, to help develop the evaluation design or to critique it;
  • outsiders may be asked to walk alongside the project in a ‘formative meta-evaluation’ (or critical friend) role
provided the process is competently run by someone with high levels of evaluation expertise (i.e., the questions, criteria, etc are not based on cultural expertise alone), the product is likely to have high levels of validity and relevance for the context; credibility and utilization are not guaranteed, but are aided by an evaluation with good validity
3. assumption blindness; undiscussability of issues that remain below the surface of the culture to help surface assumptions that may be less visible to cultural insiders; to help us discuss the undiscussables
  • outsiders used in conceptualization stage to help raise questions and surface important assumptions
  • outsiders used in evaluation design phase to help devise ways of addressing important assumptions that should be explored the evaluation report
if done well, an evaluation that ‘digs deep’ and explores important deep assumptions, both those that can really only be understood by insiders, and those that insiders may take for granted but the outsiders on the team may have helped surface for discussion
4. outsiders don’t ‘get’ our cultural context, and frequently botch general population and cross-cultural evaluations because of this lack of knowledge to help build deeper cultural understandings and give outsiders a safe context in which to gain experience in cultures other than their own
  • outsiders work in ‘apprentice’ roles; they are protected and supported by insiders to help ensure they don’t embarrass or hurt themselves or others
evaluation framed and conducted using ‘insider’ process, worldview, and methodologies (the outsider’s influence would be near-nil unless other considerations are also in play)
5. human resources challenge – “we’d like an all-insiders evaluation team, but can’t source all the needed expertise from this pool alone” to ensure the evaluation team has all important forms of expertise to draw on
  • outsiders work in areas specifically related to their expertise (likely to be technical roles, but could also be conceptual and communications roles)
evaluation framed and conducted using essentially an ‘insider’ process, worldview, and methodologies, although the outsider would have some influence depending on the role played
6. size of cultural insider pool is too small; formal training models are too slow and/or ineffective; learning by doing will work better to help build knowledge, skills and hands-on experience in evaluation among cultural insiders so that they can later fill these roles themselves
  • outsiders coach, support and facilitate specific parts of the evaluation that are related to their expertise;
  • the evaluation project as a whole would be wholly or jointly run by an experienced ‘insider’ (who has invited the outsider to take this role, and may also contribute to coaching and development)
evaluation framed and conducted mostly using an ‘insider’ process, and worldview, probably with considerable effort to make judicious use of and adapt ‘mainstream’ or ‘other culture’ methodologies to the cultural context

I’m also keen to emphasize here that a so-called ‘minority’ team could deliver so much more than advocating for the needs of recipients from their own communities. Every different culture, every different language, represents a new way of thinking of the world, a new lens and a new logic to apply to “the way we do things around here”. As Gareth Morgan’s fantastic book Images of Organization points out, the insider has insights and a way of seeing the world that outsiders don’t have. But every way of seeing is also a way of not seeing something else. This is why outsiders can often see things that have become so ‘normal’ to insiders that they never question them. [Just  think of times when you have joined a new organization as an employee, a student, or a member and have been startled by practices and thinking that the insiders take for granted and never discuss or question.]

Patricia: OK – I get it.  So the same arguments used to justify having non-Maori evaluate Maori projects, for example,  could and should be used to justify having Maori evaluate non-Maori projects.

So some reasons that might be put forward for having outsiders:

1.  they can be trusted to be honest (like auditors for Enron, oh, hang on…)

2. they have specialist expertise either in the content area or in evaluation methods

3. they will be seen to be more credible by insiders in the organisation or by outsiders (which might lead to quite different notions of who would be considered credible)

4. they will come with different conceptual frameworks and might notice different things and identify assumptions that are invisible to insiders

I had a recent experience where someone in an organisation said they wanted me to write the report as I was an ” independent external evaluator” and would therefore be more credible.  I pointed out that I did not have content knowledge or experience, nor a budget where I could hire this in,  so it would be hard to argue that my evaluative conclusions would be automatically more credible.  We agreed it was more about #4 (different perspective)  and something about #2 in terms of evaluation expertise about evidence and analysis, and framed a joint insider/outsider approach accordingly. In this case, having a totally external evaluator is less important than having a good mix of content knowledge and evaluation skills, a dash of external perspective, a sprinkle of facilitation skills to articulate tacit knowledge of insiders, and transparency around methods and data so the basis for claims is evident and can be reviewed.

Developing new skills, increasing knowledge, and being able to see one’s own assumptions, are important processes that can and should flow both ways in an insider-outsider team engaged in evaluation.

Comments are closed.