Credibility and independence in evaluation – an alternative view

Standard ‘mainstream’ belief is that one element of credibility as an evaluator comes from one’s independence and the perceived objectivity (lack of bias) that derives from that.

Here in Aotearoa New Zealand, we often find the opposite is the case: one’s credibility with the community and the provider – and with funder and external audiences who understand the implications for validity –  is enhanced where there is a long-term history of trust and competent evaluation work that fairly represents what is happening, that “speaks truth to power” when necessary, and that doesn’t (as Nan Wehipeihana puts it) “trample on the mana of the people”.

Independence (having no connection or history with the community) is actually bad for credibility and therefore for validity. If we don’t know you and trust you, if you don’t have any connection with us, then why would we share insights with you?

The following interview snippet from leading M?ori (indigenous) researcher and evaluator Dr. Fiona Cram explains how, in Kaupapa M?ori research, relationships aren’t just the means to get one-off data collection done; productive relationships are long-term, ongoing, and the very foundation on which good research and evaluation is built. [Subscribers on the email feed will need to view the video on the Genuine Evaluation website.]

[kml_flashembed publishmethod=”static” fversion=”8.0.0″ movie=”http://www.rangahau.co.nz//assets/video/fiona/methodology_fiona.swf” width=”400″ height=”300″ targetclass=”flashmovie” play=”false” loop=”false”]Get Adobe Flash player

[/kml_flashembed]

Glossary:

Kaupapa M?ori research” – research based on a M?ori worldview, conducted by, for and with M?ori (click link for more detailed explanation of principles)

“the kaupapa of the center” = the way in which the [research] center works (includes values, approach, philosophy)

tino rangatiratanga” = sovereignty, autonomy, control, self-determination and independence; M?ori being in control of their own culture, aspirations and destiny

[For more great interview snippets and insights from some of our top M?ori evaluators and researchers (both academics and practitioners), check out the excellent Rangahau website.]

It’s important to note here that, in an evaluation context, we are talking primarily (as I underrstand it) about connectedness to the community and people affected by the program (impactees), rather than necessarily to the providers of programs and services. However, in some cases the providers are  a community-based organization, in which case they may well be friends or relatives of the impactees.

One of the ways to build credibility and trust over time is to deliver any feedback in mana-enhancing ways, i.e. in ways that respect and enhance the status/influence/standing of both program/service recipients (and their communities) and the individuals managing and running the program or service.

What I take this to mean for genuine evaluation is:

  • criticize the program or the practices if necessary, not the people personally
  • criticism of the community has absolutely no place in evaluation or research – we need to understand strengths and needs that the program or service can and should be building on or addressing, not use evaluation to take deficit model ‘pot shots’ at communities that have often had the economic and social rug pulled out from under them for generation after generation
  • acknowledge the strengths, enablers, and pockets of success or promising practices – in the community as well as in the program
  • avoid causing people or communities public embarrassment or loss of face – speak truth to power; don’t trample on the relatively powerless
  • provide (or, if possible and appropriate, facilitate the creation of) insights and ideas that people can really use to change the program and the community for the better

Although many of these insights come out of the indigenous (and more specifically Kaupapa M?ori) evaluation space, I am always struck by how these ideas also have legs in general population evaluation work in this country. Perhaps this is because we have a small population. If you trample on the mana of the people in one evaluation project, you can be absolutely sure this news will have reached the next potential client or community well before you bid for or begin your next piece of work! We call this the “bush telegraph” here; friends in Hawai`i tell me the “coconut wireless” is the correct term there. With social and professional networking mushrooming all around the planet, it seems only a matter of time before the entire planet will seem like one very small village, and relationships (direct or through others) will be critically important for credibility wherever you are …

See also the following related posts:

1 comment to Credibility and independence in evaluation – an alternative view

  • Caroline Heider

    I really like this posting — many thanks — as it illustrates an example where reflective or evaluative thinking is part of the culture of people and communities who want to understand better what happened.

    The next question is: how does one develop such a culture when it is not intrinsic or when incentives exist to share information only about success/the positive (real or the “nicer message”) and fear to speak about things that may not have gone right?

    Or when critique is automatically exercised or perceived as personal rather than a reflection on a situation, events, or things?