We often see evaluation justified in terms of improving accountability (and learning). However, it is often not clear who is held to account, for what, to whom, and through what means.
Evaluation is often described in terms of holding people accountable for:
- use of inputs
- activities
- results
I’m starting to wonder how realistic these are.
Evaluation is actually not very effective in ensuring accountability for use of inputs. Auditors have many more relevant skills in tracking where the money has gone and identifying processes that have risks in terms of misuse of funds.
Evaluation can be used to check for compliance of processes, but to do this thoroughly and convincingly would require a level of surveillance that is beyond most evaluations.
Evaluation in terms of accountability for results is often not appropriate when managers do not control the entire causal chain. There is often the predictable problems of encouraging people to set safe targets, and to work to achieve the targets rather than to really solve problems, achieve goals, and respond to emerging problems and opportunities. (Anyone designing a performance management system that includes high stakes reporting on targets should understand the risks of goal displacement).
Some years ago I wrote an entry on accountability for the Encyclopedia of Evaluation, arguing that if evaluation is to be used for accountability, it needs to be for “smart accountability”, where people are required to demonstrate responsible management, including varying plans where appropriate, careful trials of new approaches, and a commitment to learning from both successes and mistakes.
Maybe it is time for evaluation to focus on this type of accountability – and to let other management support functions develop ways of ensuring accountability for inputs and processes.
Related posts:
Jane at Real Evaluation
Patricia at CIRCLE (RMIT)
As an evaluator, my thoughts on accountability have been toward ensuring administrators and managers (and others) value evaluation and are invested/committed toward the conduct of systematic evaluation in their institutions and programs. That is, being inclusive, identifying evaluative questions, using appropriate methods to collect reliable data and analyzing it, and using the findings (being informed) for decision making. Of course they might need professional evaluators to help do this, but they must view evaluation as being central. This goes along with your notion of demonstrating responsible management.
Tom, that’s a very useful description of what it responsible management involves. One of the questions that then arises is how we would know if that was actually happening? Since it will take different forms in different types of organizations, programs and situations, I think it is unlikely that we can identify indicators in advance. Instead, if we were looking for evidence of this type of accountability, as part of evaluating management, I think we would need to ask managers to describe what they do and provide evidence of these types of behaviours. Otherwise we could get the same sorts of goal displacement behavior that can happen at the level of service delivery.
I was recently involved in a multi-sector review of performance-based accountability systems here in the States and all of this rings quite true for me. I wholeheartedly agree with the proposition that many aspects of accountability are best left to the accountants and other non-evaluators. I also agree that linking evaluation to incentives/consequences is fraught with difficulties, and love the idea of focusing accountability on sound management practices, including evidence of a commitment to intelligent trial, error, and learning — “smart accountability,” as Patricia terms it. However, the march toward “less smart” versions of accountability seems likely to continue whether we like it or not — at least on this side of the Pacific. So while we need to outline a new vision for accountability, we also need to find more incremental opportunities to embed “smart” principles in existing accountability frameworks. Here in the States, for instance, the national framework for hospital accreditation includes items on whether institutions have a robust system for continuous quality improvement. Far from perfect — it’s just one of many questions on the accreditation protocol, and it’s notoriously difficult to assess whether real organizational learning is really happening or whether they’re just going through the motions. But it does provide an example of something scalable and feasible, and at least gets some of these ideas into the mainstream conversation.
Thanks for the thought-provoking posts!