How can transparency efforts adequately report on long-term and hard to measure results?

Transparency for UK international aid

One element of the new British government’s commitment to transparency was announced today. (Audio of announcement here and transcript here)

Full transparency and new independent watchdog will give UK taxpayers value for money in aid

British taxpayers will see exactly how and where overseas aid money is being spent and a new independent watchdog will help ensure this aid is good value for money, International Development Secretary Andrew Mitchell has announced.

In his first major speech as Development Secretary, Mr Mitchell said he had taken the key steps towards creating an independent aid watchdog to ensure value for money.  He also announced a new UKaid Transparency Guarantee to ensure that full information on all DFID’s spending is published on the departmental website.

The information will also be made available to the people who benefit from aid funding: communities and families living in the world’s poorest countries.

These moves come as part of a wider drive to refocus DFID’s work so British taxpayers’ money is spent transparently and on key priority issues such as maternal mortality and disease prevention.

In Mr Mitchell’s speech, delivered at the Royal Society with Oxfam and Policy Exchange, he argued that overseas aid is both morally right and in Britain’s national interest but that taxpayers need to see more evidence their money is being spent well.

Andrew Mitchell said:

“We need a fundamental change of direction – we need to focus on results and outcomes, not just inputs.  Aid spending decisions should be made on the basis of evidence, not guesswork. That is why we have taken the first steps towards creating a new independent aid watchdog.

“The UK Aid Transparency Guarantee will also help to create a million independent aid watchdogs – people around the world who can see where aid money is supposed to be going – and shout if it doesn’t get there.”

Appropriate reporting to citzens (both in donor countries and recipient countries) is an important part of genuine evaluation, and such efforts are to be applauded.

Challenges in reporting on results for accountability

There are some big challenges ahead, however.

There are some types of results that are easier to measure (because they are visible, readily measureable, and evident in the short-term) and some that are harder (because they are less visible, harder to measure, or only evident in the long term).  It is relatively easy to count the number of families with access to mosquito nets or  to clean water.  It is harder to get evidence about who is protected by the nets and how effectively, or how long new water infrastructure lasts.

And there are some situations where it is easier to attribute the results to a specific aid project – either because it is easy to rule out alternative explanations, such as a water project being the cause of improved access to water, or because the project is a discrete, strandardized intervention where an RCT is both appropriate and feasible – and some where it is harder – such as complex, adaptive interventions that vary across communities and over time, or where the results are due to a combination of interventions and other factors.

Much  of development is of the latter kind, especially under the Paris Declaration, which emphasizes aid projects that are developed and implemented in partnership with recipient countries, with a focus on capacity building.

So it will be easy to report on the effectiveness of  a new water supply project in terms of the number of families now connected to water, and hard to report on the effectiveness of a project to develop and maintain an appropriate governance structure for the water supply so that it can be maintained into the future.  The risk is only certain types of projects will be able to generate the types of evidence that are seen as credible, and ultimately these will be the types of projects supported – even though both types are needed.

The risks in ‘payment for results’

How well will the new system address these issues?  We will await the details with bated breath.  And wonder to what extent the new system will enact the Conservative party’s pre-election green paper and video ‘Using effective aid to tackle poverty’.

Part of their new plan is results-based funding where governments will receive aid AFTER they have achieved results. This risks cutting funding to projects where the results are long-term, encouraging setting performance indicators that are easy to achieve, and punishing attempts to address ‘wicked problems’ where the chances of success are low.

The video argues:

“If you need some work done you wouldn’t pay for it all up front. You’d wait and see whether the work was actually done. And then hand over the cash. Why don’t we do the same with aid?

Some money up front and the rest as reward for results. For every extra child that goes to school. For every extra person who gets healthcare. For every extra family with access to clean water.”

1 comment to How can transparency efforts adequately report on long-term and hard to measure results?

  • Robert K. Walker

    In my view, all five DAC criteria should be used for every evaluation. What impacts would be envisioned by such a project? Probably avoidance of water-borne diseases and other health problems; a negative impact to be avoided is depletion of water resources. Every project of this nature should be concerned with sustainability, and thus with institutional development (the second example mentioned; this is the core of effectiveness. I would consider connection to the water main to be an output, not an outcome, so I would classify it under efficiency, not effectiveness. But was the project really relevant anyway? Maybe they already had access to wells, or accessible underground resources would have made this a preferable alternative.

    Bureaucrats tend to like efficiency. A Brazilian state government mandated pupil-teacher ratio as a key indicator for the state university, which was to be required to have a positive delta every year. In other words, more students per professor every year, from here to eternity.