3ie have published a paper comparing the way 5 countries set up their evaluation departments.

The authors discuss some interesting questions, like: how independent should oversight bodies be from government? what powers should evaluation departments have to enforce their findings?

They look in depth at Mexico, Colombia, Chile, South Africa and China - as well as briefly summarising the situation in Spain, UK and US.

One of their insights is that in the first 5 countries the pressure for evaluation comes largely from donors, but in the West it is the public that demands evidence-based policy.

It's quite right that the institutional mechanisms should vary between countries - there's no requirement for one-size-fits-all.

But it's also worth reflecting on the many problems which have arisen from the tendency of countries which are recipients of aid to respond to their paymasters, rather than their citizens.

It's crucial that evaluation departments establish their own communication channels with taxpayers and show they are independent of donors - as well as the government.

Exciting paper from CGD on a 'collaborative market' for aid.

One particularly interesting idea is that:

Aid institutions are in a kind of political equilibrium balancing the interests of donors and recipients. Reform efforts that simply move away from equilibrium will not be implemented or sustained, and may do more harm than good. To have effective and sustainable reforms we need to try to change the political equilibrium.

A sharp break (a revolution?) is needed.

FOI on the international stage

Here's an important 'Approach paper' from the World Bank, which suggests major changes to the organisation's information disclosure policies:

This paper proposes consideration of a paradigm shift in the Bank’s approach to disclosure, from a policy that spells out what documents the Bank discloses (a “positive list”) to one under which the Bank would disclose any information that is not on a list of exceptions—a policy that would be more consistent with the Bank’s expressed presumption in favor of disclosure.

It's exciting to see an international organisation taking the initiative in favour of transparency, even if it faces short-term risks in doing so.

Mad matrices

Here's my response to the IATI consultation on the proposed standard for publishing aid information:

Dear all

I am writing in response to the IATI's Consultation on Part 1 of the IATI standard.

My comment relates to the 5th category of information: 'Results data (indicators on output, outcome and results)'. In Appendix B both of the fields in this category are listed as being free text. Appendix C indicates that they are links to the results matrix and final results matrix respectively. My reading of this suggests to me that you do not therefore anticipate that the information contained in these matrices will be published as part of the standard - only the references (URLs?) pointing to those matrices will appear.

This seems to me a missed opportunity. I would emphasise the tremendous potential value in being able to compare the results and outcomes of different projects and organisations using the information as it appears in these matrices. This is extremely important information which could be used to identify successful initiatives and those which are failing, both across donors and countries.

I would like to suggest, therefore, that the IATI standard makes an attempt to include these matrices. I presume that the perceived difficulty of making the matrices that are used by different organisations compatible with one another has precluded their inclusion in the consultation document, but this is a shame. In an ideal world, the standard would contain a form of matrix which could be used by all projects and organisations. That way, researchers would be able to compare 'like with like'.

If this is too difficult, a middle ground would be simply to include the project matrices in the standard, even if different organisations use different types. Then at a later date researchers would be able to analyse and compare the matrices without having to format them.

Simply having a link to the matrices is really the worst option, as people will in reality find themselves following links to PDFs (from which data cannot be extracted), files which no longer exist, files in a wide variety of formats and so on.

I hope you find this comment useful and relevant. I look forward to seeing the final standard.

Yours sincerely

Francis Bacon

http://thatsthewaythemoneygoes.blogspot.com/

Interesting article on UK's aid spending in today's Times.

But what changes will the Tories make to aid spending when they come in? They've agreed to keep to the 0.7% promise but I suspect they won't be able to resist the temptation to exert greater political influence over the direction of spending. Perhaps DfID will come under the FCO and more spending in Afghanistan will be marked 'aid'?

Who owns it?

Although its argument is sound, Oxfam America's recent report on the control of aid flows can't help but attract cries of hypocrisy.

One of the 3 themes is 'information'; they piously call for country donors to be transparent about what they fund.

Making US foreign aid transparent is not just important to US taxpayers. It’s fundamental to smart development. Unless recipient countries get comprehensive, accessible, timely, and comparable information from donors, intended recipients can’t hold their governments accountable, and those governments can’t plan, prioritize, or explain to their populations what they are doing.

Why can't Oxfam follow its own advice and publish details of what it funds? At the moment even donors aren't privy to a full breakdown.

Unlocking aid

Aidinfo and the Open Knowledge Foundation have launched a consultation on their International Aid Transparency Initiative (IATI). It aims to establish common standards for recording and publishing aid information.

The central message is that aid information should be open: legally open, technically open and easy to find. This has been the OKFN message on a wide range of issues.

The next and more difficult step will be to agree on what types of information should be recorded for each aid project - and in what format.

Come on GHIs

In June the Lancet, a medical journal, published this free article on the effect of Global Health Initiatives (GHIs) on country health systems.

These GHIs generally target specific diseases such as HIV/AIDS or malaria. Their influence is massive: in 2007 the three biggest (the Global Fund, PEPFAR, and GAVI) donated over $7.5 billion.

Yet this extremely detailed report tears into the GHIs for failing to evaluate their impact on country health systems.

"No robust prospective studies of the effects of GHIs on country health systems have been done."

GHIs are characterised by their focus on scaling up selected services that have proven to be effective in the past and by their use of performance-based assessment to deliver their interventions. It's sadly ironic that they have failed to collect the evidence on the effect they are having on their key partners.

Here's an episode of Radio 4's Iconoclasts on the effectiveness or otherwise of foreign aid.

Kevin Watkins (who works on UNESCO's Global Monitoring Report) says:

"It's absolutely clear that a lot of aid is not maximising its potential, in many cases you have donors who are pursuing the wrong policies, propping up the wrong governments, not having the right focus on stabilisation and growth-orientated policies...that's all a given, we all accept that."

Health warning: Linda Whetstone (International Policy Network) is shrill and misinformed throughout.

Timeless values

Great quote on the International Budget Partnership website:

“To protect the Treasury from being defrauded, let all money be issued openly in front of the whole city, and let copies of the accounts be deposited in various wards.”

(Aristotle, The Politics quoted in Shah, 2007:234).

Anti-comms

The International Policy Network has released a slightly wild-eyed critique of DfID's communications spend.

The thrust of the argument is spot on: 'Communications' shouldn't be used to covertly fund political activities, especially those of dubious benefit for development.

But doesn't this line of attack still miss the target? In 2008-09 DfID's communications budget was £140 million, but this is out of an overall spend of £6.1 billion.

To focus on the communications budget is therefore to distract attention from the more deep-rooted problems with aid transparency and effectiveness, which are of a different order of magnitude.

The DfID response to my latest FOI request says that there are over 3,000 DfID projects which have had their performance assessed and recorded on the database.

They say they won't publish it because it "supports the operational management" of the projects (rather than being "key project content").

I've asked them for a data dump of all the records relating to these hidden fields. Once it's released, it should be possible to do a mash-up with the already-published information in the PPI.

DfID divulge data

DfID have today released the first version of their Project Information Database.

There is a wealth of information there but it is important to ask straight away - what other information does DfID have which they have decided not to publish?

In my most recent FOI request I obtained a full list of the project information stored on DfID's databases.

This includes the following fields:

Performance Review
Is an Annual Review Req
Ann Review Exempt Just.
Annual Review Due Date
Annual Review Prompt Date
Annual Review Authorised
Summary Review Authorised
Is a PCR Required?
PCR Exempt Justification
PCR Due Date
PCR Prompt Date
PCR Deferral?
Deferral Justification
PCR Authorised

Output Scoring
Outputs
Impact Weight %
Output Performanc
Impact Weighted Score
Risk

Project Scoring
Review Date
Total Impact Score
Output Risk
Project Purpose Score
Purpose Justification
Overall Risk score

Method of Scoring
Sources of Information

Scoring Responsibility
Partners Involved

Knowledge Sharing/Lessons
Date
Lesson Category
Notes

Conditionality Review
Disbursement Suspended
Cause
Date Suspended
Consequences

As far as I can make out these fields have not yet been included in the online database. Instead, under 'Aid effectiveness' they have included the following, rather irrelevant, indicators:

Use of partner country public financial management systems
Are partner country budget systems used?
Are partner country financial reporting systems used?
Are partner country audit systems used?
Are all 3 partner country public financial management systems used?

Are partner country procurement systems used?

Use of programme based approaches
Does the contribution fit within a single programme/budget framework?
Is the contribution led by the host country/organisation?
Is there a process for co-ordinating donors?
Are at least 2 partner country systems being used?
Is this a Programme Based Approach?

As mentioned last time, I have another FOI to determine the criteria the department used when selecting which indicators to be published. By coincidence the deadline for replying to that is tomorrow! I wonder if they were concerned I might steal their thunder...

This is an interesting paper on different approaches to impact evaluation, published by the increasingly active 3ie.

It includes contributions from big thinkers such as Robert Chambers and Martin Ravallion - who offer widely different visions of both the philosophy and implementation of evaluation.

Self-regulation

The One World Trust have found that there are at least 50 civil society self-regulatory initiatives on development. They are listed in a new database all about self-regulation, which claims to be the first of its kind.

FOI failures

DfID have responded to my FOI request on their plans to publish aid information from their database.

Disappointingly they won't give out information about Ministerial decisions on the matter, but hopefully my latest FOI attempt will prove more fruitful.

Meanwhile, here's another tantalising glimpse of what's to come:

Publishing Project Information (PPI): As part of our commitment to improving transparency, DFID will publish summary project information on its website from early 2009. Information published will include project descriptions, dates, purposes, locations, sectors, aid effectiveness, summary financial data and whether or not conditions are attached. These items will subsequently be made available as a feed to AiDA and other external stakeholders for use in their analysis and information systems. We are also working on publishing more detailed project documentation later in 2009.

Mashing with DfID

Check out this transcript of the 1st meeting of Open Knowledge Foundations' Working Group on Open Knowledge in Development. (I'm andeggs)

We're going to mash around with DfID data that's due to be released on August 10th. I've got an FOI about it here.

One Word: Radicalism

The Tories have published their ideas for international development, called 'One World Conservatism'.

Its got a very encouraging bent towards transparency and evaluation. Here are some of the highlights:

• We will ensure the impartial and objective analysis of the effectiveness of British aid through an Independent Aid Watchdog.
  
• We will commit up to £10 million per year to the 3ie to commission impact evaluations of programmes supported by multilateral aid projects that DFID gives money to.We will volunteer bilateral DFID projects to be evaluated by the 3ie, and actively encourage all multilateral projects to which DFID gives money to have their programmes evaluated by the 3ie.
  
• We will publish full information about all of DFID’s projects and programmes – including the results of impact evaluations – on its website, and have them translated into local languages. This information will be published in a standardised format so that it can be freely used on third-party websites.
  
• We will also work for greater transparency across the whole aid system. We will require all bodies receiving DFID funds, whether NGOs, multilateral organisations or governments, to move towards greater openness and transparency.

Why are the Conservatives so far ahead of Labour on this one?

Last August, a consultancy called Agulhas had a look at DfID's progress on aid effectiveness. Overall it rated the department highly, although this paragraph is telling:

"the evaluation finds that DFID (like most donors) likes to project a positive image of its own performance, and tends to gloss over imperfections in the aid process. (Development NGOs are also guilty of overselling the impact of aid in their ‘Make Poverty History’ campaign.) In the view of a number of internal and external respondents to the evaluation, DFID is over-sensitive to external criticism, and unwilling to engage the public in a frank debate on the effectiveness of external assistance."

White flag

This paragraph is buried deep in DfID's new white paper on development:

"Part of making international aid more transparent is each donor publishing information of the projects they fund. The UK will make information about the projects we fund available to the public both here in the UK, and in the countries we work, through the launch of a new searchable database on the DFID website." (para 7.21)

Now it's up to everyone else to follow suit.