Sharpening the knife

Yesterday's Telegraph reported that DfID has conducted a study into the effectiveness of UK's aid (it is not entirely clear whether this is new or not). Andrew Mitchell seems to be softening up his department for cuts.

The department has further admitted to inaccuracies in many of the self-assessments of its initiatives – suggesting that the true proportion of failing projects could be as high as one-third.

From AidWatch

Hogging the limelight

Becky Hogge has completed an impressive study of how open data initiatives can be encouraged in middle income and developing countries.

The study starts with an interesting section on the unfolding story of how 'free our data' campaigns in the US and UK gained traction. It goes on to examine the factors that might facilitate or obstruct similar developments in other countries.

There are several unexpected findings - such as that the 'middle layer' of engaged and skilled bureaucrats was crucial to government data being opened up.

Becky also draws attention to the important differences between the political economies of developed and developing countries, such as the presence of a 'fourth tier' of donors. She quotes Ory Okolloh, a Kenyan activist:

“I think in many cases NGOs hold just as much data as the Government. And so I’ve always argued that similar efforts [should be directed at NGOs], especially in Africa where NGOs are so involved in health or water, or disease. You’ve seen the World Bank starting to release its data sets now, but they’re sometimes just as bad as Government in terms of holding onto information.”

Number 10 has today issued a press release including the following commitment:

Full information on all DFID international development projects over £500 to be published online from January 2011, including financial information and project documentation.

I'm not yet sure what the implications of this are and whether it will include performance information.

Fed up, not fed back

In lots of ways, going round the feedback loop is getting faster and faster.

• Get Satisfaction gives you real-time feedback from customers on their experience with your products.
• Five Second Test gives you real-time feedback from the public on how they interact with your designs.
• Twitter gives you real-time feedback from your friends, co-workers, customers and stakeholders on all your actions and your thoughts.

But what's happening to the feedback loop in development?

William Easterly rightly says:

We need to be getting more and more creative about systems of decentralized evaluation and more informal kinds of feedback on aid projects, taking into account the real challenges such systems face so that they are accurate and representative of beneficiaries. The internet and cell phones make such systems much more feasible now. Some NGOs are already experimenting, like my pals at globalgiving.com.

Glasspockets, a project run by the US-based Foundation Center, is a project encouraging charitable foundations to be transparent in their giving.

They've got an excellent Grants Database that exerts subtle peer pressure on recalcitrant foundations. And they've constructed a set of performance indicators that international development NGOs and other donors could learn from.

Not yet expired

If you don't already read Owen Barder's blog, you should.

Here's his concise summary of the new aid agenda. Follow the links for the full presentation too.

GOV vs NGO

Last month Martin Brookes, New Philanthropy Capital, argued for charities to be legally required to publish performance data. This is a great idea that I have highlighted on this blog previously.

The problem is there is effectively zero public pressure for this.

The Freedom of Information Act was only introduced after decades of campaigning by individuals and organisations philosophically committed and institutionally designed to keeping check on the state (aside: check out these fascinating first issues of the 'Secrets' magazine from 1984 onwards).

As long as there aren't organisations that represent charity-givers the government won't legislate in this area. What we need is an association that derives its legitimacy from a broad base of support among regular charity-givers. It would be dedicated towards improving the effectiveness of charities and advising members on where to put their money.

(New Philanthropy Capital is an interesting organisation but seem to be more of a consultancy for high-value donors than a broad association for all charity-givers).

In my opinion only a body like the one I've described could effectively mount a campaign for the legislation that Martin rightly calls for.

back-door progress

This blog began in response to the reluctance of 8 British international development NGOs to reveal what projects they fund.

In the last year we have seen the World Bank, DfID and other governmental and multilaterial donors opening up their data and providing detailed breakdowns of their expenditure.

But despite a lot of promises, NGOs have failed to follow suit.

Increasingly I think that NGOs will only release their data if they are made to do so by either governments or individual charity-givers. Without significant external pressure, they simply won't do it.

Now there is place that this pressure might come from: concern about terrorist financing.

In 2007 the Home Office and Treasury conducted a review of charity rules that called for more transparency. But the response from the sector was so ferocious that they took that line no further.

Now fast-forward to March 2009: the European Council has just published the 'Stockholm programme' which is all about anti-terrorism.

One of the measures is:

"promote increased transparency and responsibility for charitable organisations with a view to ensuring compatibility with Special Recommendation (SR) VIII of the Financial Action Task Force (FATF)"

The Financial Action Task Force is an intergovernmental organisation originally designed to stop drug trafficking and money laundering. After 9/11 it was charged with tackling international terrorist financing.

Its ominous-sounding Special Recommendation VIII includes:

"Non-profit organisations are particularly vulnerable, and countries should ensure that they cannot be misused...to conceal or obscure the clandestine diversion of funds intended for legitimate purposes to terrorist organisations"

Eurosceptics have been spooked. Statewatch have released a briefing expressing concern about NGO independence. It seems as if the idea of binding Europe-wide rules has been rejected.

It will be interesting to see if this goes any further. My hunch is that anti-terrorism measures have a life of their own. Despite his Euro-bashing, even David Cameron might adopt Recommendation VIII if there is another attack.

Then a step towards charity transparency would have been introduced by the back door.

Interesting article in the Independent about a Freedom of Information request to DfID about overseas corruption which has been refused out of fear of harming international relations.

At least the new Information Commissioner seems to be more bold in tackling the culture of non-disclosure in public authorities.

"It can't go on forever. At some point, you have to call people's bluff."

Hopefully this case will be overturned on appeal.

Really Simple Silly

DfID has finally published RSS feeds for the project information database.

For some reason they chose to provide feeds for only some of the countries in which DfID works. After 5 minutes tinkering I managed to get the country-specific feeds in Yahoo Pipes.

If you want to access this then go here, type in the country name (default is Indonesia), click 'Run Pipe' and then click 'Get as RSS'. That should let you monitor one particular country without being bombarded by information about others.

I see, oh!

I know that some of you have been following my FOI request to DfID for performance information.

Today I submitted a formal complaint to the Information Commissioner's Office (ICO) about the way it's been handled.

In my view DfID have misapplied the exemptions in the Freedom of Information Act in their attempt to avoid publishing this information (you can read the details in the complaint itself).

The complaint to the ICO has been kindly supported by Andrew Mitchell MP (Shadow Secretary of State for International Development), Karin Christiansen (Publish What You Fund) and Toby Mendel (Centre for Law and Democracy). Many thanks to them all!

Snow white and the 7 dwarfs

Today I've written a short paper outlining 7 proposals for the UK's Department for International Development (DfID).

I've tried to highlight practical ideas which haven't already had much coverage (although you might still have come across some some already, or even thought of them yourself).

All of the proposals stem from a desire to fix the 'broken feedback loop' - between aid beneficiaries - and decision-makers, tax-payers and charity-givers.

After the general election, there's a big opportunity for the new International Development Minister to tackle this problem. They're going to need all the help they can get.

The paper isn't meant to be a comprehensive solution to the aid sector's problems. Incremental improvements are usually more profitable than grand strategies.

If you have any thoughts - supportive or critical - then please do share them.

Here's a crucial exchange from the Committee of Public Accounts' grilling of DfID staff about aid to Malawi:

Chairman: We know that if you put enough money into any scheme you will achieve something, but we are a value for money committee, we are looking at efficiency...

Dr Shafik: ...I think the NAO Report does also say that the DFID programme in Malawi has contributed clearly to poverty reduction...

Chairman: I am not arguing about that. That was precisely why I said what I said. If you spend enough money you are going to achieve something, but what I want to know is how can we as a value for money committee be assured that you are achieving value for money when clearly you are lacking in data about how effective your programmes have been in terms of value for money?

But obtaining better data on aid effectiveness is two-fold:

The easy part is to demand more evaluations and to demand that they are of higher quality.

The difficult part is designing interventions which are evaluable in the first place. That is, funding projects which can be subjected to a clear, fair and unambiguous test of whether they have succeeded or failed.

How much will aid have to change to become evaluable?

Here is a startling account of how organisations resist criticism and change.

The following is an excerpt from Burt Perrin's review of DfID evaluations (p.17), which was commissioned by the Independent Advisory Committee on Development Impact.

A key issue regarding the role of committees and DFID managers concerns their authority with respect to commenting upon and approving deliverables, in particular drafts of final reports. Frequently, management and/or steering groups will provide very detailed comments, sometimes using Track Changes, to ‘suggest’ changes or even rewriting parts of drafts, deleting some critical comments and replacing these with other more positive statements. These ‘comments’ are sometimes strident and very directive.

In my view, such ‘feedback’ provided to almost all the evaluations reviewed represents a clear threat to their independence. In most cases, it appears that evaluation teams were able to deal with requests for inappropriate modifications in a responsible fashion. But this can be difficult, where even the expectation of negative reaction to critical comments can lead to self censorship. In my view, there are two studies, the Private Sector Infrastructure evaluation and the Pakistan country programme evaluation, which clearly crossed the line such that their independence was compromised...

In the former case, very strong demands for change to drafts were made, including indicating what the evaluation ‘needs’ to say. Important considerations were omitted from consideration or changed from evaluation questions to assumptions, including questions raised in the terms of reference and the preliminary literature review (whether or not infrastructure support necessarily contributes to poverty reduction, and the value of a facility approach itself).

In the case of Pakistan, some documentation was withheld from the evaluation team on the grounds of confidentiality. The evaluators also were very clearly and strongly told that they just could not say certain things that the Government of Pakistan at the time might find objectionable, out of concern for ‘sensitivity’, even though this included reference to a published article in a UK newspaper (e.g. ‘We cannot allow the sort of judgement in this sentence in what will, effectively, become a public document … The regime and media here will not make the distinction between us and our consultants.’).

For these and other reasons, it was not possible for the evaluation report to speak of the actual reasons for some of the reported findings. There were also separate internal reports that call into question the transparency and integrity of the formal published evaluation report and the management response.

Rotten Controlling Technocrats

William Easterly has co-published a new book of essays on Randomised Control Trials (RCTs). RCTs are a way of rigorously evaluating interventions in a similar way to medical trials.

In a recent talk he gave a forceful explanation of why economists should be 'thinking small' like this when assessing what works:

What we can say now is that this attempt to find the determinants of growth has failed so decisively, so comprehensively, that anyone today who makes any policy recommendation based on a growth regression has zero credibility.

Instead, he argues, RCTs should be used to measure the effectiveness of small-scale interventions. Those shown to work can then be rolled-out more widely.

Some critics have argued that a positive RCT result in one context doesn't necessarily mean the intervention will be successful elsewhere.

I think this line of argument will lose its power over time, as more and more trials are conducted. Once hundreds or thousands of trials have been done, people will have the information to adapt interventions to new contexts.

I'm in no doubt that RCTs offer an exciting new way to approach aid effectiveness. But there's one remaining concern, which few commentators have written about.

RCTs try and make development evaluation a scientific discipline. I firmly believe it is wrong to present 'science' as the only route to knowledge - especially when considering people's lives.

Let's not forget that 'scientific thinking' was at the heart of many fascist and socialist movements. Science has a self-legitimising power which can be very difficult to keep in check.

How will we cope with RCT results that suggest unpalatable or unethical interventions?

What will we do if a community rejects a project that we know will be good for them?

Just came across Leicester City Council's International Development Strategy.

Need I say more?

When the tracks diverge

New Philanthropy Capital recently held a conference bringing together the world's "nonprofit analysts" - the people who try and measure the effectiveness of charities and other social organisations.

No doubt this is a good thing. And their plans to establish an association of analysts must surely be supported.

Yet at the same time it feels slightly sad. Are the ambitions of NPC really limited only to creating a professional body of people similar to themselves?

Here's an important question: where is the redistribution of power?

In the end people's lives aren't going to be radically improved by better evaluation or analysis. We already know what the answer should be because we engage with it every time we buy something. Clients must be given choice.

Who will make them do it?

An interesting point has been raised by One World Trust in their recent study of development effectiveness initiatives run by NGO partnerships:

"self-regulatory initiatives frequently lack enforcement mechanisms...this can sometimes lead to free riding (signing up to the initiative purely to show the organisation in a good light - without any real action on the part of the NGO to implement)."

There's quite a fashion for self-regulatory NGO bodies: Global Effectiveness Framework for NGOs, G3, INGO Accountability Charter and the BOND Statement of Principles.

Few have a sanctioning mechanism.

How can donors and taxpayers effectively demand for rigorous, enforced regulation?

The Queen announced Labour's draft bill for international development in her speech today.

It contains a legal obligation on the government to spend 0.7% of GNI from 2013 on official development assistance.

In my view laws shouldn't be introduced for the sole reason that they constrict future governments. One generation shouldn't dictate to another like that.

But there's a more important point about the debate over international development, which is overwhelmed by cries for more funding.

In other sectors these voices are at least matched by calls for improved effectiveness in the way funds are spent.