Whose knowledge is it anyway?

I've just finished reading Banerjee's important little book on evaluation in international development, in which he calls for the use of randomized trials as a way of assessing the effectiveness of interventions. His argument is overwhelming: without proper experiments, what do we really ever know?

Half of the book is devoted to replies from 'aid specialists' (bravery for which the author should be applauded), many of whom criticise his proposals for one reason and another. One of their objections holds weight - much aid nowadays is delivered as sector support, which cannot be subjected to randomized testing. Some of our knowledge about the way countries develop will always have to come from sources other than field experiments.

I will add one further, rather esoteric, point. The use of randomized trials sustains a technocratic approach - it does not challenge the positions of officials in donor agencies. Wouldn't it be even better to give the beneficiaries of aid genuine customer choice and control over the service they receive, rather than relying on an analysis of what generally works and then imposing it upon everyone?

Mystery shopping

What do you make of this? On 5th February 2008 I contacted eight large international development charities, all of whom are signatory to the International NGO Accountability Charter.

I asked each of them to publish a detailed breakdown of their expenditure for the last tax year. For each project funded within 2006-07 I wanted to know a) the name of the project b) the location of the project c) a list of funders for the project and d) the total expenditure on the project in the tax year.

None of the organisations has been able to produce me with this list. None were even able to give me just the names of the projects they funded over that year.

Their excuses were as follows:

• ActionAid - "we try to balance the need for transparency with what we think readers will find digestible, useful and informative"
  
• British Red Cross - "administrative processes [must] remain cost-effective, thus maximising the money available for our front line services."
  
• CAFOD - no reply
  
• CARE International UK - no reply
  
• Christian Aid - just a link to the annual report
• Oxfam - see this post and this post
  
• Plan International - no reply
  
• Save the Children - "this is a considerable amount of information for us to prepare and will use valuable resources".
  
• Tearfund - a link to total grants made to top 50 partner organisations.

This is simply unacceptable. Producing a list of funded projects annually certainly is time-consuming, especially when financial data is included as well. But charities which take a large proportion of their income from the public should consider themselves to be under a moral duty to be transparent. Publishing a list of funded activities would obviously improve aid co-ordination as well.

No doubt these thoughts were in the chief executives' minds when they signed the International NGO Accountability Charter. This states that "By signing this Charter we seek to promote further the values of transparency and accountability that we stand for, and commit our INGO to respecting its provisions." The Charter goes on to say: "We are committed to openness, transparency and honesty about our structures, mission, policies and activities. We will communicate actively to stakeholders about ourselves, and make information publicly available."

I call on these organisations to meet their commitments.

Philanthropy without borders

New Philanthropy Capital have just published a new report for private international develoment donors. Part of their advice is that donors ensure their money follows projects that evaluate their outcomes. I found the following excerpt extremely compelling:

Despite a growing number of project evaluations, there is virtually no systemic evaluation of the impact of NGOs, which are facing some of the same incentives as official agencies to emphasise observable effort rather than focus on less observable results. Any information on results that is publicly shared is heavily biased towards success stories. As Roger Riddell notes in his recent book, Does Foreign Aid Really Work?, ‘regrettably, there has always been and remains to this day an almost complete absence of data and information with which to assess the wider and systemic impact of NGO development interventions and activities’...Imagine a world in which an effective marketplace for giving existed to ease this complexity. There was publicly-available evidence on what works and the impact of charitable, as well as private and state, projects. There was also evidence on which giving practices generated the best results. Donors were not frustrated by the lack of information or overwhelming complexities of the system. Rather they were galvanised and inspired to support life-changing initiatives. And they could see the impact that their donations had made. Glimpses of this world exist. Much more is needed. Donors need information on which organisations and projects are tackling different issues in different countries. They also need evidence of the results of these activities. Online information sites and giving exchanges are beginning to bridge the information gap. But they are fragmented and, in most instances, do not show evidence of results.

The future...and beyond!

People will not doubt focus on the recommendation that the US government's development functions be aligned with its other foreign policy objectives, including democratization and defence. But there is much else besides in the new HELP commission report, 'Beyond Assistance'.

In particular there is a strong and welcome focus on improving aid evaluation. Starting from the finding that the "foreign assistance system is broken" the authors repeatedly stresses that the government should "better evaluate projects based on the outcomes they achieve"; "programs must be innovative - which requires our nation accept and learn from failure...Interventions in states must be quantifiable, with numerical goals and timetables, and programs must have specific objectives, which could be measured, evaluated, and re-assessed."

Other notable recommendations include:

• Focussing aid on America's comparative advantages (listed as agriculture, technology, small business development and education) rather than trying to tackle all aspects of poverty
• Moving from an aid system of top-down provision to one in which programs are demand-driven
• Working with private partners and philanthropists
• Concentrating on programs which promote economic growth, rather than social programs

Stone keys

Just wanted to quickly highlight this report from Keystone on online philanthropy markets. The author, David Bonbright, quickly gets to the nub of the argument: the markets are only going to drive change when there is robust and comparable evaluation of project outcomes.

But I'm afraid the idea that these markets will collaborate to produce common evaluation standards is rather fanciful. It seems more likely that one of the more enterprising websites will develop its own evaluative approach and then capture market share. Whatever, happens, let's hope it happens soon.

In response to the report, Perla Ni of GreatNonProfits.org wrote a shameful piece in the Financial Times, arguing against the use of measures of charitable effectiveness. Amazingly, her argument seems to rest on the idea that "presenting potential donors with metrics suppresses donations", as if that were therefore a reason to resist them. Did you ever think such self-serving collusion could be possible?

The world watches

I've spent the last week in China and am becoming convinced its recent history is so impressive it warrants a complete rewrite of development theory.

Apparently, from 1981-2001, 400 million people have been lifted from abject poverty in China. That means that 80% of the world's progress in tackling poverty has, without question, been due to international trade encouraged as a result of economic reform.

International development agencies have essentially played no role in this at all.

Lange time coming

Here's a thought provoking piece on new solutions to global poverty from Mark Lange, a former speech writer to George Bush (don't let that you put you off) in the Christian Science Monitor (ditto).

Like Paul Collier, he writes that we should focus on the billion people at the bottom of the pile; for him fighting relative inequality is a distraction with an unjustifiable opportunity cost.

There are some great ideas in here. The following practical suggestions really stood out for me:

• Linking the pay of managers in donor agencies to country-based outcomes
• Using vouchers as a form of development currency; the poor could exchange them with NGOs or other organisations for basic services
• Instituting an international minimum wage based on the country's PPP price of a basket of subsistence goods.

Parsing the time

Here's an interesting site called UNdemocracy.com which might be useful for people campaigning at an international level or calling for more more representative global governance.

The developers of the site parse transcripts of the UN General Assembly and Security Council to give easy access to speeches and voting patterns.

For example, you can see at a glance that:

• The US was the only country to vote against a Resolution on the Rights of the Child
• A whole load of Pacific island states suspiciously voted against extending the mandate of the 'United Nations Relief and Works Agency for Palestine Refugees in the Near East'
• A resolution on the right to development was adopted despite significant resistance from developed countries.

According to an article in the Guardian, over time the site will develop some of the features associated with MySociety's sites (such as PublicWhip and TheyWorkForYou) which should make it even more powerful.

Inhumin

Perhaps because it is more exposed to public scrutiny, humanitarianism always seems one step ahead of development.

For example, in terms of information management there's loads of software out there for logging and sharing information about humanitarian projects. Huminman, a site maintained by Matthew Slater (see picture), tries to centralise information and resources on this subject. There are reviews of various pieces of software (some relevant to development), a library and the obligatory blog.

Get me out of this relationship

BOND says that "the quality of an NGO’s work is primarily determined by the quality of its relationships with its intended beneficiaries".

I worry about this. From the point of view of the poor, why should the quality of 'the relationship' matter? It is the outcome in terms of:

• health
• education
• wealth
• or some other predefined goal

...which matters. Whether they had a friendly relationship or not is neither here nor there.

BOND's Joni Hillman writes that:

For a sector that works with fundamental issues of power and change, we are surprisingly reluctant to address them closer to home. These debates have been moving up and down the agenda for many years but we now seem to have reached a point where internal and external pressures are combining to demand that 'something must be done'.

That's right. But instead of quality standards, organisational learning, declarations and (urgh!) outcome mapping we need to consider something which will actually work.

The only people who can really pressure government and NGOs to change are tax-payers and individual donors. A system of open, frank feedback from the 'beneficiaries' of aid to those who fund it would finally bring about the genuine accountability that the sector craves.