Evaluation of DFID’s Support for Civil Society Organisations through Programme Partnership Agreements

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We wish to undertake an evaluation of the funding of civil society organisations (CSOs) that takes place through DFID’s Programme Partnership Agreements (PPAs).¹ DFID currently has 41 PPAs, comprising a total commitment of £360 million for the period 2011-14. In operation, the PPAs are intended to improve recipients’ accountability for and the impact of their work. We wish to see whether this is the case.

1.3 KPMG is contracted to provide due diligence services to DFID for the PPAs. As a result, KPMG will have no involvement in this review,² which will be led by Agulhas Applied Knowledge, one of the other organisations in our consortium.

2. Background

2.1. CSOs, such as Oxfam, Save the Children and CAFOD, are seen by DFID as key partners for the delivery of UK aid. DFID reports that, in 2011-12, it channelled £880 million through these and similar organisations, representing 11.8% of its total programme expenditure that year. £327 million (37%) of this support was provided by country offices, £367 million (42%) by DFID centrally and the remainder (an estimated £186 million) via the UK’s contributions to multilateral organisations. £336 million of the total was provided to UK-based CSOs

2.2 DFID has five objectives for its work with civil society:

• ‘deliver goods and services effectively and efficiently;
• empower citizens in developing countries to do things for themselves;
• enable civil society to influence, advocate and hold to account national, regional and international institutions including improving aid effectiveness;
• build and maintain capacity and space for an active civil society; and
• build support for development in the UK’.³

2.3 To do this, DFID’s central departments use several mechanisms, including:

• The Girls Education Challenge Fund: begun in 2012, this fund is managed by PwC. It includes three different ‘windows’, all of which have now closed for new applicants;
• The Global Poverty Action Fund: this fund is described as ‘a demand-led fund supporting projects focused on poverty reduction and pursuit of the Millennium Development Goals’.⁴ The current request for bids notes that annual funding is over £40 million per year. It has two ‘windows’ and is administered by Triple Line Consulting;
• UK Aid Match: launched in June 2011, this is ‘a demand-led fund which will match public donations to appeals for development activities focused on poverty reduction in developing countries’.⁵ DFID matches funds raised from the UK public for individual appeals up to a total of £5 million. Its current budget is £30 million per year;
• The Governance Transparency Fund: now closed to new applicants, this is a £130 million ‘one off’ fund to encourage citizens to hold their governments to account;⁶
• The Civil Society Challenge Fund: now closed for new applicants, this will complete funding in 2015; and
• A Development Awareness Fund: again, this is now closed for new applicants.⁷

2.4 DFID’s funding commitments under these six mechanisms totalled £731 million over the period 2000-12⁸ and we will examine this area in future. As mentioned above, it also funds CSOs through its country office budgets and regional programmes and through multilateral organisations.

2.5 In addition to the funds set out above, during 2010, DFID requested CSOs to apply for medium-term funding through its PPA mechanism. In its publicity for the PPAs, DFID noted that that they were ‘aimed at CSOs with a global reach and leaders in their field who can add value to DFID’s portfolio, support realisation of its objectives, achieve real results in terms of poverty reduction and provide good Value for Money (VfM) – demonstrated through a competitive selection process’.⁹

2.6 Two DFID departments are involved in managing PPA grants: the Conflict, Humanitarian and Security Department (CHASE) and the Civil Society Department (CSD). CSD is the lead department and provides the majority of management for the PPAs.

2.7 DFID reports that the selection process for PPAs was competitive. Over 400 organisations applied, with 109 subsequently requested to provide full proposals. These 109 were subjected to an assessment process, including due diligence (outsourced to KPMG).

2.8 Proposals were assessed against the following criteria:

• the ‘strategic fit’ with the UK Government’s and DFID’s values and priorities, specifically the Millennium Development Goals, gender equality and climate change;
• the ‘vision and impact’, including whether aid is targeted to countries in most need (including fragile states) and to poor and vulnerable groups (as well as evidence of specialist contributions to specific issues);
• evidence of the organisations’ ability to ‘add value’ in their work and to the achievement of DFID’s objectives;
• their ‘transparency and accountability’, particularly to stakeholders;
• the ability of the organisations to ‘deliver results’, including the use of management information systems and the direct impacts on the lives of poor people;
• the ‘value for money’ of the organisations, focusing on economy, efficiency and effectiveness;
• the organisations’ ‘partnership behaviour’, such as working with third parties for the delivery of services; and
• arrangements for ‘monitoring, evaluation and learning’.

2.9 These selection criteria are broad and allow considerable variation in the types of organisation funded. Figure 1 on page 4 summarises the 41 successful PPAs ranked by size.

Figure 1: Current Programme Partnership Agreements

2.10 13 of the 41 PPAs are above £10 million in value, 14 are between £5 million and £10 million and 14 are budgeted to receive less than £5 million.

2.11 Performance criteria for each PPA were negotiated with the funded CSO and set out in a logical framework as part of a project plan. Coffey International, a third-party consulting firm, was given the task of providing advice and guidance to the CSOs to help finalise logical frameworks as part of a wider role in overseeing the evaluation of the PPA’s performance.

3. Purpose of the Evaluation

3.1 To assess the effectiveness and impact of DFID’s Programme Partnership Agreements with CSOs.

4. Relationship to other evaluations and studies

4.1 DFID has appointed Coffey International to develop and oversee an evaluation strategy. The terms of reference (TOR) for this role include the following:

• ‘Providing advice and guidance to PPA partners to ensure evaluability of their agreements. This will include:
  • a. Working with DFID to finalise and agree all logical frameworks
  • b. Developing proposals for establishing a PPA portfolio baseline report and finalise a PPA baseline report
  • c. Ensuring adequate baseline data against which progress can be assessed
  • d. Proposing evaluation methodologies ensuring that they are proportionate to the level and type of funding
• Developing a meta logframe (probably a nested logframe) for the PPA mechanism to synthesise the results (logical) frameworks of PPA partners
• Developing and implementing the Monitoring and Evaluation Strategy for the life span of the PPAs (3 years), taking into account the need to balance value for money with a comprehensive approach to results management
• Developing the approach to mid term evaluations and developing generic TORs for use by PPA partners and quality assuring final amended ToRs as appropriate
• Coordinating the mid term evaluations of all PPAs and developing a PPA mid term report on the results of the fund at partner and portfolio level
• Developing the approach to final evaluations and developing generic TORs for use by PPA partners
• Quality assuring final evaluations against evaluation best practice standards
• Coordinating the final evaluation of the PPAs and producing a final report of the results of the fund at the portfolio level.’¹⁰

4.2 Funded organisations report annually on their performance against the details of their agreement with DFID. This may include performance criteria and requirements to improve operational and financial management systems. In addition, organisations have been required to appoint independent evaluators to undertake mid-term evaluations of funding (termed ‘Independent Performance Reviews’). The collation of these individual mid-term evaluations has recently been completed. PPA organisations are also required to appoint independent evaluators to undertake end-of-grant evaluations.

5. Analytical approach

5.1 We will draw from an overall assessment and six case studies to address the following:

• a brief assessment of the decision-making in the selection of the funded organisations;
• the management of the agreements;
• the practical impact of the agreements on the capacity and effectiveness of the organisations, with particular reference to the quality of DFID’s mechanisms for assuring performance, notably the evaluation strategy; and
• the performance of the organisations against agreed targets.

5.2 While not seeking to undertake detailed evaluations of each organisation, we will thus test the quality and impact of DFID’s own commissioning, management and utilisation performance assessments.

5.3 We will not consider all PPAs in detail. Rather we will:

a) sample six PPAs (see 7.2 below) so that they represent:

• i. two with a value larger than £10 million, two with a value between £5 and £10 million and two less than £5 million in value;
• ii. different sources of funding from within DFID (i.e. from CHASE and CSD);
• iii. coalitions of organisations; and
• iv. organisations with different priorities and types of main activity.

b) assess the quality and impact of the engagement between the organisations and DFID;

c) assess the quality of the evaluation strategy and independent evaluations that have taken place to date. We will specifically seek to assess the quality of the organisations’ own reports, baselines and the individual independent evaluations; and

d) investigate the systems in place for the management and oversight of the PPA arrangements overall, including specifically selection, oversight and financial management.

5.4 We recognise that the PPAs have only been operational for less than two years. We also understand that there may be limits to how far we can assess delivery level impacts given the nature of PPA funding. So, although our principal focus will be on how effectively DFID has used PPAs to achieve its objectives, we will also explore what evidence is available of development impact.

6. Indicative evaluation framework

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. The questions outlined below are based on those questions in our standard evaluation framework which are of particular interest in this review. The full, finalised list of questions that we will consider in this review will be set out in the inception report.

6.2 Objectives

6.2.1 Do the PPAs have clear, relevant and realistic objectives that focus on the desired impact?

6.2.2 Is there a clear and convincing plan, with evidence and assumptions, to show how the PPAs will work?

6.3 Delivery

6.3.1 Do the PPAs’ design and roll-out involve and take into account the needs of the intended beneficiaries?

6.3.2 How do managers ensure that PPAs build the efficiency and effectiveness of the delivery chain?

6.3.3 Are the PPAs delivering against their agreed objectives?

6.3.4 What is the impact of PPAs on improving recipients’ good governance at all levels and on building sound financial management that seeks to avoid corruption?

6.3.5 Are risks to the achievement of the objectives identified and managed effectively?

6.4 Impact

6.4.1 Are the PPAs improving the delivery of clear, significant and timely benefits to the partners’ intended beneficiaries?

6.4.2 Are the programmes working holistically alongside the activities of DFID and other funders?

6.4.3 Is there evidence of a long-term and sustainable impact from the PPAs being built?

6.4.4 Is there an appropriate exit strategy from PPAs at the end of the funding cycle?

6.4.5 Is there transparency and accountability to intended beneficiaries, donors and UK taxpayers?

6.5 Learning

6.5.1 Is there evidence of innovation and use of global best practice?

6.5.2 Is there anything not done in respect of PPAs that should have been undertaken?

6.5.3 Have lessons about the design and delivery of the PPAs been learned and shared effectively?

7. Methodology

7.1 The methodology for this review will be developed during the inception phase. It will involve a number of elements, including:

• an assessment of the rigour and independence of the selection process, including an analysis of the quality of decision-making, due diligence, engagement and communication with possible and successful funders. This will include documentary analysis and interviews with successful and unsuccessful applicants;
• a summary documentary analysis of the quality and coverage of DFID’s quality assurance process, including the CSOs’ self reports, the independent evaluations for the mid-term review and the oversight work of Coffey;
• six detailed case studies of PPAs, including tracking the process of selection, oversight and management for each PPA, as well as identifying specific institutional and delivery impacts. These will be selected to represent the types of organisations funded under the PPAs; and
• detailed interviews with recipient organisations, DFID staff and independent evaluators.

7.2 It is proposed that the case studies should be as follows:

Large

• Christian Aid (£21.8 million): funded by CHASE and CSD, the purpose of this funding is ‘to achieve major, measurable and sustainable improvements in health status and livelihoods for the most marginalised communities in countries of extreme vulnerability to climate change and poverty related health issues’.
• Action Aid (£12.4 million): funded by CSD, the purpose of this funding is to ensure that ‘poor and excluded people are active in ensuring positive policy and budgetary change and that duty bearers are accountable, transparent and provide quality in universal service delivery at local, regional and national levels’.

Medium

• WWF UK (£9.3 million): funded by CSD, the purpose of this funding is described as:
  1. Communities are safeguarding the ecosystems and ecosystem services upon which they and others depend in an equitable and adaptive manner.
  2. Policy frameworks and practices relating to adaptation, REDD+ and low carbon development are climate smart, environmentally sustainable and designed to secure and/or improve the well-being of men and women living in poverty.
  3. Government and private sector policies, practices and priorities relating to investment in infrastructure and natural resource extraction/use are climate-smart, environmentally sustainable, designed to secure and/or improve the well-being of women and men living in poverty.
• Restless Development / War Child / Youth Business International (£8.3 million):¹¹ funded by CSD, the purpose is described as:
  1. A critical mass of young people in target countries benefit from programmes and services in Wealth creation and livelihoods, sexual & reproductive health practices, in particular the prevention of HIV, Civic Engagement, in particular Good Governance and Accountability.
  2. A critical mass of national youth civil society organisations in both Target and Network Countries have significantly increased the quality and scale of their work and outreach.
  3. Strategic partners in the public-sector (government, bi- and multi-lateral) and private-sector produce a multiplier effect for the work of the Consortium.

Small

• Conciliation Resources (£3 million): funded by CHASE, the purpose of this funding is set out as ‘groups and people with whom we work play an increased role in conflict prevention and peacebuilding, assist marginalised groups in regaining a say over their lives, and influence improvements in the policies and practices of governments and organisations working on conflict issues’.
• Ethical Trading Initiative (£1.2 million): funded by CSD, the purpose of this funding is ‘improved working conditions for poor and vulnerable workers, especially women, in prioritised supply chains’.

8. Timing and Deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium (led by Agulhas Applied Knowledge). The review will take place during the final quarter of 2012 and the first quarter of 2013, with the report due to be published in the second quarter of 2013.

Footnotes

1. ↑ DFID uses the broader term ‘civil society organisations’ (CSOs) rather than ‘non-governmental organisations’ (NGOs) to categorise funding that is not made through governments or multilateral organisations.
2. ↑ We are including this KPMG contract with DFID as a case study in our current review, DFID’s use of contractors to deliver programmes, which is being led by Concerto Partners LLP, an organisation which is not part of our consortium.
3. ↑ DFID Civil Society Department Operational Plan 2011-15, DFID, updated May 2012, www.dfid.gov.uk/Documents/publications1/op/csd-2011.pdf
4. ↑ See http://www.dfid.gov.uk/work-with-us/funding-opportunities/not-for-profit-organisations/global-poverty-action-fund/.
5. ↑ See www.dfid.gov.uk/ukaidmatch.
6. ↑ See http://www.dfid.gov.uk/Work-with-us/Funding-opportunities/Not-for-profit-organisations/Governance-and-Transparency-Fund/Introduction/.
7. ↑ For details, see http://www.dfid.gov.uk/Work-with-us/Funding-opportunities/Not-for-profit-organisations/.
8. ↑ DFID information from Civil Society Department.
9. ↑ See http://www.dfid.gov.uk/work-with-us/funding-opportunities/not-for-profit-organisations/ppas/.
10. ↑ Terms of reference: Evaluation manager for the global poverty action fund and the programme partnership arrangements, DFID, 2010.
11. ↑ DFID reports that the funding of this consortium was an innovation; previously only single entities had been funded.