DFID’s bilateral aid to Pakistan

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We will review the impact and value for money of the UK aid programme in Pakistan, focussing on three sectors: education, humanitarian and health. These Terms of Reference outline the purpose and nature of the review and identify its main themes. A detailed methodology will be developed during the inception phase.

2. Background

2.1 With 170 million people, Pakistan is the sixth most populous country in the world.¹ The population is increasing fast: it is expected to exceed 210 million by 2020 and nearly 40% of these people will be between 10 and 29 years old. This could yield a demographic dividend and drive economic growth over the next decade. Yet Pakistan faces many challenges. Around 36 million Pakistanis live below the national poverty line.² The country’s path to development has been punctuated by frequent crises, including the 2005 Kashmir earthquake and the 2010 floods, which affected the lives of 20 million people. The country remains off-track against the education and health Millennium Development Goals: more than 17 million children do not attend school, 14,000 women die in childbirth every year and one in ten children die before reaching their fifth birthday.

2.2 Pakistan has strong connections to the UK. The UK has one of the largest Pakistani diasporas in the world and is amongst Pakistan’s largest investors. These ties are reflected in the UK aid programme, for which Pakistan is a key partner. The country received £213 million in 2010-11, the third-largest annual amount of bilateral assistance provided by the UK, behind only India and Ethiopia. The current Operational Plan shows that humanitarian assistance accounted for almost half of this total expenditure, reflecting the scale of the response to the 2010 floods. Of the remaining sum, education and health accounted for a little over 15% and governance and wealth creation approximately 20% between them. This profile is expected to change over the next four years, as DFID focusses significantly more resources on tackling Pakistan’s ‘education emergency’³ and improving reproductive, maternal and child health. If Pakistan can demonstrate results at the federal and provincial levels, DFID will scale up its efforts to support reform and accelerate progress. It is envisaged in the DFID Pakistan Operational Plan that, by 2014-15, the Pakistan country programme may approach £450 million a year, with education and health accounting for three-quarters of this expenditure.⁴ This would make Pakistan DFID’s largest bilateral recipient of UK aid.

2.3 The overall direction for DFID Pakistan’s activities is set out in its Operational Plan. The Plan identifies four priorities: peace and stability; making democracy work; macroeconomic stability, growth and jobs; and getting the state to deliver. Within this framework, DFID aims to maintain a flexible approach, adapting its portfolio to developments on the ground. As more powers and responsibilities have been devolved from the federal government to the four provinces, DFID has concentrated its efforts at the provincial level. Punjab and Khyber Pakhtunkwha (KPK) together account for over 70% of the population and the largest share of the poor and are thus DFID’s primary focus. DFID also operates on the understanding that Pakistan is not aid dependent. Many donors are active in Pakistan⁵ but, in 2010, total Official Development Assistance represented less than 2% of the country’s Gross National Income. DFID therefore seeks to use its investments to support and catalyse reform, responding where there is evidence of positive change at federal and provincial levels.

2.4 Assistance to Pakistan is, however, challenging. Security, volatility and the incidence of natural disasters remain major concerns. These factors make it hard for DFID to undertake normal development activity. They limit the range of potential implementing partners and constrain DFID’s ability to assess the impact of its work on intended beneficiaries.

2.5 Sitting beneath the four priorities set out in the Operational Plan, DFID has six pillars to its Pakistan country programme: education; humanitarian; governance and security; health; wealth creation; and poverty, hunger and vulnerability. Table 1 gives a summary of DFID Pakistan projects by sector, with more detailed information available in the Annex. Education represents 35% of DFID’s portfolio,⁶ humanitarian 20%, health 14%, governance and security 13%, wealth creation 11% and poverty, hunger and vulnerability 7%. DFID is in the process of shifting its education, health and governance programmes towards its two target provinces.

3. Purpose

3.1 To assess whether DFID is achieving impact and value for money in Pakistan with its bilateral aid to education, humanitarian assistance and health.

4. Relationship to other evaluations/studies

4.1 The Pakistan country programme as a whole was last subject to internal review in 2008.⁹ Amongst its conclusions, this review found that:

• DFID Pakistan had an increasing tendency towards budget support, seeing it as a productive way to build the relationship between the UK and Pakistan and to assist in the implementation of the country’s Poverty Reduction Strategy Paper (PRSP). This focus on budget support was not, however, sufficiently balanced by other aid instruments: ‘the diminished range of investments and partners has led to fewer opportunities for innovative interventions and lesson learning’;
• DFID Pakistan had a reputation for producing well-regarded analytical work. The link between analysis and delivery on the ground, however, was not always apparent: ‘implementation did not sufficiently build on this platform’. In particular, it was unclear how DFID’s work would address the ‘deep-rooted obstacles to change’ in Pakistan, such as poor access to justice, corruption, weak civil society and feudal land tenure;
• DFID Pakistan had not found ‘an appropriate balance between government and non-state actors’. Engagement with civil society was considered to be inadequate, given the potential for civil society organisations to press for improved services, as well as greater government accountability and transparency; and
• the previous country assistance plan made no provision for mitigating the risk of natural disasters.

4.2 These findings helped to inform the preparation of DFID’s subsequent country assistance strategy.¹⁰ The 2008 evaluation will provide useful background for the team undertaking this review. The team will seek to establish whether the concerns raised four years ago have since been addressed.

4.3 In the aftermath of the 2010 floods, the UK Government examined the adequacy, timeliness and efficiency of the humanitarian response. DFID commissioned an independent Humanitarian Emergency Response Review¹¹ and the International Development Committee (IDC) conducted its own subsequent enquiry following the Review’s publication.¹² The resulting reports will serve as valuable reference points for the review team. They note that DFID responded quickly to the Pakistan floods, with a generous financial contribution and good leadership and co-ordination. These reports found that:

• the previous experience of responding to the 2005 earthquake in Kashmir significantly strengthened the ability of the Government of Pakistan to respond to the floods, demonstrating the value of lesson learning, local capacity and leadership; and
• DFID played a key role as a ‘network enabler’, shaping the way the UN and other multilateral organisations operate and making them more effective.

4.4 IDC’s report recommended that DFID should indicate how its increased spending in Pakistan will result in an increased focus on disaster preparedness and risk reduction: ‘predicted increases in the incidence and severity of natural disasters, coupled with demographic trends call for a step change in the [humanitarian] system. DFID must play a key role in promoting changes in the system and must itself better integrate disaster risk reduction into its own development programmes.’ The review team will, therefore, wish to understand whether DFID’s response to the 2011 floods reflected lessons learned from earlier natural disasters and whether DFID is giving consideration to disaster preparedness and risk reduction in its current programming.

4.5 In 2011, the National Audit Office (NAO) published a report on DFID’s financial management.¹³ Though not specific to Pakistan, the report’s findings will provide useful background for the review. NAO’s report acknowledged that DFID has made considerable progress in strengthening its financial systems but concluded that ‘a better information environment is needed to deal with the heightened levels of assurance required in targeting future aid at higher risk locations’. DFID’s future plans involve channelling increased funding to more fragile states, with the most significant increases in countries with relatively low scores in the Transparency International index. Pakistan ranks 134th out of 182 countries in the 2011 Corruption Perceptions Index¹⁴ and the UK aid programme in Pakistan is expected to grow significantly. The review team will, therefore, wish to investigate what DFID is doing to assure itself that it minimises the risks of fraud and corruption in Pakistan.

5. Analytical approach

5.1 The evaluation will focus on aid delivery and impact. It will examine three contrasting areas of the DFID Pakistan programme, covering a range of funding channels and delivery partners. Across these three areas, the evaluation will examine:

• the extent to which programmes have achieved sustainable impact for the intended beneficiaries; and
• the effectiveness, efficiency and value for money of the different funding and delivery channels in delivering that impact, including whether DFID has delivery chains in place that will enable it to cope with the substantial scaling-up of the UK aid programme envisaged in the next four years.

5.2 The evaluation will not attempt to assess the impact of the DFID Pakistan programme as a whole, beyond the three selected programming areas. Nor will it assess DFID’s overall approach to governance and security in Pakistan. It will, however, examine how DFID has dealt with institutional and security issues within the three focus areas.

5.3 The evaluation will examine programming over the past five years in each of the three areas, including both closed and ongoing operations. It will not examine programmes under design or which very recently commenced, as impact assessment would not be feasible in these cases.

5.4 We plan to examine three sectors: education, humanitarian assistance and health.

Education

5.5 Education is the largest sector in the DFID Pakistan programme and is expanding substantially. It is projected to account for 45% of total expenditure over the five-year period covered by the current Operational Plan. Current and recent programming includes:

• a series of projects at the national level to promote innovation, encourage voice and accountability in the education sector and build the capacity of the federal government to promote and monitor reform;
• an £80 million Punjab Education Sector Programme that is now in its fourth year of implementation, with a £260 million second phase under design. It consists of sector budget support with accompanying technical assistance. The programme supports both the public education system and the low-cost private system. In addition, the ‘Punjab Education Roadmap’, established at the request of the Chief Minister, aims to focus political attention on education to help inject pace into reform processes. McKinsey and Company provides technical support to the Roadmap. Sir Michael Barber serves as DFID’s Special Representative for Education in Pakistan, working with the Government on educational reform; and
• new provincial programmes in KPK (£203.5 million) and Sindh (£20 million), both in their first year of operation. The Sindh programme is innovative in that delivery is via the private sector, in recognition of the limitations of state capacity to provide education to marginalised groups.

5.6 The review will focus on a number of issues, including:

• the mix of aid instruments and delivery channels used, with a particular focus on the Punjab programmes;
• the use of state and non-state education providers; and
• the extent to which the assistance focusses on educational outcomes as well as enrolment.

Humanitarian assistance

5.7 Humanitarian assistance has been a significant part of DFID Pakistan’s programming in recent years, including major responses to the 2005 earthquake (£84 million), 2010 floods (£123 million) and 2011 floods (£11.4 million, through a mixture of non-governmental and multilateral organisations). In 2011, DFID terminated its general budget support programme. Around the same time, new programming in the form of the Citizens’ Damage Compensation Programme (CDCP) was approved to the value of £65 million. The CDCP provides cash payments directly to households in flood-affected areas to meet their immediate, life-saving needs, reaching some 1.6 million households in its first phase.¹⁵ The use of cash payments is a fairly new approach to humanitarian assistance that has not been extensively evaluated internationally. DFID Pakistan is considering increasing its use of cash transfers in the future through a contribution to the government’s Benazir Income Support Programme, a social safety-net programme targeting the poorest households.

5.8 As some of the earlier humanitarian programmes have already been substantially reviewed, this evaluation will focus on the response to the 2011 floods. It will examine the shift from short-term emergency support to longer-term reconstruction and development assistance, with a particular focus on the use of cash transfers. It will also explore the extent to which DFID has helped to build Pakistan’s capacity for disaster preparedness and risk mitigation.

Health

5.9 Health is an important sector for DFID Pakistan, with £122 million being spent across current and recently closed health programmes and another £111 million allocated to programmes in the planning stage. The focus to date has been on federal maternal and child health programmes, with a contribution of £69 million to a federal health programme supporting service delivery across the provinces. Focus areas for this part of the review will include how DFID engages with different levels of Pakistan’s federal system and how DFID ensures value for money when spending through government systems.

6. Indicative questions

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. For this review, we will consider all the questions from our standard framework, but we will select two of the questions for more detailed examination. These are highlighted in the list below. They will allow us to compare and contrast the effectiveness and value for money of the delivery channels used for each of our focus sectors. During the inception phase of this review, we will examine the need for additional focussed questions in respect of each of our three focus areas.

6.2 Objectives

6.2.1 Does the programme have clear, relevant and realistic objectives that focus on the desired impact?
6.2.2 Is there a clear and convincing plan, with evidence and assumptions, to show how the programme will work?
6.2.3 Does the programme complement the efforts of government and other aid providers and avoid duplication?
6.2.4 Are the programme’s objectives appropriate to the political, economic, social and environmental context?

6.3 Delivery

6.3.1 Is the choice of funding and delivery options appropriate?
6.3.2 Does programme design and roll-out take into account the needs of intended beneficiaries?
6.3.3 Is there good governance at all levels, with sound financial management and adequate steps being taken to avoid corruption?
6.3.4 Are resources being leveraged so as to work best with others and maximise impact?
6.3.5 Do managers ensure the efficiency and effectiveness of the delivery chain?
6.3.6 Is there a clear view of costs throughout the delivery chain?
6.3.7 Are risks to the achievement of the objectives identified and managed effectively?
6.3.8 Is the programme delivering against its agreed objectives?
6.3.9 Are appropriate amendments to objectives made to take account of changing circumstances?

6.4 Impact

6.4.1 Is the programme delivering clear, significant and timely benefits for the intended beneficiaries?
6.4.2 Is the programme working holistically alongside other programmes?
6.4.3 Is there a long-term and sustainable impact from the programme?
6.4.4 Is there an appropriate exit strategy involving effective transfer of ownership of the programme?
6.4.5 Is there transparency and accountability to intended beneficiaries, donors and UK taxpayers?

6.5 Learning

6.5.1 Are there appropriate arrangements for monitoring inputs, processes, outputs, results and impact?
6.5.2 Is there evidence of innovation and use of global best practice?
6.5.3 Is there anything currently not being done in respect of the programme that should be undertaken?
6.5.4 Have lessons about the objectives, design and delivery of the programme been learned and shared effectively?

7. Outline methodology

7.1 The methodology for this review will be developed during the inception phase. It will involve a number of elements, including:

• a literature review, focussing on past evaluations;
• a review of evidence from DFID’s files and information systems, including financial information;
• meetings with intended beneficiaries in Pakistan;
• interviews with DFID, counterparts from federal and provincial governments, implementing partners, civil society organisations, other donors and informed observers; and
• interviews with UK and internationally-based experts.

7.2 The review will draw on results data generated by the monitoring and evaluation systems of the various programmes being reviewed, supplemented as appropriate with data from other sources and feedback from intended beneficiaries and other stakeholders.

8. Timing and deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The review will take place during the second quarter of 2012, with a final report available during the third quarter of 2012.

Annex: Major sectors and programmes in the DFID Pakistan country programme

Footnotes

¹ The figures quoted in this paragraph are taken from the DFID Pakistan Operational Plan 2011-15, DFID, February 2012, http://www.dfid.gov.uk/Documents/publications1/op/pakistan-2011.pdf.

² Pakistan Economic Survey 2007-08 for Pakistan 1998-99 – 2005-06, Government of Pakistan, 2008.

³ In 2010, the Prime Minister of Pakistan declared an ‘education emergency’ in Pakistan, http://educationemergency.com.pk/

⁴ DFID Pakistan Operational Plan 2011-15, DFID, February 2012, http://www.dfid.gov.uk/Documents/publications1/op/pakistan-2011.pdf.

⁵ The US is the largest bilateral donor to Pakistan. The International Monetary Fund, World Bank and Asian Development Bank are significant donors. Other UK Government Departments are also active, including the Foreign Office, Ministry of Defence and Home Office.

⁶ This is calculated as a proportion of the total lifetime budgets of projects that are currently active, recently completed or in the planning stage, using data provided to ICAI by DFID.

⁷ Data provided to ICAI by DFID.

⁸ Includes projects that are in design, at tender stage or just commencing.

⁹ Evaluation of DFID Country Programmes: Country Study: Pakistan Report, DFID, April 2008, http://www.dfid.gov.uk/Documents/publications1/evaluation/ev687.pdf.

¹⁰ Development in Pakistan 2008-13, DFID, 2008.

¹¹ Humanitarian Emergency Response Review, chaired by Lord Ashdown, March 2011, http://www.dfid.gov.uk/Documents/publications1/HERR.pdf.

¹² The Humanitarian Response to the Pakistan Floods, House of Commons International Development Committee, Seventh Report of Session 2010-12, http://www.publications.parliament.uk/pa/cm201012/cmselect/cmintdev/615/61502.htm.

¹³ DFID Financial Management Report, National Audit Office, April 2011, http://www.nao.org.uk/publications/1011/dfid_financial_management_rept.aspx.

¹⁴ Corruption Perceptions Index, Transparency International, 2011, http://cpi.transparency.org/cpi2011/results/.

¹⁵ CDCP Business Case and Intervention Summary, http://projects.dfid.gov.uk/project.aspx?Project=202276.

¹⁶ Covering Punjab, KPK, FANA & FATA, Gilgit-Baltistan, Balochistan, AJK, Sindh and Federal levels.

¹⁷ Covering Punjab, KPK, FANA & FATA, Gilgit-Baltistan, Balochistan, AJK, Sindh and Federal levels.

¹⁸ DFID lists this programme under ‘poverty, hunger and vulnerability’, but the intended beneficiaries are flood victims and in this review we will consider it as part of our review of the humanitarian sector.