Evaluation of DFID’s Bilateral Aid to Pakistan

Executive Summary

The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

Green: The programme performs well overall against ICAI’s criteria for effectiveness and value for money. Some improvements are needed.

Green-Amber: The programme performs relatively well overall against ICAI’s criteria for effectiveness and value for money. Improvements should be made.

Amber-Red: The programme performs relatively poorly overall against ICAI’s criteria for effectiveness and value for money. Significant improvements should be made.

Red: The programme performs poorly overall against ICAI’s criteria for effectiveness and value for money. Immediate and major changes need to be made.

This evaluation assesses whether DFID is achieving impact and value for money in Pakistan with its bilateral aid to health and humanitarian assistance (both long-standing areas of engagement) and to education (a relatively new sector for DFID Pakistan). The evaluation looks at the results achieved in a sample of programmes in these sectors with combined commitments of over £250 million. In particular, it assesses the different delivery channels used, to determine DFID’s readiness to make effective use of the large increases in UK aid to Pakistan planned for the coming years.

Overall

Assessment: Green-Amber

The DFID Pakistan programme is dynamic and innovative, with a good range of impressive initiatives. The education programme shows a sophisticated design and some promising early results, while the humanitarian support shows good evidence of learning and a strong focus on value for money. The health programme has faced major difficulties over its life and is now undergoing substantial redesign.

We note that DFID has no track record of delivering programmes in Pakistan on the scale that is now contemplated. The ongoing devolution of power from the federal to the provincial level and the practical difficulties of working in insecure provinces magnify the challenge. This suggests that the scaling up of the country programme needs to be approached cautiously and with a very active risk management stance.

Objectives

Assessment: Green-Amber

The programmes reviewed have clear and relevant objectives closely matched to UK Government priorities and Pakistan’s development needs. The education programme is sophisticated, multi-dimensional and innovative, with a balanced approach to access, equity and quality. The humanitarian projects demonstrate strong design but are yet to engage seriously with disaster risk reduction. The health programme displays significant shortcomings but is now being redesigned.

Delivery

Assessment: Green-Amber

DFID Pakistan has been piloting a number of new approaches to managing the substantial risks associated with directing aid through country systems in Pakistan. We are, nonetheless, concerned that weaknesses in budgetary processes and public financial management systems (especially in the new devolved model) and the additional difficulties facing DFID as it expands its programming into insecure provinces are a significant constraint on the efficiency of the support. This suggests the need for caution in scaling up the programme. We welcome experiments in working with the low-cost private sector in education as a potentially cost-effective way of reaching out-of-school children.

Impact

Assessment: Green-Amber

The education programme has delivered some important early results, including improving the quality of education, but it is too early to assess whether the results are sustainable and scalable. In health, the data are inconclusive but the difficult institutional environment has clearly limited impact. The 2011 humanitarian projects show good results for the modest scale of investment but we remain concerned about Pakistan’s lack of preparedness for future disasters.

Learning

Assessment: Green-Amber

The Punjab Education Roadmap is an innovative monitoring tool which has proved to be a very good platform for policy dialogue. DFID has invested in a new framework for continuous monitoring of its humanitarian programmes, giving it the flexibility to adjust its programming in real time. The health programme, however, has not demonstrated effective learning in respect of shortcomings in delivery.

Recommendations

Recommendation 1: To manage the risks associated with scaling up funding in a difficult and volatile environment, DFID Pakistan should ensure that the conditions for scaling up its programmes are clearly articulated and that it retains the flexibility to reallocate funding away from underperforming areas. The country programme should also be better balanced across government and non-government delivery channels.

Recommendation 2: DFID should increase its support for promoting affordable and equitable private sector delivery of health and education services to poor communities. It should focus on building government capacity to regulate the private sector, improve standards and monitor delivery.

Recommendation 3: Building resilience to natural disasters at the household and community levels should become a core element of the DFID Pakistan programme. This also provides an opportunity to diversify delivery channels.

Recommendation 4: When funding through country systems, DFID programmes should include agreed standards for budget integrity as a condition of scaling up. There should be a major emphasis on promoting transparency and accountability in budget processes and on increasing the amount of beneficiary involvement in the delivery and oversight of programmes.

1 Introduction

1.1 This report assesses whether the Department for International Development (DFID) is achieving impact and value for money in Pakistan through its education, health and humanitarian programmes. We look back over the past five years of programming in these areas. We also look forward, to assess DFID’s readiness to make effective use of the much larger aid funds that the UK plans to allocate to Pakistan.

Pakistan context

1.2 Pakistan is a large country facing major development challenges. With 176 million people, it is the sixth most populous country in the world.¹ The population is expected to reach 230 million by 2030. Pakistan achieved high rates of economic growth over the past decade, with the poverty headcount falling from 34% in 2001 to 24% in 2005.² Since then, various factors, including the 2008 global financial crisis and the vast 2010 floods (affecting an area four times the size of Britain), are likely to have caused poverty to worsen. As many as 62 million Pakistanis (36% of the population) are now estimated to live below the national poverty line.³

1.3 The country is unlikely to achieve its Millennium Development Goals (MDGs) on education and maternal and child health by 2015.⁴ Around 34% of school-age children (more than 17 million) are out of school and just under half of the adult population (including 60% of women) are illiterate. Some 12,000 women die in childbirth every year and one in ten children die before reaching their fifth birthday.⁵ This is among the worst child mortality rates in the world.

1.4 In recent years, Pakistan’s development challenges have been compounded by a succession of disasters, both natural and man-made. Figure A4 in the Annex describes the history of seven major disasters and emergencies since 2000, including the 2005 Kashmir earthquake which claimed 75,000 lives and destroyed half a million houses. It also includes the 2008-09 conflict in the Swat Valley that displaced 3 million people and the floods of 2010 and 2011, which affected 20 million and 9 million people respectively. International humanitarian relief over this period has totalled more than £1.5 billion, with the UK contributing some £330 million.⁶ As well as the direct impact of these disasters on the population, they were highly disruptive of national development programmes.

1.5 Violence in Pakistan has increased significantly in recent years. This comes as a result of conflict between the Pakistan military and rebel groups in the Afghan border region and a sharp increase in sectarian violence and terrorist attacks. It creates a very difficult operating environment in much of the country for the delivery of development programmes, with some recent cases of kidnapping of development workers. Tensions with India remain high and a large proportion of national resources is spent on the military. When combined with a narrow and uneven tax base and persistent inequality in resource allocation, this has led to a long-running neglect of basic services like health and education.

1.6 Pakistani politics are in a transitional state. Since its founding in 1947, Pakistan has had three extended periods of military rule (1958-71, 1977-88 and 1999-2008) and a number of unsuccessful coup attempts. It is hoped that the forthcoming elections, to be held by May 2013, will mark the first elected government in Pakistani history to serve a full term. In April 2010, the parliament adopted an ambitious programme of constitutional reform. This reduced the powers of the president in favour of the parliament and prime minister and devolved a large share of federal resources and responsibilities to the four provinces.⁷ Known as the Eighteenth Amendment, this reform amounts to a profound reorganisation of the Pakistani state.

1.7 The country faces a potentially lengthy period of uncertainty as provincial governments acquire the resources and capacities to meet their new responsibilities. Many of the development programmes that DFID previously supported through the federal government are in the process of being transferred to the provincial level.

1.8 The Pakistan context presents high levels of fiduciary risk. Pakistan ranks 134th of 182 countries on Transparency International’s 2011 Corruption Perceptions Index, on a par with Nigeria.⁸ Various DFID assessments have rated public financial management (PFM) as weak and, as a result of devolution, on a declining trend. Both internal and external audit of public funds are notably weak. Budgets are highly unreliable, with considerable variance between published budgets and actual expenditure.⁹ Pakistan has very low rates of tax collection and a history of poor macroeconomic management has led to a succession of unsuccessful International Monetary Fund programmes.¹⁰ An uncertain macroeconomic outlook presents a continuing risk to development programmes.

1.9 Overall, Pakistan faces a daunting array of development challenges, while presenting a difficult environment in which to deliver development assistance effectively. There are, however, opportunities as well as problems. Pakistani politics are increasingly open, the media is vibrant and urbanisation is creating new social pressures for improved government performance. These positive dynamics improve the prospect of successful development programmes in the future.

UK support to Pakistan

1.10 Pakistan is of considerable strategic significance to the UK. The two countries have extensive historical and commercial ties, with over 700,000 people of Pakistani birth or descent living in the UK.¹¹ With its large population and entrenched poverty, Pakistan is also an extremely important country in respect of the UK’s commitment to tackling global development challenges. DFID’s Operational Plan describes it as ‘a country we cannot ignore’.¹²

1.11 UK aid to Pakistan has scaled up dramatically in recent years and will continue to do so. The country programme has already trebled from £87 million in 2007-08 to £267 million in 2011-12 (see Figure 1). Ministerial approval has been given for the programme to reach £446 million in 2014-15. This would make Pakistan the largest recipient of UK bilateral aid in the world. DFID is currently the second largest bilateral donor to Pakistan after the United States and the fourth largest overall after the World Bank and the Asian Development Bank. The DFID country team has expanded accordingly, nearly doubling in size over the past two years from 55 to more than 100.

1.12 To scale up the programme at this rate, DFID Pakistan is planning some of the largest individual programmes in DFID’s history. An education programme for Khyber-Pakhtunkhwa (KP) Province of £203.5 million commenced recently, with another planned for Punjab of £260 million. DFID also has new health and nutrition programmes in the pipeline totalling around £160 million. DFID has no track record of delivering programmes on this scale, with its largest individual investment in the past being a £100 million national health programme. The bulk of the scaling up will be in the education sector, which will rise from 21% of the programme in 2010-11 to more than half over the coming three years (see Figure 2 for the sectoral composition of the programme). The diagram assumes that there is no major humanitarian emergency.¹⁴

1.13 The country team faces a difficult operating environment. Due to security constraints, staff are unable to travel to some parts of the country (Balochistan and the Federally Administered Tribal Areas (FATA)), while access to KP is limited to day trips to Peshawar with extensive security support. The country team is located in the capital, Islamabad, with a small satellite office in Lahore, the capital of Punjab Province. DFID plans further decentralisation of its staff following the forthcoming election, if security conditions permit.

1.14 The country programme is in the process of responding to devolution. A general budget support operation at the federal level was discontinued, as the federal government no longer has responsibility for the bulk of development expenditure. Future budget support operations will be for the benefit of DFID’s target provinces. In the past, health programmes were mainly implemented in partnership with the federal Ministry of Health, which has now been abolished. For the time being, health funding is still provided to provinces via the federal government, while new provincial health programmes are being developed. DFID Pakistan has chosen to focus on two provinces, Punjab and KP, with which it has established relationships and relatively good physical access. They represent approximately 70% of the country’s population (see Figure 3 on page 6 for Pakistan’s population by province). DFID’s scaling up of assistance in the coming years will be mainly in these two provinces, particularly in the education field. The federal government, however, still holds the mandate for co-ordinating international assistance. DFID needs to maintain relationships at the federal level to ensure political access and policy influence.

Methodology

1.15 The purpose of this evaluation is to assess whether DFID is achieving impact and value for money in Pakistan with its bilateral aid to education, humanitarian assistance and health. Health is a long-standing area of engagement for DFID Pakistan, which has provided substantial financial assistance for national health programmes since 2003. DFID also has long experience in the humanitarian sphere, having supported seven major humanitarian operations since 2000 (see Annex, Figure A4). Education is a newer engagement, with sectoral budget support operations at the provincial level beginning only in 2011.

1.16 In each of these three sectors, we chose a selection of projects for detailed examination (see Annex, Figure A3). The sample was selected based on the scale of expenditure, feasibility of access for the evaluation team and the potential to illustrate DFID’s readiness for scaling up.

1.17 In the education sector, we examined:

• the Punjab Education Support Programme (PESP) (£80 million), a sector budget support operation in Punjab led by the World Bank, which provides £220 million; a second phase is under design, with a planned DFID contribution of £260 million and a further World Bank contribution of £220 million;
• the Education Sector Reform Roadmap (£2.1 million), a technical assistance project led by Sir Michael Barber, DFID’s Special Representative for Education in Pakistan. This supports the Chief Minister of Punjab with developing and implementing education reforms;
• the Transforming Education in Pakistan Programme (£20 million). This supports advocacy, media campaigns and support to civil society in the education sphere; and
• the Education Sector Voice and Accountability Fund (£5 million), a grant-making instrument supporting research, advocacy and innovation around education policy and delivery and the engagement of local communities.

1.18 In the health sector, we examined:

• the Maternal and Newborn Health (MNH) Programme (£69 million), a programme of financial aid via a federal government programme to boost service delivery of MNH services by provinces and districts;
• the Technical Resource Facility (£8.3 million), co-financed with the Australian government (£7 million). This provides technical assistance to the MNH programme and provincial health authorities; and
• the Research and Advocacy Fund (£11.5 million, with an additional £7 million from Australia). This provides grants for research and advocacy on public health issues.

1.19 In the humanitarian sector, we focussed on:

• the response to the 2011 floods in Sindh Province (£11.4 million, later extended by a further £10 million), with a view to identifying lessons learned from the much larger 2010 flood response. This included projects covering shelter (delivered by the International Organization for Migration and the Health and Nutrition Development Society) and agriculture (delivered by the Food and Agricultural Organization of the United Nations); and
• the Citizen Damage Compensation Programme (£65 million), where DFID is using cash transfers for early recovery as an ongoing part of its 2010 flood response.

1.20 In the case of the PESP, DFID joined an ongoing World Bank sector budget support operation as co-financier and, in the next phase, will provide more than half of the resources. The Citizen Damage Compensation Programme is also co-financed with the World Bank and other donors. In both cases, DFID has had substantial input into the design and management of the assistance. The MNH Programme is a DFID contribution to a federal government health programme. The other projects in the sample are DFID designs.

1.21 Three of our Commissioners visited Pakistan in January 2012, where we held detailed discussions with DFID and a range of its partners and visited a selection of project sites. Our observations during that week informed the design of this evaluation. We were also briefed by Sir Michael Barber on the Punjab Education Sector Reform Roadmap. Our evaluation methodology included a background literature review, detailed examination of DFID’s financial information and programme documents and a two-week visit to Pakistan, which included:

• detailed briefings from DFID Pakistan staff;
• interviews with national, provincial and district officials and other development partners;
• reviews of project and financial management processes of DFID’s implementing partners;
• field visits to sites where programmes are delivered in Punjab and Sindh, including schools, health clinics, flood-affected villages and cash transfer payment centres; and
• roundtables with non-governmental organisations (NGOs) in each of the sectors.

1.22 The team spent several days visiting schools and health facilities in Punjab and around 12 flood-affected villages in Sindh. Here we conducted scheduled and unscheduled meetings with various groups of intended beneficiaries, including schoolchildren and parents, patients at health clinics and members of flood-affected households. We used semi-structured and open interviews to test the relevance and effectiveness of the programmes and validate their results.¹⁵ In many cases, we interviewed women and men separately, in order to identify any gender bias in the delivery of the programmes. DFID implementing partners were involved in the organisation of the field visits and provided interpreters. Any bias that this may have introduced was minimised through the opportunity we had to triangulate responses between individuals and communities, using various interpreters from different partners.

1.23 The evaluation team included three UK-based and two Pakistani experts, with additional research support from the UK and the United States. In addition to the team leader, the team comprised specialists in each of the three sectors and an expert on public financial management in Pakistan.

2 Findings

Objectives

Assessment: Green-Amber

2.1 This section of the report examines the design of the DFID programmes. We assess whether the objectives are clear, relevant and realistic, whether they are based on solid evidence and whether they incorporate logical plans for how they will make a difference.

A strong service delivery focus

2.2 The statements of purpose for each of the programmes we reviewed are summarised in Figure A3 of the Annex. At the highest level, the objectives behind these three sectors are clearly relevant and important, both to Pakistan’s development needs and to DFID’s global commitments.

2.3 Pakistan’s constitution mandates free and compulsory education up to secondary level. In reality, the education system has been systematically starved of resources for many years¹⁷ and the provision of education is so poor that the Government of Pakistan has declared an ‘education emergency’.¹⁸ Pakistan ranks 160th of 187 countries on the United Nations Development Programme’s 2011 Education Index.¹⁹ Only 88% of children enter school on time, with half of those dropping out before finishing primary school. Nearly a quarter of children between the ages of 7 and 16 have no education at all.²⁰ Among those who do attend school, educational attainment is low, with two thirds of school leavers in rural areas failing to achieve basic literacy.²¹

2.4 With its high rate of population growth, Pakistan has an extremely young population, with nearly half of its citizens under 25 years of age. This is both a threat and an opportunity. If Pakistan can provide its youth with a decent level of education, this demographic dynamism will be an economic advantage, enabling it to expand its manufacturing and service sectors. On the other hand, if Pakistan is unable to provide them with opportunities for advancement, youth unemployment may become a source of instability.

2.5 Health services in Pakistan suffer from a similar pattern of underinvestment. Government expenditure on health is low by regional standards, at only 0.84% of gross domestic product or around £6 per person each year.²² While there is a substantial system of health services in place, facilities are poorly equipped and often in a state of disrepair. Rural areas are particularly poorly served, with 68% of rural women giving birth at home. Pakistan is second only to Afghanistan in its region for population growth and maternal, infant and child mortality rates. There are 12,000 maternal deaths a year in Pakistan, the fourth-highest number in the world,²³ making this a priority for DFID given its global commitment in this area. Household surveys have shown that unexpected health bills are the most severe form of shock hitting poor households, greater than either conflict or natural disasters.²⁴

2.6 As well as the clear need for improvements in health and education, the UK Government is committed to strengthening service delivery by the Pakistani state. One of the reasons for Pakistan’s poor education and health outcomes has been its unequal pattern of public expenditure, particularly a neglect of poorer rural communities. DFID seeks to change that pattern by promoting public service delivery on the basis of objective needs and citizen entitlements. DFID has also helped the Government of Pakistan to develop a mechanism for cash transfers as part of its response to the 2010 floods. This will be used to provide a social safety net to the poorest and most vulnerable households. DFID is helping to support a change in the social protection system, to introduce clear and transparent beneficiary selection processes. This emphasis on equity is one of the strengths of DFID’s Pakistan programme.

Strong design in humanitarian assistance

2.7 In August 2011, heavy rains triggered flooding that took 500 lives and left 5.2 million people in need of humanitarian assistance, mainly in Sindh Province.²⁵ A needs assessment was conducted jointly by the Government of Pakistan and the United Nations Office for the Coordination of Humanitarian Affairs (OCHA), leading to the preparation of a Rapid Response Plan for the emergency phase and an Early Recovery Framework. These contain lists of standardised packages of assistance in different sectors, from which donors can choose. Standardising the assistance is designed to facilitate a rapid and cost-effective response.

2.8 DFID chose to direct £21.4 million through three implementing partners, representing 10% of the assessed funding needs. Two of its projects focussed on shelter, providing temporary shelter kits and helping families to reconstruct damaged housing. The shelter projects helped families to adapt traditional building techniques to make their homes more flood resistant. The other humanitarian intervention was in agriculture, targeting farming communities that had lost crops through the flooding. The DFID project helped its intended beneficiaries develop a mid-season cash crop (sunflowers), giving them the resources required to restart the agricultural cycle. It included training for farmers to help them maximise the value of the crop and market it effectively. The projects were selected in accordance with the priorities identified by the government.

2.9 The provision of cash transfers in the early recovery phase of a humanitarian operation is a fairly recent innovation with growing support within the humanitarian assistance community. As compared to distributing food or material aid, cash offers a number of potential advantages. In the right conditions, it can stimulate rather than supplant local food markets.²⁶ Many of the beneficiaries we spoke to expressed a preference for cash transfers, which gave them the flexibility to meet their individual needs. They were also seen as offering greater dignity than, for example, food packages.

2.10 One of the limitations of cash transfer schemes is that an effective system for handling cash payments can take some time to put in place. Indeed, the programme supported by DFID is still providing cash transfers to the last of the recipients two years after the 2010 floods.²⁷ To address this issue for future humanitarian programmes, DFID is helping the Government of Pakistan to create a standing mechanism for cash transfers. This will use the national identity card system, a network of private banks and an electronic payment card (the ‘WATAN’ card). This system will be available for use in future emergencies and also for social safety net programmes such as the Benazir Income Support Programme, which DFID plans to support. This is a sound strategic choice: DFID has helped to introduce clear and transparent beneficiary selection criteria that target the most vulnerable and has also helped to strengthen fiduciary controls.²⁸ From our observations while visiting a WATAN centre in Sindh and talking to beneficiaries, this targeting is appropriate.

2.11 The 2011 floods were the latest in a long series of humanitarian disasters which, according to government officials, is highly likely to continue. While its humanitarian projects are well conceived, DFID has only limited engagement at present in building capacity for disaster risk reduction or management, to increase resilience to future disasters.²⁹ This is despite a recommendation from a 2008 Country Programme Evaluation that ‘[d]isaster risk reduction interventions should be a significant element of the new programme’.³⁰ DFID has acknowledged this gap and is launching a programme of analytical work, with a view to taking on disaster risk reduction as a theme in the coming period.

A health programme in transition

2.12 DFID’s engagement in health in Pakistan over the past decade has mainly taken the form of financial support to national health programmes. While health services are a provincial and district responsibility, national health programmes provide additional resources to boost the resources available for particular services, such as child vaccination or control of communicable diseases. Since devolution, these national health programmes are gradually being passed to the provincial health authorities. DFID is, therefore, in the process of redesigning its health programmes to work directly with the provinces.

2.13 The MNH programme is one such national health programme, designed to reduce maternal and infant mortality by increasing the level of skilled birth attendance. It supports the training and deployment of a network of Community Midwives (CMWs) in rural areas. It also supports the upgrading of district and local hospitals to provide basic and comprehensive emergency obstetric and newborn care services. It thereby ensures that difficult cases are identified and referred to a properly equipped health facility. This is a sound principle, based on learning from similar programmes elsewhere in the region.³¹ The DFID design document makes reference to supporting studies at the international level and practical lessons from previous MNH projects in Pakistan by other donors, although it acknowledges a general lack of robust data on MNH interventions and their outcomes.³²

2.14 While the broad choice of programme was sound, we found a range of problems with the design of the intervention. The CMWs work in parallel to an existing network of community health educators, known as ‘Lady Health Workers’, established under another national health programme to provide advice to rural women on family planning and primary health care issues. The Lady Health Worker programme was also at one stage funded by DFID. As community education is a key element of addressing maternal and infant mortality, the two networks would need to work together to be effective. In practice, however, they have operated as rival schemes. This has proved to be a significant constraint on the effectiveness of the MNH programme.

2.15 In particular, Lady Health Workers appear to have financial disincentives to refer cases to the CMWs. According to senior government health officials, Lady Health Workers receive commissions for referring deliveries to private health clinics and, in some cases, may themselves be assisting with deliveries for a fee. This is a problem that should have been anticipated and addressed in the programme design. The design should also have been grounded in a stronger analysis of how CMWs would become accepted by communities and be financially sustainable.

2.16 We are also concerned that the MNH programme focusses on complications in delivery at the expense of other key factors in MNH, such as integrated management of neonatal and childhood diseases.

2.17 Devolution has made it difficult to correct these flaws in the MNH programme design. DFID’s original counterpart, the national Ministry of Health, has now been abolished, while ownership of the federal programme at provincial level has been weak. DFID is now redesigning its health sector programming to reflect the process of devolution. It plans to provide financial support to its two partner provinces for an integrated approach to family planning, maternal and child health and nutrition.

A sophisticated education design

2.18 Compared to health, where DFID has been active in Pakistan for many years, DFID’s education portfolio is relatively young. Some of the programme elements we examined were at an early stage of implementation and therefore still untested. Nonetheless, we found the design of the education portfolio to be well conceived, with multiple reinforcing elements and a balanced focus on improving access, equity and quality.

2.19 The education portfolio compares favourably to some of those we reviewed in East Africa, which were excessively reliant on financial aid to bring about institutional change.³³ By contrast, the DFID Pakistan approach accompanies financial aid with extensive political dialogue, technical support, funding for research and innovation and a set of interventions designed to generate social pressure for change. We were particularly impressed with how DFID Pakistan has analysed the political landscape and taken on policy influence as a structured campaign. DFID is working with senior political leaders, opposition parties, the media, NGOs and the international community to build a consensus in favour of reform.

2.20 The goals of the Punjab education programme balance access (i.e. enrolment levels) with quality of basic education and equity (particularly promoting education for girls). Its programme designs and analytical work also suggest a concern for maintaining this balance. For example, in addition to a campaign to expand enrolment, the Punjab education reforms involve a series of interventions designed to improve learning outcomes. These include a drive on teacher attendance, the introduction of lesson plans, teacher performance incentive schemes, upgrading of school facilities and work with the examination board to raise the standards of assessment. There is a strong focus on gender equity, with financial incentives to encourage poor families to send their girls to school. DFID is also beginning to integrate measures to address external factors influencing education outcomes, such as parental attitudes to learning and children’s nutrition.

2.21 DFID acknowledges that the evidence base on which interventions are most likely to improve education outcomes in Pakistan is limited. The design does, however, build on earlier DFID support for the Pakistan Education Task Force. The Task Force conducted a series of roundtables with education experts, politicians, business leaders and the media to build up a better understanding of the drivers of and obstacles to reform in the education sector.

2.22 If there is any gap in this theory of change, it is that the Punjab education authorities appeared to need more hands-on support with managing the complex processes of institutional change required to achieve their goals. The Punjab Education Reform Roadmap (described in paragraphs 2.81 to 2.85 on pages 21 and 22) is an effective tool for monitoring education outcomes at district level. The information it generates, however, does not of itself assist the province with managing the complex institutional changes involved in improving education outcomes.³⁴ This element should be given more emphasis in the next phase.

Delivery

Assessment: Green-Amber

2.23 This section of the report looks at how effectively the programmes are being delivered, focussing on the choice of funding and delivery options. It considers how well DFID is managing the challenges of funding effectively through country systems and looks at the experiments with supporting non-state delivery of services. It assesses DFID’s overall approach to fiduciary risk management, anti-corruption and value for money. Finally, it considers the quality of community engagement in the design of the programmes.

Delivering through country systems

2.24 Given the UK’s commitment to strengthening service delivery by the Pakistani state, one of the central dilemmas facing DFID is how to spend money effectively through country systems. It needs to work with the responsible provincial authorities and support them in policy-making, planning and budgeting – including for recurrent expenditure like salaries. Working with and through country systems, however, means exposing UK funds to the risks associated with unreliable budgetary processes and weak public financial management systems. Public expenditure in Pakistan is routinely delayed and actual expenditure can vary substantially from published budgets.³⁵ In this context, there is a risk that donor funding will displace rather than supplement government spending in a given area. There is also a risk that development spending will be manipulated for political purposes.

2.25 Comparing the health and education programmes in Punjab reveals two contrasting strategies for managing this risk.

2.26 In the health field, the MNH programme is co-funded by the Government of Pakistan and DFID, with DFID originally planning to contribute 40%.³⁶ The funding channels are complex, with spending responsibilities shared amongst federal, provincial and district levels. The bulk of the funding, spent at district level, is subject to multiple approvals (we counted more than 20) which should ensure that it reaches its intended destination, although at the cost of extensive delays.

2.27 DFID’s contribution to the programme is earmarked (i.e. limited to agreed spending items) and ring-fenced (i.e. DFID funds are accounted for separately, at least as far as the provincial level). DFID funding is released to the provinces only when DFID is satisfied that its conditions have been met. Fiduciary risks are analysed and the programme has a fiduciary risk mitigation plan that includes independent audits and technical assistance for financial management and procurement. These measures provide additional protection for UK aid funds to supplement Pakistan’s own fiduciary controls. For example, in 2009, even prior to the introduction of continuous audit (see paragraph 2.45 on page 14), a DFID-commissioned audit detected one significant case of fraud, causing it to suspend its funding for a period (see Figure 4).

2.29 While DFID has been conscientious in safeguarding its own funds from misuse, the programme as a whole has received only 35% of its planned budget (of which DFID has provided 58%). Our enquiries of provincial and district officials confirmed that the late and unpredictable release of funds had been a significant constraint on the effectiveness of the programme.³⁸

2.30 In the education field, the PESP has employed a different funding strategy. It provides sector budget support to Punjab Province, which is spent through regular budgetary processes with no earmarking or ring-fencing.

2.31 DFID needs to ensure that the additional budgetary resources are not absorbed by higher teacher salaries (usually the largest item in any education budget) and do not displace the province’s own spending. To do this, funds are released only once the provincial government has achieved agreed milestones in ten priority areas (such as upgrading schools, buying textbooks and providing stipends for girls’ attendance). Its performance is assessed through bi-annual joint monitoring exercises. The provincial budget execution as a whole remains unpredictable. This conditionality, however, has ensured that budget releases for education are protected from cash shortages in the provincial budget.

2.32 The education financial support also contains a number of other safeguards. There are external audits of provincial spending programmes and some use of tracking studies to follow the transfer of funds down to district and school levels. The Punjab Education Sector Roadmap is also a tool for monitoring the allocation of resources across the province. This provides an opportunity for DFID to raise any inequities with the Chief Minister.

2.33 Comparing the two approaches, the sector budget support used in education offers a number of advantages. It offers less direct control over the use of UK funds but focusses instead on the overall composition of the provincial education budget. The Roadmap process is proving to be a powerful tool for monitoring and influencing provincial education expenditure. By contrast, the ring-fencing and earmarking used in the MNH programme provide greater protection against fraud. They have not, however, proved effective at ensuring that the Government of Pakistan meets its own financing commitments. Of the two funding strategies, sector budget support for education would appear to have offered greater value for money.

2.34 We note that DFID is planning to replace the MNH programme. The new programme will directly support provinces through a sub-sector budget support operation for family planning, maternal and neonatal health and nutrition. This will be an opportunity to take on board some of these lessons.

2.35 Overall, maximising the effectiveness and value for money of aid funds directed through country systems remains one of the most difficult challenges facing the DFID Pakistan programme.

Non-state service delivery

2.36 DFID Pakistan has been exploring options for non-state service delivery in education, to complement its public sector programmes. Pakistan has witnessed an extraordinary growth in low-cost private schools, which now account for a third of overall enrolment.³⁹ This has been a market-led response to the poor quality of education offered by the public school system. Often informal in nature, these schools operate at an average cost per pupil only half that of the state system. Although teachers may lack formal qualifications, pupil–teacher ratios are lower, teacher attendance is better and private schools are more accountable to the communities they serve. The available evidence suggests that, for these reasons, private schools are producing learning outcomes that are significantly better than public schools (although education standards in both systems remain very basic).⁴⁰ The low-cost private sector is, therefore, potentially both an effective and an economic means of expanding the provision of basic education in Pakistan.

2.37 While low-cost private schools are affordable for the majority of Pakistani families, the poorest households may still be unable to afford the fees.⁴¹ Low-cost private schools also tend to cluster in urban and more densely populated rural areas, leaving remote communities underserved. DFID has sought to address these issues by supporting the Punjab Education Foundation (PEF). This is an autonomous agency created by the Government of Punjab that works with the private education system through a variety of mechanisms. It provides vouchers for poor families whose children attend private schools, providing free education to 1.1 million children in 2,233 schools.⁴² (DFID is developing a similar scheme in Sindh Province through a different model, using a non-profit company to provide stipends to poor households.) These subsidies give the provincial education authorities a potential lever to address quality standards in the private sector (participating private schools must qualify to join the programme and agree to participate in regular quality assessments). PEF also runs the New Schools Programme, encouraging entrepreneurs to establish schools in remote areas of the province where private schooling is scarce or absent altogether.

2.38 We agree that the low-cost private sector must be part of the strategy for achieving universal basic education in Pakistan. At the same time, its rapid development raises some difficult policy challenges for the provincial education authorities. Is there a risk of a two-tiered system emerging, entrenching inequality? Can the private sector serve poor or remote rural communities to an acceptable standard? How can the state ensure that minimum standards are maintained? With the low-cost private sector playing an increasingly important role in education provision in many developing countries, these are important policy questions for DFID. Through its research and advocacy programming and its policy dialogue, we encourage DFID to explore them in detail.

2.39 DFID is also exploring options for using the private sector to deliver basic health services. Its experience with promoting private delivery of MNH services, however, has proved to be problematic. The programme anticipated that the CMW cadre would become self-financing, with CMWs charging 500 rupees (approximately £3.50) per delivery. For the rural poor, this charge is likely to be less than the cost of travelling to a static health facility. It nonetheless raises issues that have not been resolved, given that the better-off urban population has ready access to free MNH services in publicly funded facilities. This may be one of the reasons why uptake of the CMW services has been fairly poor (see paragraph 2.65 on page 18).

2.40 Furthermore, insufficient attention has been paid within the programme to encouraging the CMWs to become financially sustainable. CMWs receive temporary stipends for their first two years, to help them become established. In practice, however, they have become a strong lobby within the MNH programme for continuing stipends, as revealed by the Commissioners’ consultations with CMWs in Multan, in the Punjab. There should have been more attention given in the design of the programme to aligning financial incentives with the goal of achieving sustainable private-sector delivery. This is an area that will need careful assessment in the design of future health programmes.

Managing fiduciary risk

2.41 According to independent analysts, the political environment in Pakistan is not conducive to anti-corruption efforts and national anti-corruption institutions are largely ineffective.⁴³ As a result, none of the donors in Pakistan has any explicit anti-corruption programming. DFID Pakistan is a pilot for producing a country-level anti-corruption strategy.⁴⁴ It has recently undertaken scoping work to identify options for a more robust engagement in this area.

2.42 There is a lack of anti-corruption programming in the country portfolio. We found, however, that DFID Pakistan has a very active approach to monitoring and managing the high levels of fiduciary risk in Pakistan. A number of useful innovations are now being piloted.

2.43 DFID has ten active Fiduciary Risk Assessments (i.e. comprehensive assessments of fiduciary risk that are mandatory at least every three years for all financial aid;⁴⁵ annual updates are conducted in interim years to assess progress on reforms). Four are programme-specific and there are three each in the two target provinces (covering overall PFM systems and the health and education sectors).

2.44 Fiduciary risks are assessed as substantial across the board. Pakistan generally has an adequate framework of rules and regulations on PFM but these remain poorly implemented amid widespread capacity constraints. Budgets lack credibility: they can be varied in-year by the executive without parliamentary approval and expenditure outturns vary widely from published budgets. Of particular concern is the lack of adherence to procurement regulations, the absence of any internal audit function and a lack of monitoring of expenditure arrears.

2.45 DFID has been supporting PFM reform at the federal level for the past eight years. Devolution has, however, shifted the balance of risk down to the provincial level, where donors have historically been less engaged in the PFM area. In KP Province, DFID’s Fiduciary Risk Assessments have been shared with government and have formed the basis for the province’s Integrated PFM Reform Strategy, which DFID supports. In Punjab, the dialogue on PFM has not yet reached this stage. DFID’s Sub-National Governance Programme (under design) will support PFM reform in both provinces.

2.46 In terms of fiduciary risk management at programme level, DFID Pakistan is undertaking a number of innovations. One is to pilot ‘continuous audit’ across four programmes. A private audit firm has been engaged to undertake risk-based assessments using sampling and field visits to provide real-time feedback on programme implementation. The audit methodology is designed to pick up any problems at an early stage and prevent substantial losses from occurring. DFID is also piloting the use of due diligence assessments of implementing partners (a previous ICAI recommendation⁴⁶). DFID will, however, need to ensure that it has enough capacity on financial matters within the country team to follow up on the results of these audits and assessments.

2.47 In the humanitarian programme, we found that fiduciary controls were rigorous and systematic. All the projects had clear and appropriate beneficiary selection criteria and a high degree of transparency in beneficiary selection. They had introduced hotlines and complaints procedures for individuals who had not received their entitlements. Procurement processes appeared robust and checks were in place to ensure that goods and services were delivered as contracted. All three implementing partners had a range of anti-corruption measures in place, with several layers of monitoring of implementation. DFID staff also made regular visits. During their visit, Commissioners noted with approval a robust system used by Handicap International, one of the humanitarian partners, to track the transfer of funds to the intended beneficiaries. Figure 5 on page 15 describes the safeguards built into DFID support for cash transfers.

DFID’s approach to value for money in Pakistan

2.49 DFID Pakistan has recently prepared a Value for Money Strategy.⁴⁷ It includes: more rigorous assessment of delivery options during design; stronger links between programme budgets and the delivery of outputs; a more competitive procurement approach; and additional training and awareness-raising for DFID staff. While this is a new set of undertakings, we observed a high level of challenge within the country team around the design of individual programmes. Monthly Programme Board meetings involving senior advisers provide a forum for testing new programme designs and monitoring the performance of existing programmes, to check they meet quality and value for money standards.

2.50 DFID Pakistan has also undertaken an interesting value for money and results pilot in its humanitarian support. During the 2010 flood response, DFID introduced a value for money methodology, based on the unit costs of inputs and outputs and its partners’ overheads, to inform project appraisals and budget negotiations. A monthly reporting framework was also introduced whereby implementing partners could report quickly and easily on their cumulative results. This enabled DFID to identify areas of underspend and reallocate funds as necessary, allowing for a more efficient response.

2.51 When preparing for the 2011 flood response, the data collected in 2010 provided DFID with a strong basis for selecting its implementing partners. Being a smaller operation in a more concentrated geographic area, DFID was able to reduce its number of partners from 42 to three, chosen as the best performers from 2010 capable of meeting the needs of the 2011 crisis. DFID continues to maintain regular contact with a wider circle of partners, in preparation for any future large-scale emergency.

2.52 We understand that there was some initial resistance from partners in 2010 to the additional reporting burden. The three partners involved in the 2011 response, however,⁴⁸ all gave very positive feedback on the value for money and results framework. They told us that it had helped them identify cost savings and other efficiency gains in their programming. Humanitarian projects have highly standardised interventions and easily assessed unit costs. They may lend themselves more to systematic approaches to value for money than other parts of the programme. We would encourage DFID in Pakistan and internationally to look for other ways to build on this experience.

Engagement with beneficiaries

2.53 We found a mixed picture across the sectors on the quality of engagement with beneficiaries. It was best developed in the humanitarian area. Here we found that local communities had been closely involved in beneficiary selection and project implementation. They had a high level of understanding and ownership of the projects. Some of the humanitarian projects were designed with a view to promoting social mobilisation. For example, in the shelter projects, beneficiaries were provided with approximately £200 in cash to buy materials for reconstructing their houses. The communities themselves provided the labour. The cash was paid in four instalments, once certain milestones in the construction were reached (e.g. completion of adequate foundations). All the beneficiaries in any given community were required to meet each milestone in order for the subsequent payment to be made. The beneficiaries we consulted confirmed that this had helped to ensure that communities worked together.

2.54 We found that the humanitarian programmes had well designed mechanisms for potential beneficiaries to complain about any alleged mistreatment. These complaints mechanisms are extensively used and appear to be effective. Other than complaints procedures, however, we did not observe any mechanism for beneficiaries to provide feedback on the relevance and quality of the assistance they received.

2.55 DFID’s new Humanitarian Policy contains a commitment to ‘make beneficiary accountability a core element of DFID’s humanitarian work.’⁴⁹ To that end, DFID Pakistan is piloting a new project to map beneficiary feedback mechanisms currently in use in the humanitarian field in Pakistan. Using that data, the project will develop a beneficiary feedback model for use by DFID. We welcome this initiative. Rather than developing a dedicated system for DFID and its implementing partners, however, we encourage DFID to explore ways of building beneficiary feedback into the national disaster response architecture in Pakistan.

2.56 Community engagement is much less developed in the other two sectors. The MNH programme included Social Organisers to improve the uptake by rural communities of CMW services. They appear, however, to have been used more as a monitoring system over CMWs than as outreach to communities. Our inquiries suggest that not enough thinking has been done as to how to build up the local legitimacy of the CMW cadre – a point that DFID has raised with its counterparts.

2.57 In education, one of the elements of PESP is to strengthen School Councils, including giving them a role in setting school budgets. These give parents a formal say in how schools are run. Our interviews with parents, however, suggested that the Councils were in practice dominated by teachers. They are used as a vehicle for promoting education initiatives by the province (e.g. enrolment campaigns) rather than increasing beneficiary voice or accountability to communities. The new Innovation Fund plans to develop different methods of engaging communities, including in school governance and the monitoring of education budgets. We encourage DFID to ensure that its large-scale service delivery programmes include robust mechanisms to promote ownership by and accountability to the intended beneficiaries.

Impact

Assessment: Green-Amber

2.58 This section of the report considers the impact of DFID’s programmes on the intended beneficiaries in the education, health and humanitarian sectors.

2.59 In the service sectors, the causal chain from aid intervention to improvements in health and education outcomes is necessarily a long one, involving changes to policies, budgets and institutions in order to raise the quality of service delivery. In both sectors, there have been some measurable improvements in services but evidence on final impact will take time to emerge.

Education

2.60 Education is a relatively new programming area for DFID Pakistan but is already showing promising early results. DFID’s support for the Pakistan Education Task Force and the Punjab Education Sector Roadmap has helped to capitalise on a renewed interest on the part of Pakistan’s political leaders in improving service delivery. These programmes have helped to initiate an informed public debate on the ‘education emergency’ and, during the current pre-election period, strengthened political commitment to addressing it. DFID will need to continue to work hard to keep this momentum going in the post-election period.

2.61 Despite the impact of devastating floods, public expenditure on education in Punjab has increased in every year of PESP, from 59 billion rupees in 2006-07 to 92 billion rupees in 2010-11.⁵⁰ This is an important indicator, suggesting that sector budget support has not displaced national expenditure. We consider this a precondition for any scaling up of UK aid to the sector. Other results indicators are as follows:

• Enrolment: school enrolment in Punjab has been on an upward trend since the start of PESP in 2009, with enrolment targets regularly exceeded and revised upwards. In 2006, the number of children enrolled at primary level totalled 9.1 million, with 4.9 million boys and 4.2 million girls in school between grades 1 and 5.⁵¹ The most recent survey in June 2012 indicates that up to a million additional Punjabi children have been brought into the education system.⁵² The data are still being reviewed but this could signal a reduction of nearly a fifth in the number of out-of-school children. The net primary enrolment ratio has reached 70%, ahead of the PESP target of 66%, while middle-level enrolment (ages 11 to 13) has increased from 20% in 2006-07 to 37% in 2010-11. We note that the increase in enrolment rates appears to be due largely to the growth in private schooling. The DFID-supported PEF gives full-cost subsidies to 1.1 million children in private schools;⁵³
• Gender equity: approximately 500,000 girl pupils are benefiting from stipends, leading to an increase of the female–male Net Enrolment Rate ratio at middle level in rural areas from 74% in 2006-07 to 81% in 2010-11;
• Pupil and teacher attendance: the Punjab Education Sector Roadmap has been active for 18 months. Over this period, student attendance has risen from 76% to 86% and teacher attendance has risen from 78% to 82%.⁵⁴ This means a significant increase in the hours of instruction – a key variable for learning outcomes;
• Introduction of lesson plans: some 700 lesson plans have been developed and are now being used by 50% of teachers. Feedback from the teachers we spoke to was that the lesson plans were helping to raise the quality of instruction. A concern has emerged, however, that the content of the lesson plans has diverged from newly developed school textbooks, causing difficulties for teachers;⁵⁵
• Improving school facilities: investments in school infrastructure have increased the proportion of schools with a full set of functioning facilities (electricity, clean water, toilets and boundary walls) from 60% to 86%. This has involved more than 40,000 separate investments, although the programme targets were missed as a result of budgetary constraints in the province;⁵⁶
• Primary completion: the primary school completion rate increased from 55% in 2009-10 to 64% in 2010-11;⁵⁷ and
• Examination performance: PESP has been supporting the introduction of a system of independent examinations by the Punjab Examination Commission. Examination results at the end of primary school (age 11) improved from an average mark of 275 out of 600 in 2011 to 321 in 2012,⁵⁸ with the improvement seen in both rural and urban areas. A recent evaluation, however, found that the examination results are not yet considered by provincial stakeholders to be reliable.⁵⁹ A sample-based survey by the Punjab School Education Department confirmed that the average proficiency of Grade IV students in core subjects had increased from ‘poor’ to ‘basic’.⁶⁰

2.62 These are strong results, suggesting that learning outcomes are improving against a low base. DFID provides only a small share of the financing for the PESP programme, equivalent to approximately 2.3% of the Punjab education budget. In simple terms, one could conclude that in 2011-12 the UK supported the education of 133,600 Punjabi primary school children. In this case, however, the level of attribution is likely to be higher than this figure would suggest. Our observation is that DFID is providing a high level of analytical input into the provincial reform programme. Its regular dialogue with senior political leaders in the Punjab, including through the Roadmap process, has made a significant contribution to the momentum of the reforms. This momentum, however, is clearly linked to the close personal attention given by the Chief Minister. The challenge for the future will be to ensure that these early results are institutionalised. We also note that the model may not be replicable in other provinces with a different political climate.

Health

2.63 The intended results of the MNH programme are reductions in maternal, child and infant mortality, achieved by increasing the rate of skilled birth attendance. Official data on mortality rates are collected through household surveys that are conducted irregularly (the last one was in 2006-07). There is, therefore, no data at present on the programme’s impact on mortality. Estimates by the United Nations Inter-Agency Group for Child Mortality, however, suggest that no province in Pakistan is likely to achieve its health-related MDGs by 2015.⁶¹

2.64 The programme set the target of increasing the proportion of births attended by skilled birth attendants (including births attended by CMWs and births in health facilities) to 60% in 2012-13, from a baseline of 36% in 2006-07. A recent survey found that skilled birth attendance in Punjab (both rural and urban) had increased from 43% in 2008-09 to 59% in 2011-12.⁶² This suggests that maternal and infant mortality rates are likely to have reduced substantially.

2.65 Given the range of other government and donor investments in increasing skilled birth attendance, the extent to which the MNH programme has contributed to these results is difficult to assess on present data. Since its inception in 2008, the programme has recruited 8,966 CMWs, of which 5,787 have so far been trained and 2,969 deployed.⁶³ This suggests that the programme is on track to achieve its original target of 12,000 CMWs deployed by the end of 2012-13.

2.66 With each CMW serving a population of 10,000, the results to date theoretically give nearly 30 million people improved access to skilled birth attendance. We note, however, that according to the provincial co-ordinators, CMWs in Punjab are attending on average only 3.2 deliveries each month.⁶⁴ Given that a population of 10,000 will average 22 deliveries per month; this means that CMWs are covering only around 13% of deliveries in their communities. Tensions in the relationship between the CMWs, Lady Health Workers and traditional birth attendants may be contributing to this low utilisation rate. While the rate may increase in the future as the CMWs become better established in their communities, at present the value for money of this investment appears to be low. We note that CMWs also perform other duties, including the provision of care during pregnancy and referral of difficult cases to health facilities. The latter is particularly important for reducing mortality. No data, however, are being collected on the number and types of cases referred by the CMWs.

2.67 The programme has also invested in increasing the availability of emergency obstetric and newborn care (EmONC) services by upgrading facilities at district and local hospitals. It targeted 112 district hospitals and 122 local hospitals for investment, with the goal of introducing comprehensive, 24-hour EmONC services in 70% of them by 2010-11 and 95% of them by 2012-13. A 2011 Health Facility Assessment commissioned by the programme found that 76% of district hospitals and 81% of local hospitals now have the necessary facilities.⁶⁵ This indicates that the planned outputs for 2011 were achieved.⁶⁶

2.68 We found, however, that provincial MNH managers were not able to verify that 24-hour services were in fact available or to provide us with figures on their utilisation. The Health Facility Assessment also noted a series of problems that are likely to affect service quality. Many of the health facilities are severely understaffed, particularly in gynaecologists, anaesthetists and paediatricians. Equipment was deficient in many of the facilities and none had been equipped with the full set of drugs and supplies required to deliver comprehensive EmONC services. It appears that investments in upgrading facilities under the MNH programme have been partly undermined by a lack of progress on other aspects of health service delivery, including personnel and budgets.

2.69 From the data available, it appears likely that the MNH programme has made a contribution to improving mortality but problems with design and implementation suggest that the contribution is not as large as it should have been. We also have some concerns about the sustainability of the results, given that the CMW cadre for the time being rests on uncertain institutional and financial foundations. We welcome DFID’s shift of focus towards provincial health policies and systems, which have the potential to create more sustainable MNH services.

2.70 The results of the MNH programme have inevitably been affected by the uncertainties surrounding devolution. DFID is using this period of uncertainty to build up its relationships with health authorities in Punjab and KP Provinces, in preparation for its planned provincial health programme. This has included working with Punjab Province to develop its first provincial health strategy. It has also helped to develop minimum service delivery standards, now being trialled in 17 districts. These are important foundations for its future provincial engagement. At the national level, DFID has worked to promote better knowledge and recognition of the scale of malnutrition in Pakistan, including funding a national nutrition survey by UNICEF. Nutrition will be one of the themes of its future health support.

2.71 The Technical Resource Facility, which funds technical assistance to the MNH programme and to provincial health authorities, reports that 16 of its recommendations have been taken up by government. It played a useful role in supporting the development of provincial health strategies and developing minimum service delivery standards for primary health care, which are currently being piloted in 15 districts. A recent mid-term review, however, found little evidence of impact in terms of strengthening provincial health systems or management capacity.⁶⁷ This will need to be a more explicit focus of future support.

Humanitarian

2.72 DFID’s 2011 flood response appears to be delivering good results for the size of the investment. Although final figures will not be available until the projects are completed, all three appear to be on track to reach their initial targets. The results will therefore include at least 28,500 families provided with temporary shelter and roofing kits, 27,000 families receiving cash for the construction of a one-room house and 23,500 families supported in the production of a sunflower crop.⁶⁸ Additional results include knowledge transfer to the beneficiary communities on flood-resistant building techniques and ways to maximise the yield on their sunflower crops. The beneficiaries that we consulted confirmed that they had achieved yields of 700-800kg per acre, compared to only 300-400kg in previous years, with an average return of £275 per acre.

2.73 Through the CDCP, 787,691 households (or approximately 5 million individuals) have been supported with cash payments equivalent to £275 as of May 2012. DFID provided approximately 17% of the programme budget, covering 138,000 households. DFID’s interventions also ensured that the beneficiary list included 22,635 disabled heads of households and 132,033 women heads of households. Surveys suggested that beneficiaries spent the cash on a wide variety of needs, with food, repair of housing, debt repayment and medicines the most prominent. Our consultations with beneficiaries suggested that they preferred the flexibility of receiving cash.

2.74 DFID has a full impact evaluation underway to give a more complete picture of the impact of cash transfers. Early evidence, however, suggests that they increased the range of coping strategies available to the beneficiaries. As compared to non-beneficiaries, they borrowed less from relatives and were less likely to reduce their food consumption.⁶⁹ Our only concern is that high levels of indebtedness among rural communities (see paragraph 2.78 on page 21) may undermine the impact of the support.

2.75 The WATAN card electronic payment mechanism (see Figure 5 on page 15), with the many safeguards that DFID has helped to introduce, is another important outcome. This potentially improves the Government of Pakistan’s capacity to respond to future disasters. We have some doubts, however, about the sustainability of this investment. While strong capacity has been built up around the CDCP, it appears to be in temporary or ad hoc institutional forms. These were not utilised by the Government of Pakistan for its subsequent cash transfer programme in response to the 2011 floods (the ‘Pakistan card’). It is important that DFID capitalises on its investment by helping to build a mechanism that will be used for future cash transfer schemes. We applaud the rigorous checks and balances built into the mechanism. We would, however, suggest reviewing the different elements for cost-effectiveness, with a view to maximising their likelihood of being taken up by the Government of Pakistan.

2.76 While DFID’s 2011 flood response appears to have been effective, we are concerned at shortcomings in the wider humanitarian response and what they indicate for Pakistan’s capacity to handle future disasters. The UN’s funding appeal for post-emergency reconstruction (the Early Recovery Framework) remains only 11% funded as of August 2012 (much of which is the UK funding).⁷⁰ This is due in large part to poor relations between the Government of Pakistan and the UN, which led to delays in issuing the emergency appeal and disputes over the content of the Early Recovery Framework. As a result, the humanitarian response to the 2011 floods is likely to be inadequate, leaving a legacy of increased poverty and vulnerability in Sindh Province.

2.77 It is disappointing to see that, after so many years of international investment in emergency response capacity in Pakistan, confusion over institutional mandates is still undermining the impact of humanitarian operations. DFID informed us that it had been instrumental in arranging for a senior UN official from New York to visit Pakistan. This initiated a process of dialogue on basic principles of engagement between the UN and the Government of Pakistan. It is vital that clear ground rules be established for responding to future disasters.

2.78 DFID’s humanitarian projects involve some elements of building resilience to future disasters (e.g. better building techniques). The communities we visited, however, clearly stated that they were no better prepared for future flooding. Early warning systems are still rudimentary and district-level disaster management structures are largely dysfunctional. Communities themselves are poorly organised. With more flooding highly likely in the coming years, we urge DFID to press ahead with its plans to mainstream resilience into its Pakistan programme.

2.79 One of the factors we observed in our field visits was the very high level of indebtedness of rural communities. Many Pakistani farmers informed us that they need to borrow from local money lenders to purchase inputs for their crops, reportedly often on extortionate terms. The floods had caused this indebtedness to spiral, particularly where crops had been destroyed in the field. For some of the highly indebted households we interviewed, a cash transfer or standard package of livelihood support may not be sufficient to lift them out of a crisis situation. Livelihood programmes to reduce indebtedness might help to build the resilience of rural communities and households to future natural disasters. This is taken up further in paragraph 3.19 in the following section.

Learning

Assessment: Green-Amber

2.80 This section of the report looks at how well DFID measures the results of its programming and whether it is learning from experience.

Monitoring the Punjab Education Reform Roadmap

2.81 The Punjab Education Reform Roadmap is an excellent example of how a well-designed monitoring system can be integral to the design of a reform programme. A team led by Sir Michael Barber, DFID’s Special Representative for Education in Pakistan, facilitates a stock-take every two months. It uses data generated by the provincial government’s own monitoring system, which includes around a thousand monitoring officers who visit each of Punjab’s 59,000 public schools every two months. The inspectors are regularly rotated across districts to minimise the risk of false data. They collect data on key indicators such as enrolment and student and teacher attendance.

2.82 One of the outputs of the stock-take is a ‘heat map’ of school performance by district (see sample in the Annex, Figure A5). This provides the Chief Minister of the Punjab with a real-time picture of the performance of the education system across the province.

2.83 Breaking down results by district has a number of advantages. It draws attention to the continued underperformance of schools in the rural south – a result of inequitable patterns of resource allocation, as well as more challenging social conditions. This provides a platform for dialogue with the Chief Minister on how these imbalances in resource allocation can be addressed. By highlighting stronger and weaker districts, it helps the programme to highlight problems that need to be addressed and promising innovations that could be replicated. We would like to see the programme undertaking more qualitative analysis of the reasons underlying performance variations across districts.

2.84 Finally, the Roadmap is a very strong tool for promoting accountability. Officials from high-performing districts are recognised with an invitation to meet the Chief Minister, while officials from lagging districts can face censure or even dismissal. The district officials we met confirmed that this creates a strong set of incentives to improve their performance.

2.85 One weakness of the monitoring system is the lack of data on private schools, even those supported by the PEF. This leaves the province dependent on household surveys to capture data on enrolment and educational attainment. The programme should explore ways of encouraging private schools to register with and report to the provincial authorities.

Other monitoring arrangements

2.86 In the humanitarian portfolio, we found that all the partners had adequate monitoring and evaluation systems in place. DFID’s monthly value for money reporting framework (see paragraphs 2.48 to 2.51 on pages 15 and 16) provided it with real-time information on unit costs and results during the 2010 flood response. This allowed it to adjust its programming as necessary. We saw good evidence of learning from the 2010 flood response, both in terms of the design of individual interventions (see Figure 6) and DFID’s own systems for selecting and managing its partnerships. For example, the 2010 floods helped to expose the extent of the underlying nutrition problem in Pakistan. DFID Pakistan has now recruited a nutrition adviser and is working to make nutrition a theme running across its portfolio.

2.88 In the absence of an adequate management information system, it is difficult to determine whether the MNH programme’s investments in upgrading EmONC services in local and district hospitals are having the desired impact. We also have some concerns as to the interest in and capacity of those managing the programme to make use of results data. Our enquiries of the MNH Provincial Programme Management Unit suggested that basic information on the programme was not readily available and that results data were produced as a formal requirement. The data were not there to support management decisions. This may reflect the transitional nature of the programme but it suggests that the new provincial health programmes should focus on aligning with provincial information systems and ensuring that results data are used to inform policy and management decisions.

3 Conclusions and Recommendations

Conclusions

3.1 We have found many commendable aspects to DFID’s Pakistan programme. DFID has an energetic and committed country team with strong management structures. Management encourages a high degree of innovation, both in programming and in corporate systems. The Pakistan programme is piloting a series of new initiatives.

3.2 We saw strong programme designs in a number of areas, particularly education, although they remain untested. We applaud the use of complementary interventions and aid modalities. We are particularly pleased with the way that large-scale financial support in education is supported by additional investment in technical support, knowledge generation and mobilising community support for improved services.

3.3 We have significant concerns with the design and implementation of the current health programme. We believe that these should have been addressed in a more timely fashion. If this programme were being scored separately, it would rate an Amber-Red (see Figure A1 in the Annex for a breakdown of our scoring by sector). We acknowledge that the health portfolio has been heavily affected by the devolution process. We acknowledge also that DFID has since chosen to redesign its health approach entirely, rather than focus on correcting problems with the current support. Some useful foundations have been laid for the next phase of assistance, including a new provincial health strategy. We applaud DFID’s efforts at bringing the nutrition crisis into the national policy debate.

3.4 The humanitarian response shows good evidence of learning from 2010, a rigorous set of fiduciary controls and commendable attention to value for money. The implementing partners are effective and their interventions well designed. We are, however, concerned about the poor state of Government of Pakistan/UN relations and unresolved institutional mandates for disaster response. While these factors are not under DFID’s direct influence, they have led to serious underfunding of the recovery effort, detracting from the overall impact of DFID’s humanitarian support.

3.5 We support the development of a standing mechanism for cash transfers to support early recovery – an approach that seems well suited to the Pakistani context. Some care needs to be taken to make sure that the innovations introduced into the mechanism by DFID are taken up by the Government of Pakistan in future cash transfer programmes.

3.6 General: a key question arising from this review is whether DFID Pakistan has the systems and delivery channels in place to absorb the unprecedented scaling up of the country programme. Most of this is occurring in 2012-13. Pakistan is – and is likely to remain – a difficult environment in which to provide aid effectively. The political context remains volatile, with an election in the coming year. Devolution has radically changed the institutional landscape and it will take some years to work through the reassignment of resources and responsibilities. The difficult security environment limits DFID’s ability to reshape the country team to support provincial-level engagement. National budget processes are unreliable, PFM systems are weak and corruption risk is high, which makes directing aid through country systems hazardous.

3.7 We found DFID Pakistan to be well informed about the risks and to be piloting a number of innovative approaches to managing them. We welcome the introduction of ‘continuous audit’ to provide real-time monitoring of fiduciary risks and facilitate timely intervention. We also welcome the emphasis on ‘additionality’, whereby UK funds are only released when the Government of Pakistan has released its planned contribution to spending programmes.

3.8 Nonetheless, DFID Pakistan is now venturing into unknown territory. The country programme has no track record of successful delivery of programmes on the scale now contemplated. This means that DFID Pakistan will need to devote a commensurate level of effort into managing the risks around scaling up.

3.9 We have some concerns about the predominance of large service delivery programmes in the planned scaling up. Over the next three years, large investments in health and education will comprise two thirds of the programme. Clearly, given the UK Government’s strategic objectives in Pakistan, building the service delivery capacity of the state must be an important part of the programme. This means working with and through country systems, so as to help strengthen provincial institutions and budgetary processes. There is, however, a long process of challenging institutional reforms to be undertaken before we can be confident that funds channelled through the budget are being used efficiently.

3.10 We would suggest, therefore, that investments in public services are balanced by investments through other delivery channels. This is in order to spread risk and maximise the potential for impact in an uncertain environment. Our first three recommendations – on flexible funding levels for individual programmes, on developing options for non-state service delivery and making resilience a major programming area – go in the direction of diversifying the programme. The fourth relates to managing residual risk in service delivery programmes.

Recommendations⁷¹

3.11 We have the following broad recommendations for the DFID Pakistan country programme. More operational recommendations for our three focus sectors are included in the Annex in Figure A2.

Recommendation 1: To manage the risks associated with scaling up funding in a difficult and volatile environment, DFID Pakistan should ensure that the conditions for scaling up its programmes are clearly articulated and that it retains the flexibility to reallocate funding away from underperforming areas. The country programme should also be better balanced across government and non-government delivery channels.

3.12 The country programme needs the flexibility to reallocate resources away from poorly performing programmes towards more productive uses. It is important that it does not become locked into a small number of very large investments whose effectiveness will take years to become apparent. The design of the larger programmes should clearly indicate the underlying conditions for success, such as political and macroeconomic stability and level of Government of Pakistan commitment and financial support. The programmes should be designed around different funding scenarios, with the triggers for moving from lower to higher levels of funding clearly articulated. Likewise, DFID Pakistan should be ready to scale back or terminate individual programmes where the conditions for success no longer pertain. We also recommend strict adherence to the principle of ‘additionality’, with UK funds only being released once it is clear that Government of Pakistan spending commitments are being met.

Recommendation 2: DFID should increase its support for promoting affordable and equitable private sector delivery of health and education services to poor communities. It should focus on building government capacity to regulate the private sector, improve standards and monitor their delivery.

3.13 We welcome DFID Pakistan’s willingness to engage with the low-cost private sector in the education field. The private sector is currently generating significantly better education outcomes than the public system. It is therefore likely to be a key element in reaching the MDGs in education.

3.14 There is some danger, however, that a two-tier system of education might emerge, to the disadvantage of children from poorer backgrounds. It is important that the provinces begin to assume a policy-making, regulatory and monitoring role over the private sector, ensuring, for example, that quality standards are met and the national curriculum followed. Regulation should be introduced in ways that create positive incentives for private schools to register and comply with national standards. The provision of stipends for pupils from poor households attending private schools is potentially a useful tool both for addressing equity concerns and for creating those incentives.

3.15 Engaging with the low-cost private sector and developing effective state regulation of private education is relatively new territory for DFID. Pakistan would seem to offer a good opportunity for DFID to develop new polices and programming approaches in this area. We understand that the private sector also plays a significant role in health service delivery and that the issue is being explored as part of the design of the new provincial health programmes.

Recommendation 3: Building resilience to natural disasters at the household and community levels should become a core element of the DFID Pakistan programme. This also provides an opportunity to diversify delivery channels.

3.16 Pakistan has had seven major humanitarian disasters in the past 12 years. These have had a major effect on poverty, vulnerability and resilience, both through their direct impact on households and through their disruption of national development programmes. Pakistan needs to break out of this cycle of disaster and emergency response by making a major, sustained investment in reducing disaster risk and improving resilience.

3.17 DFID has committed itself internationally to taking a leadership role on resilience. It recognises that building resilience is a multi-dimensional challenge that needs to be integrated across DFID’s portfolio, requiring ‘a step change in DFID’s development work’.⁷² The Pakistan programme would seem to offer an opportunity to put this commitment into practice.

3.18 DFID Pakistan has plans to introduce resilience as a cross-cutting theme in its programmes and has begun a programme of analytical work to assess how best to do this. We welcome this commitment. Indeed, we recommend that DFID goes further and makes building resilience a substantial programming area. This is an important part of managing the risks around scaling up. Unless there is a dramatic improvement in Pakistan’s capacity to manage disaster risk, humanitarian emergencies will continue to be a major brake on national development. We also see investing in resilience through NGO channels as a means of diversifying the country programme away from excessive reliance on public sector delivery.

3.19 DFID documents recognise that resilience to disaster can take many forms, from improved macroeconomic management to investments in physical infrastructure and helping local communities become better equipped and organised. In Pakistan, we would encourage DFID to explore resilience at the local community and household levels, through initiatives that bring together its wealth creation, local governance and social safety net programmes. This should include a focus on rural livelihoods, using NGOs as the delivery channel. In our recent evaluation of DFID’s recent humanitarian response in the Horn of Africa,⁷³ we were impressed by DFID’s approach to promoting resilience among remote communities. We recommended that DFID scale up its efforts to address chronic poverty to reduce vulnerability to natural disasters. We would encourage a similar approach in Pakistan. For example, we observed that levels of indebtedness in rural communities were a significant factor in their vulnerability to disasters and shocks. Programmes that focussed on building up sustainable alternatives to local money lenders would help to promote resilience.

Recommendation 4: When funding through country systems, DFID programmes should include agreed standards for budget integrity as a condition of scaling up. There should be a major emphasis on promoting transparency and accountability in budget processes and on increasing the amount of beneficiary involvement in the delivery and oversight of programmes.

3.20 When funding through country systems, in view of the difficult environment and the unprecedented size of its planned investments, DFID will need to take a rigorous approach to fiduciary risk management. We welcome the introduction of continuous audit, which should be employed systematically across the programme. In addition to the continued use of fiduciary risk assessments, we would expect to see robust analysis of the constraints on effective and efficient spending through country systems (the forthcoming Public Expenditure Review in Punjab is an example). The design of each major programme should include a combination of short-term mitigation measures and long-term reforms to address those risks and constraints as far as is possible. DFID should continue to increase its level of supervision of its programmes in proportion to their size and risk profile. It should keep its own staffing levels and skill profile under review, to ensure that it is equipped to supervise effectively.

3.21 We welcome DFID’s exploration of options for boosting transparency and accountability around budget formulation and expenditure in education. We suggest that this be a major part of its approach to fiduciary risk mitigation. Examples might include supporting the Pakistani media and civil society at national, provincial and local levels to undertake budget monitoring. This is particularly important in provinces where direct access by DFID staff is limited.

3.22 Funding agreements with government institutions should clearly specify expected standards for budget reliability, including ensuring that budget commitments for priority programmes are met. Where continuous audits and tracking surveys reveal that funds are not being used effectively, we would expect to see a swift and robust response from DFID, including a willingness to return to lower funding levels.

Abbreviations

AusAID – Australian Agency for International Development

CDCP – Citizen Damage Compensation Programme

CIDA – Canadian International Development Agency

CMWs – Community Midwives

DFID – Department for International Development

EmONC – Emergency Obstetric and Newborn Care

FATA – Federally Administered Tribal Areas

IOM – International Organization for Migration

KP – Khyber-Pakhtunkhwa

MDGs – Millennium Development Goals

MNH – Maternal and Newborn Health

NGO – Non-governmental organisation

OCHA – Office for the Coordination of Humanitarian Affairs

PEF – Punjab Education Foundation

PESP – Punjab Education Support Programme

PFM – Public Financial Management

TRF – Technical Resource Facility

UN – United Nations

USAID – United States Agency for International Development

VFM – Value for Money

Footnotes

¹ See http://data.worldbank.org/country/pakistan.

² Pakistan Economic Survey 2007-08 for Pakistan 1998-99 – 2005-06, Government of Pakistan, 2008. The Pakistan national poverty line is based on a dietary intake of 2,350 calories per person per day or 673 rupees (£8.28) per month at 1998-99 prices.

³ Estimates following the 2010 floods: DFID-Pakistan Core Briefing: Programme Overview, DFID, January 2012.

⁴ See http://undp.org.pk/mdgs-in-pakistan.html.

⁵ World Health Organization, UNICEF, UNFPA and the World Bank, Trends in Maternal Mortality: 1990 to 2010, WHO, 2012, page 22, http://www.unfpa.org/webdav/site/global/shared/documents/publications/2012/Trends_in_maternal_mortality_A4-1.pdf.

⁶ Figures compiled from UN Appeal documents and Financial Tracking Services or provided by DFID.

⁷ Punjab, Sindh, Khyber-Pakhtunkhwa and Balochistan. Pakistan also has two autonomous territories and a group of Federally Administered Tribal Areas (FATA).

⁸ See http://cpi.transparency.org/cpi2011/results/#CountryResults.

⁹ Pakistan: Public Expenditure and Financial Accountability Assessment of Federal Government, Government of Pakistan and Development Partners, June 2009, https://openknowledge.worldbank.org/bitstream/handle/10986/2862/486520ESW0Whit1FA010Printed0Version.pdf?sequence=1.

¹⁰ International Monetary Fund lending programmes are provided for the purposes of restoring macroeconomic stability and contain conditions on macroeconomic policy that must be complied with for the lending programme to continue.

¹¹ The Pakistani Muslim Community in England, Department for Communities and Local Government, March 2009, page 29, www.communities.gov.uk/documents/communities/pdf/1170952.pdf.

¹² Operational Plan 2011-2015, DFID Pakistan, February 2011, page 2, http://www.dfid.gov.uk/Documents/publications1/op/pakistan-2011.pdf.

¹³ Solid bars represent actual expenditure; outline bars represent projected expenditure.

¹⁴ In the event of another major humanitarian emergency, DFID can make use of any unspent funds from the country programme or West Asia region and, if those are insufficient, may draw on DFID’s Central Contingency Reserve. The Secretary of State has authority to underwrite the cost of major humanitarian operations from other unspent funds.

¹⁵ We visited: 14 flood-affected villages and a WATAN centre, talking to 20-100 beneficiaries in each in group meetings as well as individual interviews (more than 500 people in total); four schools, where we spoke to four head teachers, 15-20 teachers, 15-20 parents and 50-60 pupils; and a nursing school (four instructors, 20 Community Midwife trainees), a local hospital (3-4 doctors; 5-10 patients), a community midwife in her community and around ten local women.

¹⁶ Population figures shown are the preliminary results from the 2011 census: www.census.gov.pk/census2011.php.

¹⁷ Macro Trends in Financing of Education in Pakistan: An Analysis of Public Sector Allocations and Expenditures, UNESCO, 2011, http://unesdoc.unesco.org/images/0021/002169/216912e.pdf.

¹⁸ As declared by the joint Government of Pakistan/civil society Education Task Force: http://pakobserver.net/detailnews.asp?id=85099.

¹⁹ Human Development Report. Sustainability and Equity: A Better Future for All, UNDP, 2011, http://hdr.undp.org/en/reports/global/hdr2011.

²⁰ EFA Global Monitoring Report 2010: Reaching the Marginalized, United Nations Educational, Scientific and Cultural Organization, Paris, http://www.unesco.org/new/en/education/themes/leading-the-international-agenda/efareport/reports/2010-marginalization.

²¹ Annual Status of Education Report (Rural), South Asian Forum for Education Development, 2011, http://www.aserpakistan.safedafed.org/document/aser/2011/ASERPakistan2011.pdf.

²² World Health Organization Global Health Expenditure Database: http://apps.who.int/nha/database/DataExplorerRegime.aspx.

²³ World Health Organization, UNICEF, UNFPA and the World Bank, Trends in Maternal Mortality: 1990 to 2010, WHO, 2012, page 22.

²⁴ Heltberg, R. and Lund, N., Shocks, Coping and Outcomes for Pakistan’s Poor: Health Risks Predominate, Journal of Development Studies 45/6, 2009, pages 889-910, citing data from the Pakistan Social and Living Standards Measurement Survey.

²⁵ Humanitarian Dashboard Pakistan Floods 2011, UNOCHA, April 2012, http://reliefweb.int/sites/reliefweb.int/files/resources/Full%20Report_737.pdf.

²⁶ DFID Cash Transfers Evidence Paper, DFID Policy Division, 2011, http://webarchive.nationalarchives.gov.uk/+/http://www.dfid.gov.uk/Documents/publications1/cash-transfers-evidence-paper.pdf.

²⁷ The programme was delayed in Sindh Province partly as a result of the need to repeat surveys of housing damage (one of the beneficiary selection criteria) after independent checking had revealed discrepancies.

²⁸ DFID pressed for the inclusion of households headed by women and disabled people in the beneficiary selection criteria.

²⁹ DFID made a substantial investment in building the capacity of the Earthquake Reconstruction and Rehabilitation Authority following the 2005 earthquake but is not currently active in capacity-building on disaster management.

³⁰ Evaluation of DFID Country Programmes: Country Study: Pakistan Report, DFID, April 2008, page 65, http://www.dfid.gov.uk/Documents/publications1/evaluation/ev687.pdf.

³¹ For a review of the evidence, see the DFID-commissioned research summarised in Hussein, J. et al., Systematic Review: What Kinds of Policy and Programme Interventions Contribute to Reductions in Maternal Mortality?, June 2011, http://www.dfid.gov.uk/r4d/PDF/Outputs/SystematicReviews/Maternal-mortality-2011Hussein.pdf.

³² Programme Memorandum: DFID Support for Improving Maternal and Newborn Health in Pakistan, DFID Pakistan, August 2006, page 17.

³³ DFID’s Education Programme in Three East African Countries – Ethiopia, Rwanda and Tanzania, ICAI, May 2012, http://icai.independent.gov.uk/wp-content/uploads/2010/11/DFIDs-Education-Programmes-in-Three-East-African-Countries-Final-Report-3.pdf.

³⁴ Programme Evaluation of the Punjab Education Sector Programme (PESP), Semiotics, February 2012, page 132.

³⁵ Pakistan: Public Expenditure and Financial Accountability Assessment of Federal Government, Government of Pakistan and Development Partners, June 2009, pages 24-29, https://openknowledge.worldbank.org/bitstream/handle/10986/2862/486520ESW0Whit1FA010Printed0Version.pdf?sequence=1.

³⁶ Programme Memorandum: DFID Support for Improving Maternal and Newborn Health in Pakistan, DFID Pakistan, August 2006, page 4.

³⁷ Information extracted from documents provided by DFID.

³⁸ Financial data on the MNH programme provided by DFID.

³⁹ Pakistan Education Statistics 2008-09, Academy of Educational Planning and Management, Ministry of Education, 2009, http://aepam.edu.pk/Files/EducationStatistics/PakistanEducationStatistics2008-09.pdf.

⁴⁰ Andrabi, T. et al., Pakistan Learning and Educational Achievements in Punjab Schools (LEAPS): Insights to Inform the Education Policy Debate, February 2007, pages ix-x, http://leapsproject.org/assets/publications/LEAPS_report.pdf.

⁴¹ A survey conducted in 2000 found the average cost of a rural primary school was only 60 rupees or 40 pence per month: Andrabi, T. et al., Pakistan Learning and Educational Achievements in Punjab Schools (LEAPS): Insights to Inform the Education Policy Debate, February 2007, page iv, http://leapsproject.org/assets/publications/LEAPS_report.pdf.

⁴² Second Punjab Education Sector Project – Project Appraisal Document, World Bank, March 2012, page 2, http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/04/09/000333037_20120409233739/Rendered/PDF/662910PAD0P125000official0use0only0.pdf.

⁴³ Overview of Corruption in Pakistan, U4 Anti-Corruption Resource Centre Expert Answer, August 2008, http://www.u4.no/publications/overview-of-corruption-in-pakistan/.

⁴⁴ This was a recommendation by ICAI in our anti-corruption report: DFID’s Approach to Anti-Corruption, ICAI, November 2011, http://icai.independent.gov.uk/wp-content/uploads/2010/11/DFIDs-Approach-to-Anti-Corruption.pdf.

⁴⁵ How to Note: Managing Fiduciary Risk When Providing Financial Aid, DFID, June 2011, http://www.dfid.gov.uk/Documents/publications1/how-to-fiduciary-fin-aid.pdf.

⁴⁶ DFID’s Approach to Anti-Corruption, ICAI, November 2011, Recommendation 3, http://icai.independent.gov.uk/wp-content/uploads/2010/11/DFIDs-Approach-to-Anti-Corruption.pdf.

⁴⁷ Value for Money Strategy, DFID Pakistan, May 2012.

⁴⁸ Paragraph 1.18 introduces the projects and partners.

⁴⁹ Saving Lives, Presenting Suffering and Building Resilience: The UK Government’s Humanitarian Policy, DFID, September 2011, page 16, http://www.dfid.gov.uk/Documents/publications1/The%20UK%20Government%27s%20Humanitarian%20Policy%20-%20September%202011%20-%20Final.pdf.

⁵⁰ Data taken from Punjab Education Department’s Programme Monitoring and Implementation Unit and the Punjab Finance Department, quoted in DFID’s 2011 Annual Review of PESP.

⁵¹ National Education Census 2006-07, Government of Pakistan, 2007, page 11.

⁵² Advance results from survey provided by DFID.

⁵³ Second Punjab Education Sector Project – Project Appraisal Document, World Bank, March 2012, pages 2-3, www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2012/04/09/000333037_20120409233739/Rendered/PDF/662910PAD0P125000official0use0only0.pdf.

⁵⁴ Data taken from Punjab Education Department’s Programme Monitoring and Implementation Unit, quoted in the April 2012 stock-take for the Punjab Education Reform Roadmap.

⁵⁵ Programme Evaluation of the Punjab Education Sector Programme (PESP), Semiotics, February 2012, page 31.

⁵⁶ April 2012 stock-take for the Punjab Education Reform Roadmap.

⁵⁷ PESP Annual Review 2011, DFID, 2011.

⁵⁸ Data taken from Punjab Examination Commission Results for 2011 and 2012, quoted in the April 2012 stock-take for the Punjab Education Reform Roadmap.

⁵⁹ Programme Evaluation of the Punjab Education Sector Programme (PESP), Semiotics, February 2012, page 8; Improving Education through Large-Scale Testing? A Study on Primary and Elementary Level Exams in Punjab, Society for the Advancement of Education, 2011.

⁶⁰ District-Wide Large Scale Assessment: Results Report 2011, Government of Punjab School Education Department, 2011, http://www.peas.gop.pk/Data/AssessmentReports/PEASAssessmentReport2011.pdf.

⁶¹ Annual Review for MNCH, DFID Pakistan, August 2011.

⁶² Data from Multiple Indicator Cluster Surveys carried out by UNICEF, provided to the review team by the Punjab MNH Project Management Unit.

⁶³ Data from the Punjab MNH Programme Management Unit.

⁶⁴ Data from Punjab MNH Programme Management Unit.

⁶⁵ Health Facility Assessment – Punjab Provincial Report, MNH TRF, June 2012.

⁶⁶ Annual Review for MNCH, DFID Pakistan, August 2011.

⁶⁷ Mid-term Review of Technical Resource Facility, Pakistan, Mannion Daniels Ltd., May 2012, pages 33-37.

⁶⁸ A £10 million increase in DFID funding to the three projects approved subsequent to our visit has led to an increase in the target results to 43,500 temporary shelter and roofing kits, 45,500 one-room shelters and 85,050 packages of livelihoods support. Information from DFID Pakistan.

⁶⁹ Data provided by DFID from baseline surveys carried out for an impact evaluation of the CPCD programme.

⁷⁰ Pakistan Early Recovery Framework 2012, UN OCHA, 2012, http://fts.unocha.org/reports/daily/ocha_R32_A966___27_August_2012_(02_05).pdf.

⁷¹ Operational recommendations on the programmes we examined are included in the Annex.

⁷² Saving Lives, Presenting Suffering and Building Resilience: The UK Government’s Humanitarian Policy, DFID, September 2011, pages 10-11, http://www.dfid.gov.uk/Documents/publications1/The%20UK%20Government%27s%20Humanitarian%20Policy%20-%20September%202011%20-%20Final.pdf.

⁷³ DFID’s Humanitarian Emergency Response in the Horn of Africa, ICAI, September 2012, http://icai.independent.gov.uk/wp-content/uploads/2012/09/ICAI-report-FINAL-DFIDs-humanitarian-emergency-response-in-the-Horn-of-Africa1.pdf.