Evaluation of DFID’s health programmes in Burma

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We have decided to review the impact and value for money of DFID’s aid programme directed at improving health outcomes in Burma, including the health components of humanitarian programmes. We will assess how well DFID manages the delivery of its assistance in a difficult environment, including assessing the relative merits of different delivery channels. We will review the extent to which lessons learnt from past assistance have informed future programming choices. We will also consider the extent to which DFID’s health programming contributes to promoting peace-building and state-building in Burma. These Terms of Reference outline the purpose and nature of the review and identify its main themes. A detailed methodology will be developed during the inception phase.

2. Background

Country context

2.1 Burma (also known as Myanmar) is the second-largest country in South-East Asia, sharing borders with Thailand, Laos, China, India and Bangladesh. It has a population of just over 60 million.¹ Burma was under military rule from 1968 to 2011, which included extended periods of armed conflict between the Government of Burma and a number of domestic insurgencies. More than six decades of political unrest and armed conflict have displaced an estimated 500,000 people to the east of the country and around 140,000 Burmese refugees are currently living in camps in Thailand.

2.2 Elections in November 2010, although described by independent observers as falling well short of international standards,² marked the beginning of a political transition. A largely civilian Parliament was convened in April 2011 and has since enacted a series of economic and political reforms. In early 2012, ceasefire agreements were signed with all but one of the insurgent groups. Recent months, however, have seen a significant increase in violence between different ethnic and religious communities living in Rakhine State.

2.3 Burma is one of the poorest countries in Asia. Decades of isolation and poor economic management have resulted in an under-developed private sector and very limited public spending on basic services. The population remains dependent on agriculture, which is mainly at subsistence level. Although poverty data are unreliable, a 2010 UNDP survey suggested that a quarter of the population lacks the resources to meet basic needs.³

2.4 Burma’s health indicators are among the worst in Asia and the country is unlikely to meet its health-related Millennium Development Goals by 2015 without a significant improvement in service provision for the most vulnerable people. The average life expectancy is 64 years, while the child mortality rate is 66 per 1,000 births.⁴ The country has one of highest burdens of disease from malaria, tuberculosis (TB) and malnutrition in the world. A strain of malaria that is resistant to the newest drug treatments has emerged along Burma’s eastern border, presenting a threat to malaria-control efforts in the region and, indeed, globally.

2.5 Burma’s public health system is in a weak condition. The World Health Organisation has identified the six building blocks of a functional health system as: health financing; health planning and management; a well-performing health workforce; infrastructure, drugs and supplies; health information systems; and leadership and governance. The health system in Burma displays significant challenges in all of these areas, undermining its capacity to deliver basic health care, particularly to the poorest and most vulnerable.

DFID’s work in Burma

2.6 Until recently, the UK has been bound by a European Union Council Decision that all development assistance to Burma must be implemented through UN agencies, non-governmental organisations (NGOs) and local civilian administrations, rather than the central government.⁵ Over the past decade, UK assistance to Burma has mainly been delivered in the form of humanitarian aid, in particular £46 million in response to Cyclone Nargis in 2008-09.

2.7 In April 2012, the restriction on providing development support to the Government of Burma was lifted by the EU, in recognition of progress on political reform.⁶ While DFID has re-established dialogue with the Government of Burma, it does not provide any funds through their systems. In 2011-12, DFID spent £36.3 million through the country programme, of which 14.2% was spent on health and 10% on humanitarian assistance. The country programme is projected to grow to £56 million in 2014-15.

Health programmes in Burma

2.8 DFID’s overall health goal for Burma is ‘to address the basic health needs of the poorest and most vulnerable and maximise the contribution of the programme to longer-term change that addresses the root causes of conflict and fragility in Burma’.⁷ It has three strategic objectives for its health programming:

• improve reproductive, maternal, newborn and child health and reduce the communicable disease burden;
• maintain a special focus on the global health threat posed by drug resistant malaria in Burma through a regional response, the 3MDG Fund⁸ and support for critical gaps in the response; and
• address targeted humanitarian health needs of refugees and internally displaced and other conflict affected populations where there are critical gaps in the response.

2.9 Figure 1 on page 4 summarises DFID’s health investments. The largest of these are channelled through UN-managed multi-donor trust funds. From 2007 to 2012, DFID provided £34.1 million ($57.1 million) to the Three Diseases Fund, which was 41% of the total budget of $138 million. This supported the prevention, treatment and care of HIV/AIDS, TB and malaria for the most vulnerable population groups. According to DFID, by the end of 2011, the Three Diseases Fund had: distributed nearly 60 million condoms and over 13 million needles to prevent HIV infection; provided 22,000 people with access to anti-retroviral treatment for HIV; diagnosed 180,000 people with smear-positive TB; and diagnosed and treated 1.8 million people for malaria.⁹

2.10 That fund is now being replaced by an expanded successor, the Three Millennium Development Goals (3MDG) Fund, managed by the United Nations Office for Project Services (UNOPS), to which DFID is contributing £40 million in the period 2012-14 and up to £40 million more in 2014-16. The 3MDG Fund will provide maternal and child health services for the poorest and most vulnerable and continue prevention and treatment for HIV/AIDS, TB and malaria for groups and areas not covered by Global Fund programming (see paragraph 2.13). The total budget for the 3MDG Fund is predicted to be £180 million, with contributions from Australia, the European Union, Sweden, Denmark, the Netherlands and Norway.

2.11 DFID has provided £5 million between 2009 and 2013 to another UN-managed trust fund, which provides maternal and child health services to five townships affected by Cyclone Nargis. It is also providing an accountable grant of £11.3 million, between 2011 and 2014, to the NGO Population Services International, to improve access to quality assured anti-malarial drugs in the Burmese health system. Another accountable grant of £3.3 million is to the NGO Health Poverty Action, to support maternal and child health for poor minority communities in marginalised areas of Burma.

2.12 In addition, DFID has four smaller health interventions targeting conflict-affected areas of Eastern Burma. These are accountable grants ranging from £90,000 to £834,000 to NGOs and non-profit health clinics to provide health care to Burmese refugees and internally displaced persons along the Thai-Burmese border.

2.13 The other major donors in the health sector in Burma are the two large health funds, the Global Fund to Fight HIV, TB and Malaria (GFATM) and the Global Alliance for Vaccines and Immunization (GAVI), to which the UK contributes from its multilateral aid budget. GFATM returned to Burma in 2011. The current round of GFATM grants will provide up to US$305 million over five years, taking on and scaling up some of the programmes initiated by the Three Diseases Fund. GAVI is providing $34 million for health systems strengthening over 4 years from 2011.

2.14 There are few other significant donors to Burma’s health sector, although Japan has provided some support for TB and malaria programmes and plans to continue support for communicable disease control. In addition to supporting relief and recovery in areas affected by Cyclone Nargis, the United States Agency for International Development (USAID) provides support for maternal and child health programmes in the central part of the country. USAID regional programmes on HIV, TB and malaria also include some support for Burma.

3. Purpose of this review

To assess whether DFID is achieving impact and value for money in Burma through its bilateral aid to the health sector and to humanitarian programmes with a health component.

4. Relationship to other reviews

4.1 The Three Diseases Fund has been extensively evaluated, including a mid-term evaluation in 2009, a final evaluation in October 2012 and a range of other analytical work in preparation for the design of the successor fund. There was also a DFID Internal Audit review of the Three Diseases Fund in June 2012. The mid-term evaluation concluded that the Fund had functioned satisfactorily in a constrained political and operating environment where other programmes had failed. It had successfully increased service provision and access, although only modest health outcomes were directly attributable to its efforts.¹²

4.2 Although we have not yet seen the methodology for the final evaluation, we will make as much use as possible of the available evidence and findings on effectiveness and impact, in so far as these can be validated through our consultations with intended beneficiaries and other stakeholders. This will reduce the burden on stakeholders of an additional evaluation. It will also enable us to focus particular attention on whether the successor programme, the 3MDG Fund, is based on learning from the Three Diseases Fund.

4.3 The National Audit Office is currently carrying out a value for money review of DFID programming on malaria. Burma will be one of four country case studies, with a visit planned for December 2012. This will be too early for us to co-ordinate field work. We will, however, seek to avoid unnecessary duplication, including by consulting with them in advance on the scope and focus of their work and reaching an agreement with them on information sharing.

4.4 This evaluation will have three focus areas, which will make it distinct from other evaluations, both past and on-going:

• it will compare and contrast the different delivery channels used by DFID, looking particularly at the challenge of delivering effectively in an environment where access is constrained;
• it will assess how well the programmes contribute to DFID’s objective of addressing the root causes of conflict and fragility in Burma and whether they support wider peace-building and state-building processes; and
• it will assess whether lessons learned through past evaluations have led to improved programming choices over time.

5. Analytical approach

5.1 The review will focus on the effectiveness and impact of the DFID Burma health portfolio over the past five years by examining a sample of programmes with different funding channels and delivery partners. We will assess:

• the extent to which the programmes have achieved sustainable impact for the intended beneficiaries; and
• the effectiveness, efficiency and value for money of the different funding and delivery channels in delivering that impact, including their adaptability to the changing political context.

5.2 The evaluation will address a number of key themes, including:

• whether DFID’s approach to the control of communicable diseases is technically sound and based on learning about what approaches have worked in Burma and similar environments in the past;
• DFID’s approach to building sustainable public health services in an environment where funds cannot be channelled through government systems;
• how well the health programmes have contributed to addressing the root causes of conflict and instability in Burma (one of the goals of the health programme);
• how effective UN-managed multi-donor trust funds have proved as a funding channel in a politically constrained environment; and
• how effectively lessons learned from past health programming in Burma have been reflected in the design of future programmes.

6. Indicative evaluation questions

6.1 This review will use as its basis the standard ICAI evaluation framework and criteria, which are focussed on four areas: objectives, delivery, impact and learning. We will consider all the questions from our standard framework, with those highlighted below given particular emphasis. We may add further questions during the inception phase.

6.2 Objectives

6.2.1 Does the programme have clear, relevant and realistic objectives that focus on the desired impact?

6.2.2 Is there a clear and convincing plan, with evidence and assumptions, to show how the programme will work based on the lessons of previous programme implementation and the changing context of delivering aid in Burma?

6.2.3 Does the programme complement the efforts of government and other aid providers and avoid duplication?

6.2.4 Are the programme’s objectives appropriate to the political, economic, social and environmental context?

In view of the post-conflict situation in Burma, we add the following question:

6.2.5 Does the programme support peace-building by reducing marginalisation and strengthening state-citizen relationships?

6.3 Delivery

6.3.1 Does programme design and roll-out take into account the needs of intended beneficiaries?

6.3.2 Is there good governance at all levels, with sound financial management and adequate steps being taken to avoid corruption?

6.3.3 Do managers ensure the efficiency and effectiveness of the delivery chain?

6.3.4 Are risks to the achievement of the objectives identified and managed effectively?

6.3.5 Is the programme delivering against its agreed objectives?

6.3.6 Are appropriate amendments to objectives made to take account of changing circumstances?

6.3.7 Is the programme delivered in a conflict-sensitive manner? Does it manage political and conflict-related risk effectively?

6.4 Impact

6.4.1 Is the programme delivering clear, significant and timely benefits for the intended beneficiaries through improving health outcomes for the people of Burma?

6.4.2 Is the programme working holistically alongside other programmes?

6.4.3 Is there a long-term and sustainable impact from the programme?

6.4.4 Is there an appropriate exit strategy involving effective transfer of ownership of the programme?

6.4.5 Is there transparency and accountability to intended beneficiaries, donors and UK taxpayers?

6.5 Learning

6.5.1 Are there appropriate arrangements for monitoring inputs, processes, outputs, results and impact?

6.5.2 Is there evidence of innovation and use of global best practice?

6.5.3 Is there anything currently not being done in respect of the programme that should be undertaken?

6.5.4 Have lessons from previous programmes about objectives, design and delivery been fed into the design of successor programmes, both for the UN-managed trust funds and for UK-supported health programmes more generally? Specifically, have lessons relating to programme management capacity and effectiveness been taken into account in the planning of the Three MDG Fund?

7. Methodology

7.1 The methodology for this review will be developed during the inception phase. It will involve a number of elements, including:

• a literature review covering the Burmese country context, the past record of external assistance in Burma and relevant international experience on the health challenges addressed in the programme sample;
• a detailed examination of evidence on effectiveness and impact from past programme evaluations and reviews, including collecting available health statistics;
• a review of relevant DFID files and information systems, including financial information;
• meetings with DFID and other experts in the UK on relevant health programming issues and DFID’s policy framework on basic service delivery in fragile and conflict-affected countries;
• meetings in Burma with DFID, donor partners, national and local authorities (as appropriate), civil society organisations and other key informants; and
• visits to project sites and individual meetings and focus groups with intended beneficiaries in Burma.

7.2 The review will draw on results data generated by the monitoring and evaluation systems of the various programmes being reviewed, supplemented as appropriate with data from other sources and feedback from intended beneficiaries and other stakeholders.

8. Timing and deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The main phase of the review will take place during the first and second quarters of 2013, with a report published in the third quarter of 2013.

Footnotes

1. ADB factsheet: Myanmar, Asian Development Bank, 2012, http://www.adb.org/sites/default/files/pub/2012/MYA.pdf. ↩
2. The 2010 Burmese Elections: neither free nor fair, The Public International Law and Policy Group, November 2010, http://publicinternationallawandpolicygroup.org/wp-content/uploads/2011/02/PILPG_Burma_Elections_Monitoring_Report_8_November_2010.pdf. ↩
3. Operational Plan 2011-2015, DFID Burma, August 2012, http://www.dfid.gov.uk/Documents/publications1/op/burma-2011.pdf. ↩
4. Myanmar: health profile, World Health Organisation, 2012, http://www.who.int/gho/countries/mmr.pdf. ↩
5. European Council Decision 2010/232/CFSP of 26 April 2010 renewing restrictive measures against Burma/Myanmar, Official Journal of the European Union, L 105/22, 2010. ↩
6. European Council Press Release, 23 April 2012, http://www.consilium.europa.eu/uedocs/cms_data/docs/pressdata/EN/foraff/129739.pdf. ↩
7. Information provided to ICAI by DFID. ↩
8. Three Millennium Development Goals Fund, see paragraph 2.10. ↩
9. Information provided to ICAI by DFID. ↩
10. Data provided to ICAI by DFID. ↩
11. Subject to reviews of progress and government and other donor commitments. ↩
12. Three Diseases Fund Myanmar: Mid-Term Evaluation, Verulam Associates Ltd., November 2009, http://www.3dfund.org/images/stories/pdf/3DF_Mid-Term_Evaluation_2009_&_Action_Plan.pdf. ↩