Evaluation of DFID’s work through United Nations Children’s Fund (UNICEF) – Terms of Reference

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We wish to evaluate DFID’s work through and relationship with the United Nations Children’s Fund (UNICEF). In particular, we propose to evaluate the relationship with UNICEF as a delivery partner through a review of programmes that UNICEF is managing on behalf of DFID. We will concentrate on UNICEF’s work on the ground, delivering services on behalf of DFID, whilst also considering briefly the management relationship between DFID and UNICEF headquarters. These Terms of Reference outline the purpose and nature of the review and identify the main themes that it will investigate. A detailed methodology will be developed during an inception phase.

2. Background

2.1. DFID’s view, set out in the Multilateral Aid Review (MAR), is that ‘UNICEF has a critical role in delivering DFID and HMG development and humanitarian priorities’.¹ In 2011, UNICEF spent $3.8 billion and worked in more than 150 countries.² DFID’s International Relations Division provided it with core funding totalling £165 million between 2005 and 2011. Considerably more, £713 million, was channelled through DFID’s geographical, policy and humanitarian divisions over the same period (shown below as non-core funding). After the results of the MAR were published in 2011, DFID chose to double its core funding to UNICEF. UNICEF defines regular resources (which are the same as DFID core funding) as contributions with no restriction on their use and this supports programmes implemented by country and regional offices to promote and realise the rights of children; UNICEF’s other resources (which are the same as DFID non-core funding) represents contributions that are ear-marked by donors for specific pre-determined purposes such as a particular country, theme or humanitarian emergency.³ DFID and other donors also contribute to other pooled funds (such as OCHA’s Central Emergency Response Fund, CERF), which UNICEF has accessed in the past.

2.2 UNICEF’s global headquarters are in New York. 36 National Committees, mostly located in OECD countries, undertake UNICEF’s principal fundraising. UNICEF offices in over 100 developing countries manage the delivery of global and country-based programmes in accordance with the mandate of UNICEF, given to it by the General Assembly of the United Nations, ‘to advocate for the protection of children’s rights, to help meet their basic needs and to expand their opportunities to reach their full potential’.⁵ UNICEF’s work is guided by the 1990 Convention on the Rights of the Child.⁶

2.3 In the UK, UNICEF is represented by its UK National Committee (UNICEF UK). Most of UNICEF UK’s main income is derived directly from donations and fundraising from the British public.⁷

2.4 UNICEF UK’s reporting identifies two types of programme funding; core (i.e. the continuing activities of UNICEF) and specific (focussing on particular campaigns). In 2010, £52.3 million was allocated for five specific programmes (£36.1 million for Children in Emergencies, £9.3 million for Young Child Survival and Development, £3.7 million for Basic Education and Gender Equality, £1.7 million HIV and Children and £1.5 million Child Protection). In addition, £4.7 million was spent on core UNICEF activities and a further £6.2 million was spent on UNICEF UK support activities, covering fundraising and sales, advocacy and governance.⁸

2.5 As well as DFID’s money being combined with funds raised from the British public and channelled through UNICEF UK, UNICEF offices in developing countries also receive funds from DFID directly (defined as non-core funding, most commonly channelled through DFID’s bilateral country programmes). Occasionally, some funds also flow from bilateral offices to UNICEF New York. Figure 2 below summarises the countries where such ‘geographical’ funding was channelled through UNICEF in 2008-11.⁹

2.7 This spending covers a range of activities. As Figure 3 shows below, through its non-core country spending, DFID funds UNICEF’s principal activities of responding to humanitarian emergencies, supporting health, education, water and sanitation and nutrition improvements for children, mothers and families. It also uses UNICEF to channel funding for other purposes, for instance supporting justice, democracy and conflict programmes.

3. Purpose of the Evaluation

3.1 To assess the impact and effectiveness of DFID’s partnership with UNICEF.

4. Relationship to other evaluations and studies

4.1 UNICEF programmes funded by DFID are subject to regular monitoring and evaluation. DFID funds are often combined with funds from other sources to support UNICEF’s work. Consequently, evaluations may be undertaken either by DFID or by UNICEF, which has its own Evaluation Office. UNICEF also commissions independent evaluations. In addition, the Office of the Executive Director may commission evaluations directly.¹⁰ Some UNICEF evaluations of DFID programmes have been co-ordinated with DFID’s evaluation department in the past. As with DFID, UNICEF evaluations are placed online.¹¹ We will consider all available reports, paying particular attention to evaluations undertaken after 2008.

4.2 DFID rated UNICEF as a very good performer in the 2011 MAR. It found that, while its role in meeting international and UK development objectives was ‘strong’ and cost consciousness, financial resources management and contribution to results were ‘satisfactory’, weaknesses existed in strategic and performance management, attention to climate change and transparency and accountability.¹²

4.3 We will draw on the conclusions and recommendations of: the published ICAI report on DFID’s health programmes in Zimbabwe;¹³ and our current review of DFID’s education programmes in Nigeria (due to be published by the end of 2012), where UNICEF is one of two important delivery partners.

5. Analytical approach

5.1 Our evaluation will focus on DFID’s engagement with UNICEF to deliver UK-financed activities, including:

• evidence of impact;
• how results are managed;
• whether there is evidence of clear and effective participation of intended beneficiaries in setting objectives and delivering results;
• whether the choice of delivery options and partners in particular projects delivers good value for money (with a focus in particular on relative management costs and procurement). This will cover both DFID’s decision to use UNICEF as a delivery partner and UNICEF’s selection of any sub-contractors; and
• whether DFID is providing effective management and oversight of individual UNICEF programmes and the wider partnership between the two organisations.

5.2 Following a review of options, we have decided that this evaluation will compare programmes in three countries which exhibit different characteristics of the roles and activities undertaken by UNICEF on behalf of DFID. These countries will be Sierra Leone, the Democratic Republic of Congo and Ghana.

5.3 As illustrated by Figure 3, over 20% of the DFID’s non-core funding through UNICEF in 2008-11 was in the area of health (including malaria, polio and HIV/AIDS). Given the overall weight of funding for this theme, this evaluation will concentrate on programmes that have a significant health component.

5.4 UNICEF often provides specific services to DFID, notably procurement and detailed project management. Our approach, in particular the case studies, will focus on these services.

6. Indicative evaluation framework

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. The questions outlined below comprise those questions in our standard evaluation framework which are of particular interest in this review. These will be refined on selection of the case studies. The full, finalised list of questions that we will consider in this review will be set out in the inception report.

6.2 Objectives

6.2.1 Does the engagement with UNICEF have clear, relevant and realistic objectives that focus on the desired impact?

6.2.2 Is there a clear and convincing plan, with evidence and assumptions, to show how the engagement with UNICEF will work?

6.2.3 Do the UNICEF programmes complement the efforts of government and other aid providers and avoid duplication?

6.3 Delivery

6.3.1 Is the choice of funding and delivery options appropriate?

6.3.2 Does programme design and roll-out take into account the needs of the intended beneficiaries?

6.3.3 Is there a clear view of costs throughout the delivery chain?

6.3.4 Is the programme delivering against its agreed objectives?

6.3.5 Are risks to the achievement of the objectives identified and managed effectively?

6.3.6 Is the engagement with UNICEF delivering against its agreed objectives?

6.4 Impact

6.4.1 Is the engagement with UNICEF delivering clear, significant and timely benefits for the intended beneficiaries?

6.4.2 Are UNICEF programmes working holistically alongside other programmes?

6.4.3 Is there a long-term and sustainable impact from the engagement with UNICEF?

6.5 Learning

6.5.1 Are there appropriate arrangements for monitoring inputs, processes, outputs, results and impact?

6.5.2 Is there evidence of innovation and use of global best practice?

6.5.3 Is there anything currently not being done in respect of the engagement with UNICEF that should be undertaken?

7. Methodology

7.1 The study will be deep rather than broad, focussing on the services that UNICEF provides to DFID for programme implementation. It will seek to obtain comprehensive details of specific programmes in the case study countries. We recognise that this will not necessarily provide generic lessons on the relationship between DFID and UNICEF. It will, however, provide information that may inform wider conclusions on DFID’s management and oversight of programmes. The study will seek to identify details of strategy, design, decision-making, communication and operational, performance and financial management by considering evidence from internal and external sources.

7.2 The methodology will include the following elements:

• a rapid mapping of DFID funding provided through UNICEF since 2008;
• three country visits, to look at specific programmes implemented by UNICEF (including gathering intended beneficiary views); and
• a review of available documentation on the impact and effectiveness of DFID funded activity in UNICEF since 2008, particularly assessing the findings of previous UNICEF and DFID evaluations and monitoring reports (focussing on the countries where the case studies take place).

7.3 The case studies will be of projects that are current or have finished within the past three years. This is to make sure that the projects examined are representative of DFID’s current approach to working with UNICEF and that first-hand information is still available within the DFID country teams. The case studies will thus be seen as the principal source of information for the evaluation’s findings.

7.4 The following tables set out DFID funding through UNICEF between 2008 and 2011 in three key countries of interest to ICAI. Case studies will be selected from the projects set out here. We have selected these countries because, although other countries have received more money, DFID uses UNICEF in interesting ways in these three countries, none of which have been substantively reviewed in previous ICAI reports.

a. Sierra Leone was the fifth-largest recipient of UK aid funds channelled through UNICEF over the period.

b. The Democratic Republic of Congo was the sixth-largest recipient of UK aid funds channelled through UNICEF over the period.

c. Ghana was the eleventh-largest recipient of UK aid funds channelled through UNICEF over the period.

7.5 We propose to review the largest programme in each of these three countries, as well as smaller but related programmes, where they are complementary with the main programme, in order to gain a broader perspective on the local relationship with UNICEF, as follows:

• Sierra Leone (focus on reproductive and child health):
  • support to the National Reproductive Child and Health Strategy;
  • support to the malaria prevention programme; and
  • the Improving Reproductive, Maternal and Newborn Health programme.
• Democratic Republic of Congo (focus on water and sanitation): the grant for healthy villages and schools;
• Ghana (focus on malaria prevention):
  • the Support for Malaria programme; and
  • the procurement and distribution of insecticide-treated bed nets.

7.6 These selected programmes represent total spending of £57.7 million, or 13% of DFID’s non-core country office expenditure for the period 2008-11, summarised in Figure 2.

7.7 We are aware that the focus of our study may overlap with a forthcoming National Audit Office (NAO) review and are liaising with NAO to manage this. This will involve a joint visit to Sierra Leone to minimise the burden on DFID and maximise the use of our resources.

8. Timing and Deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The review will take place during the third and fourth quarters of 2012.

Footnotes

1. ^ Multilateral Aid Review: Assessment of the United Nations Children’s Fund (UNICEF), DFID, 2011, http://www.dfid.gov.uk/Documents/publications1/mar/unicef.pdf
2. ^ Annual Report 2010, UNICEF, 2011 http://www.unicef.org/publications/files/UNICEF_Annual_Report_2010_EN_052711.pdf
3. ^ Funding modalities: quick reference, UNICEF, 2011, http://www.unicef.org/parmo/files/Funding_Modalities_Sept_2011.pdf
4. Information in this table was provided by DFID.
5. ^ For further details of UNICEF’s mission, see here: http://www.unicef.org/about/who/index_mission.html
6. ^ Convention of the Rights of the Child, Resolution44/25 of 20 November 1989, General Assembly of the United Nations, http://www2.ohchr.org/english/law/crc.htm
7. ^ The United Kingdom Committee for UNICEF, Trustees’ Report and Consolidated Financial Statements for the year ended 31 December 2010, http://www.unicef.org.uk/Documents/Publication-pdfs/TrusteesReport2010_web.pdf
8. ^ Advocacy activities are mainly in the UK and fund the promotion of the work of UNICEF with the public and decision-makers as well as supporting campaigning for specific activities.
9. The £713m of non-core funding provided to UNICEF over the period 2005 to 2011 includes funds provided directly by DFID’s bilateral country programmes to UNICEF’s offices in developing countries.
10. ^ For instance, the Independent Review of UNICEF’s Operational Response to the January 2010 Earthquake in Haiti, see http://www.unicef.org/evaldatabase/index_60396.html
11. ^ See here http://www.unicef.org/evaldatabase/index.html
12. ^ Multilateral Aid Review: Assessment of the United Nations Children’s Fund (UNICEF), DFID, 2011, http://www.dfid.gov.uk/Documents/publications1/mar/unicef.pdf
13. ^ DFID’s Support to the Health Sector in Zimbabwe, Independent Commission for Aid Impact, November 2011, http://icai.independent.gov.uk/wp-content/uploads/2010/11/DFIDs-Support-to-the-Health-Sector-in-Zimbabwe.pdf.