Results review: Assessing DFID’s results in nutrition

Rationale, purpose and scope

Malnutrition is the single largest driver of ill health in the world, affecting one in three people globally. Tackling this problem is integral to meeting Sustainable Development Goal 2 – Zero Hunger. In 2015, the UK government pledged to improve the nutrition of 50 million people by 2020. This reflected a decision made at the first Nutrition for Growth Summit in 2013 to scale up the department for International Development’s (DFID) efforts. DFID’s nutrition programming is supported by a commitment to spend £2.96 billion over the
period 2013 to 2020.

The purpose of the review is to assess the results of DFID’s work on nutrition. In its Single Departmental Plan published in June 2018, DFID is estimated to have reached a total 42.1 million children under five, women of childbearing age (15–49 years) and adolescent girls (10–19 years). The review will test and explore what is behind these claimed results, through assessing the effectiveness of DFID’s nutrition programmes, whether those reached are the ones most in need, and the impact of DFID’s portfolio on malnutrition. It will also cover DFID’s technical assistance work, designed to help end malnutrition over the longer term. ICAI last reviewed this important area of DFID’s work in 2014; however, it was too early to make any evaluative judgements on its impact.

The review period covers DFID’s work between 2015 and 2019, including the results claimed in the Single Departmental Plan (2015–2018). This timeframe is designed to capture progress since the commitments made at the Nutrition for Growth Summit in 2013.

Background

Malnutrition is a critical contributor to ill health, vulnerability, poverty and excess mortality. Worldwide, 820 million people are ‘chronically undernourished’. Malnutrition is responsible for five times the death and disability burden of HIV/AIDS, tuberculosis (TB) and malaria combined. It also holds back people’s ability to learn, be economically productive and stay healthy, making them far less resilient.

World Bank analysis highlights there is still a large gap in the investment needed to reach global nutrition targets, including the Sustainable Development Goals. Most countries are lagging behind their targets. Data from UNICEF and the World Health Organisation (WHO) show that the number of stunted children under five in Africa has risen from 50.3 million to 58.8 million between 2000 and 2018. Climate-related shocks and conflict are important drivers of this increase.

The UK government is a leading advocate of tackling malnutrition in the world’s poorest countries. The UK hosted the Hunger Summit during the London Olympics in 2012. In 2013, the UK co-hosted the first Nutrition for Growth Summit, which aimed to galvanize global efforts to tackle undernutrition, helping to secure £15.2 billion in new financial commitments. DFID’s 2015 pledge to reach 50 million beneficiaries was a significant advancement on the UK’s ambition to reach 20 million during 2011–2015. In 2015, DFID disbursed a record $1 billion of official development assistance (ODA) to nutrition, reportedly reaching a total 13.3 million children under five, women of childbearing age and adolescent girls through nutrition programmes in 2015–2016.

The share of DFID’s bilateral aid spent on nutrition was 9% in 2013, which grew to 11% by 2017 (or $895 million), reflecting a global increase in spend on nutrition. In late 2017, DFID launched a new global nutrition Position Paper aiming to tackle undernutrition. The Position Paper outlined DFID’s strategy at that point, prioritising support to the “first 1,000 day window” from conception through to two years of age, as well as preventing “the most severe forms of undernutrition” in children up to age five.

DFID’s nutrition programmes span a range of sectors, including emergency response, basic healthcare, food security and livelihoods, social protection, WASH (water, sanitation and hygiene), education, and governance and security. Whilst much of DFID’s work on nutrition is channelled through humanitarian programming, many programmes are multi-sectoral in nature.

DFID’s approach also reflects the global literature on nutrition by classifying interventions into two broad groups: nutrition-specific and nutrition-sensitive programming. Nutrition-specific interventions aim to address undernutrition directly (e.g. nutrient supplementation for women and children, support for infant and young child feeding, or treatment for acute malnutrition). Nutrition-sensitive interventions are those that address the various underlying causes of undernutrition (e.g. providing access to safely managed water and sanitation to prevent diarrhoeal disease, which contributes to undernutrition).

DFID has exceeded its nutrition-sensitive spend commitment made at the Nutrition for Growth Summit. Between 2013 and 2017, DFID spent £2.7 billion on nutrition-sensitive interventions, surpassing its commitment of £2.13 billion by 2020. DFID’s portfolio of nutrition-specific interventions is therefore much smaller than its portfolio of nutrition-sensitive programmes. The opposite is true globally, with $987 billion being spent on nutrition-specific interventions and $6 billion invested in nutrition-sensitive interventions. DFID also funds integrated programming. According to an external classification, of DFID’s 147 nutrition-related projects in 2017, 40 included both nutrition-specific and sensitive components.

DFID’s methodology to calculate the reach of its nutrition programming has also evolved. Before 2016, DFID defined ‘reach’ as “the number of children under five, and breastfeeding and pregnant women reached through DFID’s relevant projects”. However, defining reach in such terms prevented distinguishing between those people reached by a single nutrition intervention and those receiving a package of multiple, potentially higher-impact, nutrition interventions.15 Since 2016, DFID has therefore changed its methodology in order to classify reach into high, medium and low intensity. DFID defines intensity according to:

• The comprehensiveness of the package of interventions reaching a target population.
• Whether the package is directly or indirectly targeted to the population.

DFID distinguishes high, medium and low intensity reach as follows:

• High intensity reach includes those directly targeted with both a nutrition-specific and a nutrition-sensitive or hunger-sensitive programme.
• Medium intensity reach includes those directly targeted with only a nutrition-specific or a nutritionsensitive programme (or a hunger-sensitive programme with integrated nutrition-sensitive behaviour change activities).
• Low intensity reach includes those indirectly targeted with a nutrition-sensitive intervention. Examples of the type of interventions which would be categorised as low intensity are programmes which seek to influence behaviour change through mass communications campaigns.

The nutrition results claimed in the Single Departmental Plan are disaggregated by intensity and by level of country fragility. These breakdowns are presented in Figures 1 and 2.

Figure 1: DFID’s nutrition results (people reached) by intensity of intervention (2015–18)

Figure 2: DFID’s nutrition results (people reached) by fragility of country (2015–18)

As well as reaching the most vulnerable through its nutrition programmes, a further priority for DFID is supporting an end to malnutrition over the longer term. The Position Paper frames DFID’s global advocacy and technical assistance work as central to this. Through this, DFID aims to foster the leadership, capacity and action required amongst governments and other partners to end malnutrition. Programming includes Technical Assistance for Nutrition, through which DFID supports the Scaling Up Nutrition (SUN) Movement, including providing technical assistance to governments to design, cost and implement multi-sectoral nutrition plans.

In 2014, ICAI reviewed DFID’s emerging work on nutrition, awarding it a green-amber score. Whilst there were promising signs of progress, ICAI found that there was scope to enhance the potential impact of the portfolio through more targeted interventions.

This latest Results Review provides a good opportunity to review progress five years later, with a greater focus on the achievement of nutrition results and the impacts that DFID has achieved since the scale-up of its investments.

Review questions

This review is built around the criteria of effectiveness, equity and impact. Effectiveness will examine the accuracy of DFID’s immediate results, as well as progress against nutrition programme objectives, to help assess the meaningfulness of the results. This review will also analyse enabling and constraining factors. Equity will focus on whether DFID’s nutrition programming is engaging and benefitting the most hard-to-reach and vulnerable women and children. We will also assess whether DFID is maximising its longer-term impact on reducing malnutrition and its underlying causes.

Review questions and sub-questions have been developed for each of the above criteria (Table 1).

Table 1: Our review questions

Methodology

The methodology will involve five main activities. It will draw on a robust literature review, as well as donor and expert opinion, to identify learning from wider nutrition results-measurement and how others drive impact on reducing malnutrition. This will be used to create benchmarks against which DFID’s nutrition results and associated country work and programming can be compared. DFID’s nutrition interventions will then be investigated through a review of DFID’s strategies, a sample of bilateral and centrally-managed programme assessments, and country case studies. The latter will include a specific programme of consultation with nutrition programme service users and citizens.

Component A – Literature Review

We will conduct a literature review to identify areas of consensus around what works in reducing malnutrition, as well as examples from other donors. The review will explore evidence on: the factors influencing the effectiveness of nutrition programmes; the extent to which nutrition interventions reach the most vulnerable; what constitutes impactful packages of intervention; and the conditions for sustainable impact. It will also provide an overview of the different approaches to monitoring and evaluating nutrition programmes. The literature review will be based upon a synthesis of academic and grey literature. It will be used to generate a set of relevant and robust frameworks against which the review findings can be measured.

Component B – Strategic Review

To help test the validity of DFID’s results, we will conduct a detailed assessment of the analytical approach adopted by DFID when calculating its nutrition results. We will also assess DFID’s approach to evaluating impact. We will review DFID strategies, policies, annual reports, guidance documents and any relevant evaluations as part of this. We will also include a light-touch comparative assessment with the tools/ methodologies used by other donors.

We will also map DFID programme expenditure and results since 2015 against the intervention typologies in including intensity of provision, the range of nutrition-specific and nutrition-sensitive interventions, country  focus, gender split, as well as against stated targets in business cases and log-frames. This will enable us to provide an initial assessment of the breadth and depth of DFID’s nutrition results, including when compared with the evidence on what works. We will also review available external programme evaluations for evidence of key lessons around DFID’s reach and results, as well as any evidence of the impact of DFID’s nutrition portfolio.

We will complement this with key informant interviews with DFID, other bilateral donors with substantive nutrition investments (e.g. USAID), and relevant multilateral agencies (including WHO, UNICEF, and the Food and Agriculture Organisation). We will also consult with other international organisations such as the Global Alliance for Improved Nutrition (GAIN) and academic nutrition experts (including those involved in the Lancet 2013 series on Maternal and Child Nutrition). We will also engage with Development Initiatives – an organisation contracted by DFID to analyse their ODA spending on nutrition-related projects.

Interviews will be semi-structured and conducted in person and by telephone, and will explore the background, strengths and any potential areas for improvement within DFID’s results approach, DFID’s focus on equity and its claims on impact. We will hold stakeholder workshops (with relevant international nongovernmental organisations (NGOs), and other external experts) at the start and end of the review, to help shape our approach and test our findings. All qualitative interview data will be systematically analysed, using an analysis grid linked to the review questions and judgement criteria.

Through the desk review and interviews, we will also assess the influencing role played by DFID within key international forums and partnerships, such as the SUN Movement and the 2020 Global Nutrition Summit, to support the assessment of DFID’s wider and longer-term impacts.

Component C – Bilateral and Centrally-Managed Programme Assessments

DFID’s nutrition portfolio and the programmes that contribute to its results are diverse. We therefore propose to select eight countries (representing different contexts for intervention) and review a range of nutrition-specific and/or sensitive programmes operating within them. This will include bilateral and centrallymanaged programmes.

The focus will be on testing the quality of country-level data that has fed into DFID’s global results claims, given programme targets/expectations, monitoring and aggregation methods, reported outcomes and the results observed by the review team on the ground. We will also gather evidence on the engagement of vulnerable groups and triangulate evidence of impact. The latter will include looking at how an understanding of ‘what works’ has been incorporated into the design of DFID’s country portfolios and programmes. Adopting a country view as the first level of analysis will enable us to check that the risk of double counting results has been adequately mitigated, and to look at the level of multi-sectoral programming and integration across portfolios of activity. In addition, we will review the effectiveness of DFID’s global technical assistance work and its impact at a country level, including through the lens of DFID’s largest relevant programme, Technical Assistance for Nutrition (TAN).

The programme reviews will involve a desk review of programme documentation (business cases, monitoring data, annual reviews and independent evaluations), followed by in-depth interviews with DFID programme managers. This evidence will be gathered, analysed and triangulated using a detailed assessment framework for each programme, which will enable systematic assessment against the review questions.

The eight sampled countries will include two field visit countries to enable us to engage in more depth with programme partners, external stakeholders and service users. This will enable us to thoroughly assess DFID’s results claims, as well as the extent to which programmes reach the most vulnerable and deliver impact, within different contexts and for different population groups. We will also gather first-hand evidence of DFID’s technical assistance and influencing role, within the context of a multi-stakeholder approach to reducing malnutrition and its implications for sustainable impact.

For one country, we will undertake a longer visit to facilitate the engagement of service users and citizens. Service user consultation will provide more detail on the reach, benefits and equity of DFID’s nutrition programming, including for vulnerable groups, and will help with triangulating and filling any gaps in the evidence. It will help ensure that the voices of ordinary citizens are adequately reflected in our findings, and bring this important perspective to a range of review questions.

Specifically, this exercise will allow us to better understand how different intervention types reach their service users, how responsive DFID-funded nutrition activities are to differing needs and who is not being reached. We will also explore whether there have been any changes in service users’ behaviours, health and wellbeing, and economic status, amongst other outcomes. Through engaging with a range of service users across nutrition-specific and/or nutrition-sensitive programmes, with varying intensities of reach, we will further test the validity of DFID’s results at different levels.

We will work with local researchers to identify and access the most vulnerable and hard-to-reach women and young people involved in a cross-section of nutrition activities, from the sample of DFID programmes under review. We will then carry out semi-structured interviews and focus group discussions with service users in programme target areas. We will develop and provide clear guidelines to field researchers on ethical protocols including data storage and gaining permission from interviewees.

Sampling approach

We selected a sample of eight countries to be included as part of the in-depth country case studies. From this list of countries, we generated a sub-sample of 16 bilateral nutrition programmes. We also selected three centrally-managed programmes, from two different central DFID departments.

The countries were selected primarily for being within the top ten contributors to DFID’s results as well as global nutrition expenditure (2015 to 2018). The intensity of reach within each country was also considered to enable the review to assess a range of intensities. In some cases, countries contributed results in more than one intensity category. The countries selected also guaranteed a range of contexts were covered, including fragile or conflict-affected states (FCAS) as well as non-FCAS countries which require humanitarian assistance.

The programmes were also selected based on their contribution to DFID’s results claims and spend. This second-stage sampling was designed to ensure that a broadly representative range of sectors are covered, whilst excluding certain programmes (e.g. those which have yet to make a substantial contribution to results). The sample included interventions that are nutrition-specific and/or nutrition-sensitive, with the inclusion of some programmes which provide a comprehensive package of both.

Table 2: Sample of countries and programmes

Table 3: Sample of centrally-managed departments and programmes

Overall, we have identified 66 bilateral programmes and 12 centrally-managed or regional programmes that contribute to DFID’s nutrition results. Our sample of 19 programmes is sufficient for assessing the broad range of activity types and intensities of provision, and different country and regional contexts within which DFID’s programming is concentrated. The programmes span a variety of sectors including humanitarian, health, food security, water and sanitation, and social protection. The sample includes completed and live programmes, allowing for an assessment of the impact of nutrition interventions post implementation. The reported reach of all programmes included in the sample is 29 million people. The reported spend of all programmes in the sample, as of October 2019, was £1.8 billion. The breadth of this sample will allow us to extrapolate conclusions against each review question that will be relevant across DFID’s nutrition portfolio.

We also include DFID’s TAN programme, to support answering the review’s impact questions. Although this programme does not contribute directly to DFID’s overall results, it can provide key insights into DFID’s influencing work and attempts to reduce malnutrition over the longer term. The full list of programmes which will be included in the sample are in the attached annex.

Limitations to the methodology

Review scope and coverage: The multi-sector focus of nutrition programming, as well as range of intensities and fragile/non-fragile state locations, presents a challenge for the review in terms of providing a very wide potential scope of work. Whilst careful sampling can mitigate this to some extent, the review has needed to narrow its scope in order to ensure enough depth. DFID’s work focussed on the macro-level drivers of nutrition outcomes, such as climate change, trade, economic growth, food prices and land-use policies, and some of their support for the global enabling environment for nutrition will not be included within the scope of the review. The review’s main emphasis will be a retrospective focus on DFID’s achieved results and impact, although more recent developments in DFID’s strategic approach will be considered.

Data on effectiveness and equity: Aggregated results data comes from a variety of secondary sources generated by the programmes themselves. We will not be able to complete detailed verification exercises of programme results. Instead, we will interrogate the indicators, methodologies and assumptions employed, and assess the quality of DFID’s data through triangulation with other data sources. It may also be challenging to access reliable data on equity and how the most marginalised are benefitting from DFID’s nutrition programming. We will endeavour to include the voice of the poorest through the programme of service user consultation, although this will only represent a small sample of citizens and one snapshot in time.

Data on impact: Another limitation is the long causal pathways associated with reducing malnutrition, and associated indicators such as stunting and wasting (alongside the multiple drivers that make the problem difficult to tackle). This means that the available impact evidence for DFID’s recent and current programming may be limited. In some cases, we are likely to be reliant on comparing DFID’s interventions with the global literature on what works in reducing malnutrition. Alongside this we will review evidence of shorter-term programme effectiveness and outcomes (such as dietary improvements or changes in breastfeeding practice), which may contribute to reducing malnutrition over the longer term.

Risk management

Quality assurance

The review will be carried out under the guidance of the ICAI Chief Commissioner, Tamsyn Barton, with support from the ICAI Secretariat. Both the methodology and the final report will be peer reviewed by Dr Andrew Seal, University College London.

Timing and deliverables

This review will be executed over a period of ten months, beginning in October 2019.

Annex: List of countries and programmes in sample