ICAI follow-up: Assessing DFID’s results in maternal health

Individual review scores and what they mean

The Independent Commission for Aid Impact works to improve the quality of UK development assistance through robust, independent scrutiny. We provide assurance to the UK taxpayer by conducting independent reviews of the effectiveness and value for money of UK aid. We operate independently of government, reporting to Parliament, and our mandate covers all UK official development assistance.

© Crown copyright 2020

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-licence/version/3, or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gsi.gov.uk.

Where we have identified any third-party copyright you will need to obtain permission from the copyright holders concerned.

Readers are encouraged to reproduce material from ICAI reports, as long as they are not being sold commercially, under the terms of the Open Government Licence. ICAI requests due acknowledgement and a copy of the publication. For online use, we ask readers to link to the original resource on the ICAI website.

Any enquiries regarding this publication should be sent to us at enquiries@icai.independent.gov.uk.

@ICAI_UK
icai.independent.gov.uk

An inadequate score results from one or more of the following three factors:

• Too little has been done to address ICAI’s recommendations in core areas of concern (the response is inadequate in scope).
• Actions have been taken, but they do not cover the main concerns we had when we made the recommendations (the response is insufficiently relevant).
• Actions may be relevant, but implementation has been too slow and we are not able to judge their effectiveness (the response is insufficiently implemented).

An adequate score means:

• Enough progress has been made in the right areas and in a sufficiently timely manner in order to address the core concerns underpinning ICAI’s recommendations.

Executive summary

ICAI’s follow-up review is an important element in the scrutiny process for UK aid. It provides parliament and the public with an account of how well the government has responded to ICAI’s recommendations to improve spending. It is also an opportunity for ICAI to identify issues and challenges facing the UK aid programme now and in the future, which in turn helps to inform subsequent reviews. This year, for the first time, we introduced a scoring element to the follow-up exercises. For each of the reviews we follow up, we provide a score of adequate or inadequate, illustrated by a tick or a cross.

This document is a summary focused only on the results of the follow-up of our review of the Department for International Development’s (DFID) results in maternal health. The full follow-up report of all our 2018-19 reviews, including overall conclusions from the process and details of our methodology and scoring, can be found on our website.

Findings

Assessing DFID’s results in maternal health

ICAI published its review on DFID’s results in maternal health in October 2018. The review noted that DFID programmes had expanded access to family planning and some maternal health services. However, its amber-red score was based on the conclusion that a renewed effort was required to reach young women and girls and to generate lasting impacts on quality of care and maternal health outcomes. It also noted its concern over the model used by DFID to back up its claims of “maternal lives saved” through DFID-funded interventions. The review’s five recommendations were all accepted by DFID.

DFID’s response has been comprehensive. The changing shape of its strategy and programming on maternal health is already evident. DFID reacted swiftly to one of ICAI’s central concerns: how the department estimated and reported its results. However, initiatives are at an early stage and it will be some time before their full scope and influence are evident. It will be crucial to keep the momentum and to publish the delayed Ending Preventable Deaths Action Plan and Health Systems Strengthening Position Paper.

Recommendation 1: Develop a long-term approach to improving maternal health, planning through to 2030 in focus countries with high maternal mortality, focusing on quality and continuity of care, cross-sectoral interventions and empowerment of women and girls

The 2018 ICAI review welcomed the comprehensive approach to improving maternal health set out in DFID’s 2011-15 Results Framework, but found that its implementation was less balanced. Programmes, particularly on family planning, were too focused on generating short-term results, and few programmes had compelling exit strategies. ICAI therefore recommended that DFID pursue a long-term approach to investing in health and other infrastructure as well as in socio-cultural change, to maximise results in improving maternal health.

In early 2019, DFID reviewed its bilateral health programming in 18 countries in light of the ICAI review findings. Later in 2019, it began developing a new Action Plan on Ending Preventable Deaths of Mothers, Newborns and Children by 2030 (EPD). Importantly, the Action Plan will be accompanied by an internal implementation plan translating the high-level objectives into action across DFID.

The Action Plan was intended to be launched in March 2020, but was delayed due to the COVID-19 outbreak. ICAI was not shown a draft, but DFID’s description of its contents is aligned with ICAI’s recommendations. The plan focuses on a set of high-priority countries and locates DFID’s maternal health interventions within a focus on “quality health services for all”, health systems strengthening, multi-sectoral work on the wider determinants of health, and women’s and girls’ rights and empowerment. Although the EPD plan is not yet published, we saw evidence of new centrally managed programmes being developed in line with its approach.

DFID has sought to exercise global leadership on sexual and reproductive health and rights (SRHR) and quality of care for women, newborns and children over the past year. Ministerial engagement has been strong, including on topics such as safe abortion where the global consensus is in danger of being rolled back due to actions by the US administration and others. The UK government reportedly pushed for SRHR and service quality to be reflected in the UN General Assembly’s political declaration on universal health coverage (SDG 3, Target 8).

This appears to be a good response to ICAI’s recommendation. However, without sight of the EPD Action Plan, its implementation plan, and related monitoring frameworks, it is hard to assess the full potential of this work to shape future government policy and programming in relation to maternal health.

Recommendation 2: Clarify the approach to health systems strengthening, prioritising improvements in the availability and accessibility of good quality, respectful care for women and their babies

The 2018 ICAI review found that DFID programmes had a limited focus on improving the quality of maternal healthcare, and that little progress had been made in developing skills in emergency obstetric and neonatal care in many of DFID’s focus countries. DFID had not yet considered how to address discrimination, neglect or abuse in maternal health services, nor fully explored the opportunities to improve access to care (particularly for poor, marginalised and young women) through community health services.

DFID has undertaken a range of actions in response to ICAI’s recommendation. New centrally managed and country programmes are being developed. For instance, in Malawi, a new health systems strengthening programme will focus on primary and community health, quality of care, and local accountability. In the Democratic Republic of the Congo, DFID has been designing a new health sector programme that will focus on reproductive, maternal, neonatal and adolescent health, with a strong emphasis on community mobilisation and accountability, as well as quality of care.

DFID has pushed for a greater focus on quality of care within the Global Financing Facility and is funding the World Health Organisation (WHO) to help countries develop national quality policies and strategies. It is working with the UK Department of Health and Social Care to advance the patient safety agenda with the WHO. The new UK Partnerships for Health Systems programme (2019-2023) will support UK health professionals, including midwives, to provide voluntary technical assistance to developing countries in line with the EPD Action Plan.

DFID has taken many years to develop a Health Systems Strengthening Position Paper, to the frustration of external stakeholders including UK civil society organisations. DFID told us that a draft of this paper was due for consultation but had been paused first due to the 2019 general election and then by the COVID-19 emergency. The plan at the time of writing was to publish the paper together with the EPD Action Plan.

Overall, DFID has responded positively to this recommendation, although much of the work on quality of care is at an early stage. The evident collaboration across the UK government and between teams working on SRHR, violence against women and girls and safeguarding is promising. DFID has also developed good new guidance and learning resources on SRHR for its staff, placing new emphasis on respectful and dignified care. As we have not seen the draft Health Systems Strengthening Position Paper, we cannot assess its relevance to maternal health programmes.

Recommendation 3: Directly monitor the impact of sexual, reproductive and maternal health services programmes on adolescents and the poorest women, ensuring through programme design and monitoring that adolescents and the poorest women are included

The ICAI review found that DFID did not track whether its programmes reached its core target groups of young women aged between 15 and 19 and the poorest 40%. Reaching adolescents proved a particular challenge, and DFID’s own lesson learning indicated a need to expand programming to include both girls and boys aged between 10 and 14.

New centrally managed and country-level sexual health and family planning programmes now have a stronger focus on adolescents and the poorest women, with performance tracked through logframe indicators. For instance, the Women’s Integrated Sexual Health (WISH) programme (£238 million, 2018-2021), which was under procurement during the ICAI review, has a strong focus on young, poor and rural women and girls, includes efforts to improve quality of care, and seeks to engage men and boys. Five key performance indicators (KPIs) are linked to results-based funding and one relates to reaching young people under 20, against which the programme is over-performing. A third-party monitor has been engaged to verify results reporting, capture learning and support adaptation.

The new Malawi family planning programme, Tsogolo Langa (£50 million, 2018-2024), includes targeted interventions to reach young people aged between 10 and 24 with sexual health services. Indicator data will be disaggregated by location, age group (10-14 and 15-19), and disability.

DFID’s response to this recommendation is at an early stage. New programmes include interventions targeted at adolescents and poorer women, and programme documents suggest that monitoring data will be disaggregated. For most of these programmes, it is too early to see any results. However, data from the WISH programme is promising, and its strong emphasis on learning and knowledge sharing should enhance future government efforts in this area.

Recommendation 4: When using models to generate outcome data, DFID should test its assumptions and triangulate its results claims using other quantitative and qualitative data

The ICAI report found that the model DFID used to estimate departmental results on “maternal lives saved” rested on assumptions that were not robust. We considered that DFID should make more use of qualitative information, such as data on quality of care, and feedback from client surveys, community scorecards and local accountability bodies, to test and triangulate its quantitative data and results estimates.

The ICAI review stimulated swift and significant changes in the way DFID estimates and reports its results. DFID no longer recommends that country offices use the Lives Saved Tool (LiST) to generate results estimates. In December 2018, DFID published an “Update on Results achieved by the Department for International Development between 2011 and 2015”, responding directly to the concerns raised by ICAI. The department updated its Family Planning Methodology Note in July 2018 and plans to publish new methodology notes alongside its 2020 results estimates. In July 2019, DFID stated its voluntary compliance with the Code of Practice for Statistics in the generation of its results estimates. DFID no longer makes results claims of the sort ICAI assessed in our maternal health review.

Modelling is still used to generate estimates for programme monitoring purposes, but recent programme documentation is less reliant on this. There appears to be a stronger emphasis on using country-level data sets for DFID programme monitoring.

Recommendation 5: Prioritise and invest in efforts to gather data on the quality of maternal health services and outcomes, including disaggregated data on key target groups

The ICAI report acknowledged that data availability and quality are poor in many developing countries, making it difficult to monitor progress on maternal health. DFID had made investments in improving health management information systems, but many initiatives were not yet operating at scale, and data use for decision making was not yet institutionalised.

Since the ICAI review, DFID has gradually scaled up its investment in the enhancement and rollout of a health management information system platform (known as DHIS2). It is supporting national statistical capacity building in partner countries and the global rollout of maternal death surveillance and response systems, through core funding to the WHO and bilateral health programmes. This facility-level approach allows for discussion and learning by health professionals focused on how and why women have died before, during or after childbirth.

In summary, DFID has invested heavily over several years in core health data systems. While this provides the backbone for the collection and analysis of data on the accessibility and quality of maternal health services, more focused efforts are needed. Support to generate qualitative data related to maternal health has been largely exploratory to date and is only just beginning to scale up.

Conclusion

Overall, DFID has responded well to ICAI’s recommendations. It reacted swiftly to ICAI’s concern about the way it estimated and reported its results. Some changes have already been made, and a new approach to results reporting is being developed for the next spending review period. DFID has enhanced its emphasis on good quality, respectful care for women and their babies and increased focus on adolescents and poorer women within its new family planning programmes.

We are pleased with the wide-ranging actions responding to ICAI’s concerns. However, several initiatives are at an early stage: it will be some time before their full scope and influence are evident, and some results may be fragile. Importantly, neither the Ending Preventable Deaths Action Plan nor the long-awaited Health Systems Strengthening Position Paper have been published yet. We will return to our recommendations related to UK aid’s longer-term strategic planning on maternal health as outstanding issues if these two documents remain unpublished or are of insufficient quality by the time ICAI begins next year’s follow-up exercise.