ICAI follow-up review of 2017-18 reports

Letter from the chief commissioner

Within 12 months of the publication of each ICAI review, we conduct a follow-up review, to assess the government’s response to our recommendations. This follow-up is a crucial aspect of ICAI’s work, ensuring a chain of accountability from findings to remedial actions, and encouraging transparency around the delivery and value for money of UK aid. This year the follow-up review has been complemented by a separate review synthesising the findings from each of the 28 reviews issued in the four-year review period from 2015 to 2019. Together, these two reviews provide a comprehensive picture of the current state of UK aid. The follow-up process is often one of the most rewarding parts of our work as commissioners, when we see how our reviews have helped to catalyse a shift in performance or impact. It also provides a window for aid-spending departments to showcase new initiatives and improvements in UK aid. Again, this year, we report on many areas of progress and improvement, though there are also aspects where more work is required. The follow-up review is also an opportunity for dialogue between ICAI and aid-spending departments on how best to achieve desired changes – a dialogue that also feeds into ICAI’s own learning on how best to target our recommendations to have the greatest impact. This follow-up review marks the transition to the next phase of ICAI. With it we say goodbye to retiring commissioners Tina Fahm and Richard Gledhill, and to former chief commissioner Alison Evans, who left ICAI to take over the Independent Evaluation Group in the World Bank Group in December last year, and we welcome two new commissioners, Sir Hugh Bayley and Tarek Rouchdy. ICAI’s important work of independent scrutiny, through well-targeted, evidence-based reviews and systematic follow-up continues.

Tamsyn Barton
Chief commissioner

Executive Summary

This review follows up on progress made by aid-spending government departments and funds on addressing recommendations from the eight reviews we published between September 2017 and June 2018. It also revisits four issues identified as outstanding from last year’s follow-up review.

Table 1: List of Year 7 reviews and outstanding issues from earlier years covered by this followup review

There has been good progress across ICAI’s 2017-18 recommendations

While the level of progress made varied, the response of the relevant government departments to the eight reviews followed up in this report was generally strong, considered and appropriate. The report highlights a range of positive actions, leading to good progress in a number of areas of UK aid, including on strategic direction, theories of change and results management, research, analysis and diagnostic work, and the mainstreaming of cross-cutting issues such as disability and disaster resilience. Some improvements which stand out include:

• The Conflict, Stability and Security Fund’s efforts to strengthen theories of change and results frameworks. Together with the scaling up of monitoring, evaluation and learning activities, these are likely to improve both the design and the implementation of the Fund’s programmes.
• DFID’s ambitious strategic framework for mainstreaming disability inclusion in all its programming.
• Significant achievement by the Global Challenges Research Fund (GCRF) in the span of a year in developing strategic themes, setting up research hubs and appointing challenge leaders. This is highly likely to improve the coherence of the Fund’s investments and the potential for development impact at scale.
• The range of efforts by DFID, the Department of Health and Social Care and other government bodies to improve coordination and collaboration on tackling global health threats.

In many cases we can relate these improvements directly to ICAI’s findings and recommendations. It is rarely the case, though, that progress is only attributable to ICAI’s reviews, especially when other significant learning processes such as DFID’s Country Development Diagnostics work are taking place at the same time. Often ICAI’s recommendations have their strongest influence when they inform and encourage departmental initiatives and reforms that are already under way.

Cross-cutting themes

This follow-up process has highlighted issues of strategic importance to UK aid spending that cut across many of our 2017-18 reviews. We will continue to monitor these in ongoing and future reviews:

• Do no harm: Several of this year’s follow-up exercises show improvements in how UK aid actors adhere to the principle of ‘do no harm’.
• The changing profile of UK aid: Dual-purpose funds, combining development and national interest goals, now spend a significant amount of UK aid. From a low starting point, these have made strong progress in developing systems and processes to ensure value for money and compliance with official development assistance (ODA) rules.
• From targets to transformative change: After a period characterised by targets-based programming, there is now an increased focus on transformative results in DFID’s programming,  such as promoting economic transformation and building equitable and sustainable public services to help achieve the Sustainable Development Goals.

Outstanding issues

There are nevertheless areas of strategic significance where further follow-up next year will be beneficial. This year we identified three reviews for further follow-up:

• Governance: This was ICAI’s last review in the 2017-18 review cycle, with the least time available for DFID to respond to the recommendations. We found that most actions were at too early a stage for us to judge their appropriateness, so we plan to return to all this review’s recommendations in next year’s follow-up exercise.
• The GCRF: We saw significant improvements in many areas but would like to return to four issues:
  • Given the very devolved structure of the GCRF, we would like to look again at how the Department for Business, Energy and Industrial Strategy (BEIS) ensures overall accountability for the Fund.
  • We would like to revisit the block grants forwarded to the devolved funding councils via the GCRF. There have been clear improvements in the monitoring of ODA compliance since our review, but it is too early to tell how far the measures taken will align the funding councils’ GCRF expenditure with the Fund’s strategic objectives and ensure value for money. We also want to look again at how BEIS monitors and ensures accountability for how these block grants are spent.
  • The newly created research hubs are a promising innovation and we would like to follow up on how they contribute to the potential development impact of the GCRF.
  • We will also look at the effectiveness of the new cross-government Strategic Coherence for ODA-Funded Research Board in influencing the allocation and delivery of UK ODA funds for research and innovation over time.
• Irregular migration: Overall, we found good progress on the outstanding issues we raised in last year’s follow-up. However, DFID has decided not go ahead with plans for an independent evaluation of its flagship migration-related programme, the SSS II, relying instead on alternative arrangements for monitoring, rapid research and learning. Because this happened subsequent to our follow up assessment, we plan to review these revised arrangements as part of next year’s follow-up.

Introduction

ICAI provides robust, independent scrutiny of the UK’s ODA to assist the government in improving the effectiveness and impact of its interventions and to assure taxpayers of the value for money of UK aid spending. Our main vehicle for this scrutiny is the publication of reviews on a broad range of topics of central concern to the UK’s aid strategy. A crucial part of these evaluations is our annual follow-up process, where each year we return to the recommendations from the previous year’s reviews to see how well they have been received and acted upon by the relevant government department or public body.

The follow-up process is structured around the recommendations provided in each ICAI review (see Figure 1 for an illustration of the process). The government has six weeks to provide a formal response to ICAI reviews, where it sets out whether it accepts, partially accepts or rejects ICAI’s recommendations and provides a plan for how it will address the issues raised. This is followed by a hearing in the ICAI sub-committee of the International Development Committee (IDC). Then, between six and 12 months after the publication of the original review (the amount of time depends on how early in ICAI’s annual review cycle the relevant report was published), ICAI conducts a formal follow-up exercise. The aid-spending departments are asked to provide an update on what they have done, and what they plan to do but have not yet implemented, in response to ICAI’s recommendations. Based on this response, supported by documentary evidence and interviews, we investigate the extent to which the government has done what it promised to do and – also considering any additional relevant actions – determine whether we find this to be a suitable and adequate response to the original recommendation. After publication of the follow-up review, there is an IDC hearing on its findings.

Figure 1: Timeline of ICAI’s annual follow-up process

*We conduct our follow-up assessment on an annual basis, starting in January and publishing in the summer. The follow-up covers a number of reviews which are selected according to their publication date (if a review is only published a few months before the follow-up process then it will be covered in the following year). If we are not satisfied with the government’s progress on any of our recommendations, we then follow up on those areas again during the next year’s assessment.

This report presents the results of the annual follow-up exercise for 2017-18 reviews. It provides a record for the IDC and the public of how well the UK government has responded to ICAI recommendations and findings. The follow-up process is also an opportunity for additional interaction between ICAI and responsible staff in DFID and other government departments, offering feedback and learning opportunities for both parties. The follow-up process is a central part of our work to ensure maximum impact from our reviews.

ICAI published eight reviews in 2017-18 and all 43 recommendations made in these reviews were accepted or partially accepted. This report contains an analysis of three important themes that cut across many of the reviews being followed up this year, together with a series of tables summarising our assessment of progress on the recommendations in these reviews, and on four areas of concern that were outstanding from last year’s follow-up exercise. We also include a brief explanation of our follow-up methodology.

A more detailed narrative account of ICAI’s recommendations and the government’s response and actions is provided in an annex to this report. This narrative is also available on the ICAI website, alongside the relevant reports and government responses.

Methodology

When we follow up on the findings and recommendations of our past reviews, we focus on four aspects of the government response:

• Whether the actions proposed in the government response are likely to address the recommendations.
• Progress on implementing the actions set out in the government response, as well as other actions relevant to the recommendations.
• The quality of the work undertaken and how likely it is to be effective in addressing the concerns raised in the review.
• The reasons why any recommendations were only partially accepted none of the 2017-18 recommendations were rejected).

We begin by asking the relevant government department to prepare a brief note, accompanied by documentary evidence, summarising the actions taken to implement the response to our recommendations. We then check that account through interviews with the responsible staff, both centrally and in country offices, and by examining relevant documentation. Where necessary or useful, we also interview external stakeholders, including other UK government departments, multilateral partners and implementers. To ensure we maintain sight of broader developments, we also assess whether ICAI’s findings and analyses have been influential beyond the specific issues raised in the recommendations.

The follow-up process for each review concludes with a formal meeting between a commissioner and the senior civil service counterpart in the responsible department.

At the end of the follow-up process, we identify issues of continuing strategic importance where we judge the action taken by the department in question to have been inadequate or incomplete. These issues are flagged for further follow-up the following year.

We also use the follow-up review as a learning opportunity for ICAI, about the impact of our reviews on UK aid and how we communicate our findings and recommendations in order to achieve maximum traction with the government.

Cross-cutting themes

During the course of our follow-up exercise, we noted three themes of strategic importance that recurred across several of the 2017-18 reviews and remain relevant in our current review programme.

Do no harm

‘Do no harm’ is a key principle of good development practice, particularly in fragile or conflictaffected settings. It commits aid actors to avoiding causing inadvertent harm to vulnerable individuals and groups, and to considering the risk of harm and how it can be mitigated in the design and implementation of aid programmes.

If risks cannot be sufficiently mitigated, then programmes should not be launched. For instance, after conducting a risk analysis, DFID decided not to implement certain projects aimed at vulnerable irregular migrants in Libya due to the many risks involved in providing material assistance in an environment of human rights abuses, kidnappings, slavery and extortion practices perpetrated against migrants in the country’s migrant detention centres.

Risks should be monitored during the lifetime of programmes, as changing circumstances – especially in volatile conflict zones – can throw up risks that were unforeseen in a programme’s design phase.

The principle of doing no harm will continue to be central to the UK aid strategy, as the ideas from the Fusion Doctrine on national security, presented in the March 2018 National Security Capability Review, begin to inform aid programming. UK aid is an important link in the chain of political, economic and influencing activities which support the government’s national security objectives (see Figure 2 below).

Figure 2: The objectives and tools of the Fusion Doctrine

Principle:To deploy security, economic and influence capabilities to protect, promote and project our national security, economic and influence goals.

In a speech in Cape Town in August 2018, Prime Minister Theresa May described how UK aid would contribute to the joined-up way of working set out in the National Security Capability Review and outlined a series of new strategic objectives for the UK aid programme. In addition to reaffirming long-standing UK aid commitments to humanitarian relief, empowering women and girls, and tackling climate change, she set out four thematic priorities for UK aid:

• addressing the root causes of conflict and fragility
• tackling cross-border threats
• promoting the rules-based international order
• building markets in frontier economies.

She also announced a new geographical focus, including on the Sahel region of Africa and the new ‘frontier markets’ such as Côte d’Ivoire and Senegal.

This prioritisation of tackling threats, fragility and conflict makes it crucial that all departments and funds that deliver UK aid have the necessary knowledge, skills and procedures in place to assess and mitigate against the risk of doing harm to vulnerable individuals. This entails careful analysis of the drivers of conflict and human rights risks, and the design and implementation of sophisticated theories of change and strategies for mitigating the risk of harm.

This year’s follow-up exercise found significant improvements in how UK aid actors adhere to the principle of doing no harm:

• The Conflict, Stability and Security Fund (CSSF) has made considerable progress towards strengthening its ‘do no harm’ approach, elevating it from a threshold compliance question to be considered at programme design to a more broad-based and substantial endeavour to place conflict sensitivity and human rights risk management at the heart of its programming.
• Both DFID’s and the CSSF’s irregular migration-related programming in the central Mediterranean now includes more stringent risk analysis of how vulnerable migrants will be affected before programmes are rolled out, based on improved theories of change and research on what works. We would, however, have liked to see the risk of harm incorporated more explicitly in these programmes’ monitoring and evaluation approaches. Disability in development: DFID’s new Disability Inclusion Strategy has stigma and discrimination against people with disabilities as one of three cross-cutting areas “which will be consistently and systematically addressed in all of [its] work”. While this is primarily an issue of inclusive development, attention to stigma and the effects of aid interventions on people with disabilities will also contribute to avoiding harmful unintentional effects of aid interventions.
• Inclusive growth: With the development of new diagnostic tools, new programming and the Inclusive Data Charter: Action Plan, DFID has put in place a strong and credible approach to counteract trends in many partner countries of excluding women, young people and marginalised groups from the benefits of aid programmes. This is primarily about inclusion rather than ‘do no harm’, but more thorough diagnostic work on how and why individuals and groups become excluded will also help DFID ensure that its own aid programming is not inadvertently supporting harmful exclusion practices.

The changing profile of UK aid

The UK continues to be one of the few countries to spend 0.7% of its gross national income on ODA. In 2018, this amounted to more than £14.5 billion, an increase of £487 million or 3.5% from 2017. DFID is no longer the only UK government department spending significant amounts of ODA. More than 25% of all UK aid in 2018 was accounted for by other government departments and by non-departmental expenditure (see Figure 3 below).

Figure 3: Growth in UK ODA spending outside of DFID from 2014 to 2018

The proportion of UK ODA spent by DFID, other government departments and non-departmental spenders

Dual-purpose funds, combining development and national interest goals, now spend a significant amount of UK aid.

Our review of the Prosperity Fund in February 2017 was the first of a series of reviews of major aid funds managed by departments other than DFID. This was followed by reviews of the GCRF, the CSSF and, most recently, the Newton Fund. In all of these reviews, ICAI found serious shortcomings, including:

• unclear governance and accountability structures
• lack of fund-level strategies, including clarity on primary and secondary objectives
• inadequate monitoring, evaluation and learning arrangements scaling up expenditure too quickly, before robust programme designs and management processes were in place
• in the case of the Prosperity Fund, the GCRF (regarding its original approach to block grants to the devolved funding councils) and the Newton Fund, pushing the boundaries of ODA eligibility.

The GCRF, the CSSF and the Prosperity Fund have responded well to the concerns raised by ICAI. The follow-up review found that these funds had made considerable progress in putting in place the necessary systems and processes to ensure value for money and compliance with ODA rules. In particular:

• The Prosperity Fund decided to delay allocations in order to develop a fund-level strategy, a theory of change with key performance indicators, a monitoring, reporting, evaluation and learning strategy and a procurement framework.
• The CSSF has made significant progress in instituting good aid practice in fragile and conflict-affected areas, including investments in thorough conflict sensitivity analysis and ‘do no harm’ risk assessments, results management, transparency and ODA compliance.
• The delivery partners have developed a strong strategy for their GCRF portfolios, paying much closer attention to achieving development impact, particularly through the development of research hubs and priority research areas and the appointment of challenge leaders.

However, challenges and risks remain, some of which are legacies of the lack of strategic direction and ODA management structures and processes when the funds were first launched:

• Value for money risks around rapid scale-up: While we are very pleased that the funds are now putting strategies and better management processes in place, in some cases this was only after much of their budget had been allocated. This damage was mitigated in the Prosperity Fund by the postponement of its first allocation round. However, in the case of the GCRF, it was only after around £1.3 billion of the total fund of £1.5 billion had been allocated to delivery partners that priority research areas were announced and the research hubs established.
• Learning within and across funds could be improved, particularly on ODA oversight: The GCRF’s improvements since the original ICAI review have been driven by delivery partners, not by BEIS, and cross-learning from the GCRF to the Newton Fund has been insufficient, despite similar concerns about oversight arrangements and the value for money of aid spending having been raised previously.
• Improvements are at an early stage: With each of the three funds covered by this year’s followup, new governance structures, strategic directions, monitoring and evaluation arrangements and other procedures for ensuring aid effectiveness and value for money have only recently been put in place, so it is too early to assess how they have been implemented and whether they will lead to stronger programming.

ICAI has contributed to driving the progress of these funds in developing more robust structures and procedures. The next stage of scrutiny should be on how these will translate into higher-quality programming and better value for money. These funds and other cross-government aid programmes will therefore continue to be an important focus for ICAI.

From target-based to transformative results

Between 2010 and 2015, the UK government set ambitious global results targets for the UK aid programme, with country offices accountable for demonstrating their contribution to these targets. Over a number of reports, ICAI raised concerns that the targets created pressures on programmes to maximise beneficiary numbers, without proper regard to the quality, equity or sustainability of the results.

In last year’s ICAI review of DFID’s approach to value for money (which we follow up on this year), we noted that a results management system which focused on output targets would measure the direct returns on individual aid investment, but might miss out on measuring and reporting on results at the country portfolio level. Such an approach risked focusing attention on the efficiency of delivery, rather than on whether the programme was likely to achieve the intended results.

A target-based results management system makes it difficult for DFID, ICAI or the general public to assess the aid programme’s contribution to transformative change – in other words how UK aid programmes combine to contribute to broader development efforts at the country level, such as the Sustainable Development Goals, or cross-cutting objectives like resilience, disability and inclusive development.

There are now some positive signs that DFID is moving towards a greater focus on transformative results, such as promoting economic transformation and building equitable and sustainable public services. In this year’s follow-up exercises, evidence we have seen of measures to strengthen transformative results includes:

• stronger emphasis on flexible, adaptive programming
• the introduction of a new diagnostic tool, the Country Development Diagnostics process, into the business planning process to help tailor programming to the needs of each partner country
• an emerging focus on managing results at the country and sector portfolio level.

DFID’s move towards transformative results is still a work in progress. There are key technical challenges that must be surmounted for this transition from a target-based to a transformational results approach to be successful. Chief among these is the considerable challenge of finding ways of reporting results that move from attribution of results at programme level to contribution analysis at portfolio level, while still maintaining rigour and accuracy. DFID has been consulting internally on how best to measure and report on transformative results, but no standard practice has yet emerged.

Summary of findings from individual follow-ups

This section provides a short summary of the main findings of our follow-up on recommendations for eight ICAI reviews and four outstanding issues from previous follow-ups. A fuller narrative account of trends, improvements and gaps for each of the reviews is provided as an annex at the end of this report.

The Global Challenges Research Fund

Since the publication of the ICAI review in September 2017, progress has been made in all four areas covered by our recommendations, often led by UK Research and Innovation (UKRI), the GCRF’s main delivery partner. We remain concerned that BEIS is providing insufficient oversight of the Fund, particularly over the block funding allocations to the four devolved funding councils. Mechanisms to ensure ODA compliance and value for money can still be improved.

The UK aid response to global health threats

There have been positive developments in response to all four of ICAI’s recommendations. A refreshed strategic framework is under development, which will highlight health systems strengthening work and facilitate wider engagement by adopting the internationally recognised ‘Prevent, Detect, Respond’ terminology. We also saw improvements in cross-government working and learning in support of the high priority that the UK government gives to global health security.

DFID’s approach to value for money in programme and portfolio management

There is a currently a wide-ranging process of reflection on business planning and results management systems under way within DFID, including on adaptive programming, diagnostics, portfolio management, portfolio-level results management and business case  development. Most of this work remains at the design and development stage. While progress has been fairly slow, we recognise that changes to core business processes necessarily take time. We welcome plans to introduce more emphasis on portfolio-level and transformative outcomes into DFID’s approach to assessing results and value for money.

Building resilience to natural disasters

There has been a good response to ICAI’s recommendations related to the strengthening and mainstreaming of resilience across DFID programming in high-risk country settings, particularly through its Country Development Diagnostics (CDD) process. The main actions proposed and under way are relevant and useful, but much remains at the preparatory stage. DFID should provide clearer guidance on strengthening measurement of resilience at the portfolio level.

The Conflict, Stability and Security Fund’s aid spending

The CSSF has made significant efforts to address the shortcomings identified by ICAI. It is possible that the work done to date has already led to more conflict-sensitive programming with clearer and more testable intermediate outcomes. Monitoring is becoming more rigorous as quality standards for programme design have been raised. Learning has been accelerated and there are signs that some important learning mechanisms are starting to become institutionalised.

DFID’s approach to disability in development

DFID’s new Disability Inclusion Strategy is an important step forward and addresses most of the issues raised in the ICAI review. The minimum standards for all business units are specific and ambitious. Achieving them within the required deadline is likely to call for significantly more resources than currently available to departments – in particular, staff with relevant experience and skills. However, we welcome the momentum that DFID has created around the disability inclusion agenda.

The UK’s humanitarian support to Syria

Despite the dynamic and challenging operating environment in Syria, DFID has engaged proactively with ICAI’s recommendations, with many positive developments. Looking forward, the design process for the 2020 DFID Syria programme portfolio will be one of the most significant ways in which DFID Syria can ensure that further progress is made. It will be important for DFID to ensure that learning from the past seven years of working in Syria is sustained throughout the planned restructuring and relocation of the team – while at the same time overseeing the uninterrupted delivery of a complex portfolio.

DFID’s governance work in Nepal and Uganda

This was the last review in the 2017-18 review cycle, leaving little time for DFID to respond. Since many actions are at an early planning stage, ICAI will return to all the recommendations in next year’s followup exercise.

Table 2: Summary of findings on outstanding issues from earlier reports

Conclusions

Overall, the response to ICAI’s recommendations from its 2017-18 reviews has been very encouraging, from DFID, other aid-spending departments and aid-spending funds. Last year, we noted how government actors new to aid spending were facing a steep learning curve, building up aid management capacity and developing, testing and bedding down processes to ensure that their work meets the high standards expected from UK aid. In our follow-up work for the CSSF, the Prosperity Fund and the GCRF we have seen significant improvements over the past year towards meeting these standards.

We do not expect all our recommendations to have equal resonance. But there are some in areas of strategic importance where it is currently too early to assess the impact of actions taken. We have identified three outstanding issues that we will return to in next year’s follow-up exercise:

• GCRF: While we have seen many improvements at this fund, we will return to the GCRF next year to assess: (i) the role of BEIS in providing overall accountability and responsibility in managing and assuring the GCRF, (ii) the block grants that are still being forwarded to the devolved funding councils via the GCRF, (iii) progress on the research hubs as they develop, in order to assess the effectiveness of this highly decentralised delivery model, and (iv) the effectiveness of the SCOR Board in influencing the allocation and delivery of aid-funded research and innovation programmes over time.
• Governance: We recognise that our follow-up took place too soon after publication, and we will therefore return to assess progress on all the recommendations in next year’s follow-up.
• Irregular migration: Overall, we found good progress on the outstanding issues we raised in last year’s follow-up. However, DFID has decided not go ahead with plans for an independent evaluation of its flagship migration-related programme, the SSS II, relying instead on alternative arrangements for monitoring, rapid research and learning. Because this happened subsequent to our follow up assessment, we plan to review these revised arrangements as part of next year’s follow-up.

Annex: Findings from individual follow-ups

In this annex, we present in more detail the results of our investigations of the government’s responses to 43 recommendations in eight ICAI reports published in 2017-18, as well as four outstanding issues from last year’s follow-up exercise. We focus on the most significant results from, or gaps in, the response. The presentation is chronological, beginning with the earliest published review from September 2017.

The Global Challenges Research Fund

ICAI published a rapid review of the Global Challenges Research Fund (GCRF) in September 2017. With a budget of £1.5 billion from 2016 to 2021, the Fund constituted a significant increase in the UK government’s spending on development-oriented research. The review found that although progress had been made in the first 18 months of the GCRF’s existence, the rapid development of the Fund meant that many elements of its strategy, governance arrangements and procedures were unclear or weak. The review made four recommendations, summarised in the table below.

Table 3: Summary of recommendations and the government’s response

Formulate a more deliberate strategy to encourage concentration on high-priority development challenges

The GCRF’s strategy, published by the time the ICAI review came out, had a broad approach to tackle all Sustainable Development Goals (SDGs), resulting in a scattered portfolio of research projects. To increase the chances of achieving transformative research impact, ICAI recommended the development of “a more deliberate strategy that encourages a concentration of research portfolios around highpriority global development challenges, with a stronger orientation towards development impact”. BEIS only partially accepted this recommendation, but there has been considerable activity to ensure stronger strategic coherence and development impact commensurate with a research fund of this size.

There are now six thematically distinct GCRF portfolios on global health, food systems, conflict, resilience, education and sustainable cities. The Fund has also launched 12 interdisciplinary research hubs around specific development challenges and appointed nine challenge leaders. Together, these initiatives encourage a concentration of interdisciplinary research portfolios around particular global challenges. As a result, the final portion of the GCRF’s budget (some £200 million) has been allocated with greater targeting and strategic direction.

BEIS has now strengthened its analytical capability and oversight over the GCRF portfolio with the appointment of dedicated staff and the formation of a new Portfolio & Operations Management Board, whose scope specifically includes oversight of programme-level portfolios. However, we are concerned that strategic oversight is still very devolved from BEIS to the delivery partners, with a reliance on UK Research and Innovation (UKRI), which is only one of the GCRF’s delivery partners.

We also remain concerned about the GCRF block grants allocated through the devolved funding councils, of which Research England is by far the largest. After ICAI’s original review, Research England required the submission of three-year official development assistance (ODA) strategies from all UK higher education institutions wishing to receive GCRF funding. It has approved the strategies from 107 institutions. While this constitutes a clear improvement in the monitoring of ODA compliance since the ICAI review, it is unclear at this point to what extent the allocations are aligned with the GCRF strategy. It is too early to tell whether the measures taken will be enough to ensure that the block funding constitutes good value for money in achieving development outcomes. We will therefore come back to this topic in next year’s follow-up review.

Develop clearer priorities and approaches to partnering with research institutions in the Global South

BEIS accepted ICAI’s recommendation to determine the GCRF’s priorities and develop deliberate approaches towards partnering with research institutions in the Global South. UKRI, the leading GCRF delivery partner, has taken several actions in response to this. It has led a series of global engagement events in developing countries to provide information about the funding opportunities available through the GCRF. It is also drawing in partners from the Global South in the development of the GCRF challenge hubs and creating North-South research networks around each challenge. South-based principal investigators are now allowed to lead bids on some funding calls.

UKRI has set up an international development peer review college with at least 95% of its membership drawn from countries on the list of ODA-eligible countries compiled by the OECD Development Assistance Committee. It is not yet clear how this college will sit alongside existing peer review mechanisms, but we hope to see a positive impact on the quality and equity of new GCRF research calls. The academies are also developing partnerships with key institutions in the Global South, such as the collaboration between the Royal Society and the African Academy of Sciences.

Provide a results framework for assessing the overall performance, impact and value for money of the GCRF portfolio

In our original review, we noted that not enough effort had gone into specifying the GCRF’s objectives across different thematic areas or developing results indicators. We recommended that BEIS should develop a results framework for assessing the overall performance, impact and value for money of the GCRF portfolio, drawing on DFID’s guidelines on value for money in research and evidence programming. If done well, the process of developing a results framework could help to drive a higher level of strategic coherence across the Fund.

BEIS responded positively to this recommendation. It has now developed a Fund-level theory of change, led by the monitoring, evaluation and learning (MEL) service provider, but using workshops as a cocreation and collaboration mechanism. An evaluation approach has been delivered by the service provider and will shortly be published. BEIS has also awarded a contract to develop a suite of joint key performance indicators (KPIs) for the GCRF and its sister fund, the Newton Fund. However, at this late stage, their usefulness will be limited.

Separately, UKRI has plans to review the effectiveness of its policies and practices to ensure ODA compliance, not just at the award stage, but throughout the lifespan of the grant. It is considering introducing new mechanisms to ensure continued ODA compliance after grants have been awarded. Until recently, compliance assessments had only taken place at the application stage.

While important improvements are taking place, they are coming too late in the GCRF programming and funding cycle. We also remain concerned about whether the knowledge and learning generated by the GCRF’s external MEL service provider will be sufficiently taken up by BEIS.

Develop a standing coordinating body for investment in development research across the UK government

ICAI’s recommendation to develop a cross-departmental standing coordination body to clarify roles and responsibilities, avoid duplication and overlap, and facilitate exchange of learning was made to encourage the setting up of the Strategic Coherence for ODA Funded Research (SCOR) Board. The board was established in December 2017, with a senior academic leadership, to coordinate the proliferation of UK spending on development-related research across a range of different government departments. It also functions as the governance body for the UK Collaborative on Development Research (UKCDR), which brings together government departments and research funders working in international development. SCOR has approved a new strategy for the UKCDR, and the UKCDR has begun a mapping exercise and analysis of UK development research spending.

SCOR is likely to improve cross-government information exchanges, but it is an advisory body and therefore not actually empowered to coordinate between government departments. It remains unclear what effect the board will have on BEIS’s management of the GCRF and its sister fund, the Newton Fund, or on the level of coordination between BEIS and other government departments such as DFID and the Department of Health and Social Care (DHSC).

Conclusion

Since the publication of the ICAI review in September 2017, progress has been made in all four areas covered by our recommendations – often led by the GCRF’s delivery partners, and most notably by UKRI. However, we are concerned that BEIS continues to delegate a significant level of the oversight and accountability functions of the Fund, along with the majority of the delivery, to its delivery partners. We would also like to return to the issue of the block funds being allocated to Research England, to assess whether this spending is sufficiently aligned to the GCRF’s objectives and priority research themes.

ICAI’s recently published amber-red review of the Newton Fund found similar issues about governance, accountability and strategic direction from BEIS. It noted problems with the late development of value for money frameworks and insufficient attention to achieving development impact at a scale and ambition proportionate to the level of ODA spending. We plan to return to these topics in next year’s follow-up exercise, to assess improvements for the GCRF and the Newton Fund in tandem.

The UK aid response to global health threats

The UK’s aid response to the growing risk of global health threats is led by DHSC, but with strong involvement from DFID and a wide range of government departments. ICAI’s review, published in January 2018, covered the totality of the UK aid response to global health threats from 2014. It found that the UK government responded rapidly to address weaknesses in the international response system exposed by the Ebola crisis in West Africa, establishing a portfolio of relevant and often pioneering programmes and influencing activities. ICAI made four recommendations, summarised below, to help the government build on this good foundation.

Table 4: Summary of recommendations and the government’s response

The UK government should refresh its global health security strategy, with emphasis on health systems, research and mechanisms for collaboration

The ICAI review found good programming on global health threats within both DHSC and DFID, but urged the government to refresh and improve its overall ‘Stronger, Smarter, Swifter’ strategic framework. There are significant improvements under way in response to this recommendation. DHSC and DFID are in the process of refreshing the strategic framework and have committed to sharing it externally when completed. The work has been delayed due to capacity constraints caused in particular by the Ebola outbreak in the Democratic Republic of the Congo (DRC), but descriptions of the work done indicate that the concerns raised in the ICAI review are being addressed. A range of stakeholders have been included in the process of finalising the strategic theory of change. The framework is being adapted to follow more closely the internationally recognised terminology of ‘Prevent, Detect, Respond’. This will facilitate coordination with other donors and country partners, as well as engagement with the private sector.

The refreshed framework reportedly presents the government’s rationale for embedding global health security work within a health systems strengthening approach, making explicit links to the World Health Organization (WHO)’s work and internationally recognised terminology, such as health systems building blocks. We also saw signs of a practical shift in programming away from vertical interventions into particular health issues and towards health systems strengthening and capacity-building programmes.

These improvements have taken place at the same time as the UK’s close involvement in the international response to the outbreak of Ebola in the DRC (see Box 2 below). Drawing on the experience of this outbreak, lessons on the importance of having effective systems and capacity to respond to outbreaks in fragile and conflict-affected settings and ways of achieving this are being incorporated in the government’s new strategic framework. These include a focus on the knowledge and mechanisms needed to facilitate and coordinate deployment of a range of international actors (UK and others) and to keep response systems going for long periods in challenging contexts.

The Department of Health and Social Care and DFID should strengthen and formalise cross-government partnership and coordination mechanisms for global health threats, including regular cross-government simulations

The ICAI review found detailed evidence of effective cross-government mechanisms to share intelligence on disease outbreaks but noted that cross-government coordination at the strategic level and between programmes could have been improved. The Global Health Oversight Group (GHOG) was overly focused on programme monitoring, leaving gaps in strategic leadership and coordination, and there was not always a shared vision between DFID and DHSC – particularly on health systems strengthening.

The government only partially accepted ICAI’s recommendation on cross-government coordination, arguing that government departments and agencies were already working closely together. However, improvements have taken place since the publication of our review. GHOG has expanded its membership to include all government departments and agencies involved in delivering the government’s objectives for global health security, and a range of relevant government actors are holding regular coordination meetings and discussion forums. There has been a strong emphasis on close cross-government collaboration to deal with Ebola outbreaks in the DRC. These add up to significant improvements, although currently GHOG does not meet often enough to be fully able to spearhead cross-government cooperation and provide strategic leadership.

We also saw evidence of improved collaboration taking place at country level, including the development of common theories of change and monitoring and evaluation indicators by DFID and Public Health England (PHE). We found strong collaborative efforts in Sierra Leone, with weekly meetings between the international community and Sierra Leone’s Ministry of Health. After severe floods in 2017, the Ministry of Health reacted quickly, with support from the UK government, using existing data to mobilise effectively and averting serious disease outbreaks. A similar positive relationship was described in Burma, where DFID health advisers now work closely with PHE and Burma’s Ministry of Health to ensure that preparedness strategies are in place.

Ensure that DFID has sufficient capacity in place to coordinate UK global health security programmes and influencing activities in priority countries, including on strengthening national health systems

The ICAI review suggested that there was room for improving coordination of global health threats programming at country level, to maximise the collective impact of UK programming in areas such as WHO reform, International Health Regulation (IHR) compliance and health systems strengthening. DFID health advisers and other personnel are key to fulfilling this, but they are often overstretched. We recommended that the government should ensure that DFID has sufficient capacity in place to coordinate UK global health security programmes and influencing activities in its priority countries, including on strengthening national health systems.

The government accepted this recommendation. An increase in PHE activities has contributed to closer collaboration at country level, with DFID and PHE working together in countries where the IHR Strengthening programme has current or planned presence: Burma, Ethiopia, Nigeria, Sierra Leone, Pakistan and Zambia. The Fleming Fund, a UK aid programme aimed at helping low- and middle-income countries combat antimicrobial resistance (AMR), has led on establishing coordination mechanisms and cross-department working on AMR  detection, with crucial help from DFID health advisers. All stakeholders we talked to reported that the role of DFID health advisers is central to collaboration efforts. However, across DFID, and particularly in Burma, health advisers continue to be overstretched, navigating the complex political landscape of their country. DFID informed us that the UK government is recruiting seven more staff members to work on health security in Africa, which will be very helpful. However, by the end of May 2019, the recruitment process had not yet been completed.

DFID and the Department of Health and Social Care should work together to prioritise learning on global health threats across government

The government accepted ICAI’s recommendation that DFID and DHSC should work together to prioritise learning on global health threats across government. ICAI recommended that the two departments jointly oversee the development of a broad evaluation and learning framework, conduct regular reviews of what works (and represents good value for money) across the portfolio, and put in place a shared approach to the training and development of health advisers.

The new strategic framework and theory of change will facilitate cross-departmental coordination and give strategic direction to evaluation and learning. As the framework has not yet been published, we do not know whether it will include sufficient mechanisms to share learning at a strategic level and to ensure a broad programme of evaluations. However, the government’s response to the ICAI review, and a flurry of recent learning and evaluation activities among the main government global health security stakeholders, show that the need to prioritise learning is understood. We believe GHOG could take a strategic lead on this work, particularly with developing mechanisms to ensure that learning is disseminated across government and shared with external partners.

Conclusion

There have been positive developments in response to all four of ICAI’s recommendations. Although we have not seen the refreshed strategic framework, we understand that it will highlight health systems strengthening work and will facilitate wider external engagement by adopting the ‘Prevent, Detect, Respond’ terminology. We have seen clear improvements in cross-government working and learning in support of the government’s global health security objectives.

DFID’s approach to value for money in programme and portfolio management

This ICAI performance review explored DFID’s approach to value for money: the department’s progress on embedding value for money into its management processes and whether its efforts were in fact helping to improve value for money. The review was conducted in parallel with two other ICAI reviews on DFID’s procurement practice, which we will follow up on next year. Since value for money is both a process and an outcome and cuts across all aspects of DFID’s operations, we did not score the review, but we highlighted areas where DFID could do better and offered five recommendations.

Table 5: Summary of recommendations and the government’s response

DFID country offices should articulate cross-cutting value for money objectives at the country portfolio level, and should report periodically on progress at that level

For many years, DFID has not had country-level results frameworks, narrative reporting or evaluations. Country offices have been required to report on their contribution to department-wide goals, but not against the objectives in their country business plans. We therefore recommended that DFID country offices articulate value for money objectives at the portfolio level, so that they could be reflected in the design of individual programmes. We also recommended that country offices report periodically against these objectives.

DFID partially accepted this recommendation. It has begun a wide-ranging process of reflection on its results management system as a whole. This internal review has identified the need for results reporting at sector and country levels, to capture not only how much country offices contribute to global targets but also their contributions to transformational change, such as promoting inclusive growth and building sustainable public services. DFID is also putting in place new management information systems (over and above the Portfolio Quality score, which the ICAI review criticised) to help manage portfolio quality.

DFID is in the middle of a strategic planning phase ahead of the next business planning cycle and the Spending Review, so there has been no concrete action yet on this recommendation. DFID anticipates that new country-level results reporting systems will emerge after new country business plans are in place, from 2020-21 onwards.

Drawing on its experience with introducing adaptive programming, DFID should encourage programmes to experiment with different ways of delivering results more cost-effectively, particularly for more complex programming

The report found that DFID’s value for money approach was mainly focused on controlling costs and achieving efficiencies in programme delivery. This works well for straightforward, predictable interventions (such as vaccines), but is less appropriate for complex programmes in unpredictable environments, which call for a more flexible and adaptive approach. ICAI therefore recommended that DFID encourage programmes to experiment with different ways of delivering results cost-effectively. DFID accepted this recommendation and has proceeded with some useful initiatives.

Prominent among these, DFID is working with the Overseas Development Institute on a LearnAdapt initiative, to develop a value for money approach for adaptive programmes. For adaptive programmes, the best combination of activities and outputs to achieve the desired outcomes is not known in advance. Cost per output is therefore not a good measure of value for money. Instead, the focus should be on whether the programme is innovating and generating actionable learning. The shift in the LearnAdapt initiative from ‘static’ value for money (how efficient is this intervention?) to ‘dynamic’ value for money (how efficiently are we learning what works?) is broadly in line with ICAI’s recommendation.

Work remains at the preparatory or planning stage, and concrete measures have not yet been introduced. It is therefore too early for any detailed assessment of its impact.

DFID should ensure that principles of development effectiveness are more explicit in its value for money approach

‘Development effectiveness’ refers to a set of internationally agreed principles governing the quality of development cooperation. For example, working with and through local partners may be more costly in the short term, but may generate more sustainable results, making it better value in the long term. ICAI’s review found that country offices did not articulate their development effectiveness priorities or approach, and individual programmes made no effort to track their compliance with or contribution to development effectiveness goals and principles. We therefore recommended that principles of development effectiveness – such as ensuring partner country leadership, building national capacity and empowering beneficiaries – should be more explicit in DFID’s value for money approach, and that these principles should also be reflected in individual programmes.

DFID accepted this recommendation but has taken no concrete action in response to it. In interviews, some DFID staff argued that development effectiveness principles were built into programme design, even if not explicitly labelled as such. However, between 2012 and 2016 the UK slipped 12 places on the Center for Global Development’s Quality of Official Development Assistance index, from three to 15 out of 27 places. We found that development effectiveness principles are not explicit enough in DFID’s business processes, which opens up risks of deteriorating standards in development cooperation. This may be partly due to a shift of focus towards fragile and conflict-affected states, where the principles are more challenging to apply.

Be more explicit about assumptions underlying the economic case for interventions and monitor and reassess these at specific points in the programme cycle

DFID’s ability to achieve value for money is closely linked to the quality of the economic appraisal (such as a cost-benefit analysis) set out in each programme’s business case. These assessments are often sensitive to assumptions that may change or be disproved over the life of the programme. DFID does not systematically monitor and verify these assumptions, or routinely revisit its appraisal during the life of the programme or following significant changes. ICAI therefore recommended that the assumptions underlying the original value for money proposition are made more explicit and incorporated into programme monitoring arrangements, and that economic appraisals are reassessed at some point during the life of the programme. We also recommended that senior responsible owners should determine whether a reassessment is needed following material changes in the programme, results targets or context.

DFID partially accepted this recommendation. Rather than investing in rerunning the economic appraisal, DFID has instead chosen to revise its annual review process to include a reassessment of the programme’s theory of change and the overall value for money case, including by reference to assumptions set out in the business case. This is a reasonable alternative to ICAI’s suggestion. The new annual review guidance encourages reviewers to consider what depth of review is required at the current point in the programme cycle. A new section has been added, requiring an assessment of the logic and assumptions of the theory of change. The value for money assessment must now be reassessed against the assumptions in the business case, and an explicit decision made as to whether the programme should continue from a value for money perspective.

As part of our follow-up exercise, we assessed a selection of 14 recent annual reviews in Pakistan, Malawi and Nigeria that had been carried out with the new template and guidance. We found that the depth and rigour of the analysis of the theory of change varied, but were generally adequate. Importantly, there were two instances where DFID had identified that changes in circumstances required a rerun of the cost-benefit analysis in the business case and commissioned the monitoring and evaluation service provider to do the assessment. This indicates positive changes in DFID’s practice.

Annual review scores should include an assessment of whether programmes are likely to achieve their intended outcomes in a cost-effective way

The ICAI review was concerned about a lack of rigour in how DFID generated annual programme scores. It noted that logframe targets were frequently changed without strong controls or transparency, making it difficult to tell whether improvements in programme scores resulted from genuine improvements in performance or from a downgrading of the targets. We also noted that annual review scores (which are a key control point in the programme cycle) were based purely on outputs, rather than outcomes, and were therefore not a good measure of value for money.

DFID only partially accepted ICAI’s recommendation to incorporate an assessment of outcome-level achievement into the scores, and to introduce more control over changes to logframe targets. It had conducted its own internal review of the annual review process in parallel with the ICAI review, finding that annual reviews were generally fairly rigorous, but in need of some adjustments. It considered moving to outcome scoring, but decided against it on the grounds that there may not be an objective basis for scoring at outcome level early in the programme cycle. DFID was also concerned that, if programmes had to quantify outcomes right from the start in order to score well, this would affect programme design and theories of change, creating unhelpful incentives to focus on easily achievable targets rather than long-term objectives.

Given these concerns, DFID opted for a different solution. It revised the annual review template to include detailed reconsideration of the theory of change and whether the programme is on track to achieve its intended outcomes. It also introduced greater transparency over changes to logframe targets. There is now a ‘change control tab’ in the logframe format, which records all changes over the life of the programme, making it easy to see if targets have been revised downwards. The new annual review guidance states: “Ideally changes should not be made to any targets or indicators less than six months before they are being reviewed unless agreed with the Head of Department.” This is a fair response to ICAI’s request for greater transparency and quality control over target setting.

We checked a sample of recent annual reviews for this follow-up. Although quality varied, in general we found a more rigorous approach to assessing the likelihood of achieving programme outcomes. While this improvement is welcome, we are nevertheless concerned that the primary focus of accountability continues to be on the delivery of outputs by the implementing partner.

Conclusion

There is a wide-ranging process of reflection on business planning and results management processes taking place within DFID, with changes under way or in the pipeline on adaptive value for money, diagnostics, portfolio management, portfolio-level results management and programme design. Most of this work remains at the design and development stage. While progress has been fairly slow, we recognise that this is a time of uncertainty for the department and that changes to core business processes necessarily take time. While we were disappointed with the response to our recommendation on development effectiveness, there are also several positive trends, including a stronger emphasis on portfolio management. We are encouraged to see a shift from a purely target-driven approach to results and value for money towards transformative results, which corresponds with the thrust of our recommendations.

Building resilience to natural disasters

The review was concerned with how DFID embedded risk analysis and resilience throughout DFID’s country offices, across DFID’s programmes and with governments and development partners more broadly, how it disseminated evidence and supported country office staff in implementing disaster resilience mainstreaming, and how it measured and monitored its resilience results. We made five recommendations, as summarised in the table below.

Table 6: Summary of recommendations and the government’s response

In partner countries with significant risks from natural disasters, DFID should keep its risk assessments and
resilience strategies up to date

The ICAI review noted that DFID’s multi-hazard risk assessments were not updated or incorporated as a regular part of country office business planning processes and was concerned that, without an upto-date understanding of the evolving context, DFID would be less able to target its resilience work effectively. With more governments and development partners engaging on resilience to natural disasters, coordination would be difficult without a shared understanding of risks.

DFID has responded to ICAI’s recommendation by ‘baking’ risk analysis and resilience approaches into its Country Development Diagnostics (CDD) process, prompting country offices to consider risks and resilience comprehensively and coherently as part of their diagnostics work. However, there is scope for country offices to ignore these prompts, particularly when there are competing priorities. We would welcome a follow-up assessment by DFID in due course to see whether country offices have in fact implemented the results of this analysis into their portfolios.

In terms of external engagement, there are limits to the extent that DFID can influence governments and development partners, but it has made progress both practically and at the policy level. DFID has linked core funding to UN agencies to a set of progress indicators including on preparedness planning, which seems to have contributed to strengthening the quality and quantity of this. DFID has also sought to influence global policy on resilience through its participation in a range of international forums including the Global Resilience Partnership and Resilience Evidence for Decisions in Development Initiative.

DFID offices in high-risk countries should adopt a portfolio approach to resilience, articulating how their efforts in different sectors and areas will work together to build resilience

Resilience to natural disasters is best pursued as a cross-sectoral effort. Our review noted that DFID’s country-level resilience strategies were in some cases little more than a short checklist of actions. Without a thorough and up-to-date country-level strategy, DFID is not well placed to identify mutually reinforcing approaches to resilience to natural disasters across its sector programming.

DFID accepted our recommendation to adopt a portfolio approach to resilience in high-risk countries, using the CDD process as its main vehicle for responding to ICAI’s concerns. DFID told us that the CDD “ensures that each DFID country office takes stock of the current state of development, barriers and opportunities – including resilience – and considers whether programme portfolios are focused accordingly”. The CDDs are peer-reviewed and guidance notes on how to cover resilience issues were issued at the start of the CDD process. In addition, resilience is also one of the cross-cutting issues in DFID’s review of its approach to measuring results.

While these processes are ongoing, there are already signs that they are leading to a better understanding in DFID country offices of the need to address risks and resilience more systematically. However, we also saw that country offices with strong, competing demands struggle to prioritise the embedding of resilience. As noted above, since the CDD process merely encourages country offices to consider resilience as a cross-cutting issue, there is a risk that some of them will not do so.

DFID should develop its guidance on how to measure resilience results, providing options that can be adopted by country offices according to their specific contexts and needs

The ICAI review noted that although DFID embarked on the process of mainstreaming resilience across the department more than six years ago, DFID – like other donors – continued to struggle to measure resilience results. DFID accepted our recommendation to develop guidance on measuring resilience results that can be adapted to the needs and contextual demands of country offices, but progress has been slow.

For programmes that are specifically aimed at improving resilience, DFID has invested in revising its main key performance indicator on resilience, KPI4. It is also engaging in international resilience measurement forums and using lessons from its Building Resilience and Adaptation to Climate Extremes and Disasters (BRACED) programme’s approach to measurement, monitoring and evaluation.

DFID has made less progress on measuring resilience results in programmes where resilience is a secondary aim, rather than a primary focus. Given the wide range of resilience objectives of these programmes, and the very different contexts within which they operate, there is inevitably no single measure of performance. However, the literature review of measurement practices currently being conducted by DFID Nepal may offer ways forward. The use, by a growing number of country offices, of specialist monitoring, evaluation and learning units or contractors also offers the potential to capture secondary resilience outputs/outcomes or to focus specific attention on resilience.

DFID should undertake a stocktake of its work on resilience in high-risk countries to inform its country strategies – periodically, or following significant natural disasters

ICAI’s review found that DFID had not fulfilled its commitment to report bi-annually on its progress with mainstreaming resilience and recommended that DFID should undertake regular stocktakes of its work on resilience in high-risk countries. DFID only partially accepted this recommendation, arguing that it already assesses resilience and preparedness in high-risk countries through internal ‘lessons learned’ reviews in the wake of significant natural disasters and as part of the programme management cycle.

There have so far been no specific actions in direct response to this ICAI recommendation. DFID’s resilience contributions (to conflict, instability and natural disasters) have been reviewed in recent evaluations of the department’s humanitarian responses to the conflict in South Sudan, the Rohingya refugee crisis and the Somalia famine. The CDD process will also prompt routine analysis of risk and the mainstreaming of resilience, but it is too early to judge how adequately this process will ensure that country offices consider resilience as part of their country strategies (see Recommendation 1).

DFID should establish a community of practice to promote the continuing mainstreaming of resilience to natural disasters

Since resilience is a cross-cutting theme across all of DFID’s work, responsibilities for advancing different aspects of its resilience agenda have been spread across the department, and in particular split between the Conflict, Humanitarian and Security Department and the Climate and Environment Department. A central repository of knowledge and technical expertise on resilience could help ensure that dissemination of learning on resilience between programmes and across sectors takes place. We recommended, also based on DFID’s own aspirations, that a community of practice should be established to promote the continuing mainstreaming of resilience across the department.

Since the publication of our review, DFID has established an internal, department-wide community of practice, managed and moderated by a DFID staff member, which can host resources and facilitate cross-programme links and peer-to-peer support. The community of practice is supported by the development of a ‘one-stop shop’ for learning material on DFID’s intranet, including evidence generated by programmes and research from a range of policy areas. A learning strategy for resilience is in the process of being developed. The success of the community of practice will depend on sufficient resources being allocated to maintain its functions. DFID should continue to draw on good practice from similar communities of practice elsewhere (such as in the Rockefeller Foundation) to identify and replicate the ingredients necessary for success.

Conclusion

The actions proposed and under way are relevant and useful responses to the ICAI resilience review’s recommendations and concerns. However, the community of practice is not yet sufficiently established and improvements in response to our other recommendations remain mostly at the preparatory stage. There is scope for DFID to provide clearer guidance on strengthening measurement of resilience at the portfolio level.

The Conflict, Stability and Security Fund’s aid spending

The Conflict, Stability and Security Fund (CSSF) was established in 2015. It is accessible to a range of government departments and combines ODA and non-ODA funding to support the implementation of National Security Council (NSC) strategies. ICAI’s March 2018 review focused on how well this cross-government fund spent UK aid to tackle conflict, instability and insecurity and promote sustainable peace and security in fragile and conflict-affected countries.

ICAI gave the CSSF an amber-red score. The Fund’s country and regional portfolios often lacked a clear logic connecting the activities they supported to the objective of promoting sustainable peace, stability and security. Results management practices were inadequate, and there was little reliable data to assess whether projects were achieving their intended results or delivering value for money. Most of the programmes we reviewed showed design or implementation flaws. We made six recommendations, summarised in the table below. The CSSF has achieved significant progress in its response to most of them.

Table 7: Summary of recommendations and the government’s response

Introduce country or regional plans that specify how CSSF activities will contribute to National Security Council objectives

ICAI recommended that the CSSF introduce country or regional plans specifying how its portfolios of aid programmes and influencing efforts would contribute to achieving NSC objectives, the intermediate outcomes that the portfolios would achieve, and the assumptions that needed to hold for this to happen.

The government only partially accepted this recommendation. It recognised the need to set out more clearly how programmes deliver against national security objectives, but did not want to introduce CSSF-specific country or regional plans. Instead, the Fund has invested in strengthening programmelevel theories of change, which had been weak and have now improved considerably. The CSSF is now routinely developing theories of change of good quality, with clearly defined goals – including intermediary objectives – and more transparency around underlying assumptions on how to achieve these goals. Annual reviews now include the reporting of outcomes, not just outputs.

Although much of this is work in progress, there has been a step change in the quality of programme-level documentation since our original review. The feedback from CSSF staff during the follow-up exercise has been that the improved theories of change and annual reviews are also improving data collection. The annual review is also used as an opportunity to assess whether outcome targets remain adequate or relevant as contexts change.

When CSSF projects have influencing objectives, make them explicit and report progress on them to allow assessment of value for money

Our review found that there was often a discrepancy between the stated objectives of a programme and its actual objectives of achieving diplomatic access and influencing. Since these objectives were not stated, results were never tested and value for money could not be assessed. We therefore highlighted the need for the CSSF to explicate and test its influencing objectives. The government accepted this recommendation. Influencing and access outcomes and indicators have been included in the more recent theories of change (discussed above) and results frameworks, making measuring the achievement of influencing objectives now possible, if still difficult. Instead of hiding access and influencing goals, these are now presented as explicit and legitimate objectives, with annual reviews beginning to critically reflect on the issue – recognising that the results of influencing activities are not easy to measure and need a flexible, adaptive approach.

Programmes should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm

The ICAI review found that human rights assessments were not always conducted before the start of programmes (or indeed at all) and that, when CSSF interventions entailed working with counterparts that may do harm, mitigation measures were sometimes superficial. We were concerned that, since human rights risk assessments were inconsistent and programme monitoring was often weak, we did not know whether CSSF programming was causing harm. The government accepted the recommendation that the CSSF should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm.

There has been a swift and thorough response to this recommendation. The CSSF has strengthened its guidance and training, and introduced new processes to identify, manage and mitigate risks of doing harm. It has incorporated the risk of doing harm into programme risk registers and in the annual review templates and continued the development of its ‘conflict sensitivity marker’, which is currently being refined and piloted in 20 programmes (by January 2019, with another 20 in the pipeline). The marker is structured around three progressive themes – awareness, adaptiveness and accountability – and the pilot exercises have been thorough and useful: they have revealed that the conflict sensitivity of CSSF programmes is of variable quality, with some doing very well and others in need of significant improvements.

The CSSF has also conducted a large, systematic cross-programme assessment of the use and quality of 70 Overseas Security and Justice Assessments. These learning activities have been supported by centrally based conflict advisers who travel to CSSF teams and help  implement and consolidate change processes.

Ensure all new programming includes adequate results management and measures to assess value for money and rectify gaps in existing programmes

Our review found that the CSSF’s results management was weak. The tools it used were unimaginative and developed with a one-size-fits-all mentality, unsuited to a complex and fast-changing area of programming in fragile and conflict-affected areas. The CSSF did not have the evidence needed to assess whether expenditure represented value for money. Our recommendation had two layers: first, to address gaps in existing programming as soon as possible, and second, to ensure that all new programming includes adequate results management and measures to assess value for money.

The CSSF accepted this recommendation and is in the process of making significant improvements, with commitments backed up by dedicated funding, staff recruitment, the contracting of independent expertise and other action to support and ensure implementation. The CSSF is using a learning-whiledoing approach as it develops and refines its results management mechanisms. Given human resource constraints, the Fund has made sensible priorities, investing first in regional monitoring, evaluation and learning (MEL) contracts aimed at strengthening the CSSF’s own MEL capacity, as well as improving theories of change and annual reviews, as previously mentioned. It has not yet invested much in formal value for money assessments.

Create conditions that allow for the evaluation, by independent evaluators where possible, of a larger part of the CSSF’s portfolio

Evaluations are an important part of accountability and learning, but the CSSF did not in general have the requirements for programme evaluation in place. We recommended that the Fund should create the conditions to allow a larger part of its portfolio to be evaluated, preferably – where possible – using independent expertise. The government accepted this recommendation and the CSSF has since conducted a number of evaluations and has strengthened the evaluability of its newer programmes (through improved theories of change, annual reports, ongoing data gathering and results management).

There are still examples of the Fund making unsubstantiated results claims. For instance, in its most recent annual review, the CSSF claims that its work with the Syrian ‘White Helmets’ “is estimated to have saved over 115,000 lives” and that its Iraqi infrastructure work enabled “over 3.6m people to return home and access basic services, including healthcare, education, water and electricity”. Neither claim has been tested.

As the CSSF is to a large extent made up of many small projects, its decision to limit itself to only evaluating major programmes with bigger budgets means that most of the Fund will go unevaluate. It would therefore make sense for the CSSF to also evaluate a sample-based set of smaller projects. We have been told that the new MEL strategy will seek to take a more strategic approach to evaluation, but we do not yet know if this will include a strategy for evaluating smaller projects.

Do more to gather and synthesise evidence and disseminate lessons on what works in important programming areas

During our review, we saw CSSF programmes using approaches that were likely to achieve sub-optimal results, such as a disproportionate focus on training and the payment of salary supplements. We also saw gaps in thematic policy guidance, and a disproportionately modest contribution for a fund this size to global evidence on what works. We noted that implementing partners were not routinely involved in learning processes, and that programme learning was not routinely shared with the CSSF’s implementers.

The government partially accepted ICAI’s recommendation to synthesise the evidence on what works in important programming areas, and to share this learning with participating departments and implementing partners. Since then, the CSSF has made great strides in its approach to learning. Part of its global MEL plan is to consolidate Fund-level evidence, improving the understanding of what works in fragile and conflict-affected states. In December 2018, the Stabilisation Unit published a stabilisation guide,10 a comprehensively updated version of the 2014 UK Approach to Stabilisation. Other learning exercises are nearing completion – most prominently in the field of conflict sensitivity – and a few of the regions are considering opportunities to distill overall lessons from a series of thematic evaluations (such as on governance).

We saw evidence of material being more widely shared with participating departments and between countries and regions. The CSSF now has a deliberate strategy to engage more with other donors and implementing partners in order to be transparent and share learning.

Conclusion

The initial government response to ICAI’s recommendations seemed reluctant in some areas and focused on what the CSSF was already doing in others. However, since then, the CSSF has made significant efforts to address the shortcomings identified by ICAI. From the desk-based evidence and interviews we conducted, it is likely that the work done to date has already led to more conflict-sensitive programming with clearer and more testable intermediate outcomes (including influencing outcomes). Monitoring is becoming more rigorous as quality standards for programme design have been raised. Learning has been accelerated by systematic reviews, and we saw signs that some learning mechanisms are starting to become institutionalised.

DFID’s approach to disability in development

DFID was at the forefront of activities to ensure that disability was included as a central concern of the SDGs in 2015, but was slower in systematically including the concerns, challenges and contributions of people with disabilities in its own programming. ICAI‘s review looked at whether DFID had developed an appropriate approach to disability and development, and how well it identified and filled knowledge and data gaps on disability in development. As a rapid review, timed to be published ahead of the Global Disability Summit hosted by the UK government in July 2018, the review was not scored, but offered six recommendations, as summarised below.

Table 8: Summary of recommendations and the government’s response

Adopt a more visible, systematic and detailed plan for mainstreaming disability inclusion

In 2017, the secretary of state for international development committed DFID to “put disability at the heart of everything that we do”.11 The ICAI report welcomed this commitment, but called for DFID to adopt a more visible and systematic plan for mainstreaming disability inclusion. DFID accepted this recommendation and has made great strides since our review. In December 2018, it published a comprehensive Disability Inclusion Strategy which sets specific and ambitious standards for all business units, with guidance provided by the disability inclusion team. The standards cover the areas included in our recommendation: programming, human resourcing, learning and organisational culture. A new Disability Inclusion Delivery Board will meet quarterly to monitor progress, and DFID will publish an annual assessment of progress against the standards.

It is too early to know how well the strategy will be implemented, but early signs show that DFID departments are engaged and committed. That said, the strategy has an ambitious timeline, with all business units required to achieve the minimum standards by the end of 2019. The main risk to implementation is insufficient resources, both in terms of in-house expertise on disability (see Recommendation 2 below) and funding set aside for the mainstreaming effort.

Increase the representation of staff with disabilities and increase the number of staff with expertise on disability inclusion

The second recommendation addressed resource constraints from two perspectives. First, we recommended that DFID should increase the representation of staff with disabilities at least to the level of the UK civil service as a whole (9.9%). As of March 2019, 8.7% of DFID’s UK-appointed staff and only 2% of staff appointed in country informed DFID that they had disabilities (from 6.4% and 1.4% in December 2017). Second, we recommended that DFID recruit more staff with expertise in disability issues, since working on disability inclusion is a specialist area with its own skills and legal framework.

DFID accepted both parts of this recommendation. Its new Disability Inclusion Strategy has targets for increasing the proportion of its UK-recruited staff with disabilities to match that of the UK working age population. However, the strategy does not include targets for local recruitment of people with disabilities to DFID country offices overseas.

The Disability Inclusion Strategy promises to “equip DFID staff with the skills, tools and knowledge to better integrate disability inclusion into all of our policies and programmes”. Further recruitment and training is urgently required to provide the support business units need to meet the minimum standards by the end of 2019.

Country offices should develop theories of change for disability inclusion and working with national governments

The ICAI review found that DFID country offices lacked information and analysis on the barriers to disability inclusion, and how to overcome them, in their specific country setting. We therefore recommended that country offices should develop theories of change for how to foster disability inclusion in their countries, with particular focus on how to influence and work with national governments. DFID partially accepted this recommendation. Rather than requiring country offices to prepare their own theories of change for disability inclusion, DFID developed a general theory of change which it encouraged country offices to use in preparing their disability action plans.

While offering country offices the choice of a general theory of change to draw on is an improvement, this approach entails a risk that country office action plans become based on general assumptions rather than on local consultation, including with national disabled people’s organisations. However, DFID’s new Country Development Diagnostics (CDD) process, which has an ‘inclusion module’, may help to improve the quality of country-level analysis by asking country offices to consider: “What is the situation for people with disabilities in social, political and economic spheres?”.

Engage more with disabled people’s organisations

DFID recognised that effective disabled people’s organisations are critical to countries achieving disability inclusion, but country office engagement with disabled people’s organisations was mostly at the level of conversations and they have generally not been consulted in the business planning process. DFID’s main means of support for them, the Disability Rights Fund, operated in only eight of DFID’s more than 32 priority countries. We recommended that DFID should engage with disabled people’s organisations on country-level strategies, advocacy towards partner governments, capacity building and the design of programmes, including research programmes.

There has been a strong and positive response to this recommendation. The Disability Inclusion Strategy reaffirmed a fundamental principle that people with disabilities are “engaged, consulted, represented and listened to at all levels of decision-making … and empowered as powerful and active agents of change to challenge discrimination and harmful norms and to hold governments and implementers to account”.13 The Strategy committed to sufficient investments in disabled people’s organisations to promote their meaningful engagement.

In addition, the Strategy’s minimum standard states that business units must consult (at least) annually with disabled people’s organisations on the design, implementation and evaluation of programmes and office policy and strategy, and should do so in a way that builds capacity and involves groups that are sometimes excluded, such as women and people with psychosocial or intellectual disabilities.

While it is too early to see how this policy development is implemented, we were pleased to see how DFID ensured substantial involvement by people with disabilities in the July 2018 Global Disability Summit. We also note that two new research programmes have very thorough involvement of disabled people’s organisations in their governance. Practical issues remain to be tackled, such as the accessibility of DFID offices. A significant obstacle is DFID’s onerous contracting requirements, which disabled people’s organisations are often unable to meet due to their limited finances and capacity.

Tackle stigma and discrimination

The ICAI review found that few DFID programmes had major elements tackling stigma and discrimination. People with psychosocial disabilities (mental health challenges) and with intellectual disabilities (learning disabilities) are particularly vulnerable. We recommended that DFID should increase its programming on (i) tackling stigma and discrimination, including within the private sector, and (ii) inclusion of people with psychosocial disabilities and people with intellectual disabilities, noting that these are two different groups facing different sets of challenges.

Although DFID only partially accepted this recommendation, the new Disability Inclusion Strategy includes stigma and discrimination as one of three cross-cutting areas “which will be consistently and systematically addressed in all of our work”. It requires all business units by 2023 to be “supporting full participation and leadership of people with disabilities; transforming harmful stereotypes and behaviours; and ensuring policies, structures and resources [are] in place to counter discrimination”.

DFID’s rationale for only partially accepting the recommendation was that it needed to understand the issues better before designing interventions. Apart from two innovative new programmes that pilot methods of tackling stigma and discrimination, DFID’s efforts at the moment remain at the level of aspirations. We think this is unduly cautious because evidence of what works already exists, as pointed out in the ICAI report, on how to influence behavioural change.

On psychosocial disabilities, DFID has made important improvements. Beyond including these in the new Strategy, there has been a considerable amount of activity, pushed along by the appointment of a staff member dedicated to this issue. In contrast, little has changed on the issue of intellectual disabilities – highlighting the importance of dedicated, expert staff to promote DFID’s ambitious Disability Inclusion Strategy.

Create a systematic learning programme and community of practice on the experience of mainstreaming disability into DFID programmes

There is a lack of experience among DFID staff on disability inclusion, but a strong demand for knowledge about what works to achieve it. Building on the proposals for the new Disability Inclusive Development programme, with its helpdesk, we recommended that DFID should create a systematic learning programme, and a community of practice, on the experience of mainstreaming disability into its programmes. DFID accepted this recommendation and has embarked on a range of innovations. It is creating a community of internal champions on disability inclusion and designing a learning journey on the topic with five monthly sessions planned for about 30 participants. The helpdesk is now generally available for all business units.

These are welcome moves but are unlikely to be sufficient to ensure that all business units meet the minimum standards by the end of 2019.

Conclusion

The Disability Inclusion Strategy is a major step forward and incorporates policies that address most of the issues raised in the ICAI review. The minimum standards for all business units are specific and ambitious. Achieving them within the required deadline is likely to require more resources than currently available to departments – in particular, staff with relevant experience and skills. The new community of practice will need to ensure that information on challenges and successes is actively shared. The ambitious deadline for meeting the minimum standards approaches fast. However, it is more important to maintain the current sense of strong forward progress than it is to hit the exact deadline for all departments.

The UK’s humanitarian support to Syria

In response to the brutal conflict in Syria, the UK government launched its largest-ever humanitarian campaign, committing £2.81 billion to the region, with £1.03 billion allocated for humanitarian operations in Syria itself. ICAI’s review, published in May 2018, assessed the effectiveness of DFID’s humanitarian aid inside Syria since the beginning of the crisis response in 2012.

The review gave DFID a green-amber score, recognising that strong improvements had taken place over time and that vital UK aid had reached civilians in need – within the limitations of a complex and highly restrictive operational environment. We offered eight recommendations to further improve operations, as summarised in the table below. The Syrian context has changed dramatically since the publication of the ICAI report, resulting in some necessary adjustments to DFID Syria’s programming and affecting its ability to act on some of our recommendations.

Table 9: Summary of recommendations and the government’s response

A more detailed strategic approach to governance at country level, balancing risk and return across its governance portfolios

Our review had found that, particularly in Uganda, decisions were sometimes made that appeared to focus more on the short-term needs of DFID than the long-term interest of the country. We noted that there was a lack of country-level governance strategies and a tendency to be reactive to events (whether in country or changes in UK political priorities). We therefore recommended that DFID should articulate more clearly its long-term goals and strengthen its scenario-planning and risk management tools, in order to make its governance work at country level more resilient and adaptive. Its strategies should consider and articulate the desired balance of risk and return across its governance portfolios.

DFID only partially accepted this recommendation. It argued that governance was identified as the key issue in its Country Development Diagnostics process, undertaken in the second half of 2018, and would also form the core of all its new business plans. A separate country-level governance strategy would therefore be superfluous. Our follow-up exercise confirmed that the draft business plans under development for both Nepal and Uganda include governance issues at their heart.

In December 2018, after delays, DFID also agreed a new governance position paper entitled Governance for Growth, Stability and Inclusive Development. This was published in March 2019. It sets out four ‘shifts’ in DFID’s governance work: thinking and working politically across all initiatives, integrating governance for growth, stability and inclusion, being confident in the UK’s values, and keeping DFID at the cutting edge of governance work. DFID is developing guidance and tools to support the country offices in putting the approach in its governance position paper (GPP) into practice.

The guidance is currently under development and it is too early to tell what the impact of the GPP will be on country-level planning and strategy. We would therefore like to return to this issue in next year’s follow-up review, once the new business plans have been finalised.

Invest in long-term relationships with key counterparts, while maintaining the flexibility to scale individual activities up and down as appropriate, to maximise opportunities for desired results

Governance change takes time. Our review noted that where DFID had engaged consistently over time it was often able to get good results. DFID accepted our recommendation to pursue opportunities for long-term engagement with a flexible and adaptive approach. A range of improvements have taken place, including the development of guidance on adaptive programming (see the discussion on LearnAdapt in the sub-section on value for money in this report).

Beyond guidance, there has as yet been little consistent action – for instance on how to recognise and reward staff who spend time investing in long-term relationships with national counterparts. We therefore plan to return to this recommendation in next year’s follow-up review.

Rebalance how governance advisers spend their time – fewer programme management and administrative tasks and more technical inputs and external engagement

DFID accepted our recommendation that it should maximise value from its governance advisers by increasing the amount of time they have available to spend on technical inputs (including into other sector programmes) and external influencing and engagement, rather than programme management and administrative tasks. The response to this recommendation has been good in both the countries reviewed. For instance, in Uganda, the DFID governance team now has a dedicated programme manager, which has given the team more time to spend on core technical tasks. At the central level, efforts are under way to learn lessons from the use of more in-depth advisory posts. We will return in next year’s follow-up exercise to see whether this has resulted in changes to guidance or practice.

As conditions allow, DFID Syria should prioritise livelihoods programming and supporting local markets, to strengthen community self-reliance

The ICAI review found that DFID was sometimes slow to move from emergency assistance to livelihoods support, and that its delivery partners needed encouragement, support and expertise to implement livelihoods programming. DFID only partially accepted our recommendation on livelihoods programming, arguing that it was often not possible to move out of emergency mode. ICAI agrees with this, noting that the transition can only be made when conditions allow.

DFID Syria is in the process of designing its new 2020 portfolio, and its livelihoods adviser is currently reviewing its approach to livelihoods and cash-based programming, which will lead to a new Livelihoods Strategy by mid-2019. This process should better position DFID Syria to respond to contextual challenges in the livelihoods sector.

Strengthen third-party monitoring to provide a higher level of independent verification of aid delivery

DFID Syria’s third-party monitoring approach was centred on mutual learning but lacked sufficient focus on assurance and accountability, given the scale and level of risk in the Syria operations. Monitoring visits were very short, and the monitoring system relied heavily on partners to select communities for monitoring visits, which introduced the possibility of bias. Until early 2018, visits only took place in opposition-controlled areas. ICAI therefore recommended that DFID Syria strengthen the independence and rigour of its third-party monitoring approach.

DFID Syria’s response to this recommendation was thorough, with some improvements already under way before ICAI’s recommendations were published. DFID Syria has updated its methodology on monitoring, and there are early signs of positive results from its change in sampling strategy, which reduces the role of implementing partners in the selection of monitoring visit locations. DFID Syria has also increased the length of monitoring visits from one to two days (and is piloting three-day visits). Together, these two changes will provide higher-quality and less biased data for assessing how projects perform and can be improved. Two further changes, monitoring in government-controlled areas and the addition of more in-depth verification visits, are at too early a stage of implementation for ICAI to assess them. The restricted access to government-controlled areas inevitably limits the effectiveness of monitoring.

Support partners to expand their community consultation and feedback processes and ensure this informs learning and future design

DFID is committed to ensuring greater participation of and accountability to the populations it supports through its humanitarian action. The ICAI review found that the complaints mechanisms included in DFID’s Syria projects were useful, but not adequate to meet this commitment. It also noted that feedback from communities was not being collated and analysed to inform learning and the design of new activities. We therefore recommended that DFID should support and encourage its local partners to expand community consultation and feedback, and use this feedback to improve the design of future programming.

DFID Syria’s response has so far focused on information gathering on current processes. It has completed a desk-based research project that mapped beneficiary feedback mechanisms and their usage in Syria and conducted an assessment of DFID’s delivery partners’ monitoring and evaluation systems, including beneficiary feedback mechanisms. The assessment found that most of DFID’s delivery partners relied on their downstream partners’ processes – which DFID so far has not reviewed. DFID Syria indicates that it will use the information it has gathered to inform its 2020 programme portfolio, which is currently under design.

Identify ways to support the capacity development of Syrian non-governmental organisations to have more direct roles in the humanitarian response

Most Syrian non-governmental organisations (NGOs) are new, with little administrative capacity and little familiarity with donor requirements. We therefore recommended that DFID seek ways to support the capacity development of Syrian NGOs through its programming in order to strengthen the effective delivery of humanitarian aid in Syria and to further the UK’s international commitment to the localisation of aid.

DFID’s actions on this recommendation have been limited to funding an international NGO delivery partner to build the organisational capacity of one leading Syrian NGO. Unless there is more concentrated effort to address the capacity gaps of Syrian NGOs, it is unlikely that the localisation agenda can be furthered in Syria. Without stronger administrative, financial and quality assurance systems, it is unlikely that local NGOs will be able to meet the important due diligence requirements for receiving direct funding from DFID or other donors – requirements which safeguard public money and minimise opportunities for fraud and misuse.

DFID Syria should develop a dynamic research and learning strategy addressing learning needs across the international humanitarian response in Syria, and a dissemination strategy

DFID Syria accepted ICAI’s recommendation to develop an overarching strategy for producing and disseminating research, but its response to date has been limited to refining the process by which staff propose research pieces, requiring them also to identify their plans for dissemination. This may be an appropriate stop-gap in the absence of an overall learning strategy, but it does not address ICAI’s concerns. DFID Syria’s learning approach remains based on individual efforts around separate pieces of research and/or events like annual reviews. In the absence of a learning strategy, the risk remains that research pieces are siloed and wider learning does not occur.

Collect and document lessons and best practices from the Syria response, to inform ongoing and future crisis responses

This recommendation was aimed at the way that DFID as an institution can learn from efforts in Syria to inform other crisis responses. The Conflict, Humanitarian and Security Department (CHASE) has led the effort to embed lessons from the Syria response into humanitarian policy and practice. However, information sharing continues to rely largely on the efforts of individuals. DFID’s management information/data analytics team has recently introduced a search engine that will allow staff to search more easily for projects and topics. This tool may facilitate the collation of and access to lessons and experience across the organisation. All of these activities exist outside of an organisation-level learning strategy and structure, running the risk that learning remains fragmented and vital lessons are lost. In complex crises, plan for a lengthy engagement from an early stage.

ICAI found that DFID’s planning for the Syrian response remained on an emergency footing even after the crisis had become protracted. We recommended better planning for such situations, including having clear decision points for when to move beyond short-term funding instruments and staffing arrangements. DFID accepted ICAI’s recommendation and outlined several actions to be taken by CHASE.

CHASE is in the process of synthesising learning from various crises in order to streamline the government’s humanitarian response policy. This may lead to a more adaptable approach to emergencies. However, DFID has not yet taken forward the wider lessons from its Syria operation on transitioning from a short-term emergency footing and does not yet appear to have engaged at a central level on how to use different types of financial instruments at different phases of a humanitarian response.

Build on DFID Syria’s efforts to invest in reporting and data management systems that can be readily adapted to complex humanitarian operations

DFID lacks a set of data management and reporting tools that can be readily adapted to the needs of complex humanitarian operations. While there is some exchange of experience between DFID country offices, this is not done in a systematic way. We therefore recommended that DFID should build on DFID Syria’s reporting and data management approach to invest in more adaptable systems for the department more generally.

There has been little progress in the response to this recommendation. DFID Syria is still sharing its experience using its Cascade reporting tool (for results management). DFID’s information management team is currently working with the Office for National Statistics on how to build a common DFID reporting tool, but this work is progressing slowly, in large part due to the need to ensure proper electronic information security protocols, particularly when connected to external organisations such as delivery partners.

Conclusion

Despite the very difficult operating environment in Syria, DFID Syria has engaged proactively with ICAI’s recommendations, with many positive developments. DFID is currently designing the 2020 DFID Syria programme portfolio: the design process provides a good  opportunity to build on lessons learned and the progress already made. It will be important to ensure that learning from the past seven years of working in Syria will be sustained throughout the planned restructuring and relocation of the team – while at the same time overseeing the uninterrupted delivery of a complex portfolio.

DFID’s governance work in Nepal and Uganda

The review of DFID’s support to strengthening governance in Nepal and Uganda was the last in ICAI’s 2017-18 review cycle, published in late June 2018. We focused on just two countries in order to allow for in-depth exploration of the quality of assistance and results in a broad area of programming. The review found that DFID’s governance work was well tailored to the country contexts and largely achieved its intended outcomes, but needed a stronger strategic orientation and approach to learning. Our five recommendations are summarised in the table below.

A more detailed strategic approach to governance at country level, balancing risk and return across its governance portfolios

Our review had found that, particularly in Uganda, decisions were sometimes made that appeared to focus more on the short-term needs of DFID than the long-term interest of the country. We noted that there was a lack of country-level governance strategies and a tendency to be reactive to events (whether in country or changes in UK political priorities). We therefore recommended that DFID should articulate more clearly its long-term goals and strengthen its scenario-planning and risk management tools, in order to make its governance work at country level more resilient and adaptive. Its strategies should consider and articulate the desired balance of risk and return across its governance portfolios.

DFID only partially accepted this recommendation. It argued that governance was identified as the key issue in its Country Development Diagnostics process, undertaken in the second half of 2018, and would also form the core of all its new business plans. A separate country-level governance strategy would therefore be superfluous. Our follow-up exercise confirmed that the draft business plans under development for both Nepal and Uganda include governance issues at their heart.

In December 2018, after delays, DFID also agreed a new governance position paper entitled Governance for Growth, Stability and Inclusive Development. This was published in March 2019. It sets out four ‘shifts’ in DFID’s governance work: thinking and working politically across all initiatives, integrating governance for growth, stability and inclusion, being confident in the UK’s values, and keeping DFID at the cutting edge of governance work. DFID is developing guidance and tools to support the country offices in putting the approach in its governance position paper (GPP) into practice.

The guidance is currently under development and it is too early to tell what the impact of the GPP will be on country-level planning and strategy. We would therefore like to return to this issue in next year’s follow-up review, once the new business plans have been finalised.

Invest in long-term relationships with key counterparts, while maintaining the flexibility to scale individual activities up and down as appropriate, to maximise opportunities for desired results

Governance change takes time. Our review noted that where DFID had engaged consistently over time it was often able to get good results. DFID accepted our recommendation to pursue opportunities for long-term engagement with a flexible and adaptive approach. A range of improvements have taken place, including the development of guidance on adaptive programming (see the discussion on Learn Adapt in the sub-section on value for money in this report).

Beyond guidance, there has as yet been little consistent action – for instance on how to recognise and reward staff who spend time investing in long-term relationships with national counterparts. We therefore plan to return to this recommendation in next year’s follow-up review. Rebalance how governance advisers spend their time – fewer programme management and administrative tasks and more technical inputs and external engagement

DFID accepted our recommendation that it should maximise value from its governance advisers by increasing the amount of time they have available to spend on technical inputs (including into other sector programmes) and external influencing and engagement, rather than programme management and administrative tasks. The response to this recommendation has been good in both the countries reviewed. For instance, in Uganda, the DFID governance team now has a dedicated programme manager, which has given the team more time to spend on core technical tasks. At the central level, efforts are under way to learn lessons from the use of more in-depth advisory posts. We will return in next year’s follow-up exercise to see whether this has resulted in changes to guidance or practice.

DFID should increase the diversity and develop the capacity of its governance cadre

The review focused on how appropriate DFID’s governance work was to the local context, in particular whether programming was informed by a sufficiently deep and sensitive contextual understanding. We recommended that DFID should develop the capacity of its governance cadre by (i) improving the use of staff appointed in country, (ii) posting UK-recruited civil service staff for longer periods at country posts, (iii) increasing diversity in its country teams, in terms of experience, backgrounds and local knowledge, and (iv) placing more weight on practical delivery experience when recruiting staff.

DFID partially accepted this recommendation, disagreeing to some extent with the analysis that the capacity and skills of country-appointed staff were under-utilised. DFID has recently begun to take actions to address some policies that reinforced the sense that country-recruited staff were not fully integrated in country offices, such as removing the practice of having different emails for country-recruited staff. We expect to see further improvements and look forward to returning to this recommendation in next year’s follow-up review.

Improve the capturing of learning, particularly through an increased use of evaluations

The ICAI review found a very limited use of governance evaluations in Nepal and Uganda at both portfolio and programme levels. It also noted that there was clear room for improvement in how DFID captured and integrated learning – at portfolio, country and programme level. We recommended that DFID should use evaluation more to test the validity of key propositions underlying its governance programmes and portfolios, increase its investment in learning within programmes, and use these lessons to inform the management of country portfolios and programmes.

Centrally, work is under way to develop a new approach to measuring results using technology to link evidence and evaluations. At country level, both Nepal and Uganda have stepped up research and evaluation plans. DFID Nepal also plans to fund a post through the School of Oriental and African Studies and Yale University to collate and communicate lessons from its programming documentation. Both country offices have worked with DFID’s Research and Evaluation Department to improve their learning strategies. It will be worthwhile to return to this recommendation next year to see the results of country level efforts and to check on progress in reviewing DFID’s central evaluation strategy for governance programming.

Conclusion

In conclusion, it may have been a little too early to follow up on the recommendations from the governance review, which was published in late June 2018. This was the last review in the 2017-18 review cycle, leaving little time for DFID to respond, and making it too early for ICAI to assess the progress made. We will therefore return to all the recommendations in next year’s follow-up exercise.

Outstanding issues from the Year 6 follow-up review

In last year’s follow-up exercise, we flagged four reports for further follow-up. In some cases, this was because we found DFID’s progress on implementing our recommendations in areas of strategic importance to have been inadequate. In other cases, there had been some significant developments, but implementation was at too early a stage for ICAI to assess their impact.

Inclusive growth

Our original report on DFID’s approach to inclusive growth in Africa asked DFID to ensure that, in each of its partner countries, opportunities for addressing the exclusion of women, young people and marginalised groups were identified and built into programme designs and results frameworks wherever feasible. As we concluded last year’s follow-up exercise, DFID was in the process of developing new diagnostic tools and had begun a drive to disaggregate its results data. However, there was no new guidance or other concrete action to ensure that inclusion was addressed at both country portfolio and programme level. We therefore concluded that we were not yet confident that DFID was in a position to achieve the commitments in its Economic Development Strategy to reaching women, young people and marginalised groups.

One year later, we find a strong and credible approach to inclusion at both country portfolio and programme level. DFID still leaves it to country offices to decide how best to approach inclusion within each country context. But several measures have been taken to make sure that country offices take up the commitment to inclusion:

• The Country Development Diagnostics process, with a strong emphasis on inclusion (see Box 3). This is a major development, creating the foundation for the next round of country business plans and economic development programmes.
• An increased emphasis on tracking distributional impacts in programme logframes and monitoring arrangements. This includes the adoption of an Inclusive Data Charter: Action Plan which commits to disaggregating data according to distributional impacts and to monitoring the inclusiveness of programmes throughout their lifetime. The Action Plan includes various measures, such as a resource library, a guide on disability data, and updates to the smart guide on logframes. The smart guide underlines that a programme’s results indicators must enable teams to gather “real data about direct beneficiaries receiving support from DFID”, so as to better understand who get left behind (or are at risk of this), where they are located and why this is happening. Finally, the internal Guide on equity in value for money analysis, which was already available last year, is also relevant, ensuring attention to whether the obstacles to including women are properly considered and addressed.
• The launch of a number of flagship programmes with a strong focus on inclusion. For instance, the Work and Opportunities for Women programme, launched in March 2018, is designed to generate new knowledge on women’s economic empowerment and to provide technical support, including a helpdesk function, to country offices on issues such as sexual and reproductive rights and rights interventions in the workplace.

We reviewed four business cases for economic development programmes approved in 2018 (after our initial follow-up exercise). We found that each of them gives more consideration to inclusion than was the case in programmes covered under the ICAI review, showing a positive direction of travel. There are also strong commitments to monitoring distributional impact, echoing the language of the ICAI recommendation. Particular care is taken over compliance with the Gender Equality Act and with DFID policy commitments on disability. In all, we are pleased to see how this suite of diagnostics, guidance and actions adds up to a clear statement of expectation for DFID economic development programmes around promoting inclusion and monitoring the results.

Irregular migration

Our rapid review of the UK’s aid response to irregular migration in the central Mediterranean was published in March 2017. Last year’s follow-up review found useful action in response to most of our recommendations. However, irregular migration is a complex and difficult area in which to intervene, and work on developing sophisticated and credible theories of change, with monitoring and evaluation processes to match them, was progressing too slowly for ICAI to be able to assess their implementation and impact.

In this second follow-up, we assessed in particular the monitoring and evaluation arrangements for DFID’s new flagship programme, the Safety, Support and Solutions Programme for Refugees and Migrants (SSS II), which was not yet up and running last year. Now, a year later, SSS II has a strong theory of change, accompanied by a monitoring, evaluation and learning (MEL) strategy. The first three tiers of this strategy are now in place: MEL procedures for individual projects, administered by implementing partners, DFID monitoring visits to verify these, and DFID’s own programme-level monitoring. The fourth tier – an independent monitoring and rapid research and evidence facility – was at the inception phase at the time of our follow-up assessment. There had also been plans for a fifth tier, an independent evaluation. However, following preparatory work, DFID decided in 2019 not to commission this, relying instead on alternative arrangements for monitoring, rapid research and learning. Because this happened subsequent to our follow-up assessment, we plan to review these revised arrangements as part of next year’s follow-up.

Significant delays in putting in place the MEL strategy have meant that programme indicators and monitoring and evaluation contracts were not ready when the programme implementers started their work. This is a problem for the aspects of SSS II that need baselining in order to monitor progress, which fortunately is only the case for a few of the programme’s elements. The delays are partly explained by DFID’s slow procurement and partly because of DFID conducting additional assessments to ensure that SSS II has a strong ‘do no harm’ approach.

There has been general progress on the issue of ‘do no harm’ – for both DFID and CSSF-led programmes. Risk assessments have been thorough in the design phase of programmes, with a number of political economy, human rights and conflict sensitivity assessments conducted to inform programme design. These have been shared and discussed with implementing partners. In the case of SSS II, key risks have been incorporated in the programme’s central risk register, and in the case of Libya we saw evidence of activities being terminated because the risks of doing harm were deemed to be too high. Since the issue of harm is difficult to assess, we would, however, also like to see a more explicit focus on the risk of doing harm in the MEL work of the SSS II programme, and in migration-related programming in general.

The Prosperity Fund

The Prosperity Fund, which comes under the authority of the National Security Council, spends UK aid to promote economic reform and growth in developing countries. The primary purpose of the Prosperity Fund is to contribute to the SDGs through addressing barriers to growth. ICAI reviewed the Fund in February 2017, while it was still in its development phase. In last year’s follow-up exercise, we found that good progress had been made on all our recommendations, but that it was still too early for us to review some issues of strategic significance. We decided to come back to three issues: (i) developing portfoliolevel results indicators and associated systems for measuring results and learning from experience, (ii) implementing the Fund’s new procurement framework, and (iii) introducing explicit and challenging procedures to assess the ODA eligibility of programmes as part of the newly merged governance arrangements for the Prosperity Fund and the CSSF.

There has been good progress on ICAI’s recommendation to refine the Prosperity Fund’s strategic objectives and develop a set of portfolio-level results indicators. We welcome the development of 15 outcome-level KPIs, covering each of the Fund’s intervention areas and reflecting both the Fund’s strategic objectives as set by ministers and priorities identified on the ground. Twelve of the KPIs cover the Fund’s primary objectives and three are for secondary goals. The Fund’s theory of change is now updated annually and published in the Annual Report.

The Cabinet Office has contracted two monitoring, reporting, evaluation and learning (MREL) service providers, one for monitoring and reporting and another for evaluation and learning. Separate guidance for each KPI is currently being developed by the monitoring and reporting service provider. The Joint Funds Unit, which manages the Prosperity Fund and the CSSF, has committed to using these KPIs to also inform learning at the portfolio and programme level. We were pleased with the functioning of the Prosperity Fund’s two new online interactive data and knowledge management systems: Prospero, which is designed for ongoing monitoring and reporting purposes, and a learning platform, open to all government participants in the Fund. Combined, these systems give the Prosperity Fund team a consolidated overview of all elements of the Fund.

Since last year’s follow-up exercise, the Prosperity Fund has put in place a Prosperity Framework for its procurement process, which can be used by all government departments involved in the Fund’s activities. The Prosperity Framework was developed by the Foreign Office (FCO), but we were told this was informed by engagement with and input from other government departments, in particular DFID and BEIS. The FCO is also ensuring that there is regular portfolio-level market engagement (in London and at post) and horizon-scanning activities to assess market capacity constraints. Conflict of interest assessments are undertaken where potential risk exists – addressing concerns raised in ICAI’s follow-up review. Importantly, both the KPIs and the Prosperity Framework were put in place ahead of the first round of the Fund’s business case approvals.

Merged governance arrangements for the Prosperity Fund and the CSSF are now in place (see Box 4). The Joint Funds Unit is up and running, although the practical delivery arrangements are still evolving on how to provide oversight and coordination functions across both the Prosperity Fund and the CSSF. This includes the question of who will provide the service functions of audit and assurance, human resources, communications and finance (the MEL functions of the two Funds are separate and delivered by different providers).

The ownership of ODA compliance has now been formally clarified. Spending departments are responsible for ensuring that all their spending meets ODA eligibility requirements as set by the OECD’s Development Assistance Committee and, where appropriate, the provisions on poverty alleviation and gender equality set out in the International Development Act. This requirement is clearly stated in allocation letters sent to senior responsible owners. There is now a programme management training course with a dedicated session on ODA compliance, which has been completed by all staff with responsibility for Prosperity Fund programmes. Guidance has been issued on gender and inclusion requirements and relevant KPIs will be disaggregated by gender and income level.

The responsibility for ensuring ODA compliance during the lifetime of programmes sits within individual departments, but will be aided by the Prosperity Fund’s new MREL systems. The Fund’s evaluation and learning system challenges programmes to be explicit in their theories of change on how they define their primary (ODA) objectives and how they expect to reach them. The service providers will undertake annual evaluations of each programme, including assessing ODA compliance and ensuring that the primary objectives have remained primary. In addition, individual delivery departments will conduct their own annual reviews, published on DevTracker.

In conclusion, the Prosperity Fund has made good progress on the issues of concern raised in our followup last year. Once the Prosperity Fund programmes begin to be implemented on the ground, we will be able to assess the effectiveness of these interventions in achieving the Fund’s objectives. ICAI will revisit this issue under the forthcoming information note on mutual prosperity.

Transitioning from traditional aid relationships

As many of DFID’s development partners have moved from lower-income to middle-income status – and more are set to follow suit in the next few years – DFID is managing its changing relationship with partner governments and civil society organisations in middle-income countries as they ‘graduate’ from conventional bilateral or financial aid. ICAI’s original review on DFID’s approach to managing exit and transition in its development partnerships was published in November 2016. Last year’s follow-up exercise concluded that the department had only recently begun to develop working principles for successful transitions. Emphasising the importance of getting these in place before the next round of transitions to post-aid relationships, ICAI decided to return to the topic in this year’s follow-up review.

Over the course of the past year, the wider context in which DFID is operating has changed significantly. DFID has told us that it is conducting a strategic overhaul based on a horizon scan on global poverty trends, including work on ‘One Strategic Direction’ to align and expand on its strategic objectives in preparation for the next Spending Review (see Figure 4). The department proposes to draw upon a range of instruments (including trade) to broaden their development impact. It is also preparing its evidence base, particularly through use of the Country Development Diagnostics (CDD) process. The diagnostic work also fed into the business planning process for the financial year 2019-20, which informed the production of an updated Single Departmental Plan.

Figure 4: How the transition working principles fit into DFID’s One Strategic Direction

DFID has created two versions of the CDD, one of which is a Rising Powers Diagnostic targeted at seven identified countries or regions: China, India, South Africa, Brazil, Indonesia (all included as case studies in ICAI’s original review), as well as Turkey and the Gulf region. DFID has also just finalised its work on developing six working principles for transition, and is in the process of completing accompanying guidance material for how to incorporate these in the business planning process. The working principles are issued to DFID teams as part of a package of guidance when conducting diagnostics and writing strategies. They are:

• sustainability across strategies and programmes
• strengthening governance and institutions
• supporting civil society and the private sector
• using a variety of instruments and influencers
• engaging across government and UK institutions
• transparency with the UK public, governments and partners in country

DFID has also created a Learning Hub, with a SharePoint site containing resources on transition. The Learning Hub can assist country offices in applying the transition working principles in their business planning processes. Figure 4 above illustrates how the Rising Powers CDD, the working principles and the Learning Hub fit into DFID’s broader strategic planning.

The development of the new, differentiated Rising Powers CDD goes some way to addressing ICAI’s original concerns about the lack of transition-specific tools and guidance. This new diagnostic tool and the six working principles on transition and accompanying guidance are likely to bring greater coherence to DFID’s management of its changing relationships with middle-income countries. DFID shared the working principles with development leads in other government departments in May 2019. Many other government departments are involved in developing and transitioning UK relationships with rising powers and other middle-income countries, so it was a missed opportunity not to have involved them more closely earlier. We understand that other urgent government business, including Brexit preparations, delayed further consultation.

It is too early to assess whether the new working principles will be effective in addressing ICAI’s concerns around strategic direction, since they have not yet been put to use. ICAI’s upcoming review of UK Aid to Ghana – a live example of a partnership under transition – offers an opportunity to examine how this process will unfold over the coming year.