ICAI follow-up of: The Conflict, Stability and Security Fund’s aid spending

Executive summary

ICAI’s follow-up review is an important element in the scrutiny process for UK aid. It provides the International Development Committee and the public with an account of how well the government has responded to ICAI’s recommendations to improve spending. It is also an opportunity for ICAI to identify issues and challenges facing the UK aid programme now and in the future, which in turn helps to inform subsequent reviews.

This document is a summary which focused only on the results of our follow up of The Conflict, Stability and Security Fund’s aid spending. The full Follow Up report of all our 2017-18 reviews, including overall conclusions from the process and details of our methodology, can be found on our website.

Findings

The Conflict, Stability and Security Fund’s aid spending

The Conflict, Stability and Security Fund (CSSF) was established in 2015. It is accessible to a range of government departments and combines ODA and non-ODA funding to support the implementation of National Security Council (NSC) strategies. ICAI’s March 2018 review focused on how well this cross-government fund spent UK aid to tackle conflict, instability and insecurity and promote sustainable peace and security in fragile and conflict-affected countries.¹

ICAI gave the CSSF an amber-red score. The Fund’s country and regional portfolios often lacked a clear logic connecting the activities they supported to the objective of promoting sustainable peace, stability and security. Results management practices were inadequate, and there was little reliable data to assess whether projects were achieving their intended results or delivering value for money. Most of the programmes we reviewed showed design or implementation flaws. We made six recommendations, summarised in the table below. The CSSF has achieved significant progress in its response to most of them.

Table 1: Summary of recommendations and the government’s response

Subject of recommendation	Government response
Introduce country or regional plans that specify how CSSF activities will contribute to National Security Council objectives	Partially accepted
When CSSF projects have influencing objectives, make them explicit and report progress on them to allow assessment of value for money	Accepted
Programmes should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm	Accepted
Ensure all new programming includes adequate results management and measures to assess value for money and rectify gaps in existing programmes	Accepted
Create conditions that allow for the evaluation, by independent evaluators where possible, of a larger part of the CSSF's portfolio	Accepted
Do more to gather and synthesise evidence and disseminate lessons on what works in important programming areas	Partially accepted

Introduce country or regional plans that specify how CSSF activities will contribute to National Security Council objectives

ICAI recommended that the CSSF introduce country or regional plans specifying how its portfolios of aid programmes and influencing efforts would contribute to achieving NSC objectives, the intermediate outcomes that the portfolios would achieve, and the assumptions that needed to hold for this to happen.

The government only partially accepted this recommendation. It recognised the need to set out more clearly how programmes deliver against national security objectives, but did not want to introduce CSSF-specific country or regional plans. Instead, the Fund has invested in strengthening programme-level theories of change, which had been weak and have now improved considerably. The CSSF is now routinely developing theories of change of good quality, with clearly defined goals – including intermediary objectives – and more transparency around underlying assumptions on how to achieve these goals. Annual reviews now include the reporting of outcomes, not just outputs.

Although much of this is work in progress, there has been a step change in the quality of programme-level documentation since our original review. The feedback from CSSF staff during the follow-up exercise has been that the improved theories of change and annual reviews are also improving data collection. The annual review is also used as an opportunity to assess whether outcome targets remain adequate or relevant as contexts change.

When CSSF projects have influencing objectives, make them explicit and report progress on them to allow assessment of value for money

Our review found that there was often a discrepancy between the stated objectives of a programme and its actual objectives of achieving diplomatic access and influencing. Since these objectives were not stated, results were never tested and value for money could not be assessed. We therefore highlighted the need for the CSSF to explicate and test its influencing objectives. The government accepted this recommendation.

Influencing and access outcomes and indicators have been included in the more recent theories of change (discussed above) and results frameworks, making measuring the achievement of influencing objectives now possible, if still difficult. Instead of hiding access and influencing goals, these are now presented as explicit and legitimate objectives, with annual reviews beginning to critically reflect on the issue – recognising that the results of influencing activities are not easy to measure and need a flexible, adaptive approach.

Programmes should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm

The ICAI review found that human rights assessments were not always conducted before the start of programmes (or indeed at all) and that, when CSSF interventions entailed working with counterparts that may do harm, mitigation measures were sometimes superficial. We were concerned that, since human rights risk assessments were inconsistent and programme monitoring was often weak, we did not know whether CSSF programming was causing harm. The government accepted the recommendation that the CSSF should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm.

There has been a swift and thorough response to this recommendation. The CSSF has strengthened its guidance and training, and introduced new processes to identify, manage and mitigate risks of doing harm. It has incorporated the risk of doing harm into programme risk registers and in the annual review templates and continued the development of its ‘conflict sensitivity marker’, which is currently being refined and piloted in 20 programmes (by January 2019, with another 20 in the pipeline). The marker is structured around three progressive themes – awareness, adaptiveness and accountability – and the pilot exercises have been thorough and useful: they have revealed that the conflict sensitivity of CSSF programmes is of variable quality, with some doing very well and others in need of significant improvements.

The CSSF has also conducted a large, systematic cross-programme assessment of the use and quality of 70 Overseas Security and Justice Assessments. These learning activities have been supported by centrally based conflict advisers who travel to CSSF teams and help implement and consolidate change processes.

Ensure all new programming includes adequate results management and measures to assess value for money and rectify gaps in existing programmes

Our review found that the CSSF’s results management was weak. The tools it used were unimaginative and developed with a one-size-fits-all mentality, unsuited to a complex and fast-changing area of programming in fragile and conflict-affected areas. The CSSF did not have the evidence needed to assess whether expenditure represented value for money. Our recommendation had two layers: first, to address gaps in existing programming as soon as possible, and second, to ensure that all new programming includes adequate results management and measures to assess value for money.

The CSSF accepted this recommendation and is in the process of making significant improvements, with commitments backed up by dedicated funding, staff recruitment, the contracting of independent expertise and other action to support and ensure implementation. The CSSF is using a learning-while-doing approach as it develops and refines its results management mechanisms. Given human resource constraints, the Fund has made sensible priorities, investing first in regional monitoring, evaluation and learning (MEL) contracts aimed at strengthening the CSSF’s own MEL capacity, as well as improving theories of change and annual reviews, as previously mentioned. It has not yet invested much in formal value for money assessments.

Create conditions that allow for the evaluation, by independent evaluators where possible, of a larger part of the CSSF’s portfolio

Evaluations are an important part of accountability and learning, but the CSSF did not in general have the requirements for programme evaluation in place. We recommended that the Fund should create the conditions to allow a larger part of its portfolio to be evaluated, preferably – where possible – using independent expertise. The government accepted this recommendation and the CSSF has since conducted a number of evaluations and has strengthened the evaluability of its newer programmes (through improved theories of change, annual reports, ongoing data gathering and results management).

There are still examples of the Fund making unsubstantiated results claims. For instance, in its most recent annual review, the CSSF claims that its work with the Syrian ‘White Helmets’ “is estimated to have saved over 115,000 lives” and that its Iraqi infrastructure work enabled “over 3.6m people to return home and access basic services, including healthcare, education, water and electricity”.² Neither claim has been tested.

As the CSSF is to a large extent made up of many small projects, its decision to limit itself to only evaluating major programmes with bigger budgets means that most of the Fund will go unevaluated. It would therefore make sense for the CSSF to also evaluate a sample-based set of smaller projects. We have been told that the new MEL strategy will seek to take a more strategic approach to evaluation, but we do not yet know if this will include a strategy for evaluating smaller projects.

Do more to gather and synthesise evidence and disseminate lessons on what works in important programming areas

During our review, we saw CSSF programmes using approaches that were likely to achieve sub-optimal results, such as a disproportionate focus on training and the payment of salary supplements. We also saw gaps in thematic policy guidance, and a disproportionately modest contribution for a fund this size to global evidence on what works. We noted that implementing partners were not routinely involved in learning processes, and that programme learning was not routinely shared with the CSSF’s implementers.

The government partially accepted ICAI’s recommendation to synthesise the evidence on what works in important programming areas, and to share this learning with participating departments and implementing partners. Since then, the CSSF has made great strides in its approach to learning. Part of its global MEL plan is to consolidate Fund-level evidence, improving the understanding of what works in fragile and conflict-affected states. In December 2018, the Stabilisation Unit published a stabilisation guide,³ a comprehensively updated version of the 2014 UK Approach to Stabilisation. Other learning exercises are nearing completion – most prominently in the field of conflict sensitivity – and a few of the regions are considering opportunities to distill overall lessons from a series of thematic evaluations (such as on governance).

We saw evidence of material being more widely shared with participating departments and between countries and regions. The CSSF now has a deliberate strategy to engage more with other donors and implementing partners in order to be transparent and share learning.

Conclusion

The initial government response to ICAI’s recommendations seemed reluctant in some areas and focused on what the CSSF was already doing in others. However, since then, the CSSF has made significant efforts to address the shortcomings identified by ICAI. From the desk-based evidence and interviews we conducted, it is likely that the work done to date has already led to more conflict-sensitive programming with clearer and more testable intermediate outcomes (including influencing outcomes). Monitoring is becoming more rigorous as quality standards for programme design have been raised. Learning has been accelerated by systematic reviews, and we saw signs that some learning mechanisms are starting to become institutionalised.

Summary of findings

Subject of recommendation	Recent developments	ICAI's assessment of progress
Introduce country or regional plans that specify how CSSF activities will contribute to National Security Council objectives. Government response: Partially accepted	The government did not want to introduce CSSF-specific country or regional plans, but instead invested in strengthening programme-level theories of change with clearly defined objectives and more transparency around assumptions. Annual reviews now report on outcomes, not just outputs.	There has been a step change in the quality of programme-level documentation. The CSSF is now routinely developing theories of change of good quality. The improved annual review process is leading to better data collection and can be used to assess whether outcome targets remain adequate or need adaptation as contexts change.
When CSSF projects have influencing objectives, make them explicit and report on their progress. Government response: Accepted	Influencing and diplomatic access outcomes are now presented as explicit and legitimate objectives in programme documentation. Indicators of influencing and access outcomes are now included in theories of change and annual reviews are now reviewing these.	This is a strong response. Measuring the achievement of influencing objectives is difficult, but these measures should make it possible to do so. Annual reviews are already beginning to reflect critically on how to use flexible, adaptive approaches to measurement in order to capture influencing outcomes.
Programmes should demonstrate more clearly and carefully how they identify, manage and mitigate risks of doing harm. Government response: Accepted	The CSSF has strengthened guidance and training and introduced new processes to identify, manage and mitigate risks of doing harm. Risks of doing harm are now incorporated in programme risk registers and annual review templates. A 'conflict sensitivity marker' is being refined and piloted. 70 Overseas Security and Justice Assessments (OSJAs) have been assessed and revised where necessary.	This is a swift and thorough response, with much stronger mechanisms and processes in place, supported by improved learning material and conflict sensitivity analyses, to assess and mitigate against harm. The 20 pilot exercises for the conflict sensitivity marker focus on awareness, adaptiveness and accountability, and have been thorough and useful.
New programming should include adequate results management and measures to assess value for money. Government response: Accepted	Given human resource constraints, the CSSF has prioritised getting regional monitoring, evaluation and learning (MEL) contracts in place. The strengthening of the CSSF's programme-level theories of change and annual reviews (see above) will also help to improve results management.	The CSSF has made significant commitments to improving its results management processes, backed up by dedicated funding, staff recruitment, independently contracted expertise and other measures to ensure implementation of MEL initiatives. The 'learning-while-doing' approach adopted by the CSSF will allow it to refine its results management mechanisms as these commitments are transformed into action.
Create conditions that allow for the evaluation, by independent evaluators where possible, of a larger part of the CSSF portfolio. Government response: Accepted	The CSSF has conducted a number of evaluations since the ICAI review and has strengthened the evaluability of its newer programmes through improved theories of change, annual reports and better data gathering and results management mechanisms. The CSSF currently has a stated practice of only evaluating major programmes with larger budgets.	These are significant improvements, but the CSSF is still making some unsubstantiated results claims. The Fund should not focus exclusively on large programmes in its evaluation strategy. As its spending is mostly through many smaller projects, it should also evaluate these on a sample basis. We have been told that the new MEL strategy will seek to take a more strategic approach to evaluation, but we do not yet know if this will include a strategy for evaluating smaller projects.

Footnotes

The Conflict, Stability and Security Fund’s aid spending, ICAI, 29 March 2018, link.
Conflict, Stability and Security Fund: Annual Report 2017/18, July 2018, p. 19-20, link.
The guide is available online. The UK Government’s Approach to Stabilisation: A guide for policy makers and practitioners, HM Government, 19 December 2018, link.