The UK’s humanitarian support to Syria

1. Purpose, scope and rationale

The purpose of this performance review is to assess the effectiveness of UK humanitarian aid to Syria. It will explore whether UK aid is being planned, managed and overseen so as to reach the intended beneficiaries and respond to their needs in a cost-effective manner. It is expected that lessons from the review will be applicable to the Department for International Development’s (DFID) management of large-scale humanitarian operations in complex and protracted crises in the future.

The review will cover DFID’s work in Syria since the beginning of its response to the humanitarian crisis in 2011. DFID assistance to Syrian refugees in neighbouring countries will not be examined, and the review will focus on humanitarian programming and not on development, governance or reconstruction activities, or the work of the Conflict, Security and Stability Fund (CSSF). Humanitarian programming, to draw on DFID’s definition, includes “the provision of material aid including food, medical care (and personnel) and finance and advice to save and preserve lives during emergency situations and in the immediate post-emergency rehabilitation phase; and to cope with short- and longer-term population displacements arising out of emergencies.”¹

Non-humanitarian activities, including those associated with the CSSF, will only be considered if they directly affect the perception of humanitarian assistance, or the treatment (or security) of humanitarian workers in Syria in important ways. For instance, the CSSF may be considered if the review finds that UK governance- or security-oriented programming is resulting in curtailed access for humanitarian stakeholders to certain locations. ICAI is conducting a separate review of the CSSF that will be published in 2018.

Since its onset in 2011, the conflict in Syria has grown into a complex, protracted emergency with regional and global implications. Over 5 million people have fled as refugees to countries neighbouring Syria, 6.3 million have been internally displaced and over half the pre-war population of 22 million is reportedly in need of humanitarian assistance.² Syria is a highly complex and insecure operating environment with shifting frontlines and multiple armed actors. This presents obstacles for humanitarian organisations attempting to negotiate safe and secure access to populations in need of humanitarian support. The UK’s response to the Syria crisis has been its largest ever humanitarian operation, with a total commitment of £2.5 billion to Syria and the region.³

In light of its importance, complexity and scale, the Independent Commission for Aid Impact (ICAI) decided to conduct a performance review of the UK humanitarian response in Syria. As a performance review, it will examine whether the design and delivery of programmes are effective and maximise value for money, and will focus on how well DFID manages its delivery partners and their downstream partners in Syria.

2. Background

Since 2011, over half of Syria’s pre-crisis population has fled their homes. Approximately 3.9 million people live in hard-to-reach areas and around 970,000 remain in besieged regions.⁴ The United Nations Office for the Coordination of Humanitarian Affairs (OCHA) estimates that, as of December 2016, 13.5 million people in Syria required immediate humanitarian assistance.⁵

Insecurity and access or transportation challenges perpetuate extreme hardship and inadequate access to food, water and medical care for people in Syria. OCHA estimates that families spend more than 25% of their incomes on water alone, and one in three children in Syria are now out of school, with a further 1.4 million at risk of dropping out.⁶ More than 12.8 million people required medical assistance as of December 2016.⁷ Healthcare facilities and medical providers have become increasingly scarce, which complicates the aid response.⁸

Figure 1: DFID humanitarian spending in Syria (£ million), 2011-12 to 2016-17⁹

The UK’s humanitarian aid to Syria began on a small scale in the 2011-12 financial year, rose to £40 million in 2012-13 and has subsequently increased each year.¹⁰ DFID funds a number of international non-governmental organisations (INGOs), members of the Red Cross movement, including the International Committee of the Red Cross (ICRC), and several UN agencies, including the World Food Programme (WFP), the United Nations Children’s Fund (UNICEF) and OCHA. UN agencies account for 56% of DFID’s spend in Syria since 2012, and INGOs account for another 34%, with the remainder funding members of the Red Cross movement (approximately 7%), other organisations (1%), and monitoring and evaluation activities and provision of technical assistance (about 2%).¹¹

DFID’s partners cover a wide range of sectors. Between 2011-12 and 2016-17, the largest three sectors were food security (34% of DFID’s humanitarian portfolio), health (18%) and non-food items/shelter (16%).¹² DFID support is channelled both to areas controlled by the Syrian government and to areas under the control of various opposition groups.

At the onset of the crisis, DFID did not have a Syria country office. To manage the response, it established the Syria Crisis Unit within its Middle East and North Africa department in London. DFID has humanitarian advisors located in embassies and consulates in countries neighbouring Syria, but DFID staff are unable to enter Syria. The delivery challenges therefore include the remote supervision of delivery partners and their numerous downstream partners and contractors inside Syria.

3. Review questions

This performance review is built around the following evaluation criteria: effectiveness, efficiency and learning. It will address the questions and sub-questions listed in Table 1.

Table 1: Our review questions

With regards to efficiency, the review will consider, among other factors, whether and how well DFID has responded to the 2016 ICAI review of ‘DFID’s approach to managing fiduciary risk in conflict-affected environments’,¹³ an audit conducted by DFID’s Internal Audit Department and a report from the National Audit Office (NAO).¹⁴

4. Methodology

The review will use a range of methods to ensure triangulation of evidence and will be rooted in the three components outlined later. It will include primary data collection in Syria.

This review faces particular sensitivities, including a need to protect the names of several DFID delivery partners which have not been publicly disclosed in order to protect those organisations and their staff members, downstream partners and beneficiaries inside Syria.

Figure 2: Example of a delivery chain for one DFID partner in Syria

Note: This delivery chain was prepared by one DFID partner in Syria. As the figure shows, the partner reaches Syria from three hubs (Turkey, Lebanon and Jordan) and has 14 immediate downstream partners. Six of those 14 downstream partners have an additional 11 downstream partners between them.

Source: DFID, 2017

Component 1 – Delivery partner case studies

A total of seven DFID delivery partners will be selected for detailed case study analysis, using the sampling criteria discussed in Section 5. The work of these delivery partners and their supervision by DFID will be examined along the whole delivery chain, from DFID in the UK, through DFID’s and the delivery partners’ regional staff and teams, and downstream partners in Syria (see Figure 2 for an example of a DFID delivery chain in Syria). These case studies will consider each of the review questions and will involve data collection from headquarters to the closest to ground level in Syria that the review team can safely access. Other DFID delivery partners that are not selected for the case studies will still be considered over the course of the review, but with a lighter focus on data collection.

Component 2 – Geographic reviews

Given the review’s focus on DFID’s field-level performance within Syria, it is crucial to gain inputs from beneficiaries of DFID’s assistance and individuals familiar with how humanitarian programming is delivered on the ground. Working with an experienced and well-trained Syrian research team, the review will spot-check assistance provision in particular geographic areas, following the delivery chain to the lowest feasible level (community, household or individual). This component will also involve collection of beneficiary feedback on assistance provided by DFID delivery partners. In addition the review will identify past challenges in DFID programming (for example any major complaints raised by beneficiaries) and seek to identify how well DFID and its partners responded in these particular areas.

Component 3 – Thematic case studies

The review will identify field-level challenges raised in the literature, and in DFID’s own documents, to determine how DFID has addressed them within its portfolio. These will include themes that are particularly important to humanitarian action in the Syrian context, potentially including: aid access and principled humanitarian engagement; the remote management and monitoring of humanitarian programmes; and capacity building of nascent local humanitarian actors. This list of topics is provisional. The final topics will be identified through a further review of the literature and through discussion with DFID and humanitarian actors acquainted with field-level realities in Syria. The thematic case studies will consider how the selected themes are addressed by DFID, its delivery partners and their downstream partners.
Data collection for these three components will involve the methods outlined below.

Figure 3: Methodological components and data collection methods

Literature Review

The review team will prepare a review of literature on humanitarian action in Syria. This review will consider research and reports prepared by experts from academia, think tanks and humanitarian organisations, including academic and ‘grey’ literature. This will address a range of topics, including but not limited to: needs assessments, coordination, humanitarian financing, remote management and monitoring, humanitarian access, humanitarian principles, accountability to affected populations and cost effectiveness.

Documentation review

The review team will examine a wide variety of relevant documents and data sets from DFID and its partners. These will include UK government strategic documents on Syria, programme documentation, results databases, risk management and due diligence reports and any other pertinent documents. These documents will be considered for each of the three methodological components noted above and particularly for the delivery partner and thematic case studies. In addition, the review team will consider relevant materials, including research findings, programme reporting and previous evaluations, to better understand the context in which UK aid is being delivered in Syria.

Key informant interviews

Semi-structured interviews will be conducted with DFID staff currently and previously involved in managing the department’s Syria portfolio. Interviews will also be held with DFID delivery partners and their downstream partners or subcontractors; representatives of other bilateral donors; and other agencies or individuals with a well-informed perspective either on the humanitarian situation in Syria or on DFID’s performance. Some of these interviews will be conducted remotely by an Arabic-speaking researcher. Further interviews will be held with beneficiaries of DFID assistance and, where relevant, with community and civil society representatives in areas where DFID is providing assistance.

Focus group discussions

These will be held with groups of stakeholders, including representatives of aid agencies and UK aid beneficiaries within Syria. Focus groups will be guided by a small number of prompts or questions. We will aim to ensure that the focus groups capture the perspectives of both women and men. Where appropriate, focus group discussions may be organised to take place during or just after existing forums related to Syria (such as non-governmental organisation forums) in order to encourage participation.

Beneficiary consultations

Semi-structured qualitative interviews will be conducted by Syria-based researchers and will target communities where one or more of DFID’s delivery partners are currently involved in humanitarian programmes, or where they have recently been active in providing assistance. These interviews, as well as a small number of focus group discussions, will seek to verify the nature and quantity of assistance received and to assess various elements of programme quality and effectiveness. For instance, questions will address issues such as: overall satisfaction, the prioritisation of needs, vulnerability, beneficiary selection and feedback and complaints mechanisms.

Table 2: Review criteria, methodological components and data collection methods

5. Sampling approach

Sampling applies to the selection of delivery partners for case studies, the selection of areas in which to conduct geographic reviews and the selection of topics for thematic case studies.

Delivery partner sampling

We have chosen to review a sample of seven DFID delivery partners out of the 25 which have directly received DFID funding since 2012.¹⁵ Partners have been excluded from consideration (a) If they did not have ongoing DFID programmes in the 2016-17 financial year and (b) if, for UN agencies, they have not received an average of at least £4 million in DFID funding per year, or at least £3 million per year for INGOs (given that these may be termed relatively minor partners for UK aid). This approach excludes ten DFID delivery partners. From the remaining 15, the review will purposively sample seven in order to reflect major variations in DFID’s Syria portfolio, including: (i) the type of partner, including UN agencies, INGOs and members of the Red Cross movement; (ii) the areas in which they operate, including government- and opposition-held areas; (iii) the hubs (Damascus, Turkey, Lebanon, Jordan and/or Iraq) from which they reach Syria; (iv) the humanitarian sectors they cover; (v) the level of DFID funding they receive; and (vi) the number of downstream partners in Syria that deliver assistance on their behalf.

Geographic sampling

The geographic case studies will be purposively sampled based on two key criteria. The first is the number of DFID delivery partners operating with UK aid in particular districts or sub-districts of Syria. Once the review team has identified the number of DFID partners operating in different locations, it will consider which areas can be safely accessed by a team of Syrian researchers based in the country. For instance, if humanitarian activities have been paused in a particular area for security reasons, these will be excluded from consideration. The specific beneficiaries and community leaders to be interviewed or consulted in these selected areas will be chosen purposively in order to capture the perspectives of individuals with different genders, ages and levels or forms of vulnerability, including but not limited to people with disabilities.

Thematic sampling

Themes for this component are being identified through a review of DFID documents, documents produced by other major humanitarian stakeholders in Syria and research studies concerning Syria.

6. Limitations of the methodology

Results data

In assessing the effectiveness of UK aid programmes in Syria, we will rely in part on monitoring and evaluating data and documents generated by DFID’s partners and by a third-party monitoring consortium contracted by DFID in 2015. This data covers quantitative outputs but does not necessarily capture outcomes or qualitative assessments of programme quality. The review team will therefore supplement this results data with information from a range of other sources: beneficiaries and local leaders in areas where DFID operates, downstream partners, other major humanitarian stakeholders, DFID personnel and DFID partners.

Attribution

The dynamics of conflict and instability in Syria are complex, and the UN-operated Financial Tracking Service notes that 69 donor agencies contributed to the 2016 Humanitarian Response Plan for Syria. Donor funds, particularly for multilateral agencies but also for INGOs, include multi-donor arrangements to finance large-scale programmes. As such, the review may face challenges in attributing particular activities, outputs and outcomes to the UK’s humanitarian portfolio. To address this challenge, the review will work with DFID and its partners to identify which elements of their programmes are funded by DFID and how it accounts for non-earmarked funding. Furthermore, the review will transparently report where conclusions are based on programmes or activities that are (a) entirely or primarily funded by the UK or (b) co-financed by the UK alongside several other donors.

Access to information and stakeholders in Syria

Obtaining information from stakeholders in Syria, whether over the phone or in person, can be difficult for a range of reasons: stakeholders may be hesitant to trust unknown or outside actors; fears concerning information security may cause some stakeholders to be reluctant to speak openly; and insecurity can obstruct movement and field-level data collection. Furthermore, the pace and scale of work in Syria at times means that gaining access to certain stakeholders can prove challenging. To mitigate these challenges, the review team includes individuals with prior experience of conducting research and evaluations on Syria. The team will work carefully with DFID to communicate with stakeholders in Syria and encourage them to contribute to this review. Such communications will also outline the steps that the review team is taking to keep all interviewee information confidential and to withhold the names of particular individuals and organisations in order to avoid generating any additional risks.

7. Risk management

8. Quality assurance

The review will be carried out under the guidance of ICAI chief commissioner Dr Alison Evans, with support from the ICAI Secretariat. The review will be subject to quality assurance by ICAI’s service provider consortium.

The methodology and the final report will be peer reviewed by Gilles Carbonnier, professor of development economics at the Graduate Institute of International and Development Studies in Geneva. Professor Carbonnier is editor-in-chief of International Development Policy and president of the board of directors of the Centre for Education and Research in Humanitarian Action. His research and publications focus on the economics of humanitarian crises and responses.

9. Timing and deliverables

Barring major contextual obstacles, the review will adhere to the following timeline:

Footnotes

1. ↩ Glossary of terms used by the Department for International Development, UK government website, published August 2013, accessed August 2017, link.
2. ↩ About the Crisis, OCHA website, accessed April 2017, link; Syrian Arab Republic, OCHA, 2017, link.
3. ↩ UK government press release, April 2017, link.
4. ↩ ‘Hard to reach’ means: an area not regularly accessible to humanitarian actors for sustained humanitarian programming due to the denial of access, the continual need to secure access, or due to restrictions such as active conflict, multiple security checkpoints or failure of the authorities to provide timely approval. ‘Besieged’ means: an area surrounded by armed actors with the sustained effect that humanitarian assistance cannot regularly enter, and civilians, the sick, and wounded cannot regularly exit. 2017 Humanitarian Needs Overview, OCHA, December 2016, link.
5. ↩ 2017 Humanitarian Needs Overview, OCHA, December 2016, link.
6. ↩ See footnote 5.
7. ↩ See footnote 5.
8. ↩ Ongoing USG Humanitarian Assistance Syria – Complex Emergency, USAID, April 2017, link.
9. ↩ These figures exclude CSSF spending in Syria.
10. ↩ Statistics in International Development 2016, DFID, November 2016, Table A4b, link.
11. ↩ This is based on figures presented by DFID to the ICAI Syria review team and confirmed through a review of internal DFID data and materials.
12. ↩ See footnote 11.
13. ↩ DFID’s approach to managing fiduciary risk in conflict-affected environments, ICAI, August 2016, link.
14. ↩ Responding to crises, NAO, January 2016, link.
15. ↩ This figure was identified through a review of DFID internal documents and data tables shared with the ICAI review team.