The Effectiveness of DFID’s Engagement with UNRWA – Inception Report

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 In this review, we will examine the effectiveness of DFID’s engagement with the United Nations Relief and Works Agency for Palestine refugees in the Near East (UNRWA). This Inception Report sets out the evaluation questions, methodology and a work plan for the delivery of the review. It is, however, intended that the methodology and work plan be flexible enough to allow for new issues and questions that emerge over the course of the review.

2. Background

2.1 The background to this review is provided in Section 2 of the Terms of Reference.¹ It details the wider context within which DFID’s Palestine Programme is implemented, provides information on DFID’s previous work in the region and summarises its current programmes. Additional information is provided here on the impact of the Arab Spring and the UNRWA institutional context.

2.2 UNRWA has a mandate for the provision of relief, human development and protection services to refugees in five fields of operation: Jordan, Lebanon, West Bank, Gaza and Syria (see Figure 1 below).

2.3 The Arab Spring has resulted in far-reaching political, social and security impact across the Middle East and North Africa region, including the five field offices from which UNRWA operates. In the West Bank and Gaza, the impact of the Arab Spring has been limited. During 2011 and early 2012, there were a number of minor peaceful demonstrations and calls for the unification of the politically divided Hamas-led Gaza and Fatah-led West Bank. The Arab Spring has resulted in the increased political power of a number of Islamic political parties across the region. In Egypt and a number of other countries, the Muslim Brotherhood has taken on a leadership role. Hamas is a wing of the Muslim Brotherhood which is also in power in Egypt. This has had a far reaching impact on the political landscape in Gaza. Whereas in Syria the Arab Spring can be considered to be one of the triggers for the present crisis, which has also had an impact in Lebanon. Many analysts believe that the potential for the Arab Spring to lead to turmoil in Jordan remains significant. UNRWA is operating in a highly dynamic context but, as yet, this has not had a substantive impact on how UNRWA operates or delivers services in Gaza, West Bank, Jordan or Lebanon. The Syrian crisis poses significant specific challenges to UNRWA operations in the Syria field office but this is beyond the scope of the present review.

2.4 UNRWA performs a quasi-state function for Palestine refugees and this means UNRWA is a relatively unique institution. UNRWA has to work closely with the Popular Committees that represent refugees and the Staff Unions that represent the 30,000 UNRWA staff. These are extremely powerful and politically active organisations. At the same time as being a UN agency, UNRWA is also a multilateral institution. This review has, therefore, adopted a hybrid approach that takes account of the unique institutional context of UNRWA. This will require drawing on approaches typical of reviewing budget and project support to a state, as well as approaches typical of reviewing project support to a multilateral organisation.

3. Purpose of this review

3.1 The purpose of this review is to assess the effectiveness and value for money of DFID’s engagement with UNRWA from 2008-13. This will be assessed by the extent to which DFID’s assistance is meeting the needs of intended beneficiaries.

3.2 The review will, therefore, aim to develop an understanding of the specific mandate of each UNRWA field office as defined in relation to the services provided to refugees by the relevant host government or authority. On the basis of a clear understanding of the political context of each UNRWA field office, the team will be able to review how effectively DFID has leveraged reform to achieve greater efficiency, impact and effectiveness in the delivery of appropriate services to refugees.

4. Relationships to other initiatives and evaluations

4.1 Details of selected previous evaluations of UNRWA’s work are provided in Section 4 of the Terms of Reference. These include internal reviews of UNRWA,² focussed on its operational effectiveness, external reviews of its operational effectiveness³ and higher-level reviews of the overall developmental situation in the Occupied Palestinian Territories (OPTs).⁴ A desk-based review as part of this work will examine the relevant findings of these and other evaluations and assess the extent to which findings from these have informed DFID’s current assistance to Palestine refugees.

5. Methodology

Analytical Approach

5.1 Our review will examine the effectiveness of DFID’s support to UNRWA, within the context of the impact of the United Nations (UN) vote on Palestinian statehood⁵ and the regional Arab Spring. This review will focus on:

• DFID’s approach to maximising the positive impact for intended beneficiaries through its financial contribution to UNRWA;
• DFID’s approach to ensuring value for money;
• the decision-making processes that determine the scale of DFID’s contribution to UNRWA and its alignment with DFID’s objectives and other programmes;
• how effectively DFID uses its contribution to leverage positive change in UNRWA, including the role of staff unions and Popular Committee structure; and
• DFID’s strategy for helping intended beneficiaries to move sustainably from poverty.

5.2 DFID’s programming for Palestine refugees operates at two, linked levels. The first level is a response to the immediate needs of Palestine refugees for access to basic services. At this level, DFID responds by directly funding the provision of public services, such as education, health and social safety nets, through UNRWA and other routes. This level relates to the first four questions above. The second level relates to the last bullet point, the promotion of economic activities and access to livelihoods and employment to move sustainably out of poverty.

5.3 In order to answer the first bullet point, we will talk to a wide range of intended beneficiaries. This will include focus group discussions, led by our regional experts. We will triangulate these findings with the data collected during a desk review of evaluations of UNRWA’s support. Taken together, these will provide a strong assessment of the extent to which intended beneficiaries’ needs are being met.

5.4 To explore fully the value for money aspects of the second bullet point above, we will seek to examine UNRWA’s financing systems for education and food expenditure. This will provide us with an assessment of whether DFID is managing value for money through the whole delivery chain. Where possible, we will compare the unit costs of UNRWA’s impact with that of other organisations. This will include an analysis of the efficiency of UNRWA internally and in comparison to a number of host governments and/or the Palestinian Authority.

5.5 To answer the third of these bullet points, we will review DFID’s programming decisions, through its documentation and through discussions with officials. In particular, we will examine the economic appraisals of these projects, to assess whether DFID is fully considering the costs and benefits of its spending decisions and is basing its allocations on analysis of the potential impact on intended beneficiaries.

5.6 To answer the fourth bullet point, we will speak to officials from UNRWA and UNRWA staff unions, as well as to other donors. We will also examine DFID project documentation related to UNRWA spending, including the recent fiduciary risk assessment of UNRWA. This will establish the extent to which DFID assures itself of UNRWA systems and processes, as well as DFID’s effectiveness in pushing for necessary reforms.

5.7 The second level relates to the UK’s longer-term aim to create a sustainable solution to the issues facing Palestine refugees. This relates to the fifth bullet point above. Clearly, this is bound together with the broader peace process which is beyond the scope of this review. This review will, however, consider the extent to which the aims of the shorter-term assistance are consistent with – and contribute to – the UK’s longer-term objectives. The review team will be cognisant of the political context of examining issues of sustainability, in particular the direct links to the peace process.

5.8 The scope of this review covers DFID support to UNRWA which breaks down into three separate programmes of support:

• UNRWA: Funding Basic Services and Protection for Palestine Refugees (2011-15) – DFID provides £106.5 million over approximately three years, which equates to the budget support equivalent component of UNRWA as it is un-earmarked support for the organisation and the delivery of a wide range of services.
• UNRWA: Improving Food Security for the People of Gaza (Up to March 2015) – DFID provides £9.6 million of support to non-refugees in Gaza through the provision of food vouchers⁶ and £14.4 million of support to UNRWA for the Job Creation Programme that refugees in Gaza benefit from.
• UNRWA: Improving Access to Education in Gaza – DFID provides £14.7 million of support to the reconstruction of UNRWA schools in Gaza.

5.9 The business cases for the three focal projects⁷ for this review align much more closely to the first, lower-level objective. For example, the results chain below is taken from DFID’s business case for UNRWA.⁸ It shows that the direct provision (through UNRWA) of basic services is seen as the primary means to bring about long, healthy and dignified lives for Palestine refugees. The review will look at the extent to which the promotion of sustainable routes out of poverty is prioritised by DFID and UNRWA.

5.10 During our review, we will seek to understand how DFID believes these programmes contribute to the wider objectives of the DFID Palestine Programme, including the UK Government’s longer term aim of supporting a successful Middle East Peace Process (MEPP).⁹

5.11 Understanding the linkages between these three projects and DFID’s State building and Service Delivery Grant to the Palestinian Authority is central to this. This project is more clearly targeted at DFID’s higher-level aims – for example, higher economic growth and a peaceful Palestinian state and society. We will assess whether DFID is co-ordinating the projects under review with this support and is responding to changes brought about by the State building and Service Delivery Grant. This will include an assessment of the capacity and management structures within the DFID office, as well as mechanisms in place to avoid duplication of beneficiaries across programmes.

5.12 Assessing these linkages in the case of OPT spending will provide a case study of whether DFID is co-ordinating and sequencing its support to deliver long-term sustainable impact; or whether it is becoming trapped in a cycle of meeting immediate needs. In particular, we will examine the consistency between the OPT programme and the UK Government’s concept of a ‘golden thread’¹⁰ – a set of interlinked actions intended to lead to sustainable development. The review team will examine the political context of UK Government efforts to support the peaceful resolution of conflict and how DFID supports this UK Government-wide effort.

5.13 We will conduct this review in three linked phases: pre-visit analysis and discussions; field visit; and post-visit analysis and discussions.

Pre-visit activity

5.14 Pre-visit activity will be in four areas. First, a wide range of survey and other primary data and analysis exists on the support to Palestine refugees and the results of that support. Prior to the field visits, we will undertake a desk-based review of the available data, evaluations and assessments, including the work of the World Bank and of the Palestinian Central Bureau of Statistics. In particular, we will assess the methodology used in its collection and reach a view on whether it represents a robust analysis of needs among Palestine refugees in their range of locations. A bibliography of all literature reviewed and data consulted will be compiled as a draft annex to the final report.

5.15 Second, we will track money provided by donors, including DFID, through UNRWA to intended beneficiaries in order to determine the efficiency of service delivery methods. We will use this methodology to carry out two case studies which track DFID financing to UNRWA. The final selection on the case studies will be made by the review team during the field visit. The selection will prioritise choosing case studies under different funding modalities (budget support and project support) and in different locations if this is feasible.

5.16 Third, we will review DFID’s documentation related to the programmes, to understand the analysis underpinning the original designs, as well as how the projects have evolved over time in response to lessons learned as the programme has been implemented.

5.17 Fourth, we will engage with key stakeholders in the UK. This will include interviews with key DFID staff directly related to the programme, as well as DFID officials from evaluation and audit departments. It will also include discussions with the Foreign and Commonwealth Office (FCO), UN organisations, refugee host authorities and representatives in the UK of a range of non-governmental organisations (NGOs), including those providing services and support to Palestine refugees.

Field visits

5.18 During the field visits, we will prioritise speaking to intended beneficiaries to address the issues set out in the evaluation framework. We will interview intended beneficiaries during four field visits to refugee camps in Jordan, Lebanon, West Bank and Gaza.

5.19 We will also hold discussions with other key stakeholders in the following locations: Jerusalem, Jordan, Lebanon, West Bank and Gaza. Stakeholders will include:

• Palestinians who are the intended beneficiaries of DFID support in Gaza, the West Bank, Jordan and the Lebanon;
• UNRWA officials at their headquarters in Amman, Jordan;
• UNRWA project delivery staff in food security, health and education programmes;
• UNRWA staff union representatives;
• representatives of NGOs who are delivering services to Palestinians;
• representatives of the Palestinian Authority in the West Bank;
• journalists and other public figures who have expressed views on UNRWA;
• DFID officials in Jerusalem; and
• the United States Agency for International Development (USAID), the European Commission (EC) and Swedish officials, who perform a liaison role with DFID and UNRWA.

5.20 These discussions will include structured focus groups, discussions with beneficiaries and workers in hospitals, schools and other public service institutions, individual one-on-one discussions with refugees inside the camps and visits with families. These discussions will triangulate the data analysis of impact from the desk-based review.

5.21 The focus group discussions will include between 10-12 beneficiaries of a range of the UNRWA services and projects. At least one focus group discussion will be held in each of the four field locations being visited (West Bank, Gaza, Jordan and Lebanon). Furthermore, individual interviews will be held with refugees during visits to refugee camps. This number will allow for an open and frank discussion of the key topics whilst ensuring a degree of representation of diverse opinions and perspectives.

5.22 The detailed design of the focus group discussions, including locations, specific questions to be posed and the selection of beneficiaries will be undertaken by the Qualitative Data Expert based in the region. The beneficiary selection for the focus groups and with beneficiaries in hospital, schools and with refugee families will be conducted directly by the review team. This will be done through a random process of selection, whilst ensuring a degree of balance between ages, gender and beneficiaries from different programmes. Discussions will be focussed on impact question in the evaluation framework presented below.

Post-visit analysis

5.23 After the visit, we will collate and analyse all of the data collected. We will follow up by phone or videoconference on any outstanding issues. In particular, we will meet again with representatives of DFID and NGOs in London to check our understanding of any unresolved issues.

Evaluation Framework

5.24 The evaluation framework for this review is set out in the Annex. This has as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. It also incorporates other pertinent questions we want to investigate in this review. The questions which are highlighted in bold are those on which we will focus in particular.

5.25 The evaluation methodology will comprise the following elements.

Literature review

5.26 There is a wealth of analysis and research related to the Palestine refugee situation, carried out by DFID and a range of other organisations. We will undertake a detailed literature review, including scanning the various relevant knowledge blogs on this issue, to assess this information and to enable us to understand the dynamic regional context in which UNRWA is operating, the delivery challenges and the past record of external assistance to refugees. This examination will consider the evidence on effectiveness and impact from these past programmes and reviews and provide an indication of how DFID has utilised the available evidence to develop its programme plans and in its engagement with UNRWA.

Review of programme management systems

5.27 We will review all available documentation related to the programmes under review, including project concepts and management processes, business cases, financial information, economic appraisals, terms of reference and annual reviews. We will also review broader strategic documents on DFID’s country policy and objectives and the guidance on programming in conflict-affected countries. This will include a detailed examination of DFID’s State building and Service Delivery Grant to the Palestinian Authority, to assess the linkages to the three projects under review.

5.28 The analysis of these documents will be used to inform lines of questioning during our consultations with DFID and identified stakeholders both in the UK and during country visits.

Preliminary consultations with peers and stakeholders

5.29 In order to maximise the benefit of face-to-face meetings during country visits, we will gather as much information as possible prior to the visit, through face to face consultations with those based in London and by telephone for others.

5.30 We will identify a range of experts in the provision of refugee support to discuss methodological approaches to delivery programmes and long term sustainability, with particular emphasis on the direct delivery method implemented by UNRWA. These will include meetings and telephone calls with individuals in international think tanks and research organisations. These will include: the Institute for Palestine Studies (IPS); Refugee Studies Centre Oxford Department of International Development, University of Oxford; and the Overseas Development Institute.

5.31 We will identify key advisers within DFID and FCO who provide thinking and policy on Middle and Near East strategy, humanitarian aid to refugees and UNRWA engagement and interventions. These are likely to be within the FCO and DFID’s Middle East and North Africa Departments based in London. Interviews will be undertaken with relevant individuals to gather information about the context and policy framework within which DFID’s humanitarian work in the OPTs is undertaken; as well as how DFID’s wider policy objectives have informed the design of these programmes. In addition, information regarding other related work in the OPTs that DFID supports will be gathered, including evidence of learning and co-ordination across programmes.

Field research

5.32 We will carry out country visits to Jordan, the West Bank, Gaza and the Lebanon. These will take place over a two-week period starting on 6 May 2013. We will visit Jerusalem to meet DFID and other Jerusalem-based stakeholders. From Jerusalem we will visit the West Bank and Gaza to speak to intended beneficiaries and officials responsible for delivery. We will then travel to Amman for meetings with UNRWA centrally and to visit a refugee camp in Jordan, then on to Lebanon, before returning to the UK, via Jerusalem, to hold a second round of discussions with DFID and other Jerusalem-based stakeholders.

5.33 During these visits, we will prioritise discussions with intended beneficiaries and speak with UNRWA, DFID, FCO, donors, NGOs and Palestine Authority officials. This will provide the opportunity to discuss, in detail, project delivery and impact of the UNRWA projects with implementing partners and the direct beneficiaries.

5.34 In addition to discussions with individual Palestine refugees, we will conduct a series of well-organised focus group discussions held on UNRWA premises but organised independently of UNRWA by the review team. The qualitative data expert from the region will interrogate the databases in each Field of Operations Office and randomly select a sample of approximately eight refugees for each focus group. Each sample will contain at least 30% females. The focus group discussions will address each of the key questions to be addressed in this review. A semi-structured approach to the focus group discussions will be designed by the qualitative evaluation team member to meet the review methodology requirements by trying to capture the perceptions of refugees regarding the key issues identified. This will build on and be shaped by the data collected in the pre-visit phase and the other data collection methods used during the field visit.

5.35 The situation in Gaza can change rapidly, as the recent demonstration and evacuation of UNRWA staff and the security operation in November 2012 illustrates. The review team, therefore will only be able to confirm the Gaza visit at the last minute. Contingency plans will be in place for additional research and meetings to be held in Jerusalem and the West Bank, if the Gaza visit is not possible. These contingency plans will include organising meetings with UNRWA’s Gaza Field Office staff outside Gaza, most likely in Jerusalem.

6. Roles and responsibilities

6.1 This review will be led and managed on a day-to-day basis by the Team Leader who will be the primary point of contact with DFID. We have prioritised regional awareness over ICAI experience. This should be mitigated by the presence of the Lead Commissioner and a member of the ICAI Secretariat on the field visit.

6.2 The review will also draw on input from a second core team member, who will be responsible for setting the methodology and conducting two case studies to assess how efficiently funding passes from DFID, through UNRWA, to intended beneficiaries.

6.3 The team will be supported by two researchers. The first will conduct a thorough literature review on the evidence available on the impact of UNRWA on Palestine refugees. The second researcher will assist with the literature review and will also be responsible for the analysis and collation of project-related material from DFID.

6.4 The team also includes two Palestinian experts. Both experts are experienced facilitators and are based in Gaza and have a very in-depth understanding of the refugee situation and the role of UNRWA. One will lead on the qualitative data components, in particular the refugees’ interviews and focus group discussions. The qualitative data experts will be responsible for the design of the focus group discussion methodology, random sample selection of beneficiaries and the collection and analysis of data. The regional experts will also provide context, support field visit planning and be responsible for identification of beneficiaries. Additional translator/interpreter support will also be available to the review team.

6.5 The peer reviewer, who is familiar with DFID programmes in conflict-affected countries, will support the literature review in the area of the analytical framework. He will also provide advice throughout to ensure prior ICAI experience is reflected within the review team.

6.6 While lead responsibility for answering sections of the framework is shown below, all team members will contribute to the analysis supporting the findings for each section.

7. Expected outputs and time frame

7.1 The following timetable is based on the assumption that the report will be finalised in Quarter 3 2013, to meet ICAI’s requirements.

8. Risks and mitigation

8.1 The following sets out the key risks and mitigating actions for this review:

9. How will this review make a difference?

9.1 DFID has committed £343 million to the OPTs over the period 2011-15. Its financing through UNRWA totals £136 million, around 40% of its total Operational Plan expenditure. The review will provide an assessment of how a multilateral organisation that receives significant DFID funding but was not part of DFID’s Multilateral Aid Review, is meeting the needs of the intended beneficiaries (Palestine refugees). There is a wealth of material available on the situation of Palestine refugees. We will assess the extent to which DFID is using this material. We will also use a focus group methodology to conduct a ‘reality check’ of the data that DFID is using in its decision making.

9.2 UNRWA is a unique body as it is a multilateral organisation that effectively performs the functions of a quasi-state body, delivering services to the refugee population. This status underpins how the review has been structured and the detailed questions that have been devised and included in the Evaluation Framework. The overall questions for this review, as set out in paragraph 5.1, also provide the basis for examining four issues of more general interest to the impact of the UK’s development expenditure.

9.3 First, UK Government policy towards development has been expressed as a golden thread – a series of necessary, sequenced actions that lead to sustainable development. We will assess the extent to which funding through UNRWA is consistent with the golden thread concept and whether it is creating unsustainable dependence. In particular, we focus on the linkage between DFID’s support to UNRWA and its State building and Service Delivery Grant to the Palestinian Authority. The golden thread and value for money analysis will provide a valuable insight into UNRWA; an agency that is not widely understood outside the region.

9.4 Second, this review is a strong case study for value for money through the delivery chain. We will examine this through a ‘follow the money’ exercise, to establish how DFID’s funds are spent and the impact it is delivering for intended beneficiaries. We will also seek to compare end-to-end delivery costs through UNRWA with alternative mechanisms and develop an understanding of how DFID reaches its decisions for selecting a delivery model.

9.5 Third, this review will demonstrate an innovative model of analysing how donors hold multilateral organisations to account against an agreed reform agenda. The model will focus on understanding how DFID balances demands for reform to achieve greater results whilst adhering to OECD DAC principles on aid effectiveness. This will, thereby, support DFID in leveraging reform of UNRWA as well as providing accountability for UK taxpayers on a large spending stream within the DFID Palestine Programme.

9.6 Finally, we will build on our understanding from other reviews of DFID’s spending through multilaterals and budget support. In particular, we will assess whether DFID is sufficiently robust in demanding impact in spending through UNRWA as a multilateral organisation and as a quasi-state body.

10. Annex: Evaluation Framework

The detailed evaluation framework is provided in a separate annex to this document. It includes the specific evaluation questions, criteria for assessment, and sources of evidence to be used in the review.

Footnotes

¹ Terms of Reference: The Effectiveness of DFID’s Engagement with UNRWA, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-OPT-TOR-Final-08042013.pdf.

² Terms of Reference: UNRWA sustaining Change: Relief and Social Services Department, UNRWA, November 2011, http://www.unrwa.org/userfiles/201201154647.pdf.

³ Multilateral Organisation Performance Assessment Network, Organisational Effectiveness Assessment, UNRWA, December 2010, http://www.unrwa.org/userfiles/201204291514.pdf.

⁴ The Humanitarian and Development Situation in the Occupied Palestinian Territories, Eleventh Report of Session 2007-08, Volume 1, International Development Committee, 24 July 2008, http://www.publications.parliament.uk/pa/cm200708/cmselect/cmintdev/522/522i.pdf.

⁵ BBC Questions and Answers on Palestinians’ upgraded UN status, BBC, November 2012, http://www.bbc.co.uk/news/world-middle-east-13701636.

⁶ The WFP food voucher component of this programme is for non-refugees and it is not administered by UNRWA. Therefore, it is beyond the scope of this review.

⁷ These three projects are set out in the Terms of Reference paragraph 2.18. They are: core support to UNRWA; financial support to improve food security in Gaza through UNRWA; and financial support to improve education in Gaza.

⁸ Intervention Summary: United Nations Relief and Works Agency (UNRWA): funding basic services and protection for Palestine refugees, DFID, undated, http://projects.dfid.gov.uk/iati/Document//3717451.

⁹ Operational Plan 2011-2015, DFID Palestinian Programme, DFID, July 2012, http://www.dfid.gov.uk/documents/publications1/op/occupied-palestinian-territories-2011.pdf.

¹⁰ See, for example, David Cameron’s June 13 2012 speech on vaccines and development, http://www.politics.co.uk/comment-analysis/2011/06/13/cameron-development-speech-in-full.