Sudan-ToRs-FINAL – Evaluation of DFID’s Water, Sanitation and Hygiene programming in Sudan

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We have decided to review the Department for International Development’s (DFID’s) water, sanitation and hygiene (WASH) programmes in Sudan.¹ These Terms of Reference outline the purpose and nature of the review and identify its main themes. A detailed methodology will be developed during an inception phase.

2. Background

Country context

2.1 Since independence in 1956, Sudan has been beset by protracted conflict that has contributed to years of underdevelopment and has resulted in some of the most severe humanitarian crises in recent history. In Darfur, one of the two main conflict zones, 4.5 million people have been directly affected by conflict and 1.9 million people remained displaced in 2011.² Following decades of civil war, a Comprehensive Peace Agreement (CPA) was signed in January 2005, leading to the establishment of a government for South Sudan and its secession in July 2011. While Sudan and South Sudan are now two separate countries, the two governments have a number of unresolved issues and there is continued conflict in Southern Kordofan, Blue Nile and the disputed area of Abyei.

2.2 Sudan is resource-rich, endowed with oil, natural gas and minerals. The secession of South Sudan, however, where many of the oil fields are located, has deprived Sudan of three-quarters of its oil revenues, causing a spiralling budget deficit and foreign currency shortages that threaten macroeconomic and political stability.³

2.3 Conflict and population displacement have hampered the Government of Sudan’s ability to deliver basic services. Conflict has disrupted harvests and agricultural production. With most of the population dependent on agriculture, 46.5% of Sudan’s 31 million people live below the national poverty line, with wide disparities within and between states.⁴

2.4 In 2009, Sudan was the world’s ninth-largest recipient of development aid (at US$2.4 billion) and the largest recipient of humanitarian aid (at US$1.3 billion).⁵ In recent years, the UK has been the second-largest Organisation for Economic Co-operation and Development (OECD) bilateral donor in Sudan (figures predate the secession of the South). UK aid to Sudan between 2011 and 2015 will be around £146 million. Sudan received £31 million in 2011-12. This will rise to £71 million in 2012-13, as a result of a large humanitarian programme, falling back to £44 million by 2014-15.

DFID’s work in Sudan

2.5 DFID published a revised Sudan Operational Plan 2011-2015⁶ in May 2012 following the secession of South Sudan. The plan aims to support Sudan during this transitional phase. DFID’s intention is to move away from humanitarian assistance towards longer-term development initiatives. According to the plan, DFID objectives include:

• ‘A gradual transition from life-saving humanitarian assistance towards support for sustainable livelihoods, particularly in conflict-affected areas’;
• ‘Peace-building between Sudan and South Sudan; in the East; in Darfur; and between Sudan and its neighbours, including through support to reduce the underlying causes of conflict and the community-level impact by improving access to services, and enabling currently excluded groups to influence decision-making’;
• ‘Increased security, peace and justice; democratic accountable governance; and a reduction in corruption’; and
• ‘More equitable and sustainable development through better use of the national budget; the extension of basic services; and a focus on economic diversification, increased livelihood opportunities, and employment.’

2.6 The Operational Plan seeks to deliver the following headline results by 2015:

• 800,000 people with access to clean drinking water;
• 20,000 young people receive education and training;
• 80,000 people and firms have access to financial services;
• 10,000 square kilometres of land returned to productive use;
• 250,000 women have improved access to justice;
• 3 million people have access to health and nutrition-related programmes; and
• 1.5 million people achieve food security and receive livelihood assistance.

2.7 DFID does not provide any financial assistance directly to the Government of Sudan, at either the federal or state level; it works through UN agencies and the World Bank, national and international non-governmental organisations (NGOs) and the private sector. DFID faces significant challenges in scaling up aid and delivering results, as operational partnerships are often constrained by insecurity and government restrictions on the work of international agencies in fragile and conflict-affected areas.

2.8 The security risks and highly challenging operating environment lead to persistent delays in implementation. As a result, the operational costs and risks to the achievement of planned results are generally higher than in other settings. The financial and technical capacity of local authorities in areas such as water management and maintenance is often very low, making it difficult to ensure sustainable results.

Water, sanitation and hygiene aid in Sudan

2.9 DFID is committed to helping developing countries to achieve their Millennium Development Goals (MDGs), including on water and sanitation. Around the world, it supports interventions aimed at providing poor communities with access to safe drinking water supplies, promoting access to and use of basic sanitation and encouraging behavioural change to reduce health risks associated with poor hygiene.⁷

2.10 Sudan is off track on its water and sanitation-related MDGs. The limited data available suggest 61% of households have access to an improved water source (67% in urban areas and 58% in rural areas) and 28% have access to improved sanitation (65% in urban areas and 25% in rural areas).⁸ There are wide regional discrepancies, however, with access to water and sanitation at 73% and 51% in Khartoum but at only 27% and 24% in Red Sea State. These figures are lower than the 1990 baseline, suggesting a deteriorating situation.⁹

2.11 DFID is one of a number of international development partners supporting water sector activities in Sudan, including Japan, UNICEF, the Islamic Development Bank, the Kuwait Fund and the African Development Bank. China and India are also active in the sector.¹⁰ DFID has invested £14.4 million in WASH in Sudan and South Sudan over the past five years, representing about 4% of its overall investment in WASH.¹¹

2.12 A large proportion of DFID expenditure has been in Darfur. With growing water scarcity recognised as a major cause of conflict in many sub-Saharan African countries, DFID WASH programming also supports peacebuilding. In Sudan, 4.5 million people have been directly affected by conflict in Darfur region alone.¹² There are many drivers of conflict in Darfur, including water shortages and inadequate and inequitable access to basic services. The importance of water to providing peace and secure livelihoods in Darfur was acknowledged in an international conference on Darfur (Water for Sustainable Peace) in Khartoum in June 2011, at which the Government of Sudan and the United Nations requested international support of nearly a billion pounds over the next six years to reverse the rapid decline in Darfur’s water supply.¹³

2.13 DFID’s WASH programming uses three different delivery channels: UN agencies; UN- and World Bank-administered trust funds; and international NGOs.

i) WASH sector support via UN projects

• The Darfur Urban Water Supply (DUWS) project is a £6.7 million initiative implemented by the United Nations Office for Project Services (UNOPS),¹⁴ launched in November 2010 and due for completion in December 2012. The project’s objective is to provide access to clean drinking water to 150,000 people by increasing the water supply to four major urban centres in Darfur. A complementary project to construct a distribution network in three of the towns is being managed by the federal government.
• The Sudan Integrated Environment Programme (SIEP) is a £20 million programme implemented by the United Nations Environment Programme (UNEP), with support from UNOPS, launched in 2009 and due for completion in 2013. Its aim is to improve the management and use of natural resources through investments in more effective environmental governance. The programme is constructing or rehabilitating a series of dams in Darfur to improve the sustainability of the water supply there. Some SIEP dams contribute to urban water facilities developed under DUWS. As of May 2012, four construction projects have been completed, four are undergoing design or procurement and two feasibility assessments are underway.¹⁵

ii) WASH sector support via multi-donor trust funds

• Since its commencement in 2005, DFID has contributed nearly £300 million to the Sudan Common Humanitarian Fund (CHF) – a multi-donor trust fund administered by the United Nations Office for the Coordination of Humanitarian Affairs (OCHA) and the United Nations Development Programme (UNDP). The 2011 contribution was £50 million. Other donors include the Netherlands, Norway, Sweden, Spain, Ireland, Denmark and Australia. CHF funds are programmed through a joint process under the leadership of the United Nations Humanitarian Coordinator and implemented by UN agencies and NGOs. In 2012, the CHF has 17 WASH projects for a total contribution of £5.1 million (all projects are also co-financed from other sources). Of these, 15 are in Darfur and two in the disputed border areas.
• DFID has also contributed £27 million to a World Bank-administered Multi-Donor Trust Fund (MDTF), which funded activities identified through a Joint Assessment Mission carried out jointly by the World Bank, the United Nations and the Sudanese authorities following the 2005 CPA. The MDTF will build or rehabilitate 451 water points before it is completed in the middle of 2013. It was designed to benefit half a million people, with a particular focus on improving water facilities for school-age children.

iii) WASH sector support via international NGO humanitarian projects

• A grant of £9.1 million to Medair between 2006 and 2012 for a project supporting the handover of basic health care and WASH services from humanitarian organisations to local authorities in West Darfur (now complete), of which approximately £3.6 million was spent on WASH.
• A grant of £6.7 million to Tearfund between 2007 and 2012 to meet emergency and early recovery needs of conflict-affected communities in Darfur (scheduled for completion in August 2012), of which approximately £3 million has been allocated to WASH.

2.14 Looking forward, DFID is currently developing a business case for a new water programme in eastern Sudan, which is planned for submission in September 2012. The initial budget is likely to be £20 million, with the aim of providing 500,000 people with access to clean drinking water over a three- to four-year period and developing plans to renovate Port Sudan water and sanitation systems. The implementing partners will be selected through a competitive process, with bids expected from UN agencies, NGOs and private contractors. DFID is also developing a business case for its support to humanitarian projects in Sudan over the next three years, which is planned for submission in September 2012.

3. Purpose of this review

3.1 To assess the effectiveness and value for money of DFID support for water, sanitation and hygiene programmes in Sudan.¹⁶

4. Relationship to other evaluations/studies

4.1 In March 2010, DFID’s Evaluation Department published a Country Programme Evaluation for Sudan covering the period 2005-08.¹⁷ The evaluation examined the operations of the CHF and the MDTF. It found that the CHF was slow and unpredictable and therefore was not used as initially envisaged for critical needs but rather for top-up funding of projects with access to more reliable support. Despite a complex allocation process, quality control of projects was found to be weak and monitoring and evaluation poor. The MDTF was found to have been slow to become operational and a lack of sufficient in-country management staff resulted in slow disbursement; it had, nonetheless, delivered some good results on community development, health and the conduct of the national census. Overall, the DFID Sudan programme was found to involve a proliferation of relatively small and inefficient aid instruments. The evaluation recommended rationalising and consolidating the delivery channels and instruments.

4.2 There have been no prior external evaluations of DFID Sudan’s WASH programmes but a number of other studies and reviews are relevant:

• the Overseas Development Institute (ODI) was commissioned to carry out a study¹⁸ of the context of DUWS investment in water supply to the town of Nyala, Darfur, which examined the impact of an expanded urban water network on the livelihoods of poor households and internally displaced persons (IDPs), particularly informal water sellers;
• ODI also carried out research on urbanisation and vulnerability in Sudan in 2010-11, looking at the effects of rapid urbanisation on infrastructure, basic services and livelihoods.¹⁹ The study found that the aid community is predominantly focussed on supporting rural populations and has not yet risen to the humanitarian and development challenges of the urban centres;
• UNEP carried out a post-conflict environmental assessment²⁰ in 2007, examining environmental risks, looking both at the impact of conflict on the environment (through population displacement, poor governance, conflict-related resource exploitation and under-investment in sustainable development) and at the impact of environmental issues on conflict dynamics (including competition over oil and gas reserves, Nile waters, timber and agricultural land); and
• a DFID Internal Audit visit to Sudan in February 2012 included a management audit of DUWS.

4.3 DFID is currently planning an external evaluation of SIEP (to be completed by the end of 2012) and is undertaking an external evaluation of its direct support to international NGOs (including Medair and Tearfund) for humanitarian projects. We will take this into account in planning our work.

5. Analytical approach

5.1 This evaluation will examine DFID WASH programming in Sudan over the past five years. While covering the full ICAI evaluation framework, the evaluation will focus in particular on two themes.

5.2 First, we will evaluate how well the WASH projects have been designed and implemented to deliver a sustainable impact for communities. This will include an assessment of whether:

• the projects are based on sound analysis of conflict dynamics and seek to anticipate and monitor risks of unintended consequences;
• the choice of delivery channel is based on an assessment of options and a sound evidence base;
• adequate measures have been taken to ensure longer-term sustainability of water supply both in terms of quantity (taking into account social and environmental projections and considering different water uses, e.g. agricultural irrigation) and quality (including managing pollution);
• intended beneficiaries and their representatives are adequately included in the design, implementation and monitoring and evaluation of the projects;
• the investments in WASH are owned by the local authorities, with the necessary technical and management capacity developed for future operations and maintenance; and
• there is good co-ordination amongst the different agencies running WASH projects in Darfur, including government agencies, private sector firms and local communities.

5.3 Second, the evaluation will examine the efficiency and value for money of the three main delivery channels used by DFID. We will examine the adequacy of financial management, procurement, fiduciary risk mitigation, project management and monitoring and evaluation undertaken by the implementing partners and their contractors. This will include ‘following the money’: assessing the efficiency with which UK aid funds reach the intended beneficiaries in Darfur.

5.4 By comparing the different delivery channels and partners used in the WASH portfolio and the strategies and methods they employ for managing the difficult operating environment, we will seek to draw lessons on effective programme delivery in conflict-affected environments. A detailed methodology for conducting this assessment will be developed during the inception phase.

5.5 As far as possible, we will also aim to assess the level of impact of the WASH programmes on the recovery and livelihood opportunities of conflict-affected communities. According to DFID Sudan, however, the pool of primary data on conditions in Darfur is very limited and we will have very limited capacity to supplement it through our own fieldwork due to security constraints. We will, nonetheless, compile and analyse the results data that are available, assess the strategies used by DFID to measure impact and meet with intended beneficiaries or, where that is not possible, representative groups to get their views on impact.

6. Indicative questions

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. The questions outlined below comprise those questions in our standard evaluation framework which are of particular interest in this review, as well as other pertinent questions we want to investigate. The full, finalised list of questions that we will consider in this review will be set out in the inception report.

6.2 Objectives

6.3 Delivery

6.4 Impact

6.5 Learning

7. Outline methodology

7.1 While the programmes to be examined in this evaluation are managed from Khartoum, the majority of their activities are located in the Darfur region. An assessment as to what level of access to Darfur is possible will be made based on security conditions at the time of the visit. The evaluation methodology must therefore be flexible enough to allow the balance between field work in Darfur and headquarters-level consultations in Khartoum to be adjusted as necessary.

7.2 The methodology will have two main components. First, choosing one project from each of the three different channels, we will assess how well the WASH projects have been designed and implemented to deliver a sustainable impact for communities. We will also make an assessment of impact, as far as the data allow. It will involve the following elements in Khartoum:

• a literature review on the causes and consequences of conflict in Sudan, in particular its effect on infrastructure, basic services and livelihoods;
• a review of project and programme documentation and reporting, monitoring results and any external reviews and evaluation; and
• interviews with DFID, implementing partners and their contractors, other donors, federal and state government officials and national and international NGOs.

7.3 The second component will involve a detailed examination of financial and project management arrangements in each of the three projects in the sample. The assessment will cover cost control and value for money, fiduciary risk management, procurement, project management processes and monitoring and evaluation. It will also involve assessing the efficiency with which UK funds reach the intended beneficiaries. During the inception phase, a detailed assessment framework will be developed for assessing the quality of management of UK funds by third parties. This element will be carried out in Khartoum.

7.4 A field study to be undertaken in Darfur will involve:

• a series of visits to project sites for technical assessment of ongoing and completed works and engagement with sub-contractors;
• interviews with local authorities to assess institutional and financial sustainability of results; and
• contact with a range of beneficiary community representatives and individuals.

7.5 The sample size for project visits will be determined during the inception phase, in consultation with DFID Sudan and the implementing partners. Given the logistical difficulties associated with travel in Darfur, a random sample is unlikely to be feasible. The sample, therefore, will cover a range of project types and implementing partners, taking into account the feasibility of travel. The sample will be large enough to enable flexibility in response to security conditions at the time.

7.6 In the event that travel to Darfur is not possible at all, alternative methods will be used to collect the data required for the evaluation. These may include reviewing photographs and other physical evidence of works, commissioning local partners or consultants able to travel in Darfur to collect data, telephone interviews with local officials and community representatives, inviting local officials or community representatives to meet the evaluation team at a third location or identifying Khartoum-based individuals with personal knowledge of the projects and communities.

8. Timing and deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The country visit is scheduled for September 2012, with a final report available in the first quarter of 2013.

Footnotes

1. ↑ This is an evaluation of DFID WASH programming in Sudan and does not cover WASH programming undertaken by DFID in South Sudan, which is now a separate country programme.
2. ↑ Operational Plan 2011-2015, DFID Sudan, May 2012, page 14, http://www.dfid.gov.uk/documents/publications1/op/sudan-2011.pdf.
3. ↑ Operational Plan 2011-2015, DFID Sudan, May 2012, page 2, http://www.dfid.gov.uk/documents/publications1/op/sudan-2011.pdf.
4. ↑ In 2009, the national poverty line was equivalent to 114 Sudanese pounds, then around £30 per month:Sudan National Baseline Household Survey 2009: North Sudan – Tabulation Report, Central Bureau of Statistics, http://ecastats.uneca.org/aicmd/Portals/1/Publications/Others/POVERTY%202009.pdf.
5. ↑ Development Initiatives, Sudan Aid Factsheet 1995-2009: Trends in Overseas Development Assistance, http://www.globalhumanitarianassistance.org/wp-content/uploads/2011/02/Sudan-Aid-Factsheet-2011.pdf.
6. ↑ Operational Plan 2011-2015, DFID Sudan, May 2012, http://www.dfid.gov.uk/documents/publications1/op/sudan-2011.pdf.
7. ↑ Water, Sanitation and Hygiene Portfolio Review, DFID, 2012, http://www.dfid.gov.uk/Documents/DFID%20WASH%20Portfolio%20Review.pdf.
8. ↑ ‘Improved water source’ is defined by the UN as any of the following: piped water, public tap, borehole or pump, protected well, protected spring or rainwater. It does not include vendor-provided water, bottled water, tanker trucks or unprotected wells and springs. ‘Improved sanitation’ means facilities that hygienically separate human excreta from human, animal and insect contact, including sewers, septic tanks, poor-flush latrines and simple pit or ventilated improved pit latrines, where correctly constructed and properly maintained. Indicators for Monitoring the Millennium Development Goals: Definitions, Rationale, Concepts and Sources, United Nations, 2003, pages 64 and 66, http://mdgs.un.org/unsd/mdg/Resources/Attach/Indicators/HandbookEnglish.pdf.
9. ↑ Data provided by DFID.
10. ↑ L. Attree, China and Conflict-Affected States, between Principles and Pragmatism, Saferworld, January 2012, http://www.saferworld.org.uk/downloads/pubdocs/China%20and%20conflict-affected%20states.pdf.
11. ↑ Water, Sanitation and Hygiene Portfolio Review, DFID, March 2012, page 38, http://www.dfid.gov.uk/Documents/DFID%20WASH%20Portfolio%20Review.pdf.
12. ↑ Voices of Darfur, United Nations Mission in Darfur, June 2011, http://unamid.unmissions.org/Portals/UNAMID/Voices/vod_june_en_web.pdf.
13. ↑ See http://www.darfurwaterforpeace.org/.
14. ↑ UNOPS is a service agency within the UN system that provides support in procurement, contract management and civil works to other organisations in the humanitarian and peacekeeping areas, including other UN agencies, donor and recipient governments, NGOs and the private sector: http://www.unops.org/english/whoweare/Pages/Mandate.aspx.
15. ↑ Information provided by DFID Sudan.
16. ↑ This does not include programmes in South Sudan.
17. ↑ Evaluation of DFID Country Programme: Sudan 2005-2008, DFID, March 2010, http://www.dfid.gov.uk/Documents/publications1/evaluation/evaluation-summary-sudan708.pdf.
18. ↑ Pipelines and Donkey Carts: A Social Risk Analysis of Water Availability, Access and Use in Nyala, South Darfur, ODI, 2012, www.odi.org.uk/resources/docs/7634.pdf.
19. ↑ City Limits: Urbanisation and Vulnerability in Sudan, ODI, January 2011, http://www.odi.org.uk/resources/docs/6511.pdf.
20. ↑ Sudan Post-Conflict Environmental Assessment, UNEP, June 2007, http://postconflict.unep.ch/publications/UNEP_Sudan_synthesis_E.pdf.