Evaluation of DFID's livelihoods work in Western Odisha - Independent Commission for Aid Impact

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We wish to undertake a post-implementation evaluation of the DFID-funded Western Orissa Livelihoods Project (WORLP), which sought to improve the livelihoods of 124,692 households in the State of Odisha, India.¹ These Terms of Reference outline the purpose and nature of the review and identify the main themes that it will investigate. The review is intended to build on the previous evaluations. A detailed methodology will be developed during an inception phase.

2. Background

2.1. India’s 1.16 billion people and the 28 States and 7 Union territories they live in are diverse. Whilst overall income per person remains low at US$1,389 per year (around one-thirtieth of that for the UK), the country is in the process of rapid economic and social development. India has seen economic growth rates in excess of 8% for much of the last decade and the percentage of the population living in extreme poverty fell from 60% in 1981 to 42% in 2005. Yet, although overall wealth has increased, huge inequalities remain: 456 million people (equivalent to the combined populations of Russia, Germany, Turkey, France, the UK and Poland) still lived on less than US$1.25 a day in 2005.² There are also considerable regional inequalities. At current rates, India will only achieve its Millennium Development Goal (MDG) poverty targets by 2043. One fifth of all child deaths in the world are of Indian children. As Save the Children has noted, ‘if India fails to achieve the MDGs so does the world’.³

2.2 In the past, DFID has operated under an overall Country Strategy and then, for each of its partner states, had a State Strategy. It is increasingly focussing work at the state level. DFID’s current priority states are Bihar, Madhya Pradesh and Odisha.

2.3 Odisha’s 42 million people live in 30 districts and comprise 3.47% of the overall population of India. The State’s population remains under considerable pressure: while in the decade prior to 2011, Odisha’s GDP grew by 15%, the population grew by 14%. The Census of India reports that overall poverty only dropped by 2% to 46.6% (among the highest in India) over that period. 83% of the population live in rural areas.⁴ The state is governed from Bubhaneswar.

2.4 WORLP began in August 2000 and was completed in March 2011. The two-phase project focussed on the poorest tribal districts. It was implemented through a state-level body, the Orissa Watershed Development Mission, with DFID providing funding and advice. The project was planned to be wide-ranging and sought to reduce poverty in 870 villages (this figure reportedly was exceeded) by building sustainable livelihoods. DFID reports that £31.68 million was spent out of a budget of £32.75 million (96% of the total).⁵

2.5 In 1999 DFID defined a livelihood as: ‘…the capabilities, assets (including both material and social resources) for a means of living. A livelihood is sustainable when it can cope with and recover from stresses and shocks and maintain or enhance its capabilities and assets both now and in the future, while not undermining the natural resource base.’⁶

2.6 The project sought to build up the assets of (particularly poor) households in Western Odisha through a variety of methods such as developing community-based planning and resource management, increasing employment, providing access to finance, improving agricultural production methods, improving access to drinking water and reducing morbidity. The project worked at the household, village and watershed⁷ levels, through the Government of Odisha’s District and Block structure.⁸ The approach was to be as integrated as possible, dealing with complete communities at a time. The project was explicitly aimed to demonstrate approaches that could be replicated.

2.7 Western Odisha’s population is particularly vulnerable to climate shocks (rainfall is highly variable, drought and dry spells take place every two years, with a major drought every five to six years, flash floods are common in the rainy season). It is notable that the programme ‘…was not designed with any climate change objectives or indicators to measure this, and indeed no major environmental impact was envisaged other than that which might be expected through the enhancement of natural resource assets’.⁹

3. Purpose of the Evaluation

3.1 To assess the impact and long-term sustainability of DFID’s Western Orissa Livelihoods Project.

4. Relationship to other evaluations and studies

4.1 WORLP was subject to an independent impact evaluation during 2011 that was presented to both DFID and the Orissa Watershed Development Mission.¹⁰ In addition, the Planning Commission of the Government of India commissioned an impact assessment of WORLP alongside two other projects that had operated at the same time: the Orissa Tribal Empowerment and Livelihoods Programme (OTELP) and the Targeted Rural Initiatives for Poverty Termination and Infrastructure.¹¹ Both of these evaluations used quasi-experimental approaches to impact assessment. They undertook comprehensive sampling and comparison of project impact in villages with controls; the former using a sample size of 300 villages (100 of which were controls), the latter using 40 villages in total. Both evaluations took place during 2011, at the end of the project’s life.

4.2 Findings from both studies were positive, showing villages had increased access to physical and natural assets, as well as more effective community institutions, better health care and incomes. Resilience was also seen to increase and the dedicated WORLP impact study noted an economic rate of return of 25.44%.¹²

4.3 The focus of our evaluation will be to validate these studies’ findings, both in terms of the approaches taken and in assessing whether the observed benefits have proved sustainable. We will also seek to identify whether the lessons from these impact assessments and other evaluations have been used to influence other activities and evaluations in Odisha, across India and in other parts of DFID’s global programme.

5. Analytical approach

5.1 Our evaluation will, therefore, focus on three areas:

review of the evidence of impact of the project gathered to date;
assessment of the sustainability of the project’s outcomes; and
assessment of the impact of the lessons learnt.

5.2 The approach of this study will be to prioritise effort on the impact of the programme. While not seeking to undertake primary research, it will test the quality of DFID’s own commissioning, management and utilisation of impact assessments. It will specifically test how well such data has subsequently been used by DFID.

6. Indicative evaluation framework

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. The questions outlined below are based on those questions in our standard evaluation framework which are of particular interest in this review. The full, finalised list of questions that we will consider in this review will be set out in the inception report once a decision is made on the programmes to be covered.

6.2 Objectives

6.2.1 Did the WORLP have clear, relevant and realistic objectives that focus on the desired impact?

6.2.2 Did the programme complement the efforts of government and other aid providers and avoid duplication?

6.2.3 Were the programme’s objectives appropriate to the political, economic, social and environmental context?

6.3 Delivery

6.3.1 Was the choice of funding and delivery options appropriate?

6.3.2 What was the evidence base for DFID’s decisions and cost analysis?

6.3.3 Did programme design and roll-out involve and take into account the needs of the intended beneficiaries?

6.3.4 Did managers ensure the efficiency and effectiveness of the delivery chain?

6.3.5 Were risks to the achievement of the objectives identified and managed effectively?

6.3.6 Did the programme deliver against its agreed objectives?

6.3.7 Were appropriate amendments to objectives made to take account of changing circumstances?

6.4 Impact

6.4.1 Did the programme deliver clear, significant and timely benefits for the intended beneficiaries?

6.4.2 Did the programme work holistically alongside other programmes?

6.4.3 Was there a long-term and sustainable impact from the programme?

6.4.4 Was there an appropriate exit strategy involving effective transfer of ownership of the programme?

6.4.5 Was there transparency and accountability to intended beneficiaries, donors and UK taxpayers?

6.5 Learning

6.5.1 Was there evidence of innovation and use of global best practice?

6.5.2 Was there anything not done in respect of the programme that should have been undertaken?

6.5.3 Have lessons about the design and delivery of the programme been learned and shared effectively (in Odisha, in India and across DFID)?

7. Methodology

7.1 The methodology for this review will be developed during the inception phase. It will involve a number of elements, including:

a literature review, focussing on project reports and evaluations and evaluations of other DFID sustainable livelihoods programmes in India¹³ and similar programmes supported by other development agencies (e.g. World Bank);
face-to-face and telephone interviews with DFID staff and consultants, who worked on the design, implementation, monitoring and evaluation of WORLP;
field visits in project districts to review the findings and conclusions of the WORLP Impact Assessment completed in early 2011. The visits will be conducted by a team comprising the rural livelihoods consultant on the team, supported by three national consultants. The field team will visit a random sample of 20-30 villages covered by the Impact Assessment, including both WORLP villages and control villages not included in the project. The team will conduct focus group discussions and interviews with key individuals at each location;
analysis of the financial reports of partner organisations and the overall project to identify costs and the proportion of allocated funds reaching beneficiaries;
a workshop in Odisha, with 20-30 key WORLP stakeholders (e.g. partner NGOs, local consultants, government officials) to reflect on the achievements and impact of the project and draw key lessons;
discussions with senior government officials in Bhubaneswar and Delhi (e.g. Planning Commission, key ministries), DFID staff (in London and Delhi) and key researchers and policy advisors (in Delhi); and
analysis of the design of DFID’s portfolio of livelihoods projects to identify the wider impact of the WORLP (if any).

8. Timing and Deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The review will take place during the third and fourth quarters of 2012.

Footnotes

1. Odisha state was until recently called Orissa.

2. The developing world is poorer than we thought, but no less successful in the fight against poverty, World Bank Development Research Group, Policy Research Working Paper 4703, August 2008, http://www-wds.worldbank.org/external/default/WDSContentServer/IW3P/IB/2010/01/21/000158349_20100121133109/Rendered/PDF/WPS4703.pdf.

3. Written evidence submitted by Save the Children to the International Development Committee report ‘The Future of DFID’s Programme in India’, March 2011, www.publications.parliament.uk/pa/cm201011/cmselect/cmintdev/writev/616/m15.htm.

4. Dr C Chandramouli, Rural Urban Distribution of Population, July 2011, http://censusindia.gov.in/2011-prov-results/paper2/data_files/india/Rural_Urban_2011.pdf

5. Project Details: WORLP, DFID, http://projects.dfid.gov.uk/project.aspx?Project=107874.

6. Sustainable Livelihoods Guidance Sheets, Numbers 1–8, DFID (1999a, 2000d, 2001). Department for International Development (also available on www.livelihoods.org).

7. A watershed is the area that drains to a common outlet (such as a river or into a lake). It is often seen as the basic building block for land and water planning.

8. India’s States are subdivided into Districts (Odisha has 30 Districts) which are then further divided into Sub-Divisions and then Blocks (represented by Panchayats, or councils). There are 314 blocks in Odisha.

9. Policy Brief Sustainable Livelihoods and Climate Change Adaptation in WORLP, DFID.

10. Sambodhi Research and Communications Ltd (for DFID), Impact Assessment of WORLP, 2011, Summary of findings available at: http://community.eldis.org/.59cc6781/Files/.

11. Center for Rural Development for Government of India Planning Commission, Impact Assessment of Externally Aided Project Interventions on Livelihood of the poor and Marginalized in KBK Districts of Orissa, 2011, http://planningcommission.nic.in/reports/sereport/ser/ser_2901.pdf.

12. Sambodhi Research and Communications Ltd (for DFID), Impact Assessment of WORLP, 2011, Summary of findings available at: http://community.eldis.org/.59cc6781/Files/.

13. For example, the Western and Eastern India Rainfed Farming Projects (WIRFP and EIRFP), the Andhra Pradesh Rural Livelihood Programme (APRLP) and Madhya Pradesh Rural Livelihoods Programme (MPRLP).

Evaluation of DFID’s livelihoods work in Western Odisha