ICAI-Afghanistan-growth-and-livelihoods-inception-report-FINAL

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We have decided to undertake a review of selected growth and livelihoods projects in DFID’s Afghanistan programme to assess the impact it delivers for intended beneficiaries and the extent to which it has made preparations to continue operating projects under likely future scenarios. Perhaps the most important challenge DFID faces in Afghanistan is its operational context as security control is fully handed over from the International Security Assistance Force (ISAF) to the Afghan National Security Forces (ANSF) by the end of 2014. We will examine how DFID manages its growth and livelihoods projects in this difficult environment and how it is responding to the current and future needs of the Afghan people.

1.3 This Inception Report sets out the questions, methodology and work plan for the review. The methodology and work plan are intended to be flexible enough to allow for new issues and questions that emerge over the course of the review.

2. Background

2.1 The background to this review, including the wider context within which DFID’s growth and livelihoods work in Afghanistan is implemented, is described in the Terms of Reference.¹

3. Purpose of this review

3.1 To assess the impact of DFID’s growth and livelihoods projects on intended beneficiaries and examine how DFID identifies and responds to their changing needs during and after the drawdown of international troops.

4. Relationship to other reviews

4.1 The relationship to other reviews of DFID’s growth and livelihoods work in Afghanistan is described in the Terms of Reference.

5. Methodology

5.1 This section sets out our intended approach, the selected focus projects and our rationale for their selection, the process for undertaking the review and the assessment framework to be used.

Approach

5.2 This review has two approaches: an overall strategic review and a more detailed assessment of case study projects:

• At the strategic level: we will assess how DFID is prepared to deliver sustainable impact throughout and beyond the period of drawdown of international troops in 2014 and how DFID has made preparations to provide a sustainable impact for its growth and livelihoods projects under likely future scenarios. We will assess how the evidence of the current context and challenges, learning from previous projects in Afghanistan and other relevant challenging locations, have been used by DFID in designing its portfolio of growth and livelihoods projects in Afghanistan. We will take particular interest in how recently started and currently planned projects have been designed for potential post-2014 futures. Projects we will look at in this context might include Strengthening the Agriculture Sector in Afghanistan (SASA), Road Rehabilitation and Maintenance Programme and Support to Demining (phase II); and
• At the project level: we will select a sample of project case studies (see Figure 1 on page 5) to review at each stage of the project lifecycle, from setting objectives to capturing lessons from project delivery and impact. By comparing the different delivery channels and partners used in DFID’s growth and livelihoods portfolio and the strategies and methods employed for managing in a difficult context, we will seek to draw lessons on effective project delivery in conflict-affected environments. We are particularly interested in assessing how, through these case study projects, DFID is achieving impact for intended beneficiaries and to what extent this impact is likely to be sustained – and indeed the extent to which intended beneficiary ownership is one of the ways that sustainable impact in the new environment is being ensured.

5.3 There is limited availability of reliable results data for DFID’s projects in Afghanistan (see Section 9, Risks and Mitigation). We will therefore, apply a variety of techniques to measure progress against indicators, to assess impact. The appropriate approach will vary by project and stage of the review.

5.4 We intend to understand the impact of DFID’s bilateral growth and livelihoods programmes on intended beneficiaries by reviewing a number of selected case studies. A livelihood comprises ‘the capabilities, assets (including both material and social resources) and activities required for a means of living. A livelihood is sustainable when it can cope with and recover from stresses and shocks, maintain or enhance its capabilities and assets, while not undermining the natural resource base’.²

5.5 We will review documentation and existing reviews (if they exist) using a desk-based research approach supported by in-depth interviews with DFID staff, delivery partners, beneficiaries and/or their representatives (where possible) and other stakeholders and subject experts.

5.6 For three projects – in addition to the steps stated in paragraph 5.5 – we will conduct qualitative survey fieldwork with intended beneficiaries. This additional fieldwork will be carried out by a local Afghan research partner (see paragraphs 5.10-5.11).

5.7 For all selected case study projects we will attempt to:

• Establish a baseline and the current status of relevant indicators: by reviewing existing surveys or, for example, through recall questions in our interviews with intended beneficiaries;
• Identify and assess major intervening variables: factors that are likely to have significantly interacted, positively or negatively, with project activities. An example of a negative intervening factor might be a drought or military activity that forces parts of the intended beneficiary population to leave a region. Such factors might eliminate the impact of an otherwise well-designed and well-implemented project. Positive intervening variables might be projects with a similar focus (either by DFID or another donor) that enhance the impact of our case study project. Properly identifying intervening variables is essential to making any statements about attribution of observed change to a certain project. We will use interviews, literature review and assessment of project documentation to identify possible intervening variables;
• Strengthen our claim regarding the attribution of project impact using comparisons: if we can identify two very similar villages or regions, one of which has benefitted from a selected case study project while the other has not, observed changes between the two locations are likely to be the result of the project we investigate; and
• Trace the exact causal mechanisms that were triggered by the reviewed projects or project components (known as process tracing): given the weakness of existing data and the qualitative emphasis of our review, such process tracing, based upon interview findings and analysis of existing data, is necessary to support any claims of causation.

Selected projects

5.8 Five case study projects have been selected from DFID’s portfolio of bilateral growth and livelihoods projects (see Figure 1 on page 5). This ensures we can review fully each case study project and that there is a fair balance against the criteria set out in the Terms of Reference.³ The selected case study projects constitute a good proportion of DFID’s total and bilateral portfolio. Their aggregate expenditure budgets total £97.75 million, an annual average of £24.9 million. This amounts to almost 20% of DFID’s total planned programme expenditure and 30% of annual bilateral programme expenditure.⁴

5.9 The fieldwork will also provide first-hand feedback from intended beneficiaries. Three projects have been selected for a more detailed review using additional surveys. We selected these using the following criteria:

• The project has not been recently evaluated: a re-assessment of a recently evaluated project is not likely to reveal much additional information on the project’s impact on intended beneficiaries; and is therefore not a sensible use of resources;⁵
• The project is either ongoing, in its final stages or has been completed in 2012 or later: we exclude projects with an older completion date because intended beneficiaries might not be able to remember the project inputs and would have even greater difficulties in recalling the baseline situation; and
• The project takes place in a clearly identifiable (and accessible) geographic location, with clearly identifiable intended beneficiaries or their communities.

5.10 We will conduct the additional fieldwork by commissioning a local research organisation to assist us. This local team is able to operate in the difficult and changing security environment, mitigate cultural constraints (for example, interviewing women) and have the additional resources we need to cover the size of the projects selected. To illustrate this point, the problematic security situation in Helmand does not allow international consultants to leave the provincial capital Lashkar Gah to visit local villages where most intended beneficiaries live.

5.11 The research organisation is a non-governmental, non-political and non-profit organisation led by Afghans with the aims of working in the field of survey, monitoring and evaluation, disaster risk management, sustainable development and agricultural capacity building. It has extensive experience in conducting quantitative and qualitative research throughout Afghanistan and has also repeatedly worked in Helmand, where its researchers can access large parts (though not all) of the province. See paragraphs 5.16-5.18 for a detailed description of the approach we will take for this intended beneficiary consultation.

Review process

5.12 The process comprises two parts, a pre-visit assessment and field research.

Pre-visit assessment

5.13 This first phase of the review has four components:

• Literature review: we will investigate the theories of change or similar project design methodologies on which our selected projects are based to assess the extent to which they are supported by evidence. We will draw from literature on comparable fragile and post-conflict environments. This, along with findings from interviews with subject experts from the UK and elsewhere, will be used to inform our fieldwork;
• Review of programme management documentation: for our strategic review we will examine DFID’s programme documentation, evaluations, reviews etc and investigate work on scenario planning in detail. In doing so we will assess the extent to which scenarios are informed by available evidence (e.g. expert interviews, recent research, trend analysis, risk analysis, understanding opportunities and threats) and reviewed regularly. We will also interview DFID staff to understand what processes exist for scenario work to be incorporated into existing and new projects. In addition, we are interested in looking at evidence as to whether existing projects are adapting to changing circumstances.

  For our project review, we will review DFID’s project documentation, including monitoring and other routine inputs from delivery partners. From these documents we will establish the extent of strategic and options appraisal, commercial and financial cases and identified risks to the successful delivery of each project. Within our assessment of risks we will be interested to update our progress assessment of DFID’s steps taken to avoid corruption in Afghanistan; to examine how mitigating actions relate to best practice and to ICAI’s earlier recommendations to DFID on this topic.⁸ We will consider how the full range of monitoring and evaluation activity, which is both undertaken and available to DFID Afghanistan, is used to inform project planning. We will examine project plans to identify chosen indicators of progress, the availability of baseline data, milestones and targets as well as any assumptions that underpin these plans. These will then be compared to actual outputs and measured outcomes in order to assess progress against targets, project effectiveness, sustainability and impact;

• Consultations with peers and stakeholders: we will identify a range of experts in growth and livelihoods and related poverty impacts to discuss methodological approaches to the design and implementation of livelihoods programmes, with particular emphasis on ensuring impact on the poor. These will include meetings and telephone calls with individuals in academia and think tanks, as well as with research organisations in Afghanistan. These will include: DFID Livelihoods Head of Profession, the Overseas Development Institute (ODI), the Afghanistan Research and Evaluation Unit (AREU), the Institute of Development Studies (IDS) and other institutions where a relevant body of knowledge about pro-poor policy and practice in Afghanistan has been accumulated. From these discussions, we will gather evidence about the latest thinking on growth and livelihoods impacts and poverty linkages; and
• Preliminary consultations: in order to maximise the benefit of meetings during our visit to Afghanistan, initial consultations with DFID’s country office and delivery partners will take place via telephone or video conference to develop early themes for investigation following the review of management documentation. Time in country can then be used efficiently to target important issues.

Field research

5.14 We will conduct field research in Afghanistan to assess the impact of DFID’s projects on intended beneficiaries and develop an understanding of how its work translates into improved livelihoods for the Afghan people. The field research is made up of two components: (a) interviews by the ICAI team with DFID staff, delivery partners, officials of the Government of Afghanistan, politicians (where appropriate), intended beneficiaries and their representatives, donors and NGOs; and (b) interviews with intended beneficiaries, carried out by a local research organisation who can visit areas inaccessible to the ICAI team. Field research by this organisation will begin prior to the arrival of the core ICAI review team.

(a) Field research by the ICAI team

5.15 The review team will visit DFID-funded projects in Afghanistan to assess directly the level of impact of the programmes on the livelihood opportunities of conflict-affected communities. Security and practical considerations permitting, we will conduct a mixture of announced and unannounced visits to collect a range of evidence and hold meetings with target groups in workshops. We will arrange focus groups with intended beneficiaries at each pre-arranged site visit. Interviews and field visits will be closely coordinated with DFID’s security management. Interviewees are likely to include:

• DFID programme staff in country, including Helmand programme staff;
• delivery partners engaged in project delivery;
• representatives of the Government of Afghanistan, local leaders and dignitaries;
• civil society organisations (CSOs), including agricultural intermediaries and farmer-based organisations; and
• intended beneficiaries and their representatives in project areas, for example, Community Development Councils (CDCs), District Development Assemblies.

(b) Field research by a local research organisation

5.16 To gather additional evidence of programme impact, we will arrange a survey of intended beneficiaries of three projects: Support to Demining (Herat), Road Improvement in Central Helmand and the Helmand Growth Programme. Given the need for flexibility and timely outputs, the research organisation will adopt qualitative guideline interviews as the main research methodology. This approach will provide us with in-depth information from a sample of intended beneficiaries, whom the core ICAI review team would otherwise not be able to reach. The qualitative interviews will provide us with reliable information on how intended beneficiaries experienced the selected DFID projects and how this impact relates to the intended impact of the projects. It should also allow us to identify possibly unintended positive or negative consequences of the selected DFID projects.

5.17 The local research organisation will conduct up to 160 qualitative interviews (of 60-90 minutes each in duration) and approximately 30 focus groups (with an average of five people in each group, for approximately 90 minutes each). Up to 310 intended beneficiaries will therefore be interviewed, with each meeting transcribed and translated. We will use six teams (two in Herat province and four in Helmand). Two of these teams will be led by a woman in order to specifically interview female intended beneficiaries.⁹

5.18 Current security restrictions prevent the core ICAI team from visiting Herat. Where possible, implementing agents and other project stakeholders will travel to Kabul for interview by the ICAI core team. In addition, the local research organisation will provide one senior qualitative interviewer, as a proxy for the ICAI core team, to carry out any remaining interviews in Herat. At this stage, we expect these to be with representatives of the Government of Afghanistan and of intended beneficiaries. The findings from these interviews will be relayed to the core team during their time in-country, to be followed up where necessary in Kabul.

5.19 The local organisation’s research involves the following steps:

• establishment of agreed terms of reference and budget;
• preparation of a training programme: we will prepare a training programme for survey staff and, in consultation with them, will establish the most effective survey approach;
• design of qualitative questionnaires: we will design and test questionnaires to be used to measure and establish impact on intended beneficiaries, taking into account social, logistical and security conditions on the ground at the time of the surveys;
• determination of project locations and intended beneficiaries: as a first step we will use project documentation and intelligence gained from DFID project managers and each project’s delivery partners. In addition, we will consult Community Development Councils (CDCs), the task of which is to facilitate any project implementation and so they are familiar with which individuals benefitted from each project in their community.¹⁰ Given the volatile security context and time constraints, security and logistic factors will also have to be considered when selecting project sites and target respondents. In other words, sites and target respondents might have to be dropped if they are located in highly insecure or inaccessible areas – even if, from a point of view of representativeness, we would like to include them in our survey;
• selection of target respondents: the selection of target respondents must be specific to the surveyed project and/or project component. As a general rule with our respondent selection, we will try to depict a project’s or project component’s intended beneficiary profile as best as we can. Respondent selection will be criteria based, i.e. we will try to select a sample of intended beneficiaries that best reflects the profile of total intended beneficiary communities.¹¹ We will consider, among other criteria, geographic and, if available, demographic (including gender if appropriate) and socio-economic factors. We will also attempt to interview beneficiary representatives (such as members of CDCs and District Development Assemblies) to understand their views on the projects and their implementation. If appropriate and if possible, we will also try interview individuals, who have not participated in a project to get their understanding of the selected projects and to spot possible biased selection of intended beneficiaries;
• advance training: one of the ICAI team will visit Kabul nine days in advance of the rest of the review team to train the research partner’s survey staff in ICAI policy, refresh qualitative interview techniques required for the research, communicate the data needs of the review, sampling methods and selection of target respondents. We will then pilot the questionnaire instrument and interview guidelines during the initial days of fieldwork and make adjustments if necessary;
• liaison with and remote supervision of the local research organisation throughout the rest of the fieldwork (in particular during its work in Helmand);
• debriefing of local researchers by the ICAI team in Kabul: the debriefing should ideally take place when the ICAI team arrives in Kabul to begin its research. Security or logistical difficulties, however, might delay researchers in completing their work in time for the arrival of the core ICAI review team. In this case the de-briefing would be held towards the end of the main Afghanistan visit; and
• translation of interviews and follow-up.

Assessment Framework

5.20 The assessment framework for this review is set out in the table below. This is based on the standard ICAI guiding criteria and assessment framework, which focus on four areas: objectives, delivery, impact and learning. It also incorporates other pertinent questions we want to investigate in this review. Our focus is on the questions highlighted in bold.

Note: The detailed assessment framework table has been omitted for brevity.

6. Roles and responsibilities

6.1 KPMG provides oversight of this review under the overall leadership of the ICAI Project Director.

6.2 This review will be undertaken by a core team of four. Research partners in Afghanistan will be responsible for local surveys and identification of intended beneficiaries. All will contribute to the analysis supporting the findings for each section.

Team Leader (Independent)

He has a solid academic grounding with degrees in economics and political economy and has worked on public policy issues for over ten years. He led projects for six years in the Prime Minister’s Strategy Unit, providing high quality and objective research and policy analysis in crime, housing, transport, climate change and family policy. He worked with the Government of South Sudan in 2007 leading a detailed evaluation of progress since the cease-fire; and was Country Head of the Africa Governance Initiative in South Sudan setting up a long-term governance capacity building project in the Office of the President. He spent two years as Head of Strategy for the Department for Transport advising ministers and the board on long-term policy options across aviation, shipping, rail, road and public transport; and developing a highly respected economic analysis team.

He will use his experience in government to interact effectively with and listen to the needs of stakeholders, undertake analysis and give a clear lead on evaluation issues in a difficult environment.

Team Member (Independent)

He is a specialist in private sector development, capacity building and evaluation with over 30 years’ general management and consultancy experience in the UK private sector and DFID advisory experience. He has international consultancy and project management experience in Afghanistan, Southern and East Africa, Asia and the Caribbean.

He will use his international development, project evaluation and Afghanistan experience to provide direct input to the team to help support planning and delivery.

Team Member (Independent)

He is a specialist in evaluation and social and economic survey research, analysis and reporting in conflict environments. He has developed a conflict sensitive strategy for the post-2014 period in Afghanistan for GIZ and has designed project evaluations with conflict sensitive approaches in conflict and fragile states.

He will use his international development evaluation and Afghanistan survey experience to provide direct input to the surveys, managing the survey partner and supporting project planning and delivery.

Team Member (Independent)

He is educated to Master’s level in Civil Engineering and Economics, with particular knowledge of economic growth (including the role of infrastructure development and the damaging effects of political and violent conflict), the evaluation of poverty reduction strategies and the role of institutions in supporting economic growth and livelihoods. He is familiar with ICAI, having worked on the recent ICAI reviews of DFID’s Trade Development Work in Southern Africa¹² and of DFID’s Support to Palestine Refugees through UNRWA.¹³

He will support the team in their methodological design and analysis and will serve as a researcher for the review period.

Peer Reviewer (KPMG)

He is a member of KPMG’s International Development Services team. He is a development economist with extensive experience in the world’s poorest countries, gained with DFID and with consultancies including PwC, Atos and Coffey International. He worked as an economic adviser with DFID in India, Iraq, Pakistan and Southern Africa. He has also worked on projects in Afghanistan and Bangladesh.

He has led the design and management of complex public sector reform programmes in difficult and conflict affected environments, developed economic analysis to inform government decisions on aid allocation and appraised aid interventions for effectiveness and value for money. He is a specialist in Asia and in conflict affected countries. In Pakistan, he has taken part in a number of projects; he recently led a team designing a sub-national governance programme, he also led a team reviewing a conflict-related multi-donor trust fund and led agreement on the final stages of the UK’s budget support to the Government. His most recent work in Bangladesh involved developing business cases for key sector support programmes in health and education. In Afghanistan, he led the analysis of value for money of support to the Ministry of Interior.

7. Management and reporting

7.1 A first draft report for review by the ICAI Secretariat and Commissioners will be produced for 21 October 2013, with time for subsequent revision and review prior to completion and sign off in January 2014.

8. Expected outputs and time frame

8.1 The following timetable is based on the assumption that the report will need to be finalised in January 2014 to meet ICAI requirements.

8.2 The following survey fieldwork timetable is based on probable dates of completion but is subject to final planning and approval by KPMG and DFID.

9. Risks and mitigation

9.1 The following table sets out risks and mitigating actions for this evaluation.

10. How this ICAI review will make a difference

10.1 The UK Government has committed to long-term support to the Government of Afghanistan through the Enduring Strategic Partnership agreement. DFID Afghanistan has committed to a high level of spending (£714.4 million over the planning period 2011-15) on humanitarian and development assistance, of which £340.4 million is allocated to wealth creation (which contains growth and livelihoods projects). This review will offer independent analysis and conclusions that will assist DFID with feedback on its strategy for ‘a more peaceful, stable, viable and prosperous Afghanistan’.¹⁴ It is intended that this review should be of value to other development assistance stakeholders in Afghanistan and in conflict states more generally.

10.2 Assessing DFID’s preparedness for the transition to 2014 and the post-2014 period is an explicit objective of this review. DFID’s humanitarian and development assistance programmes will require progressive adjustment to cope with the changing needs of the Afghan people as the handover of security from ISAF to full Afghan control gathers pace and the potential for large-scale disruption of economic life accompanied by displacement of rural communities becomes more probable. DFID’s preparedness for such adjustment and the sustainability of its programmes are a matter of vital importance to DFID’s partners in government, the donor community, NGOs and the private sector. The review can add value by providing an independent assessment of DFID’s project sustainability and adaptability to future scenarios and by making recommendations for the design of new projects and adjustment of existing ones, to respond more effectively to changing circumstances.

Footnotes

1. Terms of Reference: DFID’s Bilateral Support to Growth and Livelihoods in Afghanistan, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Afghanistan-ToR-final.pdf.
2. Scoones, I., Livelihoods Perspectives and Rural Development, Journal of Peasant Studies, Vol 36, No 1, 2009, http://community.eldis.org/?233@@/.59b9a649/15!enclosure=.59c20af7&ad=1.
3. Visiting projects where aid is provided to intended beneficiaries presents a number of logistical difficulties as the point of delivery is often in remote, conflict-affected areas, therefore we will also select two reserve projects in secure areas for close assessment should the first choices prove to be unsuitable for fieldwork.
4. Proportions are calculated from DFID Operational Plan 2011-2015 and project documents.
5. If available evaluations give detailed consideration to impact on intended beneficiaries, we will re-assess and review this material, rather than gather further qualitative evidence.
6. Other projects and project components will be examined where they provide insights into how DFID’s learning on a theme/sector feeds into objective setting and the delivery of more recent activities.
7. Increased income opportunities include (i) increased sales by a programme’s intended beneficiaries, such as micro, small and medium enterprises, Enterprise Groups and Savings Groups; and (ii) increased access for consumers to low cost products/services.
8. In 2011, ICAI reported on The Department for International Development’s Approach to Anti-Corruption, in which it highlighted the risk to DFID of increasing its aid disbursements to fragile and conflict-affected states. This report recommended that DFID develop explicit anti-corruption strategies in countries assessed as having a high risk of corruption. DFID responded positively to ICAI’s recommendations. See: The Department for International Development’s Approach to Anti-Corruption, ICAI, 2011, http://icai.independent.gov.uk/wp-content/uploads/2010/11/DFIDs-Approach-to-Anti-Corruption.pdf.
9. It should be noted that, at this stage (without full access to DFID’s project documentation), the actual number of interviews might vary depending on the details contained within DFID’s documentation. These details will determine the precise tasking of the teams and locations and also on the amount of monitoring required from the researchers. The local security situation – and the possibility to access selected research areas – will also determine the geographical distribution of intended beneficiary consultations.
10. CDCs represent communities and are therefore familiar with contact from outsiders. Any research team arriving at a village, wishing to conduct fieldwork, must first ask permission of the local CDC representative.
11. Criteria-based selection is common if it is possible to conduct only a limited number of qualitative probes, i.e. we will only be able to sample a limited number of intended beneficiaries’ settlements and within these only a limited number of intended beneficiaries (not sufficient for representatively depicting the surveyed villages).
12. DFID’s Trade Development Work in Southern Africa, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2013/12/DFIDs-Trade-Development-Work-in-Southern-Africa-Report.pdf.
13. DFID’s Support to Palestine Refugees through UNRWA, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2013/12/ICAI-UNRWA-report-FINAL-110913.pdf.
14. DFID Afghanistan Operational Plan 2011-2015, DFID, 2013, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/209876/Afghanistan1.pdf.