DFID’s Bilateral Support to Growth and Livelihoods in Afghanistan 2
Executive Summary
Afghanistan is an extremely poor country that has suffered decades of conflict and instability. Half the population requires humanitarian assistance and four-fifths rely on agriculture for their living. 2014 begins a period of even greater uncertainty: all international troops are expected to have withdrawn by December and there is a presidential election in April.
This review assesses the effectiveness of DFID’s bilateral growth and livelihoods projects, which account for approximately 30% of DFID’s £190 million annual aid budget in Afghanistan. We examine the impact on intended beneficiaries of five case study projects that aimed to strengthen infrastructure and give people the necessary skills and resources to improve their livelihoods. We also assess how DFID is preparing to deliver sustainable impact this year and beyond.
Overall Assessment: Amber-Red
Afghanistan is one of the most difficult places to deliver aid and DFID’s staff work hard under demanding conditions. Although the five projects we reviewed were, on the whole, well delivered, we found mixed results. The more ambitious and multi-faceted projects were less successful than those with more limited scope. Our fieldwork provides evidence that a positive difference is being made to the livelihoods of intended beneficiaries in the areas we surveyed. It is not clear, however, how positive impacts will, in all cases, be sustained in the long term.
DFID faces a challenge to ensure that its future growth and livelihoods portfolio is sufficiently coherent and flexible, given an increasingly uncertain future. The on-going international military drawdown is both an important challenge and an opportunity for DFID to focus its future strategy solely on poverty reduction and to reposition itself as the ‘lead operator’ of the UK’s presence in Afghanistan.
Objectives Assessment: Amber-Red
DFID’s overall goal, flowing directly from the UK’s overall strategy for Afghanistan, is to help to create a ‘viable state’. While individual projects meet critical needs, the growth and livelihoods portfolio lacks strategic coherence. Weaknesses in design – particularly a lack of direct consultation with intended beneficiaries and unproven theories of change – have made it harder for DFID to meet and assess its intended targets.
Delivery Assessment: Green-Amber
DFID chose effective delivery partners and managed them well, particularly for the smaller, less complex projects we reviewed. Engagement by these implementing agents with intended beneficiaries was positive. DFID’s use of ‘umbrella’ structures for some projects, which group together different components, was intended to reduce management costs and provide greater flexibility. In fact, they led to a lack of clarity and made management more difficult than was necessary. Fiduciary controls were good: DFID continues to respond positively to our previous review of Afghanistan.
Impact Assessment: Amber-Red
Results reported by DFID for our five focus projects were mixed. Two exceeded their targets (the demining and road construction projects), one partly achieved them; one is currently off-track and one has undetermined results. Our independent fieldwork verifies the positive results, showing significant improvements to the lives of intended beneficiaries. It also shows that useful information on the impact of DFID’s projects can be generated cost-effectively and independently in Afghanistan. There was variable quality of planning and design for sustainable impact across the projects. The credibility of results reported by DFID was weakened by an inappropriate use of overall impact indicators and target setting and lack of validation by DFID of the results provided by implementing partners.
Learning Assessment: Green-Amber
We support the activities underway to prepare for the post-2014 period and recommend additional measures. DFID has learned from experience in Afghanistan but should do more to document and share this, as well as ensure that lessons are learned from other conflict and fragile states. DFID must implement planned improvements to monitoring and evaluation in order to have a meaningful impact on the design of its future project portfolio and to contribute to the weak evidence base of ‘what works’ in Afghanistan.
Recommendations
Recommendation 1: DFID should formally review current and future projects to focus its portfolio more firmly on reducing poverty using evidence-based interventions. This should be completed within six months using a consultative and evidence-led process.
Recommendation 2: DFID should ensure that intended beneficiaries are, as far as practicable, directly consulted when new projects are being designed, so that their needs are clearly addressed and their ownership is enhanced.
Recommendation 3: DFID should enhance its approach and commitment to independent monitoring in order to assess current and future project performance and to allow it to assess the impact of its programme.
1 Introduction
1.1 This review focusses on the impact on intended beneficiaries of the Department for International Development’s (DFID’s) growth and livelihoods1 projects in Afghanistan. These projects aim to positively impact infrastructure, rural and urban livelihoods and jobs. We examine the performance of five case study projects and assess the preparations that DFID has made to maintain impact in 2014 and beyond.
1.2 This introduction puts DFID’s current work in context, considers how an uncertain future may affect DFID’s programme and explains our methodology and focus. It also reflects on progress made since our 2012 report, when we carried out a review of DFID’s Programme Controls and Assurance in Afghanistan.2 That report found that DFID’s financial and risk management processes were insufficiently robust.
Afghanistan has experienced prolonged conflict, remains unstable and is extremely poor
A long history of conflict leaves Afghanistan highly unstable
1.3 Afghanistan has suffered decades of instability and conflict since the pro-Soviet coup of 1978. It is estimated that by 1985 over half the country’s population was displaced by war.3 In 1989, the Soviet army pulled out but civil war continued. The US-led invasion of 2001 followed the 11 September attacks by al-Qaeda. The International Security Assistance Force (ISAF) began a protracted offensive against insurgents in 2002.
1.4 Conflict and its associated risks affect delivery of aid, with an especially severe restriction on the ability of staff to move freely and meet directly with intended beneficiaries and project implementers.4 Since 2008, insecurity has increased again and spread, especially, to the east and north.5 At least one Taliban attack is reported every day in 13 of Afghanistan’s 34 provinces.6 The UN reported that civilian casualties rose by 14% from 2012 to 2013.7
1.5 Political stability has remained low since 2005.8 The Government of Afghanistan’s central and provincial capacity is weak, despite large numbers of donor-funded advisers filling posts in the most important ministries. Accountability and oversight structures are poor.
Afghanistan suffers poverty equivalent to some of Africa’s poorest nations
1.6 Afghanistan has a population of 27 million (2012-13)9 and a Gross Domestic Product (GDP) per head of around £400, similar to Tanzania (£331) and Rwanda (£356).1011 (As noted below, however, GDP figures in Afghanistan are notoriously unreliable.12) It ranks 175th out of 186 countries reported in the 2012 United Nations’ Human Development Index.13
1.7 Nearly half the population is in need of humanitarian assistance14 and more than 80% of the population relies on agriculture for its livelihood.15 Agricultural production is sustained predominantly by water from snow-melt and is subject to severe droughts and surges in prices of basic commodities. Nearly a third of the population is food insecure.16 Natural disasters affected around a third of households in 2012.17 It is predicted that climate change will increase the frequency and duration of drought and flooding.18
1.8 Afghanistan has one of the world’s youngest populations, with 48% of citizens under the age of 15.19 The World Bank estimates this ‘youth bulge’ adds 400,000 entrants to the workforce each year, contributing to a growing problem of underemployment and unemployment.20
1.9 Significant constraints to private sector development exist, including lack of human capital and infrastructure, weak institutions and land management and ownership issues.21 The World Bank’s 2013 Doing Business Report ranks Afghanistan last for ‘investor protection’ and at 168 out of 185 countries on ‘ease of doing business’.22 Corruption is endemic: Afghanistan ranks last, jointly with North Korea and Somalia, in Transparency International’s 2012 Corruption Perceptions Index.23
1.10 Women in Afghanistan face considerable inequality: 41% are married before the age of 24.24 Only 17% of females over the age of 15 can read, which is less than half of the male literacy rate.25
1.11 Programmes to support crop diversification, livestock and orchards through improved access to inputs and markets are important to Afghanistan for rural income generation and the elimination of poppy cultivation. There are also powerful UK domestic interests in reducing the heroin trade. This was especially powerful from 2002, when the UK took on the G8’s ‘lead nation’ role in counter-narcotics in Afghanistan.26 This policy priority resulted in a particular focus on encouraging alternative licit crops at the centre of DFID’s livelihood programming.
1.12 Terrorism and conflict, however, have prevented the Government from establishing effective control in all areas of the country, particularly in Helmand. Poppy cultivation and the production of opium pose a serious challenge to Afghanistan’s security by providing funds to insurgent groups. The high level of corruption that surrounds the narcotics industry also jeopardises foreign assistance to development.
1.13 Opium is Afghanistan’s most important agricultural crop by value and provides livelihoods for many in rural areas.27 Despite years of efforts to eradicate and provide alternatives, 2013 marked a high point of poppy production, with a 36% increase from 2012. In total, 209,000 hectares are under production in 2013 (74,000 in 2002).28
Since 2001, military and civilian support to Afghanistan has grown enormously, despite the difficulties of delivering aid
1.14 In 2001, the US-led Operation Enduring Freedom began.29 The United States was supported by a broad coalition of international forces, including the Afghan Northern Alliance,30 Canada and the UK. The Taliban’s rule in Afghanistan ended in late 2001 and, in 2002, they began the insurgency which continues today. In 2006, control of military operations in Afghanistan switched to NATO’s International Security Assistance Force (ISAF).31
1.15 Since the fall of the Taliban in 2001, Afghanistan has experienced rapid economic growth, driven largely by military and donor spending. Real GDP growth averaged 9.1% from 2003-04 to 2010-11.32 In 2011, Afghanistan received £10 billion (over 55% of GDP) in military and civilian aid; of this, approximately £3.7 billion was development spending.33 A stark example of the effect of so much international expenditure is that services are now the majority sector of the economy.34
1.16 Figure 1 shows that the United States dominates aid expenditure. The UK makes the fourth biggest contribution after Japan and Germany. Aid largely focusses on security and state-building, with projects often having both objectives,3536 and is often directed where conflict (and hence military presence) is greatest – not where poverty is most prevalent. This has caused considerable resentment in the more peaceful provinces.37
Source: OECD Development Assistance Committee.
1.17 Aid has often been used as a direct part of military operations, particularly interventions aimed at reducing opium poppy growing38 and in the delivery of quick-impact projects aimed at winning the ‘hearts and minds’ of the local population.39 It is estimated that only 40% of aid reaches the Afghan economy; the rest is passed through contracts with international providers of goods and services, imports and the expatriation of profits.40
1.18 The scale of international investment has, however, resulted in notable successes, especially in improvements to basic services. For example:
- from about one million in 2001, nearly six million children were attending school in 2012.41 As opposed to virtually no girls attending school under the Taliban, almost two-fifths of pupils are now girls;42 and
- from 16% in 2003, the most recent survey shows half of pregnant women received antenatal care.43
In the period 2006-16, DFID will spend over £1.2 billion on growth and livelihoods in Afghanistan
1.19 The UK’s aid expenditure in Afghanistan is small, compared with its military spending. DFID’s annual expenditure in Afghanistan has risen from around £100 million in 2006 to nearly £180.5 million in 2012-13 and it is budgeted to reach almost £190 million this financial year.44 In comparison, UK military expenditure, since 2009-10, has averaged about £3.85 billion per annum. Total spending by the UK on the direct cost of the military intervention, excluding salaries, will be around £21 billion (2001-02 to 2012-13).45
1.20 The UK Government has committed long-term support to the Government of Afghanistan and will maintain a level of non-military spending similar to today until at least 2017 (although this is a reduction in both real terms and as a proportion of DFID’s total budget).46
1.21 DFID Afghanistan allocates over half of its funding to multilateral development institutions. In the period 2006-14, DFID’s largest contribution (£506 million) is to the World Bank-led Afghanistan Reconstruction Trust Fund (ARTF) – a co-ordinated financing mechanism for the Government of Afghanistan’s budget and priority national investment projects.47 As Figure 2 shows, over 85% of DFID Afghanistan’s annual budget is allocated to growth and livelihoods – an important reason this review focusses on such projects. Of this, around 40% is bilateral aid.
Note: figures based on disbursements to date and projected future expenditure.
Afghanistan’s future is highly uncertain
Security, economic and political changes will occur simultaneously with highly uncertain outcomes
1.22 The next two years are critical to Afghanistan’s future:
- Security: security transition is underway. By the end of 2014, nearly all ISAF troops will have left and full control will be handed over to Afghan forces. While it is extremely difficult to know what implications this will have, territorial losses by the state are predicted;48
- Economic: drawdown of troops is being accompanied by a very large reduction of direct military funding. International donors are further expected to reduce financial support to Afghanistan.49 These reductions could lead to a large fiscal deficit, driven by the need for the Government of Afghanistan to maintain military spending. Donors are concerned about the government’s ability to fund future spending. For example, a leaked IMF report stated that the government was failing to reach even the target of covering 40% of its annual non-security spending of $5 billion because of ‘widespread tax evasion abetted by government officials, the increasing theft of customs revenues by provincial governors and softening economic growth’.50 Intensified violent competition for scarce resources is also expected;51 and
- Political: April 2014’s presidential election will result in the first new head of government in over a decade.52 An election that is fair and representative of the people’s will is likely to be rejected by insurgent groups; if there is widespread fraud, it is likely to lead to a greater level of conflict. The election could also further weaken the state if sub-state actors, especially power brokers from northern and western Afghanistan, lose faith in the central government and accelerate efforts to rearm.53
1.23 The cumulative effect of these changes is hard to predict but may have widespread impact on DFID’s and other donors’ ability to deliver aid. Potential effects include even greater barriers to accessing project sites and intended beneficiaries than at present, difficulties hiring and motivating staff and more challenges to working with the Government of Afghanistan. Living conditions for staff may also become more restricted if the risk of attacks, such as that in January 2014 on a popular restaurant in Kabul, which killed 21 people including two Britons, continue.54
1.24 Responding to such difficulties may involve a greater need for flexibility to shift resources to where they are most needed; a different mix of implementing partners with a greater access to intended beneficiaries; and an allocation of more resources to influence other donors to maintain their spending levels.
Our approach to this review
We reviewed DFID’s overall strategy for growth and livelihoods and considered five projects in detail
1.25 We assess how DFID’s strategy to deliver growth and livelihoods projects in Afghanistan links to the overarching objectives of the UK Government. We also assess the robustness of the theories of change – or similar project design methodologies – which underpin relevant interventions.55 We examine DFID’s preparedness to deliver sustainable impact throughout and beyond the period of drawdown of international troops in 2014.
1.26 We undertook a review of the literature on growth and livelihoods interventions in Afghanistan and comparable fragile and post-conflict environments. In addition, we interviewed subject experts in Afghanistan, the UK and the United States, including the Special Inspector General for Afghanistan’s Reconstruction (SIGAR).56 This work informs our assessment of the evidence that DFID has available in terms of strategic options, project choices and detailed design decisions.
| Helmand Agriculture and Rural Development Programme The ‘Rural Development Programme’ 2006-10 £19.3 million |
‘Reduce poverty and increase legal economic activity in Helmand through more visible national and provincial government service delivery.’57 (Capacity building, microfinance, market support, infrastructure, water and sanitation.) |
| Herat Demining Project The ‘Demining Project’ 2008-18 £20.6 million |
‘Enable increased legal livelihoods opportunities for the poorest and most vulnerable communities.’ (Landmine and other unexploded ordnance removal.) |
| Supporting Employment and Enterprise Development The ‘Employment and Enterprise Programme’ 2009-14 £36.0 million |
‘Increase employment and income opportunities by strengthening the private sector’s ability to invest and compete.’ We focus on the three bilateral elements: The Afghanistan Business Innovation Fund (ABIF); Zardozi – design support, technical advice and market information for chronically poor female home workers; and Turquoise Mountain – training in traditional Afghan crafts. |
| Road Improvement in Central Helmand The ‘Road Project’ 2010-13 £7.7 million |
‘Contribute to significant growth in Helmand’s licit economy by improving road links.’ (Upgrading of five sections of road totalling 46 kilometres.) |
| Helmand Growth Programme The ‘Growth Programme’ 2010-14 £14.2 million |
‘Support farmers, agri-businesses and the workforce in Helmand’. We focus on five bilateral elements: improving youth employment skills; DASTGAH – skills training for farmers and a voucher scheme; funding the growth of new agribusiness ventures; an innovative Islamic loan product to encourage growth of small- and medium-sized enterprises (SMEs); and support for establishing a small business membership association. |
Note: project descriptions are quotations from DFID’s project documents. Main activities are added in parentheses.
1.27 We chose five bilateral case study projects to examine in detail, with a combined budget of £97.7 million, implemented from 2006 to 2018. Figure 3 above sets out the chosen projects giving their cost, duration, intended impact (as set out in DFID’s project documents) and a summary of the activities involved. Figure 6 on page 16 details intended targets and reported results.
1.28 Projects were selected with a set of objective criteria, including how directly they focussed on the poor; our ability to access intended beneficiaries and project staff; and implementation dates to facilitate beneficiary recall.58 We chose our shortlist to ensure that we had fair coverage in terms of location, size and thematic focus.
1.29 In 2006, the UK took control of the joint military and civilian Helmand Provincial Reconstruction Team (PRT), which closes in 2014, with support from the governments of Denmark, Estonia and the United States. UK troop levels in Helmand rose from 3,300 in Spring 2006, peaking at around 10,000.59 As a result, the UK decided to increase its development programme and hence the number of its growth and livelihoods projects based in and around Helmand. To ensure fair coverage, three of our five case studies are Helmand-based. These projects aim to support both DFID’s strategic objectives and the jointly owned Helmand Plan, which guides the work of the international partners working in close co-operation with the Government of Afghanistan.
Our methodology includes interviews in Afghanistan and engagement with intended beneficiaries
1.30 Meeting directly with intended beneficiaries is an important part of our assessment process. Our core team met with intended beneficiaries and their representatives, implementing agents, officials of the Government of Afghanistan and other local stakeholders in Kabul and Helmand.
1.31 In addition, we engaged a local Afghan research team to conduct qualitative interviews with intended beneficiaries in Helmand and Herat Provinces (see Figure 4 on page 8). This fieldwork focussed on the Road Project, the Demining Project and the Developing Access to Skills and Trade for Growth of Agri-Business in Helmand (DASTGAH) sub-component of the Growth Programme. Fieldwork projects were selected based on the following criteria: a recent evaluation had not taken place; the project is on-going or recently completed (to ensure beneficiary recall) and it was delivered in a safely accessible location, with identifiable intended beneficiaries.
Note: Interview feedback data; map from the United States Central Intelligence Agency World Factbook.
1.32 We interviewed 420 individuals across three provinces in Afghanistan, as shown in Figure 5, using both individual interviews and focus groups (around 20 people in each). Focus groups were used to understand impact at the village level and individual interviews assessed household impact.
| Case Study Project | Focus Groups (people) | Individual Interviews (male) | Individual Interviews (female)60 |
|---|---|---|---|
| Growth Programme | 46 | 55 | 40 |
| Road Project | 46 | 73 | – |
| Demining Project | 70 | 65 | 25 |
| Total | 162 | 193 | 65 |
1.33 A full description of our fieldwork methodology and findings for each project can be found in Annexes A2-A6.
2 Findings
Objectives
Assessment: Amber-Red
2.1 This section reviews the role and relationship of DFID’s strategic aims and those of the five case study projects. It considers whether objectives are realistic and relevant given the operational circumstances in Afghanistan.
DFID’s strategy in Afghanistan
DFID’s current strategy derives from the UK Government’s overall approach to Afghanistan and has changed over time
2.2 DFID’s current strategy is set out in its Operational Plan, published in 2010 and updated annually.61 This plan derives from the ‘viable state’ pillar of the UK Government’s current National Security Council’s (NSC’s) Strategy for Afghanistan (unpublished), for which DFID is responsible. The NSC’s plan is soon to be refreshed to give DFID greater prominence following UK military drawdown.
2.3 The Operational Plan specifically aims for a ‘more peaceful, stable, viable and prosperous Afghanistan’, implemented, at the time of our fieldwork in September 2013, through 32 active projects.62 This ambitious plan is set out under three ‘priority areas’:
- supporting peace, security and political stability;
- promoting economic stability, growth and jobs; and
- helping the state to deliver improved services.
2.4 The current strategy has changed repeatedly since the Transitional Country Assistance Programme was published in 2003, at which time approximately half of funds were spent on humanitarian assistance. Following elections in 2005, DFID produced an interim strategy supporting the Government’s strategy of that time. Although initially designed for one year, it was extended to 2008. This focussed on state building and on poppy reduction. A similar focus was set out in 2008 in a short Country Plan – developed from a set of detailed context studies.
2.5 Some trends across these plans are discernible. Humanitarian assistance fell in relative importance as counter-narcotics and governance (particularly in central government) objectives increased. Counter-narcotics was a very large – and a very ambitious – part of DFID’s strategic focus for many years. The current plan and future thinking reverses many of these trends. In particular, the reduction of poppy growing does not feature in any major way, humanitarian support will grow and there are plans for a major infrastructure programme (£130 million over five years) – a sector which is becoming increasingly important for DFID in Afghanistan.
DFID’s current strategy has an unclear role in determining the growth and livelihoods project portfolio
2.6 DFID Afghanistan must deliver its strategy in a context of substantial political, military and headquarter pressures. We did not, therefore, expect to find a mechanical system through which the ‘viable state’ objective is translated into growth and livelihoods projects – this is often an iterative process. We do, however, question whether DFID’s strategy adequately informs the project selection and design process. Staff were not clear, for example, about how additional money would be spent if it became available.
2.7 We did not find any robust processes for translating the high-level strategy into project options and decisions or any analysis of how the growth and livelihoods portfolio is, in combination, contributing to the overall aims of the strategy. This was compounded by the inability to track the impact of projects (particularly the smaller ones) and, thus, the feedback into project selection and design was weak (see Learning section on page 22). Few DFID staff had an awareness of the full portfolio – unsurprisingly, given it consisted of 32 projects at the time of our fieldwork – which also limits the opportunities for staff to identify synergies and to co-operate effectively.
2.8 We heard in the DFID Afghanistan office of plans by central teams to start to think more coherently about the portfolio and a desire to rationalise the approach into fewer, larger projects. If DFID’s growth and livelihoods projects are to be fully coherent with the Operational Plan (and the NSC strategy), then either the strategy itself ought to be more specific or more planning needs to take place at the intermediate level between projects and the ‘priority areas’.
2.9 More planning would also enable DFID to assess how individual projects contribute to its strategic priorities (and ensure that all parts of the strategy have corresponding projects) and to design its portfolio to take better advantage of opportunities for synthesis and scale and be suitably robust for an unknown future.
DFID co-ordinates well with other donors and supports multilateral systems in support of its strategic aims
2.10 DFID is well-regarded in the donor community in Afghanistan. The UK has contributed to and has influenced various important donor co-ordination processes, such as the recent Tokyo Mutual Accountability Framework that established an initial approach to delivering aid based on mutual commitments.63
2.11 DFID is a major contributing donor to multilateral funds and is active in their management (see Figure 2 on page 5). DFID also works in partnership with other donors, including Australia, Denmark, New Zealand and Sweden.
Project objectives and design
The objectives of our case study projects were often suitable for their contexts and innovative, yet sometimes too ambitious
2.12 The objectives of three of our case study projects were suited to their contexts. The Demining Project had clear, relevant and realistic objectives with (in phase one) a strong humanitarian focus and (in phase two) a livelihoods focus. The Enterprise and Employment Programme was well suited to the Afghan context, given both the ‘youth bulge’ and that private sector development is a clear presidential priority.64 The Road Project also had a realistic and desirable goal to impact economic growth as a result of the intended improved roads. It also had security benefits such as reducing vulnerability to Improvised Explosive Devices (IEDs) and enabling faster incident response times.
2.13 The remaining two projects, the Growth Programme and the Rural Development Programme, were complicated and ambitious. They have multiple objectives targeted at multiple beneficiaries making them difficult and costly to design, manage, monitor and evaluate effectively and efficiently.65 Both projects were intended, in part, to reduce opium poppy production, which was a clear UK political priority at the time of project design. Opium poppy production is part of a complex and deeply embedded system that provides substantial incomes.
2.14 While we identified some interesting attempts to innovate in the Growth Programme, including the design of a new Islamic loan product suited to the needs of growing SMEs, the programme’s overall objective of a ‘vibrant’ and ‘licit’ economy was unrealistic and vague and lacked specific, measurable, attainable, relevant and time-bound indicators. This complexity was one of the underlying causes of delivery and impact weaknesses and was not properly recognised as a risk by DFID in programme design and during delivery.66
2.15 Two components of the Employment and Enterprise Programme were also innovative:
- Zardozi has introduced a simple membership-based guarantee mechanism for the women that it supports. Intended beneficiaries told us that this fairly simple process has significant impact in relieving a major constraint to the establishment of supplier relationships with shops; and
- ABIF is pioneering a risk-adjusted approach to grant allocation which has the potential to substantially improve the efficiency of grant provision.67
2.16 Of our focus projects that will continue into 2014, none have made any specific preparations for drawdown, even though it is likely that they would be affected by more instability and greater risk. For example, the beneficiaries of the Zardozi project (part of the Enterprise and Employment Programme) reported a fall in shopkeeper confidence – a probable response to uncertainties about future business prospects – reducing the availability of advances for handicraft goods.
2.17 Of the documentation we were able to review for projects in development, very little planning has yet taken place to test projects against likely future scenarios.
Weak project design limits DFID’s ability to realise ambitious objectives
2.18 As a basic requirement for good project design, beneficiary consultation brings greater ownership of project objectives and encourages sustainability of benefits. Joint ownership of objectives reinforces local accountability, which is particularly important in Afghanistan given the regular combination of military and development objectives.
2.19 The Enterprise and Employment Programme, the Growth Programme and the Rural Development Programme did not sufficiently consult intended beneficiaries during the project design phase. Although difficult in a context like Afghanistan, this is not impossible to achieve.
2.20 A good theory of change – the sequence of events that is expected to lead to a desired outcome and long-term goal68 – focusses attention on the needs and behaviours of intended beneficiaries. It makes explicit, evidence-based links between activities and impacts and relies on sound assumptions. We found incomplete and poorly articulated theories of change. Examples of theory of change weaknesses in the projects include:
- the Employment and Enterprise Programme’s theory of change was based on multiple empirical linkages (for example, using microfinance provision to stimulate enterprise development; or creating jobs as ‘critical for combating the insurgency and tackling the illegal narcotics industry’).69 The programme design did not, however, provide evidence to show how assumptions about such linkages would be sustainable in the long term in Afghanistan; or, in some cases (job creation and insurgency, for example), sufficient evidence to justify the activity; and
- transformational events in the Growth Programme’s theory of change were not identified or appraised adequately, which affected sustainability of the intended benefits. For example, incentivising farmers to engage systematically and sustainably with input markets by providing highly subsidised agricultural inputs was already a discredited approach at the time of designing the programme.70
2.21 More clearly basing project design on a well-evidenced theory of change would have made projects more realistic and their objectives more likely to be met. In areas with a less clear evidence base, projects should have clearly stated that an objective of the project was to test a hypothesis of the theory of change and then ensure adequate monitoring and evaluation took place.
Delivery
Assessment: Green-Amber
2.22 This section examines the delivery of DFID’s growth and livelihoods programme in Afghanistan. We assess whether the needs of intended beneficiaries were appropriately taken into account and whether DFID had appropriate arrangements for financial and risk management.
There is evidence of both good and poor delivery by implementing partners
Some delivery partners perform to high standards
2.23 The Demining Project was delivered very competently by The HALO Trust. The organisation demonstrated an ability to work closely with national and international co-ordination bodies – in part through DFID’s intervention – as well as with the Ministry of Rural Rehabilitation and Development (MRRD). The HALO Trust works under very risky circumstances. For example, it has had staff abducted and equipment stolen – in January 2014, 57 of The HALO Trust’s staff were kidnapped in Zarghun district in Herat as they drove to an area that needed to be cleared of mines; they were then released unconditionally. Such risks are managed, in part, through very good community relations, as well as through The HALO Trust’s own competent risk management systems.71
2.24 The UK’s Specialist Teams Royal Engineers (STRE) delivered mainly good results implementing the Road Project. They used local Afghan companies to undertake construction and hired and trained local staff to monitor and inspect progress, as well as undertaking some spot-checks themselves. The Afghan Provincial Department of Public Works was also involved in the implementation of the project and regularly inspected progress of the road works. They sign off on completed roads only when satisfied with the condition.
2.25 Our fieldwork respondents were very satisfied with the quality of the roads in two of the three areas we surveyed, in Central Marjah and in the area west of Lashkar Gah. Given the quality assurance processes that we were told about, we were surprised to find that respondents along the road section between Chah-e Anjir and Spin Masjed criticised the quality of the road construction. Both our researchers and DFID confirmed that approximately four kilometres of road has been seriously degraded.
2.26 The contract that DFID specified had only a six-month retention period to fix problems. This was too late for DFID to ask the contractor to fix these problems, as they became aware of the damage after the retention period.72 Our view is that DFID contributed to the likelihood that poor quality construction (and hence degradation) could take place for at least two reasons. First, during the early stages of construction STRE’s quality assurance process was still being developed. Second, STRE’s resources were overstretched – having implemented more than 120 construction projects in a volatile security environment.
Other attempts to deliver, especially through the Government of Afghanistan, were less successful
2.27 The Rural Development Programme aimed to demonstrate the Government of Afghanistan’s competence to deliver services but the ministries involved failed to deliver as planned.73 In particular, there was no operational budget and weak budget and management control, leading to significant monitoring and project redesign costs to DFID.74 This lack of capacity caused DFID to consider closing or suspending the Rural Development Programme. Neither action was, ultimately, pursued but DFID mitigated its exposure to under-performance by limiting the role of government in the following Growth Programme.75
2.28 Some private sector implementing partners proved unreliable. For example:
- DFID decided to drop one of the components of the Growth Programme – an Islamic loan product – because the implementing partner was unable to sell the loan product to the targeted business community. The target market, small agribusinesses, was financially unsophisticated and could not understand the loan benefits;76 and
- DASTGAH, another component of the same project, suffered a series of roll-out problems, including weak leadership, cash-flow difficulties and a long period of inactivity during negotiation with DFID. The independent consultant engaged to review the project concluded that little was done to expedite the project.77
Implementing partners often engaged well with intended beneficiaries and the government of Afghanistan
2.29 Our fieldwork shows that the implementers of the Demining Project demonstrated excellent co-operation with local communities, which was both transparent and responsive to need. As a result, the community had confidence in The HALO Trust’s ability to remove all mines, while previous attempts by other demining organisations were incomplete. Through beneficiary consultation, The HALO Trust was able to direct its work to where communities most needed land to be made safe. Staff were also praised for their culturally sensitive approach. For example, they always asked for permission to enter land or houses, regardless of whether they were abandoned. They prayed together with villagers and recited the Koran ahead of community meetings.
2.30 The Road Project was delivered with ongoing consultation between the implementing partners and the provincial government in Helmand. A regular infrastructure working group was established, placed first at the PRT and recently transferred to the provincial government’s offices. This contributed to building local capacity and developing Afghan ownership. Our Afghan interview partners favourably contrasted DFID’s approach to working with the provincial government to that of other donors.
2.31 As a result of this co-operation and the desire of DFID and STRE for the Government of Afghanistan to be seen to be leading this work most fieldwork survey respondents saw provincial government authorities as having led the project.
2.32 The Growth Programme’s DASTGAH project, implemented by the international non-governmental organisation (NGO), Mercy Corps, co-operated well with local village representatives and Community Development Councils but failed to co-ordinate adequately with district government authorities, in particular with the Department of Agriculture, Irrigation and Livestock. The result was a delay in project implementation.
‘Umbrella’ programmes may provide organisational efficiencies and flexibility but the way they were used in Afghanistan reduced DFID’s ability to deliver impact
2.33 An ‘umbrella’ programme incorporates a range of components with a common thematic focus, grouped and managed as a single project to deliver collectively the desired impact. Generally, each component could otherwise exist as a standalone project.
2.34 Umbrella programmes can be more cost effective and efficient than stand-alone projects, particularly if their design were to take into account the following criteria:
- coherence of individual projects (and fit with the overall growth and livelihoods strategy);
- focus on the same intended beneficiary or a homogenous group of intended beneficiaries (to avoid complexity);
- single or narrow range of objectives (to avoid complexity);
- sound internal logic (mutually consistent objectives);
- potential synergies between projects (for instance, co-ordination of monitoring and evaluation resources); and
- alignment of start and end dates of components (particularly to allow effective performance management through the annual review process).
2.35 DFID Afghanistan’s use of umbrella programmes stemmed from a corporate desire to reduce management costs in project operations by reducing budget lines; permitting flexibility to introduce and remove components as required by external changes; and seeking economies of scale. (DFID Afghanistan’s total number of projects would be much greater if the individual components of the umbrella projects were counted as separate projects). Specific problems that we observed in the use of umbrella programmes include:
- long delays in the design and delivery of programme components with substantive reconsideration to approaches and activities (project components were successively cancelled, amended or added as it became evident that they would not achieve performance targets or political priorities changed). This was most notable for the Growth Programme and the Employment and Enterprise Programme. For example, the Growth Programme was originally planned to cost £28 million, then rose to £36 million and, with successive revisions, was reduced to £14.2 million as components were accepted, found to be non-performing and subsequently rejected;
- the introduction of a funding envelope which welcomed small components – that DFID would otherwise not have supported – and allowed resources to fund ad hoc activities and which did not contribute directly to targets;78
- ability to allow project additions and subtractions with minimal justification – for example, the removal of the Employment and Enterprise Programme’s £3.5 million support agent and advisory council – without redesign. This strategic governance structure was presented as critical to maximising value for money, enabling engagement with the Afghan private sector and managing the multiple, varied risks of an umbrella programme in Afghanistan. Yet, within a year, DFID considered this structure to be unsuitable to the context (in which time nothing material in the context had changed) and it was discontinued, to be replaced with one private sector adviser;
- when major changes took place to programmes (such as the removal of the Bost Business Park core component of the Growth Programme), there was insufficient strategic review to ensure that the proposed objectives and impact could still be met with the remaining components; and
- multiple components of varying length with start and end dates that created difficulties in monitoring the performance of a project, using DFID’s quarterly implementing partner reporting and annual review processes.
2.36 DFID acknowledges that its umbrella programmes are ‘complex and overstretched’ and is ‘unclear whether they are cohesive or mutually reinforcing/leveraging’.79 We found that the use of an umbrella structure in the Growth Programme and Employment and Enterprise Programme damaged performance and negatively affected impact. While these programmes contain thematically similar projects, each individually lacks a convincing argument as to why it is the best option to contribute to the intended impact.
Financial management, risk and corruption
DFID continues to respond positively to our previous review
2.37 In response to our first review, Programme Controls and Assurance in Afghanistan,80 DFID has translated our recommendations into nine clear and relevant management actions. DFID established a team independent of its Afghanistan office to develop actions to address ICAI’s recommendations for further controls and assurance, detailed in a published follow–up report.81
2.38 DFID has created new posts to focus on contracts and financial control; reviewed its risk register; started a due diligence assessment of implementing partners; produced an Anti-Corruption Strategy (reviewed every six months) and established a Portfolio Assessment Tool (PAT) to measure and track risk, performance, expenditure and management cost of all projects.
2.39 Across our case study projects, we found a good standard of financial and anti-corruption controls, especially for the projects with a shorter delivery chain. We saw evidence of best practice, such as the use of contractor blacklisting. DFID was alerted to several cases of potential fraud in our focus projects, including the alleged misuse of funds, corruption and bribery. Cases were passed onto DFID’s Fraud Unit; enhanced due diligence, restructuring of project components and additional controls were added as a result.
Fiduciary controls and risk management are working well
2.40 A risk register correctly identifies and records most programme risks and mitigating actions. Each project has a full risk matrix, setting out both probability and impact of all identified risks. The Rural Development Programme, for instance, identified nine risks as high or medium probability, including deteriorating security, ineffective provincial reporting structures, expanding poppy cultivation, supplier shortages and government corruption.
2.41 Mitigating actions to address these risks included robust financial management controls (to mitigate leakage risks) and capacity building support to government (to mitigate weaknesses in government decision-making). The security threat was correctly rated as high risk. It proved to be a major hindrance to project implementation and to effective monitoring and record keeping.
2.42 The Growth Programme identifies 16 risks, including ineffective beneficiary targeting by implementing partners and under-performance by the Government of Afghanistan.
2.43 The Roads Programme rated security as the main risk and STRE, DFID’s implementing partner, mitigated this risk by training local staff members to undertake monitoring and by building the capacity of Afghan line ministries. STRE was aware of corruption by engineering contractors and security providers and held a blacklist of the fraudulent contractors. A forensic background check was conducted to check for links with insurgents.
2.44 The risks and realities of operating in Afghanistan continued to affect project performance, in spite of effective risk assessments. In the case of the Growth Programme, for example, the risks associated with the difficult operating environment called for projects to be tailored to the context but implementing partners did not consistently provide workable solutions.82
Impact
Assessment: Amber-Red
2.45 This section assesses the impact of our case study projects on intended beneficiaries. We report results against DFID’s intended targets and our independent fieldwork, discuss monitoring data and consider sustainability of the focus projects.
Results of our case study projects
Our case study projects report a range of performance for intended beneficiaries relative to intended targets
2.46 This section assesses our five focus projects using the data reported by DFID and our interviews with stakeholders, intended beneficiaries and their representatives. The following section details the results of our fieldwork for the Demining and Road projects and for a component of the Growth Programme, undertaken in Helmand and Herat in Autumn 2013.
2.47 Project performance relative to planned targets, reported by DFID and its implementing partners, is set out in Figure 6 on page 16. We have also set out project-specific scoring and more detailed findings in Annex A1 and Annexes A2-A6 respectively. In summary, the reported results show:
- the Demining and Road projects exceeded their target results (we refer to the achievement at DFID’s outcome level as ‘results’);83
- the Rural Development Programme partly achieved its target results;
- of the projects still being implemented, the Growth Programme has undetermined results due to lack of suitable data; and
- the Employment and Enterprise programme is off track with two of its three targets.
| Project Name, Value Period, Implementer | Description of Activities | Objectives | Achievements against selected key targets (assessed to time of review, unless project has completed) |
|---|---|---|---|
|
The Rural Development Programme84 £19.3 million 2006-10 Government of Afghanistan; NGOs, donors, private sector |
|
Results Provide economic opportunities for the rural poor, including alternatives for poppy farmers Impact Reduce poverty and increase legal economic activity through more visible service delivery |
Target 180 km roads rehabilitated or constructed 50 jobs created through Microfinance Investment Support Facility loans 70,000 people benefit from infrastructure improvements Achievement 97 km 190 jobs 64,000 people Undetermined: there is no evidence of long-term sustainable change; no poverty data. Poppy cultivation initially decreased in Helmand (65,045 hectares in 2010 from 69,324 in 2006) but increased thereafter. |
|
The Growth Programme £14.2 million 2010-14 Private sector and NGOs |
|
Results Increase the potential for licit economic growth in selected districts of Helmand Impact A vibrant and licit economy in a stable Helmand |
Target 177 full-time employees created by project interventions Three projects fully transitioned to national programmes Achievement No data on jobs created One project transitioned Undetermined: poppy cultivation increased rather than decreased (from 65,045 hectares in 2010 to 100,693 in 2013); vibrancy measured by proxy data: 53% saw the central bazaar improve and 40% saw opportunities improve.85 |
|
The Road Project £7.7 million 2010-13 Specialist Teams of Royal Engineers |
Upgrade poor quality gravel roads to all-weather two-lane asphalt roads (47 km across five road sections) |
Results Improve road links between district centres and key provincial strategic economic locations Impact Significant growth in Helmand’s economy and provide growing opportunities for the population |
Target 20% traffic increase in vehicles (480 from 400 annual average daily vehicle count) Achievement 610 vehicles Undetermined: indicator used by the project poses question about local government legitimacy and is, thus, irrelevant with regard to ‘economic growth’ and ‘growing opportunities’. |
|
The Employment and Enterprise Programme £36.0 million 2009-14 (with extension) Government of Afghanistan; private sector and NGOs |
|
Results Increase employment and income opportunities by strengthening the private sector’s ability to invest and compete Impact Increased economic opportunity, delivered by the private sector |
Target 9,173 (of which, 7,555 for women) jobs created 118,000 people (of which, 22,267 women) experience better value for money and/or improved income opportunities Draft 20 pieces of legislation, regulation or policy Achievement 6,610 (5,914) 4,625 (1,907) 16 Undetermined: indicators (GDP growth rate and private investment as a percentage of GDP) are inappropriate to measure performance and they suffer major measurement difficulties. |
|
The Herat Demining Project86 £20.6 million (£11.4 million and £9.2 million) Phase I 2008-12 (Phase II 2012-18) The HALO Trust |
|
Results Increased legal livelihood opportunities for the poorest and most vulnerable communities Impact Reduce the threat from landmines and unexploded ordnance to Afghan lives and livelihoods |
Target 129,000 families benefit from cleared land 70% reduction in casualties caused by landmines or unexploded ordnance Achievement 133,770 76%87 Undetermined: milestones for ‘all emplaced antipersonnel mines cleared’ and ‘all Afghan communities unaffected by presence of antipersonnel landmines’ were not achieved. |
84 Both the Rural Development Programme and the Growth Programme focussed on poppy eradication. Positive impact was claimed when poppy cultivation was shown to have declined from 69,000 hectares (2006) to 65,000 hectares (2010); subsequently, production has increased, especially onto land in the Helmand desert (the ‘dasht’).
85 Perception survey results are compared to DFID targets of 35% and 25% respectively. See paragraphs 2.14 and 2.55 for a discussion of the weaknesses of these indicators to assess a ‘vibrant and licit’ Helmand economy; and paragraph 2.35 for comment on insufficient revision of objectives following programme modifications.
86 It is too soon to measure Phase II results so the table reports only on the completed Phase I.
87 This figure was supplied to us by the Herat Demining Project, based on casualty reporting that came through after the DFID project completion report was finalised.
Our independent fieldwork found positive changes to intended beneficiaries’ livelihoods
2.48 We conducted fieldwork for the Demining Project in Herat, the Road Project and a subcomponent of the Growth Project in Helmand. We surveyed project results and, to the extent that qualitative fieldwork allows, sought to understand whether projects achieved their stated objectives. Figure 7 sets out three case study highlights, Figure 8 on page 19 summarises our main findings and Annexes A2-A6 give details for each project and survey site.
2.49 While our fieldwork reflects the views of a sample of intended beneficiaries (locations were, in part, determined by secure access), it credibly demonstrates that valid insights into results can be collected relatively inexpensively – through carefully planned engagement – and suggests that more beneficiary consultation at the design stage is and has been possible. The information gained through such exercises is useful for DFID to gain a sense of what is working and why and provides insights for future project design.
2.50 We heard of significant changes to intended beneficiaries’ livelihoods. Highlights include:
- Herat Demining Project: this demonstrated the greatest changes to livelihoods of our fieldwork. Through both our focus groups and individual interviews, we learned how The HALO Trust has improved physical security and employment opportunities (especially in agriculture and livestock keeping), provided greater access to services (such as education and health) and increased ease of movement;
- Road Project: a significant impact on the daily lives of intended beneficiaries was found. These include improved access to markets (to buy and sell goods and services), new job opportunities (particularly day labour), more and faster transportation, increased access to public services and greater security. Our interviews did, though, make us aware of a section of road that has already significantly degraded and where positive changes to livelihoods are less prominent; and
- DASTGAH (component of the Growth Programme): this introduced intended beneficiaries to plastic tunnelling that allows farmers to plant crops in winter (when they would traditionally only cultivate low-profit wheat). Nine months after completion, we found significant improvements to household incomes.
Herat Demining Project: the village of Saray Naw, located 7-8 kilometres from Herat City, benefited enormously from demining
The village was one of the worst affected with mining of most agricultural land. As a result, a large part of the population left; others turned to wage labour in Herat City. Following demining, villagers took advantage of the possibilities offered by the fertile clay soil and their vicinity to the rapidly growing city of Herat.
One opportunity was making bricks. Eight brick kilns have been established on demined land, offering seasonal work during the summer months for some 300 villagers. The bricks are sold in Herat City, which is experiencing a construction boom. An abattoir was also set up on demined land, offering jobs to 15-20 people, as well as three poultry farms employing seven workers. Additional employment opportunities are offered by NGO-funded projects, including tailoring and literacy courses, greenhouses and assistance to establish saffron production.
Road Project: freedom of movement and transportation services improved around the village of Nad-e Ali
Travel fares dropped markedly. For example, from Pir Abad, located halfway between Marjah Junction and Bolan, to Lashkar Gah used to cost AFN 50 and is now AFN 30:
‘Before, fares were high but they have dropped and we can now go to the bazaar anytime. We can easily bring necessary groceries from the bazaar to cater for guests. For example, I had guests the day before yesterday who arrived early afternoon. I was able to hire a taxi from the road, travel to the market and buy meat and other necessities. Without the road, I wouldn’t have been able to find a car but even if I had, I wouldn’t have been able to get to the bazaar in time.’
Growth Programme: when asked about their future plans, half of the interviewed farmers (20 respondents) stated that they would continue to use provided plastic tunnelling next winter and pay fully for the necessary inputs
The seeds and plastic sheeting must be bought new each season, while metal rods can be used repeatedly. Some farmers even wanted to use plastic tunnelling on more land. A farmer from Kangai Village (Nawa District) explained:
‘I want to apply the plastic tunnelling technology on more land so that I can obtain higher yields and increase my income even further. I want to spend the money [I’ve earned in the first year] on purchasing more plastic tunnelling materials.’
| Evidence of need findings | Impact findings | |
|---|---|---|
| The Road Project |
|
|
| Herat Demining Project |
|
|
| DASTGAH component of the Growth Programme |
|
|
Poor monitoring, validation and target-setting severely limit DFID’s ability to assess impact
DFID Afghanistan’s weak monitoring systems reduce its ability to show how its spending is affecting livelihoods
2.51 We found poorly designed monitoring methodologies and inappropriate indicators to be common features of our case study projects. The design of systems to measure intended results was poor and sometimes demonstrated a lack of understanding of what, if measured, would reliably reflect the performance of a project. These shortfalls in DFID Afghanistan’s monitoring were found, despite several previous similar findings:
- DFID’s 2009 Country Evaluation stated that ‘DFID is keenly aware of the difficulties of assessing and demonstrating impact in the Afghan context’;88
- other reviews by the National Audit Office (NAO) and DFID on working in fragile and conflict-affected states report similar findings. For example, in 2008, the NAO found that ‘learning has been hindered by incomplete project reporting… quicker and fuller dissemination of the lessons from front line experience will help DFID improve the effectiveness of its aid’;89
- a 2010 DFID synthesis of evaluations, reviews and monitoring made several recommendations for improvement; and90
- the 2012 International Development Committee report makes several references to the problems of monitoring and welcomed that DFID was ‘considering the use of ‘third party verification and continuous audit”91 which DFID set out in its response to our first report on Afghanistan.92
Few incidents of validating reported monitoring means that the reported results are questionable
2.52 Results for the case study projects were, in most cases, reported directly by implementing partners. We saw very few validation mechanisms that sought to confirm the accuracy and reliability of those reports. Weaknesses include the following:
- DFID has just begun a trial survey to verify job creation figures, delivered by the bilateral components of the Enterprise and Employment Programme. This comes more than three-quarters of the way through the project’s implementation period, in spite of a requirement for independent verification of results at the time of approval;93
- validation is important to deter misreporting, especially in places like Afghanistan where data are hard to collect and often unreliable. This was a problem for the Growth Programme, where one implementing agent reported excellent results that did not hold up to closer scrutiny by independent review;94
- there are insufficient data to assess whether the Growth Programme is meeting its intended outcome to increase ‘potential for licit economic growth’;95 and
- the proposed baseline survey and annual assessment for the Rural Development Programme did not take place, which means no accurate assessment of intended job creation and income increases can take place.
2.53 The absence of validation was taken into account in our scoring of impact because we are unable to determine, credibly, whether DFID’s targets have been met.
Inappropriate indicators and targets further limit our ability to fully assess impact for the case study projects
2.54 DFID states that ‘impact is not intended to be achieved solely by the project. This is a higher-level situation that the project will contribute towards achieving’.96 A well-defined indicator should, however, track whether positive changes are occurring at the impact level (with the assumption that a project has contributed to these changes) and indicate when adjustments to the project need consideration.97
2.55 Largely due to inappropriate use of indicators and their targets, we are severely limited in our ability to assess the progress against DFID’s impact targets for our case study projects. DFID selected inappropriate indicators for four of our case study projects. Specific problems include:
- Rural Development Programme: DFID had no mechanism for measuring the programme’s impact on poverty or on the illicit cultivation of poppies in Helmand. Poverty data were neither collected before, during or after the programme and the effects of global wheat and opium prices eclipsed any programme effects and ensured that the beneficial effects could not be attributed solely to the programme;
- Growth Programme: impact indicators are not specific, relevant or measurable. DFID’s plans to commission consultants to collect periodic milestone data did not happen.98 Only perceptions of improvements in economic opportunities were measured, neglecting quantified measures of self-assessed income or livelihood status (which were available from the perceptions survey used by the Growth Programme);99
- Road Project: the impact indicators measure neither ‘economic growth’ nor ‘growing opportunities’, which were the stated goals.100 Instead, one indicator measures local government legitimacy,101 using the DFID-funded Helmand-wide perceptions survey.102 The second indicator was for the size of population living near the new road, which is also inappropriate to measure economic growth in Helmand. In addition, for this indicator, a baseline and targets were never populated, therefore this indicator was never used;
- Employment and Enterprise Programme: the chosen indicators of GDP growth rate and private investment, as a percentage of GDP, are influenced by, amongst other things, the changing security context. The use of GDP as an indicator by donors, more generally in Afghanistan, has been widely criticised – both because it is a poor choice of indicator and because the quality of the reported metrics is very low, variable and opaque.103104 DFID is left unable, therefore, to separate the impact of its project from the influence of other factors at this high level. DFID cannot learn whether these activities are effective tools to achieve the desired impact in Afghanistan. ABIF, a component of the same programme, has recorded no improved income opportunities; and
- Demining Project: this is the only project for which we can assess DFID’s intended impact and only because a livelihoods survey was conducted in year five of the project not from the project’s intended monitoring.105 The late decision to conduct a survey, while commendable, meant that there was no livelihoods baseline available against which to measure change (i.e. the knowledge of the situation before demining took place is less reliable, as it had to rely on the memory of interviewees).
2.56 We also found poor choices of results indicators and targets (the level below DFID’s definition of ‘impact’). For example, while the Zardozi project, (part of the Enterprise and Employment Programme) delivered income benefits to intended beneficiaries, interviewees in Kabul reported no change in their time contribution to the labour market – the basis of DFID’s job creation calculation.106 In another case, DFID failed to agree on the outcome targets for the ABIF project (part of the same programme); results are yet to be measured.
2.57 The inappropriate nature of the indicators that were selected by DFID contributes to our scoring of impact because we are unable to determine whether these case study projects are contributing, meaningfully, to growth and livelihoods in Afghanistan. As a result, DFID loses the possibility to attribute impacts to its efforts in Afghanistan and, with this, loses an opportunity to learn what is effective in this challenging environment. Further analysis of existing data, establishing validation processes and additional fieldwork could fill these gaps.
More should be done to ensure sustainability
Our case study projects show variable quality of planning and design for long-term sustainable impact
2.58 The Road Project should provide long-term benefits. The head of Helmand’s Department for Public Works told us that he is confident these roads can be maintained, so long as the security situation holds and state funding is available at current levels. We heard from the intended beneficiaries of the Herat Demining Project how they have been provided with long-term freedom of movement to access services and develop livelihoods. The HALO Trust has shown it is likely to be able to continue operations under worsening scenarios. The Growth Programme implementing agents stated that they will also continue under worsening conditions but will be forced to withdraw further into urban strongholds if insurgent activity precludes field activities with beneficiaries.
2.59 There are questions, however, about the long-term sustainability of the Employment and Enterprise Programme. The lack of adequate long-term planning raises two future concerns:
- the mid-term review of Zardozi (the support to female workers component of the Enterprise and Employment Programme) recommended that the implementing agent step back to enable beneficiaries to manage their own business affairs and become self-sufficient. From our interviews with implementing partners, it is evident that this is unachievable in the timeframe of the current project and would require an extension;107 and
- the likelihood that ABIF (the challenge fund component of the Enterprise and Employment Programme) will meet its already extended deadline to complete all awards by December 2014 is low. Even if this is achieved, which is unlikely based on round one experience, DFID will require monitoring capacity, post-2014, to assess the impact that this innovative tool is capable of delivering in Afghanistan.
2.60 As well as these specific examples, it will be increasingly important that all growth and livelihoods programmes are able to deliver sustainable impact in a post-drawdown state. This may require rethinking approaches to delivery, creating more flexibility and having appropriate review points and options available to adapt to conditions.
Learning
Assessment: Green-Amber
2.61 This section reviews the preparations being made by DFID to ensure that it can continue to tackle poverty in Afghanistan in 2014 and beyond. It also reviews the areas where we observed learning at strategic and project levels.
Preparations for post-2014 impact
Earlier preparations would have given DFID more flexibility to respond to an uncertain future
2.62 International military drawdown was announced in January 2010 and began formally in July 2011 but DFID has only recently begun, formally, to consider how an uncertain future will affect its project portfolio. DFID’s projects can take up to 24 months to be designed and approved. Earlier planning would have allowed DFID to have a clear understanding as to whether and how current projects can be delivered under the most likely futures.
2.63 The projects being designed that we were able to review do not yet adequately include considerations as to how they would operate under different futures or do not set out how the projects will need to adjust to different scenarios. This is something DFID is aware of and it was highlighted in a recent workshop, convened to consider likely futures.
2.64 It is likely to become increasingly difficult to design a new project portfolio in a future of greater risk and uncertainty. For example, reduced access will limit further DFID’s ability to interact with intended beneficiaries and provincial governments to determine future needs.108 These changes make it all the more important that the projects currently being designed are made sufficiently flexible and robust to be able to operate under likely futures.
DFID is taking initial positive steps to consider its post-2014 approach and portfolio
2.65 DFID has recently started a process to adapt to a post-2014 future. Many of the actions underway reference our previous review of Afghanistan in 2012 as the catalyst for action.109 DFID intends to assess the current situation, develop a set of scenarios and consider priorities and constraints. Work underway or completed includes:
- Scenario forecasting: internal UK Government workshops have produced a range of potential scenarios. DFID’s intention is to ensure that all future projects state clearly what they would do under these likely futures. The workshop revealed, however, that staff were not clear how to modify projects to meet future scenarios. We encourage DFID to fill such capacity gaps. Scenarios, alone, have limited use. It is, therefore, important that DFID uses them to inform both the design of new projects, so it is clear how they would be required to operate under different futures; and for project reviews, to test whether further adjustments are needed;
- Review of humanitarian aid: DFID’s current preparation of a new business case to double its allocation to humanitarian aid suggests that it is preparing for the potential of a worsening future. This can also be considered a response to known problems and past criticism that DFID Afghanistan has been underfunding humanitarian projects.110111 We support DFID’s plans to move its humanitarian aid to a multi-year, multilateral approach, based on learning from the past and from an assessment of future risks;
- Conflict sensitivity: DFID Afghanistan intends to audit projects for conflict sensitivity – which we welcome – and has recently increased its advisory capacity on conflict issues. In our view, however, this should have been routine throughout the period of DFID’s activity in Afghanistan. The process aims to understand better how interventions interact with the operating context, in order to avoid negative impacts and maximise positive results;
- Monitoring: a study has been commissioned to determine whether the UK needs to do more to improve its monitoring in Afghanistan. Many donors have already concluded that more should and can be done (and will need to be, given the likely reductions of access),112 for example, using both more local engagement and innovative use of technology.113 DFID intends its study to learn from these examples;
- Evaluation: DFID Afghanistan produced an ambitious Evaluation Strategy in May 2012, which aims to ‘put evaluation at the heart of DFID Afghanistan’s portfolio development’;114 and
- Resources and skills: we saw evidence that DFID Afghanistan has considered how different futures might impact staffing. There was also a recognition that it may get much harder for staff to ‘get beyond the compound’ and improve their understanding of the operating environment. DFID has also started to improve staff skills115 and awareness of monitoring, evaluation and methodological design (such as through training in ‘logframe’ design). High turnover and relative inexperience of staff are, however, ongoing challenges.
2.66 DFID Afghanistan also has plans to refresh its strategy. We were presented with the following set of potential strategic aims:
- A focus on equity: in terms of gender, location and ethnicity (other documentation reviewed also suggested a greater emphasis on poverty reduction, as a central determinant of equity);
- A focus on jobs: in line with current objectives and DFID’s corporate aims;
- A more ‘balanced portfolio’: fewer, larger projects and a greater focus in particular areas, such as infrastructure, as well as more options that will not rely on input from the Government of Afghanistan; and
- Projects that tackle violence against women: a priority of the current Secretary of State.
2.67 This set of aims does not yet appear to derive from a top-down, clearly defined goal. Importantly, it does not draw from the available evidence base which, for jobs, growth and livelihoods interventions in Afghanistan, is very weak. The evidence that does exist is usually of low quality, with ‘outputs privileged over impacts’ and focussed on design and not on the experiences of intended beneficiaries.116
2.68 While it is now widely accepted that conflict retards economic growth and that higher poverty levels are linked to a greater likelihood of conflict, relatively little is known about the specific mechanisms though which these processes operate. More complicated still is that, while livelihoods and growth share a complex and slightly ambiguous relationship, they are not necessarily positively linked or mutually supporting.117
2.69 Job creation, in particular (although a seemingly sensible response to the large numbers of young people entering the workforce), has no evidence base to support the link that employment creation will promote poverty reduction or increase stability in conflict-affected states.118 In addition, given how difficult job creation is in non-conflict affected states, it may be wise to exercise caution over the extent to which this guides a future strategy, especially relative to other options with a stronger evidence base.
Learning from experience in growth and livelihoods
Our focus projects demonstrate little formal learning from other comparable contexts but there were good examples of learning within Afghanistan
2.70 Opportunities for our focus projects to learn from the experiences in other fragile and post-conflict states, at either the project design or delivery stages, were not taken. Little was formally learned, for example, from aid delivery in Iraq: an acknowledged missed opportunity.119
2.71 DFID has, however, demonstrated a willingness and ability to learn from its activities in Afghanistan. There are examples of responses to design flaws and delivery weaknesses in our case study projects. In these instances, DFID took appropriate and measured action to change direction and closed or moved specific components following annual reviews, especially for the Growth Programme.
2.72 At a strategic level, DFID has learned that single crop substitution for poppy cultivation does not work. Detailed learning from counter-narcotics programmes dates from at least 2008 and, therefore, while not explicit, it seems the design of both the Rural Development Programme and the Growth Programme were trying to avoid past mistakes and go beyond crop substitution to promote alternative livelihoods to opium poppy growing. For example, the Growth Programme uses vouchers to allow farmers to purchase inputs at a subsidised rate.
2.73 We found interesting examples of learning in the Helmand PRT. Both military and civilian staff talked about how they had, through trial and error, markedly improved the way they worked together, particularly within the Helmand PRT base. We also recognised the innovation that was attempted in establishing the Helmand Monitoring and Evaluation Programme (HMEP). Based in the PRT, this Helmand-wide perceptions survey monitored the implementation of the Helmand Plan (the shared plan for international donors and the military). HMEP data helped to inform a substantial revision of the plan. Figure 9 describes HMEP’s main features.
2.74 Few of our case study projects demonstrated or documented any formal learning. The Demining Project is a notable exception. This project clearly built on learning between its two phases, in particular, expanding its objectives to include livelihood and economic growth targets following criticism by an independent evaluation. We observed good collaboration between DFID and its implementing partner, The HALO Trust, in conducting a livelihoods survey to identify and define explicit targets in response to the evaluation. To monitor future impact, a livelihoods monitoring survey, with a baseline and follow-up survey, is being developed. In addition, The HALO Trust’s performance is reviewed on a quarterly basis by the national demining agency.
The Helmand Monitoring and Evaluation Programme (HMEP) is a highly innovative attempt by DFID to measure progress against the Helmand Plan using a quantitative perception survey. The survey baseline was undertaken in 2009 and is added to on a quarterly basis. The project will continue until 2014 when the PRT closes.
Whilst suffering some initial problems, it has played an important role in the redesign of the Helmand Plan and has provided some useful insights into the perceptions of the population of Helmand.
HMEP staff consists of contractors working for Coffey International Ltd.120 Staff learned useful lessons that should be shared more widely. For example, independent validation should have been used at an earlier stage, indicators designed to measure a province-wide, multi-agent plan should not be used to monitor the impact of individual projects and more control should have been exercised on the use of the data (particularly political pressure to generate ‘good news stories’).
3 Conclusions and Recommendations
Conclusions
3.1 DFID has implemented a range of projects as part of its role in the UK presence in Afghanistan. These had a considerable focus on growth and livelihoods, especially within Helmand province. The projects have been conducted against the backdrop of a very difficult security and political situation in a culturally and geographically complex environment. We have, however, found less strategic perspective to the balance of the portfolio than expected and limited evidence that the tight coherence of the programmes to integrated outcomes, which meet beneficiary needs, has been in place. Top-down policy directives, pre-existing programmes and limited on-the-ground options have skewed and reduced the focus.
3.2 Once international military forces have left Afghanistan by the end of 2014, meeting the humanitarian and development needs of the poor is likely to become even more difficult. It is vital, therefore, that DFID selects the right mix of projects for this context – projects that focus on the needs of intended beneficiaries and that are based on sound evidence. This, in turn, requires up-to-date information on the performance of existing projects to learn what is working best and why.
3.3 We reviewed five case study projects which aimed to support infrastructure and to improve livelihoods of the poor in Afghanistan. While the projects had strong development-oriented objectives, their weak project design and insufficient attention to country context – especially consultation with intended beneficiaries – diminished the prospects of success. The projects’ combined role in DFID’s aim of a viable state was also unclear.
3.4 We found mixed success in our case study projects, including some very good results. The Demining Project and the Road Project exceeded their targets. The Rural Development Programme partly achieved its targets. We could not determine the results of the Growth Programme, due to lack of suitable data. The Employment and Enterprise programme is off track with two of its three targets.
3.5 The positive results were supported by our independent fieldwork. We saw evidence of significant changes to intended beneficiaries’ livelihoods. The Demining Project has improved physical security and employment opportunities, provided greater access to services and increased ease of movement. The Road Project has improved market access, created new job opportunities and brought more (and faster) transportation, with a resulting increased access to services and greater security. One component of the Growth Project has enabled farmers to grow higher-value crops and improved household incomes.
3.6 DFID has not always been able to supply us with reliable, up-to-date targets and results. Most of the results published by DFID were unverified. This opens them to challenge and limits DFID’s ability to know how its money was spent. In addition, DFID’s use of inappropriate indicators made it difficult to ascertain whether the projects are contributing, meaningfully, to DFID’s high-level impact targets to improve growth and livelihoods in Afghanistan. With this, DFID loses an opportunity to learn what is effective in this challenging environment.
3.7 The HALO Trust, which delivers DFID’s Demining Project in Herat and STRE, which delivered the Road Project in Helmand, stand out for their strong delivery performance in high-risk environments. Both organisations work effectively with government and have developed trusted community relations, confirmed by our independent fieldwork.
3.8 Other projects were delivered less effectively. The Rural Development Programme required significant monitoring and project redesign as a result. Also, some private sector implementing partners proved unreliable, particularly those implementing components of the Growth Programme. This demonstrates the importance of ensuring that implementing partner competencies are fully assessed in relation to operations in conflict situations and that assistance is provided to build capacity, if needed.
3.9 We were not convinced that the way DFID used umbrella programmes in Afghanistan was, on balance, positive. Although such methods of delivery should facilitate streamlined project management they, instead, affected programme coherence and complementarity of activities, ultimately reducing their potential impact.
3.10 Our focus projects demonstrate little formal learning from other comparable contexts but there were good examples of strategic-level learning within Afghanistan, including how civilian and military staff can best work together.
3.11 DFID has begun to think about how it needs to adapt both its approaches and strategy after the forthcoming security transition is complete. Initial steps are encouraging but incomplete and there is a risk that this work is unco-ordinated and inadequate to the scale of the challenges ahead.
3.12 It will be increasingly important that all growth and livelihoods programmes are able to deliver sustainable and verifiable impact in a post-drawdown state. This requires rethinking approaches to delivery, creating more flexibility and having appropriate review points and options available to adapt to conditions.
Recommendations
3.13 Three recommendations address the areas we consider most important to ensure that future projects have the best chance of changing the lives of the poorest in Afghanistan. Collectively, they aim to help DFID to find the right mix of projects which are focussed on intended beneficiaries’ needs, assess impact and provide the information required to learn and to design better projects.
Recommendation 1: DFID should formally review current and future projects to focus its portfolio more firmly on reducing poverty using evidence-based interventions. This should be completed within six months using a consultative and evidence-led process.
3.14 We understand that there are plans to review DFID’s overall strategy for Afghanistan. We support the intended approach to narrow DFID’s remit and focus on those in greatest need. This will, in turn, make fulfilment of this recommendation easier (see paragraph 2.66).
3.15 Building on the work DFID has already begun, it should map current and pipeline projects against its strategic objectives to identify limitations, gaps and opportunities to focus specifically on reducing poverty in Afghanistan, subject to access restrictions.
3.16 Using the results of this exercise, DFID should analyse how projects relate to and complement one another. It should identify which projects are at risk of failing to deliver under worsening scenarios and where gaps in coverage exist for new projects to focus on reducing poverty in Afghanistan.
3.17 Existing projects should be adapted to ensure that they are able to operate under likely futures. This will require rethinking approaches to delivery, ensuring adequate flexibility to changes in the operating environment and building in appropriate review points that are informed by scenario planning. This process would enable DFID to consider future project options as combinations that support one another, spread risk and gather evidence for future project design.
3.18 All future project designs should ensure, for a range of likely scenarios, that project objectives are realistic and flexible. They should be based on a clearly articulated theory of change which uses evidence to demonstrate that the chosen design is the most appropriate way to reduce poverty.
Recommendation 2: DFID should ensure that intended beneficiaries are, as far as practicable, directly consulted when new projects are being designed, so that their needs are clearly addressed and their ownership is enhanced.
3.19 Given the probable future in Afghanistan, it is all the more important that DFID focus future projects on the needs of the poor.
3.20 Because its staff are likely to be subject to continued and greater movement restrictions, DFID needs to engage competent Afghan research and development institutions to build a network of local partners, who have the confidence and ability to access rural and urban beneficiary groups, collect qualitative and quantitative data to an acceptable standard and inform and advise on project design.
3.21 DFID should consider not moving ahead with a project if it has not been able to engage sufficiently with beneficiary groups and secure local ownership.
Recommendation 3: DFID should enhance its approach and commitment to independent monitoring in order to assess current and future project performance and to allow it to assess the impact of its programme.
3.22 Given that monitoring projects will very probably become harder over the next few years, it is even more important for DFID to make the necessary improvements to its monitoring systems. A pragmatic, balanced approach should be taken to improve independent, real-time monitoring, as well as longer-term evaluations. Enhanced monitoring comes at a cost. It is, therefore, essential that sufficient resources are allocated to meet this recommendation.
3.23 In its 2012 Evaluation Strategy, the use of local partners for future monitoring is recommended by DFID to overcome problems of access. DFID should publish a plan, within the next six months, that places Afghan organisations at the heart of its future monitoring strategy and, with it, make a long-term commitment to build the monitoring capacity of its implementing partners.
3.24 New projects should not be approved until they assess how performance against their intended targets will be robustly measured. These results will be important, not just to enable DFID to track progress and prove impact but also to build an evidence base of what works in aid programmes in fragile and conflict-affected states.
Annex
This annex provides more detailed background information to the review. This includes:
- Annex A1: Assessment of our case study projects;
- Annex A2: The Rural Development Programme;
- Annex A3: The Growth Programme;
- Annex A4: The Employment and Enterprise Programme;
- Annex A5: The Herat Demining Project;
- Annex A6: The Road Project; and
- Annex A7: List of interviewees.
Sections A2-A6 summarise the five case study projects, within which each of the following are covered:
- details of the project’s purpose, activities, targets and reported results;
- details of our fieldwork (where applicable); and
- additional recommendations (where applicable).
Footnotes
1 There is no generally accepted definition of ‘livelihoods’ but we have used the following by Young et al (2002), building on Scoones (1999), as a useful guide: ‘the ways in which people access and mobilise resources that enable them to pursue goals necessary for their survival and longer-term wellbeing and thereby reduce the vulnerability created and exacerbated by conflict’.
2 The Department for International Development: Programme Controls and Assurance in Afghanistan, ICAI, 2012, http://icai.independent.gov.uk/wp-content/uploads/2010/11/ICAI-Afghanistan-Final-Report_P1.pdf.
3 For a clear timeline of conflict-related events, see http://www.bbc.co.uk/news/world-south-asia-12024253.
4 Sarah Collinson and Mark Duffield, Paradoxes of presence, Risk management and aid culture in challenging environments, Overseas Development Institute (ODI), 2013, http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8428.pdf.
5 Livelihoods, basic services and social protection in Afghanistan, Working Paper 3, Adam Pain, Secure Livelihoods Research Consortium, 2012, http://www.refworld.org/pdfid/523ad6124.pdf.
6 The Insurgency in Afghanistan’s Heartland, Asia International Crisis Group, 2011, http://www.crisisgroup.org/~/media/Files/asia/south-asia/afghanistan/207%20The%20Insurgency%20in%20Afghanistans%20Heartland.
7Afghanistan Annual Report on Protection of Civilians in Armed Conflict, United Nations Assistance Mission in Afghanistan, 2013, http://unama.unmissions.org/Portals/UNAMA/human%20rights/Feb_8_2014_PoC-report_2013-Full-report-ENG.pdf.
8 World Bank Governance Indicators, http://info.worldbank.org/governance/wgi/index.aspx#countryReports.
9 Statistical Yearbook 2012-13, Afghanistan Central Statistics Office (CSO).
10 In this report, we have used pounds sterling figures where possible. If figures are only available in a foreign currency, we have converted using the applicable average annual exchange rate from http://www.oanda.com/currency/average.
11 The World Bank, GDP per capita (current US$), 2011 (latest data available), http://data.worldbank.org/indicator/NY.GDP.PCAP.CD.
12 A good review of the difficulties and wide variation of existing estimates is: Anthony Cordesman, The Afghan War in 2013 Meeting the Challenges of Transition: Volume II Afghan Economics and Outside Aid, Center for Strategic and International Studies, 2013.
13 2012 Human Development Index, as reported in Human Development Report 2013, United Nations Development Programme, 2013, http://hdr.undp.org/sites/default/files/reports/14/hdr2013_en_complete.pdf.
14 Based on UN estimates: Common Humanitarian Action Plan 2013, Mid-Year Review, https://docs.unocha.org/sites/dms/CAP/MYR_2013_Afghanistan_CHAP.pdf.
15 Afghanistan Common Humanitarian Action Plan 2013, United Nations Office for the Coordination of Humanitarian Affairs (OCHA), 2013, https://docs.unocha.org/sites/dms/CAP/2013_Afghanistan_CHAP.pdf.
16 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
17 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
18 National Capacity Needs Self-Assessment (NCSA) for Global Environmental Management, United Nations Environment Programme, 2009, http://www.unep.org/dgef/Portals/43/publications/Afghan_NCSA_and_NAPA_2009.pdf.
19 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
20 Data provided by DFID, based on World Bank estimates.
21 Afghanistan in 2011: A Survey of the Afghan People, The Asia Foundation, 2011, http://asiafoundation.org/resources/pdfs/TAF2011AGSurvey.pdf.
22 Doing Business 2013, World Bank, 2013, http://www.doingbusiness.org/~/media/GIAWB/Doing%20Business/Documents/Annual-Reports/English/DB13-full-report.pdf.
23 Corruption Perceptions Index, Transparency International, 2012, http://www.transparency.org/cpi2012/results.
24 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
25 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
26 In 2003 the international community committed to implementing Afghanistan’s National Drug Control Strategy: An Updated Five-Year Strategy for Tackling the Illicit Drug Problem, 2006, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/36443/fco_nationaldrugcontrolstrategy.pdf.
27 Afghanistan in Transition: Looking beyond 2014, World Bank, 2013, https://openknowledge.worldbank.org/bitstream/handle/10986/13107/758480PUB0EPI0001300PUBDATE02028013.pdf?sequence=1.
28 Afghanistan Opium Survey 2013, United Nations Office on Drugs and Crime (UNODC), 2013, https://www.unodc.org/documents/crop-monitoring/Afghanistan/Afghan_report_Summary_Findings_2013.pdf.
29 For an overview of the history of the military intervention in Afghanistan, see: The War in Afghanistan, BBC History, http://www.bbc.co.uk/history/the_war_in_afghanistan.
30 The Afghan Northern Alliance, known officially as the United Islamic Front for the Salvation of Afghanistan, was a multi-ethnic group united in its desire to overthrow the ruling Taliban. The Northern Alliance was backed by the United States and, after the US-led invasion and establishment of the new administration in 2001, the Northern Alliance broke apart, with different political parties forming.
31 The ISAF is a NATO-led security mission in Afghanistan, established by the United Nations Security Council in December 2001 to train the Afghan National Security Forces (ANSF) and assist Afghanistan in rebuilding key government institutions. On 11 August 2003, NATO assumed leadership of the ISAF operation. See: http://www.isaf.nato.int/.
32 World Economic Outlook 2013: Transitions and Tensions, International Monetary Fund (IMF), 2013, http://www.imf.org/external/pubs/ft/weo/2013/02/pdf/text.pdf. Other estimates exist and as such figures should be treated as low-quality.
33 Afghanistan in Transition: Looking beyond 2014, World Bank, 2013, https://openknowledge.worldbank.org/bitstream/handle/10986/13107/758480PUB0EPI0001300PUBDATE02028013.pdf?sequence=1.
34 DFID Understanding Afghanistan: Growth Diagnostic: Technical, DFID, 2008, http://www.hks.harvard.edu/fs/drodrik/Growth%20diagnostics%20papers/DFID%20Understanding%20Afghanistan%20Final%20Technical%20%28word%202003%29.pdf.
35 Livelihoods, basic services and social protection in Afghanistan, Working Paper 3, Adam Pain, Secure Livelihoods Research Consortium, 2012, http://www.refworld.org/pdfid/523ad6124.pdf.
36 Country Programme Evaluations Conducted in Fragile States, ITAD, 2010, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67709/syn-cnty-prog-evals-frag-sts.pdf.
37 For example, see DFID’s detailed political economy review: Understanding Afghanistan, Strategic Conflict Assessment Final Report, The Recovery and Development Consortium, November 2008, http://www.york.ac.uk/media/politics/prdu/documents/publications/pub.Afghanistan%20Conflict%20Assessment%20Nov2008.pdf.
38 Eyes Wide Shut: Counter-Narcotics in Transition, AREU, 2013, http://www.areu.org.af/Uploads/EditionPdfs/Opium%20BP.pdf.
39 Winning Hearts and Minds? Examining the Relationship between Aid and Security in Afghanistan, Feinstein International Center, 2012, http://fic.tufts.edu/assets/WinningHearts-Final.pdf.
40 Afghanistan in Transition: Looking beyond 2014, World Bank, 2013, https://openknowledge.worldbank.org/bitstream/handle/10986/13107/758480PUB0EPI0001300PUBDATE02028013.pdf?sequence=1.
41 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
42 Education Management Information System Statistical Analytical Report 1390, Islamic Republic of Afghanistan, 2011-12, http://www.moe.gov.af/Content/files/MoE_1390_Stat_Analysis_Final.pdf.
43 National Risk and Vulnerability Assessment 2011-12, Afghanistan Living Conditions Survey, Central Statistical Office of the Government of the Islamic Republic of Afghanistan, 2014, http://cso.gov.af/Content/files/NRVA%20REPORT-rev-5%202013(1).pdf.
44 DFID Afghanistan Operational Plan 2011-2015, DFID, 2013, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/209876/Afghanistan1.pdf. Note: we have subsequently been informed by DFID that its budget for 2013-14 was reduced, in December 2013, by £11.25 million. The figure in the text takes into account this reduction.
45 House of Commons Library, The cost of international military operations, Standard Note SN/SG/3139, 2012.
46 Based on the budget of 2013, the year of commitment.
47 For details, see http://www.artf.af/.
48 Contested transitions: International drawdown and the future state in Afghanistan, Norwegian Peacebuilding Resource Centre (NOREF), 2012, http://www.peacebuilding.no/var/ezflow_site/storage/original/application/b327769d97adb4c5c1376febaf6b3579.pdf.
49 ‘Drawdown’ refers to the withdrawal of international military troops from Afghanistan.
51 IMF World Economic Outlook Database, 2013, http://www.imf.org/external/pubs/ft/weo/2013/02/weodata/index.aspx.
52 Hamid Karzai was interim President from December 2001-04.
53 Graeme Smith and Seth G. Jones, Prospects for Afghanistan in 2014, Council on Foreign Affairs, 2013, www.cfr.org.
55 Throughout this report we will use ‘theories of change’, or its singular variant, as shorthand for this range of design methodologies.
56 The Special Inspector General for Afghanistan’s Reconstruction (SIGAR) conducts audits, inspections and investigations to provide oversight of the United States’ expenditure on relief and reconstruction programmes in Afghanistan. See http://www.sigar.mil.
57 The illicit economy is dominated by opium cultivation for heroin production.
58 Terms of Reference: DFID’s Bilateral Support to Growth and Livelihoods in Afghanistan, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Afghanistan-ToR-final.pdf.
59 See https://www.gov.uk/uk-forces-operations-in-afghanistan.
60 We were unable to conduct female interviews for the Road Project, as selected villages were too far from the comparatively safe provincial capital and security of the female researcher could not be guaranteed. Helmand is significantly more conservative with regard to gender relations and more insecure than Herat.
61 DFID Afghanistan Operational Plan 2011-2015, DFID, 2013, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/209876/Afghanistan1.pdf.
62 We have subsequently been informed by DFID that, as of February 2014, its portfolio in Afghanistan consists of 25 projects.
63 Senior Officials Meeting Joint Report, Tokyo Mutual Accountability Framework (TMAF), 2013, http://mof.gov.af/Content/files/TMAF_SOM_Report_Final_English.pdf.
64 Afghanistan National Development Strategy 2008-2013, Islamic Republic of Afghanistan, 2008, http://www.undp.org.af/publications/KeyDocuments/ANDS_Full_Eng.pdf.
65 DFID was warned of the high costs of the Growth Programme, which were a function of over-ambitious objectives (Growth Programme Approval Email 19/08/2010).
66 Helmand Growth Programme Project Document, DFID, 2010.
67 ABIF is an example of a challenge fund. This is a competitive grant-making facility that DFID uses in Afghanistan to support private enterprises that are deemed able to deliver goods and services that provide benefits to intended beneficiaries.
68 DFID formally introduced the ‘theory of change’ in December 2010. Before this, DFID developed a results chain to demonstrate the links between project activities and impact for intended beneficiaries. For simplicity, we use the term ‘theory of change’ throughout.
69 Project Document, Supporting Employment and Enterprise Development, DFID, 2009, http://devtracker.dfid.gov.uk/projects/GB-1-200898/documents/.
70 By the time the Helmand Growth Programme (HGP) was designed, Making Markets Work for the Poor (M4P) principles were well established in DFID’s policy thinking.
71 See http://www.theguardian.com/world/2014/jan/21/kidnappers-seize-57-afghan-mine-clearance-workers.
72 A six-month retention period appears to be common for DFID road construction projects in Afghanistan; for example, the Road Rehabilitation and Maintenance Programme applies a similar retention period. See Annual Review, Road Rehabilitation and Maintenance Programme, DFID, 2013 (review undertaken 2 June – 21 July 2013).
73 Helmand Agricultural and Rural Development Programme Project Memorandum, DFID, 2006.
74 HARDP Project Completion Report, DFID, 2010.
75 Cost and Time Submission, HARDP, DFID, 2012.
76 Review of the Helmand Improved Livelihoods and Economic Opportunity Programme, Coffey International Development, 2013.
77 Final Evaluation Report: Developing Access to Skills and Trade for Growth of Agri-Business in Helmand, AA International, 2013.
78 The Employment and Enterprise Programme, for example, includes support to Turquoise Mountain. The project’s budget from DFID, for a three-year period, was £121,000. Interviews confirmed that this is not the scale of project that DFID would have supported as a single budget line.
79 Cost and Time Submission (HGP), DFID, 2012.
80 The Department for International Development: Programme Controls and Assurance in Afghanistan, ICAI, 2012, http://icai.independent.gov.uk/wp-content/uploads/2010/11/ICAI-Afghanistan-Final-Report_P1.pdf.
82 Examples include weak component design and assessment for the Bost Agri- Business Park, the Flexible Fund, the hybrid Sharia compliant loan product and the biomass project, all of which were, ultimately, cancelled or transferred to other programmes. Helmand Growth Programme Annual Review, DFID, 2012.
83 A project has a target outcome (which it is expected to deliver) and a target impact (to which it is expected to contribute). Each of these targets, if achieved, represents the provision of benefits to intended beneficiaries. We discuss the difficulties in assessing impact in paragraphs 2.52-2.57.
84 Both the Rural Development Programme and the Growth Programme focussed on poppy eradication. Positive impact was claimed when poppy cultivation was shown to have declined from 69,000 hectares (2006) to 65,000 hectares (2010); subsequently, production has increased, especially onto land in the Helmand desert (the ‘dasht’).
85 Perception survey results are compared to DFID targets of 35% and 25% respectively. See paragraphs 2.14 and 2.55 for a discussion of the weaknesses of these indicators to assess a ‘vibrant and licit’ Helmand economy; and paragraph 2.35 for comment on insufficient revision of objectives following programme modifications.
86 It is too soon to measure Phase II results so the table reports only on the completed Phase I.
87 This figure was supplied to us by the Herat Demining Project, based on casualty reporting that came through after the DFID project completion report was finalised.
88 Country Programme Evaluation: Afghanistan, Evaluation Report EV696, DFID, 2009, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67734/afghan_eval.pdf.
89 Department for International Development: Operating in insecure environments, NAO, 2008, http://www.nao.org.uk/wp-content/uploads/2008/10/07081048es.pdf.
90 Synthesis of Country Programme Evaluations Conducted in Fragile States, Evaluation Report EV709, ITAD, 2010, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67709/syn-cnty-prog-evals-frag-sts.pdf.
91 Afghanistan: Development progress and prospects after 2014, International Development Committee, 2012, http://www.publications.parliament.uk/pa/cm201213/cmselect/cmintdev/403/403.pdf.
93 The original project document stated that the project would ‘monitor progress […] independent from implementing partners’.
94 The implementing partner reported that the (self-) employment rate was 84% for training graduates but an independent verification survey produced lower results (47%). See HGP Annual Review, DFID, 2012, http://devtracker.dfid.gov.uk/projects/GB-1-201023/documents/.
95 UNODC reported an increase in net poppy cultivation from 123,000 hectares in 2010 to 154,000 hectares in 2012. World Drug Report 2013, UNODC, 2013, http://www.unodc.org/unodc/secured/wdr/wdr2013/World_Drug_Report_2013.pdf.
96 The results chain must be evidence-based to take account of lessons learned, evaluation and research evidence. This allows DFID to identify realistic targets and to determine how much change can be expected from an individual project. How to Note, Guidance on using the revised Logical Framework, DFID, 2011, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/253889/using-revised-logical-framework-external.pdf.
97 For example, if a project is designed to increase employment then the unemployment rate could be a suitable impact indicator. As the rate moves DFID is provided with a sense of project performance and a trigger to consider whether it should increase or reduce its efforts in this area.
98 DFID planned to commission monitoring and evaluation at the outset for six-monthly monitoring reports, 12- and 24-month project reviews, a post-project evaluation and a full impact evaluation 12 months after project completion. DFID HGP Project Memorandum, July 2010.
99 Ian Robinson, Final Evaluation Report: Developing Access to Skills and Trade for Growth of Agri-Business in Helmand, AA International, 2013.
100 Intervention Summary: Road Improvement Central Helmand (RICH), DFID, 2010.
101 The term ‘legitimacy’ refers to people’s beliefs about the rightfulness of political authority. Survey respondents were asked to what extent they agreed with the statement: ‘Over the last 3 months in your District, the District Government has improved infrastructure and conditions to roads in my area.’
102 See http://www.coffey.com/our-projects/helmand-monitoring-and-evaluation-programme-hmep-future.
103 This is a problem reported in the academic literature of many projects by different donors in Afghanistan. See: Richard Mallett and Rachel Slater, Growth and Livelihoods in Fragile and Conflict-Affected Situations, ODI, 2012, http://r4d.dfid.gov.uk/pdf/outputs/slrc/slrc-wp9.pdf.
104 Anthony Cordesman, The Afghan War in 2013 Meeting the Challenges of Transition: Volume II Afghan Economics and Outside Aid, Center for Strategic and International Studies, 2013.
105 Project Year Five (of Five): Livelihood Report, HALO Trust, 2013.
106 Intended beneficiaries, interviewed in Kabul, stressed that they worked no more and no fewer hours than before Zardozi support. DFID will soon begin to verify Zardozi’s results.
107 DFID has received a concept note for a second phase of support to Zardozi. If DFID supports a second phase, there is a possibility of good impact sustainability.
108 Our discussions with the Special Inspector General for Afghanistan’s Reconstruction reinforced this point: access to emergency medical facilities falls with a declining military presence. SIGAR has begun to map potential future areas of operation for US staff, based on maximum accepted distances from remaining medical facilities. It is clear from its early work that, as the drawdown proceeds, access will fall rapidly.
109 The Department for International Development: Programme Controls and Assurance in Afghanistan, ICAI, 2012, http://icai.independent.gov.uk/wp-content/uploads/2010/11/ICAI-Afghanistan-Final-Report_P1.pdf.
110 Humanitarian Emergency Response Review, March 2011, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67579/HERR.pdf.
111 The International Development Committee report (2012) recommended that ‘DFID do much more to meet humanitarian needs and address the underlying causes of the crisis. More of DFID’s budget should be spent on disaster mitigation’.
112 For example, USAID personnel can only travel within a specified distance of hospitals providing adequate trauma care, which are currently mainly based in PRTs which are closing, severely reducing the ability for staff to travel, irrespective of conflict.
113 USAID, for example, has very ambitious technology-focussed plans, while CIDA (Canada) and GIZ (Germany) are establishing independent monitoring units. The World Bank, in its delivery of the ARTF, has demonstrated effective and continued use of a third-party monitoring agent; a third-party supervisory agent (that uses smart phones, satellite imagery and innovative technology to systematically monitor developments financed under key ARTF-funded projects); and three independent evaluations of the fund have been conducted since its inception.
114 DFID Afghanistan Evaluation Strategy, DFID, 2012.
115 The 2012 International Development Committee report recommended that ‘DFID create a cadre of experts with knowledge of Afghan language and culture’.
116 A very useful resource is the following systematic review which has a very wide-ranging bibliography of further reading (this paragraph draws from these findings): Richard Mallett and Rachel Slater, Growth and Livelihoods in Fragile and Conflict-Affected Situations, Working Paper 9, ODI, 2012, http://r4d.dfid.gov.uk/pdf/outputs/slrc/slrc-wp9.pdf.
117 Richard Mallett and Rachel Slater, Growth and Livelihoods in Fragile and Conflict-Affected Situations, Working Paper 9, ODI, 2012, http://r4d.dfid.gov.uk/pdf/outputs/slrc/slrc-wp9.pdf.
118 What is the evidence on the impact of employment creation on stability and poverty reduction in fragile states, a systematic review, Working Paper 9, ODI, 2013, http://www.odi.org.uk/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8386.pdf.
119 Similarly, Rt. Hon. Andrew Mitchell MP, when shadow Secretary of State for International Development, called for a full investigation into all aspects of British involvement in Iraq, including the aid and reconstruction effort, in order to learn lessons for their involvement in Afghanistan. Birmingham Post, July 2009.
120 See http://www.coffey.com/our-projects/helmand-monitoring-and-evaluation-programme-hmep-future.
Abbreviations
| ABIF | Afghanistan Business Innovation Fund |
| ACCI | Afghanistan Chamber of Commerce and Industry |
| ANDS | Afghanistan National Development Strategy |
| ANSF | Afghan National Security Forces |
| AREDP | Afghanistan Rural Enterprise Development Programme |
| AREU | Afghanistan Research and Evaluation Unit |
| ARTF | Afghanistan Reconstruction Trust Fund |
| CARD-F | Comprehensive Agriculture and Rural Development Facility |
| CDC | Community Development Council |
| CIDA | Foreign Affairs, Trade and Development, Canada |
| CSO | Central Statistics Office, Afghanistan |
| DAIL | Department of Agriculture Irrigation and Livestock |
| DASTGAH | Developing Access to Skills and Trade for Growth of Agri-Business in Helmand |
| DFID | Department for International Development |
| DPW | Department of Public Works |
| EU | European Union |
| GDP | Gross Domestic Product |
| GIZ | Deutsche Gesellschaft für Internationale Zusammenarbeit (formerly GTZ) |
| HALP | Helmand Alternative Livelihoods Programme |
| HARDP | Helmand Agriculture and Rural Development Programme |
| HASP | Helmand Agriculture and Rural Development Support Programme |
| HGP | Helmand Growth Programme |
| HMEP | Helmand Monitoring and Evaluation Programme |
| HNIA | Helmand National Investors Association |
| ICAI | Independent Commission for Aid Impact |
| IDP | Internally Displaced People |
| IED | Improvised Explosive Device |
| IMF | International Monetary Fund |
| ISAF | International Security Assistance Force |
| M4P | Making Markets Work for the Poor |
| MACCA | Mine Action Coordination Centre of Afghanistan |
| Ministry of Agriculture, Irrigation and Livestock | |
| MPW | Ministry of Public Works |
| MRRD | Ministry of Rural Rehabilitation and Development |
| NAO | National Audit Office |
| NATO | North Atlantic Treaty Organization |
| NCSA | National Capacity Self-Assessments |
| NGO | Non-governmental organisation |
| NOREF | Norwegian Peacebuilding Resource Centre |
| NRVA | National Risk and Vulnerability Assessment |
| NSC | National Security Council |
| OCHA | United Nations Office for Coordination of Humanitarian Action |
| ODI | Overseas Development Institute |
| OECD | Organisation for Economic Cooperation and Development |
| PAT | Portfolio Assessment Tool |
| PDP | Provincial Development Plan |
| PRT | Provincial Reconstruction Team |
| RICH | Road Improvement in Central Helmand |
| SEED | Supporting Employment and Enterprise Development |
| SIGAR | Special Inspector General for Afghanistan’s Reconstruction |
| SME | Small and Medium Enterprise |
| STRE | Specialist Team of Royal Engineers |
| TMAF | Tokyo Mutual Accountability Framework |
| UN | United Nations |
| UNAMA | United Nations Assistance Mission in Afghanistan |
| UNDP | United Nations Development Programme |
| UNMAS | United Nations Mine Action Service |
| UNODC | United Nations Office on Drugs and Crime |
| US | United States |
| USAID | United States Agency for International Development |
| UXO | Unexploded Ordnance |
| WOCCU | World Council of Credit Unions |