DFID’s Approach to Delivering Impact – Terms of Reference

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement.

1.2 It has always been our intention to look in depth at the complex issue of impact once we had built up a strong body of evidence. Our reports always focus on impact but now with 30 reports published, we believe we have the evidence we need. In our 2012-13 Annual Report,¹ therefore, we stated that, as part of our Year 4 programme, we would undertake a study looking specifically at how best to deliver and monitor impact. We noted the number of internal and external discussions that we have had about the concept of impact, what it means in practice and how to measure it in an international development context. The topic was also highlighted in a number of the responses to our consultation on the Year 4 work plan.

1.3 ICAI has always had a particular focus on ensuring that impact is considered from the standpoint of the intended beneficiaries of the Department for International Development’s (DFID’s) aid programme, those who live in poverty. We have always identified the intended beneficiaries as those whom the aid is ultimately intended to benefit and not those who are sometimes referred to as ‘beneficiaries’ but who are, in fact, other stakeholders or recipients who are involved at various stages in the delivery of the aid. We are also concerned to ensure that these impacts are sustainable and that the aid programme avoids negative impacts. This report builds on all the studies we have conducted to date, looking at the different facets of impact and how DFID has set out to achieve it.

1.4 These Terms of Reference outline the purpose and nature of the review and the main themes that it will investigate. The issue of the impact of the aid programme is a large and complex one. This review is limited to the aid programme provided by DFID, not the broader support provided to the process of international development through, for example, the UK’s trade relationships with developing countries. Within this large topic, we will need to focus our lines of enquiry as set out in these Terms of Reference and future ICAI reports will complement the analysis contained in this report. There are various themes which we will not seek to address in this report. For example, we will not address the specific question of whether aid delivered via multilateral organisations is more or less impactful than aid delivered via other channels. Multilateral organisations are used to deliver some DFID-managed interventions as part of the bilateral programme – this will be covered in the review. We will not, however, address aid delivered via core funding to multilateral organisations. We will be looking specifically at the theme of multilateral aid in a forthcoming report in Year 4 and will compare different delivery channels as appropriate in our future reports. We are also not assessing humanitarian aid as part of this review, given the particular characteristics of this form of assistance. A detailed methodology will be developed during the inception phase.

2. Background

2.1 As set out in the International Development Act (2002), the purpose of development assistance is poverty reduction; furthering sustainable development and improving the welfare of poor people. While achieving impact is a central theme of DFID’s work and it indicates the absolute number of people it intends to assist in different ways in various sectors, the Act itself does not describe what the impact of development assistance should be. DFID subscribes to the definition provided by the Development Assistance Committee of the OECD.² This states that the impacts of development assistance are ‘positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended’. This definition embraces the possibilities of both doing harm and doing good. One of the issues for this review will be to draw a picture of what DFID means by impact and how this relates to the needs of the intended beneficiaries of the aid programme. In addition, we will ask whether the demand for short-term, visible results is at the expense of long-term, structural and sustainable impacts.

2.2 DFID’s current work on impact takes place within a wide global debate around the impact of aid programmes: what the term means for different stakeholders and particularly intended beneficiaries; the different paths that can be taken to deliver aid; and the different ways in which impact can be assessed. What value for money means in an aid context is a related part of this debate. The debate has involved a number of different communities: bilateral and multilateral aid programmes, government and non-governmental organisations and – increasingly – the ultimate beneficiaries of aid themselves. It is important that the way DFID manages its resources to achieve impact responds to this debate and that the approach it takes is relevant, rigorous, effective and provides value for money across different timeframes, types of aid and contexts, including fragile states. The following sections set out what we understand by this.

Relevance: the changing nature of development co-operation

2.3 DFID delivers its aid through a mix of relationships: bilateral (country to country), regional and multilateral (such as via the UN agencies). It does this within a highly complex and changing global environment. The processes of international development help to improve social welfare, to promote economic growth and trade and to alleviate discrimination by ensuring that the provision of global public goods, such as a healthy environment, is equitable. The aid programme, which is provided through a mix of government and non-governmental channels, is only able to address these issues to a limited degree. For instance, the design of social welfare programmes may be supported by UK aid but they are delivered through national governments. Aid can play a part in supporting policies to enhance economic growth and trade but brokering specific trade relationships is outside the official aid programme and economic activity is more likely to be provided by the private sector than the public sector. While the aid programme can be delivered in a way that attempts to reduce discrimination, making policies to reduce inequity over the long term is ultimately a political process that relies on transparent and accountable relationships between citizens and their governments and between governments themselves.

2.4 The official aid programme is, therefore, only a part of this broader picture of development assistance. Nevertheless, there can be close ties between the two, as the aid programme can provide the seedcorn for these wider relationships to develop.

2.5 How aid is delivered is changing. Different aid instruments have been used in different combinations with debt relief, budget support, project aid, multilateral aid and projects, programmes and grants all playing their part in financing the development process. Further changes, however, are taking place: the BRICS³ are developing trade and infrastructure relationships with developing countries, for example, offering counterweights to the traditional ‘northern’ aid programmes and bringing a greater level of Foreign Direct Investment (FDI) with them. Overall, the balance between FDI and aid is shifting, changing recipient governments’ views on what types of external support can most effectively contribute to economic growth. South-South ties have the potential both to complement and to challenge what the UK’s development co-operation programme delivers (the mix of technical assistance, development finance, policy engagement and capacity building), how it delivers aid (the governmental and non-governmental channels through which aid is provided) and where that aid is targeted (towards specific populations or to global, regional or local institutions).

2.6 The geography of poverty is also changing. On current trends, by 2025, five-sixths of the world’s poor are likely to live in Africa, with poverty concentrated in fragile states. Fragility breeds mobility: whether displaced by conflict, economic deprivation or natural disasters, the movement of people within and across national borders is likely to increase, bringing with it both challenges (in the form of social welfare provision) and opportunities (in terms of employment and growth). Understanding what impact means in the context of fragile and conflict-affected states will be an important focus for this review.

2.7 With the shift in economic power to the south and east, the roles of the multilateral institutions on which much aid has depended are increasingly being questioned. Issues of global public goods (for example climate change, financial stability and water) rise up the political agenda in response to current or foreseen crises. Connectivity is changing the ways in which non-governmental and civil society organisations interact with national debates over key poverty-related issues. Furthermore, the rise of private giving (directly or via philanthropic organisations) is offering different opportunities for transmitting development finance from individual donors to individual recipients and communities.

2.8 There is a final factor to consider. Middle-income countries are beginning to exit from aid dependency and develop more equitable trade and political relationships with traditional donor countries. Increasing wage differentials between middle-income and low-income countries could begin to boost manufacturing in low-income countries. Current work on economic growth in low-income countries shows a combination of optimism and pessimism, with islands of rapid growth being seen in otherwise sluggish sectors and regions.

2.9 These global forces, that are transforming the full breadth of development co-operation, will shape the way DFID delivers its aid programme. In this changing world and given the complexity of the endeavour that is international development, aid needs to achieve a real and positive impact for the poor. In addition, the changing media environment means that there is greater pressure for impact to be visible to all of DFID’s stakeholders. UK taxpayers need to be assured that their donations are having a real and lasting effect. In the context of the global developments we have already identified, the recipients and beneficiaries of aid – developing country governments and their citizens – need to know that aid adds value to what they are already doing.

2.10 The central question we want to ask is: what value does DFID’s aid programme bring and how does it create that value, in terms of the impacts it has on its intended beneficiaries (i.e. the poor)?

The different dimensions of impact

2.11 A thorough and careful analysis of impact needs to consider several different elements: the timescale over which impacts are considered, the levels at which impacts are measured, the directness of the impacts and their nature as a public or private good. These are set out below. The question for DFID is how much it chooses to do of each; how that choice is reflected in its strategies and plans for delivery; and how it monitors, evaluates and learns to improve what it chooses to do in future.

2.12 Short- versus long-term timelines, global versus local levels. Discussions about the timelines over which impacts take place tend to get bogged down in definitional debates over ‘inputs’, ‘outputs’, ‘outcomes’ and ‘impacts’. It is important to understand the nuances but, for this review, it will be more important to step back and understand the different timeframes for impact given the important changes in development co-operation identified above and the need for impact to be sustainable in the long term. Similarly, the review offers an opportunity to reflect on the tension between impacts at global and local levels – reflecting DFID’s broad remit to deliver impacts to individual beneficiaries and in the form of global public goods.

2.13 Systemic or direct, public or private impacts. A further issue to consider is the relative weight that DFID should place on delivering impacts directly to beneficiaries or via systemic changes to economies, societies or environments. To what extent should aid be targeted to specific locations where the poorest of the poor are found or would aid be better used to support the provision of some form of national safety net? How far could technological change be harnessed to deliver aid directly to private individuals, as opposed to sharing the benefits of aid across communities? The answers to these questions are complicated by the aid programme (and development support more broadly) being delivered through a huge variety of government and non-governmental channels. This raises the question of how the different channels can best be used to deliver a range of public and private impacts that are deeply embedded in society and government and are, therefore, sustainable.

2.14 Cutting through all of these issues are the questions of who assesses the impacts and what is the role of the aid programme’s intended beneficiaries in that assessment. ICAI reports have already indicated the importance of involving beneficiaries to improve accountability, engagement and sustainability of impacts and the potential for learning. For some aspects of the aid programme, there may be a long ‘impact pathway’ and this review will ask how intermediate organisations are best involved in achieving and assessing impact over time. In addition, robust impact assessment requires highly technical approaches. We will also ask whether the incentives to deliver snapshots of impact at regular intervals foreshorten the understanding of the different types of impact that could be delivered in the long term.

2.15 Finally, there is an important question of how to balance the three-legged stool of accountability for public funds, learning from past mistakes and offering sufficient space for risk-taking and innovation. We will review the approaches DFID uses to deliver the aid programme to see how it balances these different tensions.

DFID’s business tools for achieving impact

2.16 The preceding sections set out some of the tensions DFID needs to navigate as it appraises, implements, monitors and evaluates its impacts. DFID has a long history of implementing and adapting its business tools to improve the impact of its work. The sequence of tools it uses is as follows:

• the results framework, developed in 2013. This is a central part of DFID’s business processes, setting out the development outcomes DFID seeks to contribute to, the results it aims to deliver and the measures it will use to assess organisational effectiveness and efficiency. It comprises four levels of indicators at which delivery can be assessed: development outcomes (linked to the Millennium Development goals), outputs and intermediate outcomes (linked to projects and programmes), operational effectiveness and DFID’s internal corporate processes. The outcomes in the results framework drive the lower-level objectives which DFID programmes and projects seek to achieve;
• strategy and resource allocation processes, which have important implications for how development assistance is targeted and how resources are spent to achieve impact;
• regional and country operational plans. These set out the vision, forward plans and resource mechanisms to achieve impact. Operational plans are updated annually;
• business cases, introduced in 2011 as part of an effort to strengthen the evidence on value for money underpinning investment decisions. Business cases set out the rationale for selecting a particular project, programme or approach to funding following a standard recommended by HM Treasury. We note that the format for business cases is currently under review;
• logical frameworks, used since the early 1990s as a basis for planning, monitoring and evaluating projects and programmes;
• theories of change, introduced in 2011-12 to strengthen the intervention logics set out in business cases and logical frameworks;
• annual and project completion reviews, which are applied to projects and programmes as part of DFID’s ongoing monitoring effort; and
• evaluations which have been taking place throughout DFID’s history at project, programme, sector and thematic levels. Guidance on evaluations has been continually updated over the past twenty years.

2.17 DFID continues to refine and adapt these tools, for example in a current internal ‘End to End’ review of business processes. It also maintains its work to strengthen the organisational attributes that enable its business processes and ensure that DFID uses its resources wisely to achieve impact. These include staff skills and capabilities, transparency of decision-making and the wider culture within the Department. Other ongoing activities include the effort to improve the strength of its evidence base, as set out in the recently issued How-To Note.⁴ We will build on this ongoing work to embed the lessons from the review in our narrative about how DFID engages with the impact agenda.

2.18 Figure 1 on page 6 lists some of the main events that have shaped how DFID uses its business tools. It is not a detailed chronology but does indicate that much of the effort has taken place in the past decade. We will deepen our understanding of this chronology during the review, noting that the development and use of these tools has been influenced both by the global debate on how to measure aid effectiveness and by approaches used in other UK Government departments, which do not, necessarily, meet DFID’s particular needs.

3. Relationship to other reviews

3.1 Figure 2 below sets out our overall report ratings given in the 29 reports we have published to date (excluding our first, which discussed our overall approach and, therefore, did not contain ratings). The standard ICAI guiding criteria and assessment framework are focussed on four areas: objectives, delivery, impact and learning. The criteria are designed to reflect the typical cycle of programme planning and delivery and identify the reasons why impact has or has not been delivered, providing an analysis that supports recommendations for improvement. ICAI’s overall assessment always reflects the impact in the round of the programme under review – the impacts of the programme and how and why these impacts have been achieved through assessment of the clarity of objectives and the effectiveness of delivery and learning.

3.2 There are a number of themes that emerge from our assessments of impact that we will want to explore in more depth as part of this review. These include but are not limited to:

• the nature of impact:
  • defining impacts on the intended beneficiaries;
  • sustainability of impact and avoiding negative impacts;
  • understanding gender impacts; and
  • identifying and overcoming challenges to impact.
• DFID’s instruments:
  • the types of instrument used to deliver aid; and
  • the types of impact these instruments deliver to different groups of beneficiaries.
• DFID’s processes:
  • theories of change and programme design;
  • monitoring throughout the delivery process;
  • evaluations of projects and programmes; and
  • strengthening beneficiary engagement throughout the project or programme cycle, from identification to evaluation.

3.3 This review will also use and develop the findings of our current review of How DFID Learns.⁶ The learning review seeks to identify the way DFID learns, what inhibits it doing better and how the impact of DFID’s work can be improved through learning. We will build on the results of our learning review to look in particular at how DFID learns about impact. This includes looking at DFID’s internal ‘End to End’ review of business processes through the cycle of planning, monitoring and reviewing programmes.

3.4 This review is happening at a time of renewed global interest in the impacts of aid at various levels. As well as the intense public interest referred to earlier and DFID’s internal reviews, there are several ongoing or recently-completed related studies of impact. These have been commissioned by DFID, the European Commission, the Bill & Melinda Gates Foundation, International Development Research Centre (IDRC) and many different project and programme impact evaluations commissioned through international and national organisations such as the International Initiative for Impact Evaluation (3ie), ITAD, Abdul Latif Jameel Poverty Action Lab (JPAL), Innovations for Poverty Action (IPA) and others. None of them share the exact same focus as our review but we will draw on their reports and interviews with key evaluators to situate DFID’s understanding of impact – and the tools and techniques they use – within this wider knowledge.

4. Purpose of this review

4.1 The purpose of this review is to address the questions: from the perspective of the intended beneficiaries – including the most marginalised – how does DFID articulate, define and prioritise impacts; how does DFID try to achieve those impacts; how well does it deliver them and how well does it learn about what is most effective in achieving impact? Within the context of the complexity set out in Section 2, our key questions are:

• How can DFID maximise the impact of the aid programme for its intended beneficiaries?
• How effective are the approaches DFID uses to defining, delivering, monitoring, evaluating and learning about impact? How do they compare to the approaches used by other donor agencies?
• How well are these approaches supported by the business tools and practices DFID uses?
• What part does DFID play in the global debate around impact and aid effectiveness?

5. Analytical approach and outline methodology

5.1 Our overall approach is to consider DFID’s clarity of vision about impact; how well it uses the best practices, tools and techniques at its disposal to address that vision; how this maximises impact for intended beneficiaries; and how it learns from past experience to refine both the vision and the tools.

5.2 Our review will be built in four blocks of work. As well as looking at what is currently happening in DFID to strengthen its impact focus, we will also briefly look at the history and track record of how DFID has engaged with the impact debate to put this in a long-term context.

5.3 A literature review will summarise the state of the global debate on impact, taking a forward look at what impact is likely to mean in future and addressing the tensions and challenges outlined in Section 2. We will use the findings of this review to engage a variety of stakeholders in consultations about impact. This will be an opportunity to ensure that the voices of both government and non-governmental stakeholders and the different types of organisation that deliver the aid programme are able to contribute to the study. A second part of the literature review will summarise the different business tools and processes used in other donor organisations, for comparison with DFID.

5.4 A synthesis of ICAI reports to date will summarise such common themes as have emerged, the good practices identified and the role of DFID’s process tools in delivering impact. They represent a considerable body of knowledge from which this review can draw as it assesses the definitions, tools and delivery channels that the aid programme currently uses.

5.5 A review of documentary evidence from DFID’s own files will give us an overview of the definitions, best practices, insights and process tools that DFID currently uses and how they have changed over time. This will be complemented with interviews and focus groups with staff, to understand how the different approaches have changed over time and how they have worked. We may include a desk review of a random sample of DFID documents to assess the use of the process tools in practice using a checklist methodology and rating system developed by the team.

5.6 The literature review, synthesis of ICAI reports and review of documentary evidence will set the framing for in-country case studies which will be used to examine how DFID tools have been used in practice. We will explore the practicality of including rapid retrospective assessments, seeking to identify projects and programmes that ended between 5 and 10 years ago and tracing the timeline to impact, using a similar methodology to that employed in our review of DFID’s livelihoods work in Western Odisha, India.⁷ This sort of analysis is not routine within DFID to our knowledge. We believe, however, that it could be an important part of our review, complementing our more recent assessments in existing ICAI reports and the forward look at the components of impact in a changing context.

5.7 We will work with DFID to identify practical in-country case studies in addition to our existing body of findings and explore the opportunity to focus on cases where DFID considers the impact to have been successful and lasting. We will seek to identify common issues that limit the achievement of impact, although we recognise that proving the absence of impact is problematic.

6. Indicative assessment questions

6.1 This review will use as its basis the standard ICAI guiding criteria and assessment framework referred to earlier, covering objectives, delivery, impact and learning. A detailed methodology will be developed during the inception phase, setting out the assessment questions and the methods to be used for answering them. Likely questions include:

6.2 Objectives: how does DFID define impact and how does this relate to the needs of intended beneficiaries?

6.2.1 How has DFID defined impact in practice, for different groups of poor people (including the most marginalised) and across different strategies and programmes? Is there an internal consistency to the definitions or do they vary across the Department? If they do vary, why is that?

6.2.2 How are the definitions rooted in the experience of intended beneficiaries? Do the definitions focus on impact over the right scales and time frames?

6.2.3 Do the definitions of impact change with gender, the context (e.g. fragile states), sector (e.g. health, agriculture or governance) or the delivery mechanism (e.g. public / private / third sector)?

6.2.4 How well do the definitions of impact provide the vision that helps DFID develop effective programmes?

6.2.5 How well do these definitions of impact help DFID address the shifting landscape of development outlined in Section 2 above?

6.3 Delivery: given the definitions of impact as outlined above, what are the key components of aid delivery (including good practices, process tools and insights) that help DFID achieve impact for intended beneficiaries and are they fit for purpose and used effectively?

6.3.1 What are the strengths and weaknesses in the different process tools and good practices and what are the implications for how DFID delivers impact?

6.3.2 Are intended beneficiaries involved in the most effective way throughout the development, implementation and evaluation of projects and programmes? Where this does happen and what difference does it make? Does it vary between contexts?

6.3.3 How well are the tools and good practices applied in practice? Does this vary and, if so, why? How do they support or distort effective behaviours?

6.3.4 What balance of effort does DFID put into the different process tools, how has this changed over time and what are the implications for the future?

6.3.5 How effectively do the tools and practices help DFID plan for, manage and assess tangible and intangible impacts and address negative as well as positive impacts (e.g. market distortion)?

6.3.6 How does the corporate results agenda help maximise the impact of the UK aid programme? Where are its strengths and weaknesses?

6.4 Impact: how do DFID’s definitions, business tools and practices help it maximise the impact for intended beneficiaries?

6.4.1 What evidence is there that DFID’s tools and practices have helped deliver impact? Have they contributed to improving impact over time and, if so, how?

6.4.2 How transferable is the evidence over different contexts and sectors? Are there common themes about what works best and least well?

6.4.3 Does DFID’s approach attempt to maximise wider, long-term development results for the citizens of the partner country?

6.4.4 What trade-offs are necessary to balance a focus on impact with recognition of the need for risk-taking and innovation; or to balance the need to demonstrate traceable impact with the need to foster local ownership and local decision-making?

6.5 Learning: how have the experiences of implementing DFID’s practices and tools helped to improve its focus on impact?

6.5.1 What has DFID learned from its experiences of implementing its practices and tools? How has it applied these experiences to its forward thinking?

6.5.2 How will DFID’s current end-to-end review of business processes help improve its focus on impact? What else might be required?

6.5.3 To what extent is DFID leading and learning from the global debate around aid effectiveness and impact?

7. Timing and deliverables

7.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The lead Commissioner will be Diana Good. The review will take place up to the third quarter of 2014 and will be published by the end of 2014.

8. Potential impact of our report on DFID

8.1 Our intention is that this report will help DFID to improve by:

• helping to sharpen the way DFID achieves impact – what it means and how it uses the concept of impact to achieve positive change for its intended beneficiaries;
• identifying specific changes that could be made to DFID’s business processes to help it maximise impact; and
• providing clear examples of the good practices that have helped maximise the impact for intended beneficiaries.

Footnotes

¹ ICAI Annual Report 2012-13, ICAI, June 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Annual-Report-2012-13.pdf.

² Glossary of Key Terms in Evaluation and Results Based Management, OECD/DAC, http://www.oecd.org/dac/evaluation/18074294.pdf.

³ Brazil, Russia, India, China and South Africa: an acronym used to indicate middle-income countries that have graduated or are graduating from aid dependency.

⁴ How to note: Assessing the Strength of Evidence, DFID, February 2013, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/158000/HtN_-_Strength_of_Evidence.pdf.

⁵ Green: The programme performs well overall against ICAI’s criteria for effectiveness and value for money. Some improvements are needed.
Green-Amber: The programme performs relatively well overall against ICAI’s criteria for effectiveness and value for money. Improvements should be made.
Amber-Red: The programme performs relatively poorly overall against ICAI’s criteria for effectiveness and value for money. Significant improvements should be made.
Red: The programme performs poorly overall against ICAI’s criteria for effectiveness and value for money. Immediate and major changes need to be made.

⁶ Terms of Reference: How DFID Learns, ICAI, http://icai.independent.gov.uk/wp-content/uploads/2014/01/ToRs-for-ICAI-Review-on-How-DFID-Learns.pdf.

⁷ DFID’s Livelihoods Work in Western Odisha, ICAI, February 2013, http://icai.independent.gov.uk/wp-content/uploads/2013/12/ICAI-Report-DFIDs-Livelihoods-Work-in-Western-Odisha.pdf.