The role of cash transfers in reducing poverty and vulnerability

Purpose and rationale for review

This impact review will assess the results that DFID has achieved with its cash transfer (CT) programmes, and the extent to which these results are likely to be sustainable. The review will probe the credibility of DFID’s results claims and assess their significance. It will assess DFID’s use of and addition to the evidence of CTs contribution to reducing poverty and vulnerability. Value for money of DFID’s CT programmes will also be examined.

The review will provide insight into whether DFID’s CT portfolio represents a credible response to target 1.3 of the Sustainable Development Goals, which is to ‘implement nationally appropriate social protection systems and measures for all, including floors, and by 2030 achieve substantial coverage of the poor and the vulnerable’.

Type of review

This portfolio was chosen for an impact review because it is mature enough to have generated a substantial amount of results data. There is also a large body of evidence of what works available in the literature. In impact reviews, ICAI conducts a thorough assessment of the validity of DFID’s results claims and of the significance of a portfolio’s development impact. This review also explores the quality of the systems that DFID uses to capture evidence, and whether the portfolio is using this and other evidence to maximise its results over time.

Scope

Defining ‘cash transfers’

In line with DFID’s own definition, we define cash transfers as “all regular cash transfer payments made to individuals and households to tackle poverty and vulnerability”. This includes child-support grants, non-contributory pensions, wages from employment on public work schemes, seasonal payments and other transfers to poor households and particularly vulnerable categories of people (eg disability grants, widow’s allowances, universal basic income grants). CTs are non-contributory. They can be unconditional or conditional upon other development objectives, such as school enrolment, health clinic visits or work on labour-intensive infrastructure programmes.

Definition from: DFID (Oct 2014) Indicator methodological note template: Cash transfers

This review examines the provision of CTs for the purposes of reducing poverty and vulnerability. Both terms are interpreted widely to do justice to the range of DFID’s CT objectives, which includes those related to schooling, health, nutrition, resilience and women’s empowerment. The review also examines DFID’s support to national social protection systems and programmes. Most of DFID’s system support relates to CT delivery systems. It is a portfolio review covering the period since the beginning of DFID’s Results Framework 2011 (DRF)1. The review will also take account of elements of previous CT work undertaken, insofar as they helped to shape CT work done in the review period.

CTs are used increasingly as a form of humanitarian assistance. We hope to address humanitarian cash transfers in the future, but have excluded them from the scope of this review as our primary interest is in the development of sustainable national social protection systems.

DFID provided us with a list of 28 development programmes that focused on CTs during the 2011-2015 review period,2 plus its current portfolio. Of these, 19 reported results against DFID’s DRF indicator on CTs, which is the ‘number of people benefiting from DFID-supported cash transfer programmes’. This indicator is grouped under the DRF pillar of ‘poverty, vulnerability, nutrition and hunger’, and DFID’s target is to reach six million people in at least one of the years of the review period. Of the other nine programmes, six are limited to providing technical assistance and other forms of systems strengthening support to social protection programmes and/or systems. Two programmes are too recent to be included in the DRF results, and the results of one programme are captured under a different DRF pillar. These 28 programmes provide the sampling frame for this review (hereafter ‘the portfolio’).

The review will look in particular at DFID’s achievements in two areas:

  • The impact of CT programmes on CT recipients, their households and communities.
  • The development of sustainable national social protection systems. DFID’s work may have contributed to building the capacity, scale of coverage, scalability in response to shocks, targeting or inclusiveness of such systems. This may have been achieved by establishing a supportive policy and legal environment, attracting national government resources and building broad political backing.

Background

The importance of reliable social protection systems that reach the poorest and most vulnerable people and communities was underscored at the Addis Conference on Financing for Development in July 2015.3 It has been adopted as target 1.3 for achieving Goal One of the Global Goals.4 The commitment to ‘leave no one behind’, which is embedded within the Global Goals and endorsed by DFID, also adds weight to the social protection agenda.

This formal high-level recognition comes at a time when the volume and reach of CT (an important component of the social assistance part of social protection – see Figure 1)5 is growing fast in both low and middle-income countries.

Figure 1: Types of social protection: Social Care Services, Social Insurance, Social Assistance, Labour Policies. Cash transfers fall within 'social assistance' which is one form of social protection.
Figure 1: Types of social protection

DFID’s support for social protection has focused on CTs. Following the 2006, 2008 and 2011 White Papers, but without an explicit CT strategy, DFID’s CT contributions have grown from £53 million in 2005 to an annual average of £219 million over the review period.6 This equates to 2.6% of DFID’s overall spend in this period (2010/11-2014/15 figures), which funded operations in 19 countries. Most of this funding directly financed the delivery of CTs, but there was also a significant allocation to technical assistance and systems-building work to improve national CT and wider social protection systems.

DFID is a major contributor to CT-related work of the multilateral development banks, the European Commission and the UN (not covered in this review), and has played a significant role in the funding, commissioning and dissemination of CT-related research and evaluation. Over the review period, DFID’s research plans prioritised the development and promotion of strong monitoring and evaluation (M&E) practices; the embeddedness of M&E within national strategies for collecting and analysing socio-economic data; the strengthening of cost-benefit analysis to allow for an effective comparison across different instruments and design options; and tackling frontier issues in research on CTs. In response to changing priorities of DFID and its partners, these frontier issues changed over the course of the 2011-2015 period and included CT application in fragile and conflict-affected contexts, the role of social protection in building effective states, financial inclusion and the role of social protection in responding to crises.

Theories of change

DFID does not have an overall theory of change for its CT or wider social protection portfolios. Instead it develops theories of change for individual programmes, on the basis of national host government and DFID priorities.

A 2015 literature review drew on several of DFID’s programme-level theories of change. It presented one that is neither final nor comprehensive, but that does help to understand some of DFID’s common thinking underpinning its CT work.

The starting point of this theory of change is a framework that lists livelihood-related constraints and enablers at household, local and national level. At individual and household level, CTs cause or contribute to three sequential ‘orders of outcomes’: first-order outcomes such as ‘expenditure on education’; second-order outcomes such as ‘school enrolment, attendance and retention’; and third-order outcomes such as ‘school learning, performance and progression’. The theory states that these outcomes are facilitated by CTs through household-level changes in time and risk preferences, household dynamics and gender relations.

At the community level, the theory of change states that a system of CTs can cause or contribute to changes in local labour markets and the economy, and in social networks.7 The aggregate level results are ‘poverty and inequality reduction; productivity and growth; social relations and cohesion’.

The framework acknowledges that all these results are influenced by design and implementation factors, such as the level of transfers, associated conditionalities, targeting, payment systems, links to complementary interventions and social accountability mechanisms.

This theory of change reflects some of the thinking behind several of DFID’s programme-specific theories of change. In addition, some programmes have theories of change that focus on the development of national social protection systems. Others have incorporated CT and the national social protection system development within bigger change processes that lead to increased well-being for particularly disadvantaged populations.

Existing evidence

There is a substantial and growing empirical body of evidence related to social protection in general and to the impact of CT in particular. Much of the research is publically available and has been captured in a number of literature reviews.8 The latest – a forthcoming Cash transfer rigorous literature review from ODI – identifies 199 credible studies.9

From 1990 to 2010, most evidence came from Latin America and the Caribbean. However, the rapid growth of CT programmes in Africa in the past five years has changed this. This is set to continue as CT programmes and systems in Africa continue to generate evidence.

The evidence suggests that CTs can have a range of positive impacts on school enrolment and attendance, use of prenatal and postnatal health care, children’s physical development, and food security, income and productivity. There is compelling evidence to show that the size and coverage of transfers, and the quality of targeting, are key choices affecting impact. However:

  • There is some evidence of negative impact as well (eg CT programmes increasing spousal abuse and child abuse and neglect).
  • The strength of evidence varies considerably across types of impact. In part, this reflects the changing objectives of programmes implemented since the 1990s, as well as the developing interests of development agencies, academics and national governments. By far the largest number of impact studies examines savings, investment and production, as well as changes to income poverty. Fewer studies and evaluations look at the design and implementation features of programmes that helped to create such impacts, or at targeting mechanisms or levels of accountability in CT programmes.10

Review questions

The review is built primarily around the OECD-DAC evaluation criteria of impact and sustainability. It will consider the following questions and sub-questions:

Q1: Impact: To what extent have DFID’s CT programmes contributed to reductions in poverty and vulnerability?

Q1.1: To what extent has DFID ensured maximum value for money for its CT programming?
Q1.2: How well has DFID made use of and contributed to evidence on what works to enhance the impact of CT programmes?

Q2: Sustainability: To what extent does DFID maximise the sustainability of its impact?

Q2.1: How well is DFID ensuring that its work has maximum and sustained impact on CT recipients?
Q2.2: How successfully is DFID supporting the development of sustainable, nationally-owned social protection systems?

Core ICAI issues

As this review explores the impact and sustainability of cash transfer programmes and social protection systems as well as the targeting of development programmes on the extreme poor and marginalised, it is closely linked to the ICAI theme of ‘leaving no-one behind’. In addition, the review explores two core ICAI issues.

  1. Approaches to programme delivery. This review will provide insights into:
    • The manner in which DFID uses theories of change, data and contextual analyses to identify programmatic entry points, select delivery channels, and shape the design, implementation and evaluation of programmes.
    • The nature of DFID’s guidance for and practice of its programme cycle management, and programme implementation, support and learning.
  2. Quality and use of evidence to enhance development impact. As per review question 1.1, this review will provide insight into the manner in which DFID makes use of and contributes to evidence on what works to enhance the impact of a programme portfolio.

Overview of methodology

Four main methodological elements will jointly provide the data and insights required to answer the review questions at both programme and portfolio level (see Figure 2). This section covers them sequentially.

Figure 2: Overview of methodology

Figure 2 shows how the different elements of the methodology interact. The desk reviews provide broad insight into how CT programmes are designed and managed so as to maximise impact, sustainability and value for money. The country case studies allow for in-depth exploration of CT programming, in particular national contexts, to gain insight into how CT fits within broader national poverty reduction strategies and other programming. They also assess DFID’s contribution to building sustainable national systems. The literature review and external interviews provide an external perspective, to challenge DFID’s approaches and choices. Together, these methodological elements allow for an assessment of the three levels of analysis: 1. a review of a sample of CT programmes; 2. a review of the CT portfolio as a whole; 3. an assessment of the way in which this portfolio is embedded in higher DFID agendas. We present these methodological elements in turn.

1. Desk reviews

These consist of DFID interviews and a review of documents such as business cases, logframes, baseline studies, political economy analyses, annual reviews, project completion reports and independent evaluation reports. We will capture data and insights in a review assessment framework that is grounded in the review questions, and analyse DFID’s views on CT-related approaches, choices, work and results. The desk reviews will contribute to each of the three levels of analysis. Specifically:

A review of a sample of CT programmes. These programme desk reviews focus on the programmes’ impact on poverty and vulnerability and other selected objectives and on the likely sustainability of this impact.11 In addition, we will assess issues such as the use of and contribution to data, evidence and knowledge; the causalities assumed in implicit and explicit theories of change; learning and reporting; and the guidance behind and practice of the management of the various elements of the programme cycle. We will review 15 programmes, covering approximately a third of DFID expenditure of the CT programme portfolio that DFID has provided us (see section 11 for the sampling approach).

A review of the CT portfolio as a whole includes:

  • Choices, trends and current practice around strategic thinking, theories of change, policies and guidance.
  • The division of responsibilities for managing the portfolio and the staff capacities for doing so.
  • Knowledge management and learning systems and processes around CTs, including DFID’s CT-related research portfolio management, the way DFID collects, synthesises and disseminates evidence from internal and external sources on what works, and the way DFID uses this evidence to learn and improve practice.
  • The provision of technical support to country offices.
  • The guidance around DRF indicators and other results and value for money measurements.

An assessment of the CT portfolio against higher DFID agendas. Through interviews and the comparison of CT and higher-level documents, we will assess the match between choices made in the CT portfolio and DFID’s high-level agendas such as the prioritisation of the rights of girls and women, and the ‘leave no one behind’ principle.

2. Literature reviews

These reviews will draw on systematic evaluations and syntheses of those evaluations. In combination with the team’s own expertise, these reviews will help to assess two levels of analysis:

A review of a sample of CT programmes. We will assess the alignment of programme design and modalities with good practice principles; the way programmes use and contribute to CT’s knowledge bank; and the credibility of results claims.

A review of the CT portfolio as a whole. We will assess the way the portfolio aligns with, and contributes to, evolving insights on what works.

3. External interviews

On the basis of an initial list and subsequent snowball sampling, we will conduct interviews with a range of stakeholders from academia, civil society, programme implementers, partner governments, other supporting governments, and multilateral partners. These interviews will add depth and context to the literature reviews. They will ensure that critical voices are heard and that alternative practices are considered. They will also support the contribution analysis by triangulating causal links and by identifying and assessing rival theories that may explain results claimed.

Depending on the stakeholders, these interviews will support the review of a sample of CT programmes, and/or the review of the CT portfolio as a whole. At both levels, these interviews will help to assess DFID’s knowledge contributions and stakeholders’ use of these contributions.

4. Country case studies

We will conduct two country case studies (see section 11 for the selection criteria). The subject of each is DFID’s entire CT-related country portfolio. Both country case studies will provide in-depth insights that expand and complement the review of a sample of CT programmes. The case studies will enable us to:

  • Assess the role of country-level CT portfolios within DFID country strategies.
  • Assess DFID’s support to national social protection programmes and systems. The assessment includes DFID’s policy advocacy, technical assistance and financial support to such systems, and a probing into the sustainability and level of national ownership of these systems.
  • Test the quality, accountability and results claims of DFID-funded CT programmes.

These country case studies will provide in-depth insights through gaining access to data that are only available locally, and feedback that can only be obtained through face-to-face interviews and discussions. Specifically, these case studies will enable us to:

  1. Understand the role of country-level CT portfolios within higher DFID agendas and in-country operational plans. We will also assess where CTs fit within high-level strategies for tackling extreme poverty and marginalisation, such as the SDGs and the ‘leave no one behind’ agendas. We will explore links between CT programmes and complementary interventions.
  2. Explore the quality of DFID’s policy advice and technical assistance. As part of a light-touch contribution analysis, we will gather feedback from government counterparts and external observers on the relevance, effectiveness, impact and staging of DFID’s policy advice and technical assistance to support the development of national social protection systems.
  3. Probe the sustainability of national social protection systems and the quality of the data that underpin these systems. Through a review of country-specific data and literature, and through interviews with a range of in-country stakeholders, we will explore the robustness of and remaining bottlenecks for sustainable national systems. This will include exploring financial sustainability, the level of political support for sustaining the required resource allocation and the feasibility of the proposed institutional model in the national context. Governance and accountability mechanisms and the adequacy of national poverty statistics in supporting effective targeting of CTs will also be examined. We will assess the extent to which DFID has identified the key shortcomings and to which DFID or other stakeholders are taking credible action to help national governments to address them.
  4. Assess the credibility of results claims, issues of inclusiveness, and value for money measurements. On a sample basis, we will:
    • Trace the reported number of CT recipients against the DRF target back to results data from individual programmes, and vice versa.
    • Assess the extent to which DFID has adequate monitoring arrangements in place to verify data reported by its implementers.
    • Explore the validity of programme-specific results claims by triangulating them against stakeholder feedback from non-recipients, recipients and frontline workers, and nationally available data sources.
    • Assess programmes against the ‘leaving no one behind’ principle by examining how programmes manage issues of remoteness and intersectionality (ie the way in which individuals and groups are subject to overlapping and reinforcing vulnerabilities and systems of discrimination).
    • If these analyses exist, validate and check the programmes’ Economy, Effectiveness, Efficiency and Equity analyses, cost-effectiveness analyses and/or cost-benefit analyses, on the basis of DFID’s value for money guidance and considerations of economies of scale.
  5. Assess the quality of accountability mechanisms. During visits to CT-receiving communities, in which we will interview CT recipients, non-CT recipients and front-line workers, we will assess the extent to which end-of-pipeline accountability and complaints handling mechanisms are functioning. We will also test the implementation of targeting and CT modalities. This will support the overall analysis on value for money.

Review framework

Four main methodological elements will jointly provide the data and insights required to answer the review questions at both programme and portfolio level (see Figure 2). This section covers them sequentially.

Evidence requiredComponentsMethodological notes
Question 1: To what extent have DFID's CT programmes contributed to reductions in poverty and vulnerability?

In relation to DFID's Results Framework:

  • Results measurement protocol
  • Evidence related to quality of data underlying reported results.
  • Reported results, compared to DFID's overall CT reach target.
  • Evidence of relation between the reach indicator in DFID's Results Framework, and impact on poverty and vulnerability and other selected objectives.
  • Evidence related to the programmes' positive and negative effects that are covered in impact evaluations.

Literature review and interviews to assess the rationale that underpins DFID's measurement protocol.

Document review and country case studies to:

  • Assess the extent of data protocol adherence, reliability and validity of the data, and data gaps; assess the aggregation of programme results and performance against overall reach target.
  • Assess the broader impact of the CT portfolio on poverty and vulnerability, by reviewing impact evaluations from sample programmes and drawing out recurring impacts and analysis on why programmes have been more or less impactful.
  • Explore positive and negative effects that are not captured in the DRF.

Compare the results measurement protocol with evidence from literature, and assess the practical challenges related to protocol adherence.

In the countries visited: use upwards and downward tracing techniques to test the data sets underlying the reported contributions to the DRF results and their aggregation into a single reach number.

Consider the impact on people reached, using evidence from a programme sample that includes each of DFID's most common CT-related objectives.

Question 1.1: To what extent has DFID ensured maximum value for money for its cash transfer programming?

DFID's 2013 document titled Guidance on measuring and maximising VfM in social transfers.

Programmes' value for money exercises and programme-level data related to the various metrics mentioned in this guidance document (for sample programmes for which these exercises exist).

Evidence related to the extent to which value for money issues receive on-going and systematic attention throughout programme life cycles.

Evidence related to the various ways in which programmes minimise and mitigate risks.

DFID and external interviews, document reviews and country case studies to:

  • Assess the rationale behind DFID's guidance on value for money assessments.
  • Validate and check DFID programmes' value for money (including economies of scale) 3E and 4E calculations and conclusions, where such calculations exist.
  • Check to what extent value for money is an on-going theme in the course of project life cycles and in the treatment of risks and governance and accountability mechanisms.

In the sample programmes, where they exist, validate and check of DFID's 3E and 4E analyses, cost-effectiveness analyses and/or cost-benefit analyses, on the basis of DFID's own value for money guidance and considerations of economies of scale.

Interrogate decisions taken in the course of programmes' life cycles against value for money principles and concerns.

Map programmes' governance and accountability arrangements, and test them on a sample basis.

Question 1.2: How well has DFID made use of and contributed to evidence on what works to enhance the impact of CT programmes?

DFID's use of evidence:

  • Evidence of use of evidence in business cases, theories of change and programme design.
  • Evidence of DFID's evidence-based contextualisation of programmes, including geographical data and key documents of partner governments.

DFID's contributions to evidence:

  • Evidence of an appropriate level of resourcing for M&E and learning components of CT programmes.
  • An overview of DFID's own and DFID-financed knowledge contributions related to CT programmes and systems.
  • Evidence on the external use of these contributions.

Literature review to identify:

  • DFID's and the broader sector's key evidence-based learning.
  • Where evidence is strong, and what key evidence gaps still exist.
  • DFID's knowledge contributions.

Document reviews to assess the ways in which, and the extent to which contextualisation and use of evidence are incorporated in DFID's CT guidance and programmes.

DFID and external interviews to:

  • Assess the nature of DFID's knowledge contributions (eg the extent to which they fill knowledge gaps, create or consolidate insights, and/or highlight innovation).
  • Assess use of these contributions by DFID and stakeholders such as international finance institutions, other governments and implementing agencies.

Outline the conclusions of key evidence-based reports, from DFID and the wider sector, and assess DFID's absorption of these conclusions by using textual analysis and interviews to compare these conclusions with:

  • The internal DFID discourse.
  • DFID guidance and programme documents.
  • The way DFID's CT programmes have evolved in the course of the past decade.

Make an inventory of DFID's knowledge contributions, assess the nature of these contributions (eg focus on innovation, meta-evaluations), and compare this with the evidence gaps as they existed at the time the research was commissioned or conducted.

Assess the use of these contributions by interviewing peers in international finance institutions, other governments and implementing agencies.

Question 2: To what extent does DFID maximise the sustainability of its impact?
Question 2.1: How well is DFID ensuring that its work has maximum and sustained impact on CT recipients?

DFID's high-level agendas.

The rationale behind choices (eg the choice of CTs as part of country portfolios, target groups, CT types, delivery channels); and evidence of cross-fertilisation, positive sum games, positive externalities and influencing/advocacy in DFID's CT portfolio.

Programme-specific business cases and theories of change, and evidence underlying the causal links assumed in these theories of change.

Evidence of links between (1) the programme-specific theories of change and contextual data (including vulnerability and price data) and analyses (including community needs and capabilities), and (2) the design (including the choice of transfer modalities), implementation and evaluation of programmes.

Evidence related to programme governance, support, M&E and learning systems.

Interviews and reviews of DFID's higher level agendas to understand DFID's portfolio choices.

Literature review to assess the research evidence underlying the causal links in the various programmes' theories of change.

DFID interviews and document review to:

  • Assess the extent to which DFID's CT portfolio composition and portfolio choices maximise opportunities for cross-fertilisation, positive sum games, positive externalities and influencing/advocacy.
  • Assess the manner in which DFID has used theories of change, secondary data and contextual analyses to identify programmatic entry points and to shape the targeting, design, implementation and evaluation of programmes.
  • Assess, against common good practice principles, the nature of programme governance, support, M&E and learning systems.

Seek to understand what evidence and which agendas have shaped DFID's CT portfolio. Use Venn diagrams and related tools to map the connectedness and cross-fertilisation and advocacy-related potential of the CT portfolio components.

Categorise information on the basis of the theories of change causal links, risks and assumptions.

Use the iterative approach described by Mayne to assess the strength of the causal links of the explanatory chain, and the extent to which this chain reflects evidence and learning in relation to impactful CT design, risks and assumptions.12

Understand learning and support systems and map actual learning and support by tracing examples of learning and their upward and downward dissemination, and of the provision and use of support.

Map programme choices (eg governance arrangements, objectives, target groups, transfer levels and frequency, conditionalities, M&E design) and trade offs (eg higher transfer costs to reach particularly vulnerable people; the trade off between reach and transfer size).

Compare these choices and trade offs with the focus countries' own priorities and with DFID's higher-level agendas (eg the leaving no-one behind agenda; the promotion of gender equality).

Where possible, conduct sustainability assessments on the basis of post-intervention data. Where data are not available, base the analysis on proxy-methods (see section on limitations, point 4).

Question 2.2: How successfully is DFID supporting the development of sustainable, nationally-owned social protection systems?

Evidence of the extent to which the countries covered by capacity-focused DFID programmes have recently developed or are working towards nationally-owned social protection systems. Evidence of trends in relation to country governments committing HR and/or financial resources to these systems; of the system's capacity development; and of the extent to which social protection has broad political backing and/or is enshrined in legislation or key policy documents.

DFID's theories of change and role in risks and assumptions in relation to the development of sustainable, nationally-owned social protection systems.

Evidence, from literature and DFID assessments, on the strengths and weaknesses of the causal links that involve DFID.

Evidence of the use of political economy analysis and comprehensive institutional capacity assessments to inform the type and scope of support.

Evidence in relation to other influencing factors and rival explanations (ie evidence that explains progress irrespective of DFID's work).

Literature review to assess evidence on causal links in DFID's theories of change related to social protection system development.

DFID and external interviews, document reviews and country case studies to:

  • Explicate the theories of change of national system development (and DFID's role therein) if these theories of change are not explicit already.
  • Assess the extent to which the sampled target countries have recently developed sustainable, nationally-owned social protection systems, and assess the political economy underpinning progress (or lack thereof).
  • Assess the extent to which DFID has made a contribution to the sustainable progress in the development of any such social protection systems.
  • Assess DFID's choices and staging in relation to the nature of the support provided (ie technical, financial or multidimensional support).
  • Assess DFID's influence on CT design and, if relevant, assess DFID's choices in relation to trade offs between evidence-based good practice and national ownership.

Develop one or more sample programmes that provide technical assistance and/or financial support to national social protection systems (see section on sampling strategy) if they do not have an explicit theory of change. This should be undertaken jointly with DFID stakeholders, with the help of a standard visualisation tool. This tool explicates the strategic approach (and DFID's role therein), 1st, 2nd and 3rd order outcomes (and an indication of the causal links from one type to the next), objectives, risks, assumptions, non-DFID influencing factors, and rival explanations.

When the evidence is sufficiently robust, conduct contribution analyses that assess the strength of the causal links in the postulated theory or theories of change and the risks to these links, and explore other influencing factors and rival explanations.

Ideally, a programme knows about a planned contribution analysis from the start, as it requires real time capturing of data that are otherwise unlikely to be captured. This is particularly important for programmes that have advocacy and capacity building components.

Some of the programmes will not have captured the data needed for a thorough assessment of all causal links. The review team will manage this by assessing the available data, and focusing on the causal links for which the data are likely to be most robust.

Sampling strategy

The programme sample criteria

The distinct characteristics of each of DFID’s CT programmes confirm that a ‘standard DFID CT programme’ does not exist. To gain insight into DFID’s CT work the review team therefore needs to assess a sample of programmes that is selected purposively to reflect the full spectrum of programme objectives, modalities, support, types and sizes.

Looking across DFID’s CT work, the team identified the most important distinctions among programmes. These were related to the objectives that DFID’s programmes seek to achieve, the modalities used, the types of support that DFID provides, and the range in size of financial support programmes. These characteristics became the sampling criteria.

To avoid conclusions based on outliers, our sample will consist of programmes that together, cover each characteristic at least twice (eg at least two education-focused programmes, at least two conditional CT programmes). The criteria are as follows, with the number of programmes befitting each criterion added between brackets:

Sampling selection criteriaReview questions
Primary objective: poverty graduation (2), schooling (3), nutrition & health (9), climate resilience (2) and women's empowerment (3).1, 2.1
Modality: seasonal (3) and not seasonal (10); conditional (5) and unconditional (8)1, 2.1
Type of social assistance system support: provision of technical assistance only (2), or provision of CT financing, with or without technical assistance support (13).1.1, 2.2
Significance of DFID contribution: a proportionately large (7) or small (5) DFID contribution (defined as DFID covering more than 80% and less than 20% of total programme funding respectively); and a relatively large (5) or small (2) monetary contribution (defined as DFID contributing more than £50m or less than £15m respectively).1.1

Secondary criteria were as follows13:

  • Implementation period. Programmes that ended before mid-2011 or started after mid-2015 were excluded as they cover an insufficiently meaningful part of the review period.
  • Previous UK government assessments. Programmes that have recently been assessed by ICAI, the IDC or NAO were excluded to avoid duplication of efforts.

The result is a sample of 15 programmes that will be subject to a desk review. Four of them will also be covered by a country case study. Together, these 15 programmes cover approximately 63% of DFID’s spend on CTs and approximately 50% of its results towards the DRF.14 They are as follows:

Sample selectionProgrammes selected
BangladeshEconomic Empowerment of the Poorest (EEP)
BangladeshStrengthening Government Social Protection Systems for the Poor (SGSP)
BurmaLivelihoods and Food Security Trust Fund
EthiopiaProductive Safety Nets Programme Phase 4 (PSNP)
KenyaSocial Protection Programme Phase II
NepalRural Access Programme 3
NigeriaChild Development Grant Programme
PakistanPakistan National Cash Transfers Programme
RwandaTechnical Support to Social Protection in Rwanda
RwandaSocial Protection Support to the Poorest in Rwanda
SahelBuilding Resilience in the Sahel through Adaptive Social Protection
UgandaExpanding Social Protection Programme
UgandaExpanding Social Protection Programme Phase II
ZambiaSocial Protection Expansion Programme
ZimbabweChild Protection Fund for NAP II

The country case study criteria

There are 19 countries in which DFID has undertaken CT-related work in the review period. The team considered which of them would be suitable for country case studies. The criteria were that the country portfolio must be sizeable and include both technical assistance and financial support to the national country’s social protection system. It must have been operational for at least a few years and must not have been the focus of a recent UK government review. The team found that:

  • DFID does not provide significant system support in Burma, Nigeria, Mozambique, Occupied Palestinian Territories, Yemen and Zimbabwe.
  • DFID’s work in Kosovo, Egypt and Vietnam does not include financial support.
  • DFID’s programme in Tanzania is too new for analysis.
  • The cash transfer component in Nepal’s multi-component programme is modest.
  • Other UK government reviews have recently visited Ethiopia, Pakistan and Zambia.

Five countries remained viable options for country case studies: Bangladesh, Ghana, Kenya, Rwanda and Uganda. Of these, the team opted to visit Bangladesh, as it has by far the largest CT portfolio of these five countries (irrespective of the size unit used). The team will visit Rwanda, as it has had the longest system development history of the remaining four countries. This provides an opportunity for an in-depth exploration of the review questions concerning the sustainability of national social protection systems. Both country case studies fulfil the key selection criteria. In addition, the different aspects of DFID’s CT programming in each country that would be analysed would also complement each other in helping to cover a wider spectrum of DFID’s CT work, see Figure 3.

Figure 3: Comparison country case studies

Limitations to the methodology

This review has four key limitations.

  1. This review will identify but not fill gaps in credible evidence. Significant gaps will affect the robustness of the review’s findings. This review was not built into any CT programme’s design, and does not generate substantial primary data to test the causal link between interventions and results. Instead, this impact review draws primarily on existing data. It could potentially mitigate this limitation by conducting primary research – but any such research will be small-scale, and the review team will only consider it in case of data deficits that are both serious and easy to fill.
  2. Findings may not be easy to generalise across DFID’s CT portfolio. There is no ‘standard’ CT programme. The sample selected for this review covers at least two programmes in each category of programmes (see the section on the review’s sampling strategy) but none of these sets of two can be assumed to be truly representative.15 (They are, however, sufficient to enable us to identify recurrent patterns in the portfolio that influence delivery of impact.) The review will mitigate this limitation by considering key findings from the review of the sampled programmes in the light of key findings from the wider CT literature. It also aims to provide a typology of what worked well, where and why.
  3. The assessment of the success with which DFID is supporting the development of social protection systems is likely to be biased towards the effects of financial assistance and the ‘hard’ side of DFID’s technical assistance. The effects of financial support and assistance with, for example, legislative support or support with the system architecture, can probably be captured with reasonable rigour. However, a contribution analysis of DFID’s advocacy and influencing work is challenging, especially in a complex multi-stakeholder context, given that programme teams are unlikely to have collected data on influencing systematically. The evidence of DFID’s role in reaching trigger, tipping and turning points – which could potentially be among DFID’s biggest achievements – can not be captured in real time, and therefore become lost or unverifiable. The team will manage this by assessing the available data, and focusing on the causal links for which the data are likely to be most robust.
  4. Our assessment of the sustainability of impact will, at least in part, be based on theory and evidence from comparable, older programmes. Where possible, sustainability assessments will be based on post-intervention data. In most cases, this data will not (yet) exist, and we will therefore be required to make predictions based on whether programmes are giving sufficient evidence-based attention to sustainability in their design and implementation.

Ethical considerations

This review will use an appropriate, proportional and sound set of methods. It will use them in close consultation with DFID in order to ensure cost effectiveness by selectively testing but not duplicating work that DFID has already done.

The review team is guided by the OECD DAC Quality Standards for Development Evaluation. The team will undertake all interviews on the basis of informed consent and will ensure that the opinions and information incorporated in the review report will not be traceable to individuals, unless the interviewer has obtained prior explicit consent of the individuals in question.16

The perspectives of CT recipients will be collected indirectly, through key stakeholder interviews and by drawing on secondary sources. These perspectives will substantially inform the review findings and the team will ensure that key CT recipients’ perspectives are voiced in the review report. In addition, the team will conduct a number of interviews with CT recipients and non-CT recipients. When conducting these interviews, the team will respect cultural sensitivities and will ensure that it seeks a diversity of perspectives, with an inbuilt bias towards those who are often invisible and unheard. In the unlikely event that the team interviews children, it will conduct these interviews in line with UNICEF’s child-focused ethical evaluation guidelines.

Research tools

We will develop relevant research tools in the early stages of the review. These include:

  • A mapping of DFID and external stakeholders for interview and consultation.
  • Interview guides for key stakeholders, indicating relevant areas to explore.
  • A review framework for desk reviews that allows for a comparison and aggregation of findings.
  • Country case study frameworks covering contextual analysis, list of key stakeholders, sampling of delivery partners and programmes, and detailed plans for field research (to be completed three weeks in advance of the country visit).

Quality assurance and peer review

This review will be carried out under the guidance of ICAI Lead Commissioner, Alison Evans. The Peer Commissioner is Tina Fahm. The quality of the review will be assessed by Pam Vallance, Review Unit, ICAI Secretariat, using OECD DAC evaluation standards.

This review will be externally peer reviewed by Professor Wendy Olsen, a statistician and professor in Socio-Economics at the School of Social Sciences of the University of Manchester. Professor Olsen has reviewed a draft of the Approach Paper (and her feedback has been incorporated in this final version) and will review:

  1. The thematic overviews of the various types of interviews that the team will conduct, once the team has gained sufficient insight in DFID’s CT work to develop these overviews.
  2. The sampling principles that will guide the country visits, once the team has formulated them. This is to ensure it is based on as much team insight as possible.
  3. The draft report.

Risk management

RiskMitigation and management actions
Security concerns cause the cancellation of a country visit.The team has identified Uganda and Pakistan as 'back up' country case studies, in case the Rwanda or Bangladesh country case study is not possible. This could delay the publication of the report, as DFID requires up to eight weeks to prepare for a country case study.
A reliance on DFID documents and DFID staff is perceived by external stakeholders as causing a favourable bias.

The team will prioritise external evaluations in the desk-based programme assessments, and will triangulate DFID's claimed results against other data sources wherever possible.

When arranging interviews, the team will include potentially critical voices from amongst CT recipients (eg non-eligible people) and from amongst other stakeholders (eg from partner governments, academia, civil society).

The team will use non-DFID interpreters, conduct interviews without DFID staff present, use a range of interview techniques to avoid a favourable (or unfavourable) bias, and emphasise confidentiality.17

Timeline and deliverables

Key stagesIndicative timeline
Inception phaseFebruary - April 2016
Data collection and field workMarch-June 2016
Analysis and emerging findingsJune -July 2016
ReportingAugust-September 2016
Publication and disseminationOctober - November 2016

Footnotes

  1. The DRF is a tool that DFID uses to manage and monitor progress for a select number of indicators, and to report publicly on its delivery against these indicators.
  2. The measurement protocol around the CT-related DRF indicator is covered here.
  3. UN General Assembly (2015) Addis Ababa Action Agenda of the Third International Conference on Financing for Development, UN General Assembly A/RES/69/313, 27 July 2015.
  4. Sustainable Development Knowledge Platform (undated), Sustainable Development Goals, United Nations.
  5. Other components of social assistance are in-kind transfers (mostly food), vouchers, and subsidized access to goods and services.
  6. We note that these are tentative figures and may be revised throughout the course of the review.
  7. In the visualisation of this theory of change, there is no line from the CT insertion to these meso-level changes, so perhaps there is no assumed CT contribution to these changes – though the rest of the framework implicitly suggests that there is.
  8. E.g. World Bank (2015) The State of Social Safety Nets 2015, Washington, DC, World Bank; Arnold, C. with Conway T. and Greenslade, M. (2011) CT: Literature Review. London, DFID; and Baird, S.J., Ferreira, F.H.G., Ozier, B., and Woolcock, M. (2013) “Relative effectiveness of conditional and unconditional CTs for schooling outcomes in developing countries: A systematic review”, Campbell Systematic Reviews, 8.
  9. Bastagli, F. et al (forthcoming 2016) Cash Transfers Rigorous Literature Review, ODI.
  10. Bastagli, F. et al (forthcoming 2016) Cash Transfers Rigorous Literature Review, ODI.
  11. The likely sustainability is assessed on the basis of post-intervention data when possible and estimated on the basis of evidence from comparable programmes in other cases – see the section on ‘limitations to the methodology’.
  12. See Mayne, J. (2012) “Contribution analysis: coming of age?” Evaluation, 18:3, 270-280. Note that this iterative methodology may cause modifications in the original theories of change.
  13. The team also considered evaluative evidence as a selection criterion, but was unable to operationalize this criterion because at the time of writing, this information was not yet available.
  14. These are tentative figures, based on incomplete data, and may have to be revised.
  15. The value which can be generalised is likely to be particularly limited in the case of DFID’s support to country governments’ system development. Such support may not have many commonalities across countries, and achievements are likely to be highly dependent on personalities and relationships.
  16. The evidence pack is a deliverable from the Service Deliverer to ICAI. To maximise its value as a quality assurance tool, it will be fully referenced and it will not be anonymised. It is possible to do this whilst maintaining confidentiality, because the evidence pack will not be published. As part of the team’s interview protocol, the team will make this clear to each respondent.
  17. ‘Confidentiality’ defined as ‘only ICAI commissioners, the service provider and secretariat will be able to access the notes of this interview’.