DFID’s Support to Agricultural Research inception report

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We have decided to review the impact and value for money of the Department for International Development’s (DFID’s) aid programme supporting agricultural research. The programme aims to develop new agricultural products, increase the understanding of agricultural innovation and ensure proven new technologies and products are used to improve the livelihoods, food security and nutrition of the poorest people.

1.3 This Inception Report sets out the evaluation questions, methodology and work plan for the evaluation. It is, however, intended that the methodology and work plan be flexible enough to allow new questions and lines of inquiry to emerge over the course of the evaluation.

2. Background

2.1 The background to this review is described in the Terms of Reference.¹

3. Purpose of this review

3.1 To assess the value for money and effectiveness of DFID’s support to agricultural research and its impact on poor people in developing countries.

4. Relationship to other reviews

4.1 The relationship to other reviews is described in the Terms of Reference.¹

5. Methodology

Analytical approach

5.1 The aim of this review is to assess the impact of DFID’s agricultural research on intended beneficiaries, that is, smallholder farmers and poor and malnourished people in developing countries. We will assess what works and what doesn’t work and how poor people benefit from the programme. We will do this by examining:

• the objectives of the current research portfolio and choice of delivery channels, bidding processes, the approach to risk management and the resultant balance of risk;
• how well DFID uses agricultural research findings to encourage uptake and good policies, both internally and by other organisations;
• the extent to which the agricultural research has led or is likely to lead to improved food and nutrition security for poor people through the scaling up of new technologies and their widespread adoption by smallholders; and
• the extent to which the research agenda targets the priorities of poor farmers in developing countries and is set by them.

5.2 We will undertake:

• a portfolio review to assess DFID’s overall approach to agricultural research;
• a detailed review of seven projects to assess value for money and trace the impacts that follow from generating new technologies in the laboratory to improved livelihoods and nutrition for poor people in developing countries;
• an assessment of impact, through available evidence and by revisiting two previously completed impact assessments; and
• an analysis of how DFID applies learning from earlier programmes, as the current programme builds on decades of earlier DFID support.

Portfolio review

5.3 This will be a review of DFID’s portfolio of agricultural research projects, to assess how DFID balances the need to research and develop products for use by farmers in the short to medium term (two to five years: putting research into action), with higher level research, which takes longer to generate benefits for farmers (5-15 years) but has potentially high pay-offs. We will consider DFID’s use of evidence, its assessment and management of risk and objective setting across the portfolio of projects. We will also assess the rationale for DFID’s agricultural research strategy and how it contributes to – and is guided by – the overall strategy of DFID’s Research and Evidence Division.

Detailed review of current projects

5.4 The DFID agricultural research programme comprises 13 on-going projects. Another large project, the £42.4 million Research Into Use project, was recently completed. We will undertake a detailed review of seven of these 14 projects (see Figure 1 on page 7). The selected projects will enable us to trace the impacts that follow from generating new technologies in the laboratory to improved livelihoods and nutrition for poor farmers. For each, we will assess value for money by reviewing the processes of objective setting, bidding and market testing and the efficiency and effectiveness of chosen delivery channels. We will also examine financial management and performance assessment and the ways in which risk is managed. The detailed review projects have been selected to cover:

• different types of research project (advanced research on global scientific priorities; adaptive research translating known science into technologies for use by researchers and farmers in developing countries; and near-market research and development (R&D) to develop new products for use by farmers);
• different delivery channels (multi-donor trust funds managed by the World Bank, memoranda of understanding with research institutes, accountable grants with not-for-profit organisations; and research contracts with for-profit companies);
• projects of varying size (from £5 million to £120 million; in the case of the grant to the Consultative Group on International Agricultural Research (CGIAR), a global consortium of 15 research centres); and
• high, medium and low risk projects.

5.5 This ICAI review will look explicitly at how DFID’s agricultural research programme supports and influences the CGIAR reform process. Approximately 50% of DFID’s agricultural research expenditure is channelled to this organisation. We will examine the on-going reform programme, examine DFID’s management of the grant and assess the impact of CGIAR research on the ground in Africa.

5.6 Advanced science projects are generally undertaken in developed countries. Adaptive research and near-market R&D takes places in developing countries. Details of each of the selected projects are given in Figure 1 on page 7.

Impact Assessment

5.7 We will examine the evidence on the impact of completed agricultural research projects implemented by DFID’s partner organisations, including CGIAR. We will also assess current efforts to strengthen the capacity for robust impact assessment of CGIAR and other partners, including the International Centre for Insect Physiology and Ecology (ICIPE) and the African Agricultural Technology Foundation (AATF).

5.8 We will revisit previously completed impact assessments of two projects. The first will be the International Food Policy Research Institute (IFPRI) impact evaluation of CGIAR’s HarvestPlus programme, a large-scale intervention to improve nutrition through cultivation of nutritionally-improved sweet potato varieties in Uganda.² The second impact evaluation to be studied will be a smaller one, of the Farm Inputs Promotion Service, which aims to provide advice and inputs to farmers.³ We will verify the findings of the two studies and compare the different approaches used.

Applying learning

5.9 We will assess how well DFID’s agricultural research creates policy-relevant evidence and how well DFID and other organisations integrate these findings into their agricultural development programmes. We will also assess whether the lessons learnt through past evaluations have led to improved programming choices over time.

5.10 The evaluation methodology will comprise the following elements:

Phase 1: Pre-site Assessment

5.11 We will conduct:

• a literature review, focussing on research and evaluations of DFID’s agricultural research programmes and similar research programmes supported by other agencies, including CGIAR and the World Bank;
• interviews, either in person or by telephone, with:
  • DFID’s agricultural research team members and consultants that worked on the design, implementation, monitoring and evaluation of DFID-financed agricultural research projects;
  • scientists and other staff of implementing organisations, including CGIAR, other international research organisations, UK-based research organisations and national researchers in developing countries;
  • DFID agricultural advisors in different countries, to understand how they incorporate research findings into their regular agricultural development programmes;
  • technical assistance consultants who have been engaged by DFID or its partners in the design, implementation and/or evaluation of DFID-supported projects; and
  • third-party experts in agricultural research and development, who have not been involved directly in DFID’s agricultural research work;
• a desk-based assessment of the use of evidence in the design of DFID’s agricultural research portfolio. Using the literature review, we will assess how well project documentation (especially project planning documents⁴) incorporates knowledge and lessons from earlier evaluations. We will assess whether or not there is an appropriate evidence base to support the current design of DFID’s agricultural research portfolio and whether DFID used all available evidence appropriately; and
• a desk-based assessment of the previous evaluations and reviews of the DFID agricultural research portfolio. This will give a view on the quality of the evaluative process in each case, the certainty of results and the utility of the reports, including helping to establish baseline data and how this is used to inform subsequent programming and future evaluations. It will consider how the full range of monitoring and evaluation activity that is undertaken and available to DFID’s agricultural research team is used to assess the likely impact of its entire agricultural research portfolio.

Phase 2: Field Work, including site visits to Kenya and Uganda

5.12 DFID’s agricultural research programme focusses largely on Africa. Many of its African partners (e.g. AATF), two of CGIAR’s global research centres and other international research organisations (e.g. ICIPE) are based in Kenya, although they have continental or global remits. These organisations also have active field programmes in Kenya and other East African countries. In view of this, we will visit Kenya and Uganda. We will interview selected DFID agricultural research partners and their intended beneficiaries in rural areas.

5.13 We will:

• conduct interviews with:
  • DFID staff in country offices;
  • the scientists and staff of implementing agencies of selected DFID-supported research programmes (see Figure 1 on page 7);
  • senior government officials in Kenya, where DFID has made significant investments in agricultural research;
  • third-party experts in agricultural research and development, who have not been involved directly in DFID’s agricultural research work; and
  • the intended beneficiaries of the programme – smallholder farmers and other poor people;
• assess directly the level of impact on DFID’s agricultural research programme on the intended beneficiaries by:
  • conducting a mixture of announced and unannounced visits to interview intended beneficiaries of DFID-supported agricultural research projects and collecting a range of evidence. We expect to interview more than 250 intended beneficiaries in the course of our field visits; and
  • reviewing the findings and conclusions of two completed impact evaluations. We will re-visit the areas involved to assess the sustainability of results and the quality of the impact assessment. The review team will work with local researchers and visit randomly selected villages to conduct focus group discussions and interviews with intended beneficiaries and others at each location. The other stakeholders interviewed will include poor consumers, small agri-business firms and local government farming advisors.

5.14 In addition to the focus projects outlined in Figure 1, the review team will examine documentation for those projects listed in Figure 2 on page 9. Although not chosen for detailed review, these will provide contrast to the core projects and will support our assessment of DFID’s strategy and overall agricultural research portfolio.

5.15 We will examine a range of documentation, as set out in the evaluation framework from page 10. Our assessment will include a review of documentation in-country and a detailed examination of DFID’s operational files and evaluations related to its agricultural research programme.

5.16 We will assess the agricultural research programme’s financial information, focussing on the last five years. This will include analysis of the financial reports of selected partner organisations to try to identify costs and the proportion of allocated funds reaching intended beneficiaries.⁹ It will consider the flow of funds, accounting and reporting systems, audit and the costs at each stage of the delivery chain. By comparing the different delivery channels and partners used in the DFID agricultural research portfolio, we will seek to draw lessons on effective programme delivery.

5.17 The full list of interviewees will be drawn up by the review team based on discussions with DFID’s agricultural research team and other key informants. We will conduct face-to-face interviews with DFID staff who worked on the design, implementation, monitoring and evaluation of the agricultural research programmes in the portfolio, to inform analysis of the use of data, impact and performance management. We will review, in particular, how these evaluations have used the lessons identified to inform future programmes. We will also conduct face-to-face and telephone interviews with partner organisation staff who deliver DFID’s agricultural research programmes.

Evaluation framework

5.18 The evaluation framework for this review is set out below. It is based on the standard ICAI guiding criteria and evaluation framework, which cover four areas: objectives, delivery, impact and learning.

6. Roles and responsibilities

6.1 The Team Leader will be the primary point of contact with DFID. KPMG will provide oversight of this review under the overall leadership of the ICAI Project Director. Supplementary analysis and peer review will be provided by KPMG staff.

6.2 The team will comprise the following members:

Team leader (Independent)

He is a rural livelihoods expert with over 30 years’ experience in research and consultancy on agricultural and rural development. He led the design of a number of DFID’s and the World Bank’s flagship rural poverty programmes and has also worked on more than a dozen impact evaluations of rural livelihoods programmes in Asia and Africa. He has a good knowledge of agricultural research. He was a member of DFID’s Plant Sciences Research Programme’s advisory committee from 1995 to 2005. He also led the evaluation of the International Rice Research Institute’s poverty-focussed rice research programme in 2001. He was an expert on the ICAI evaluation of the Western Orissa Rural Livelihoods Project, in 2012.

Team member 1 (Independent)

He has over 40 years’ experience in agricultural research and development in Africa and Asia. From 1990 to 2005, while employed by the United Nations Food and Agriculture Organisation (FAO), he served on the CGIAR independent Technical Advisory Committee, which FAO manages on behalf of CGIAR donors. While there, he co-ordinated the external programme and management reviews of CGIAR Centres. He also led reviews of CGIAR research priorities and strategies. He earlier developed FAO’s approach to planning agricultural land use. He is currently a visiting professor at Reading University and has also done research at Ahmadu Bello University, Nigeria. He is an adviser on agricultural and rural development with Aga Khan Foundation programmes in Asia and Africa.

He will be the agricultural research expert on the mission. He will assess the quality of DFID’s agricultural research and its relevance to the needs of smallholder farmers. He will also contribute to the review team’s assessment of the reform process in the CGIAR and other research partners.

Team member 2 (KPMG)

She is a Chartered Accountant with a Masters in Development Studies and has over six years’ experience with KPMG working across public sector audit. She has also worked at the European Commission’s Humanitarian Aid Office and at the Institute of Development Studies. Within KPMG, she has worked for two years as part of the internal audit team at King’s College Hospital NHS Foundation Trust and has experience of auditing charitable funds at a range of healthcare organisations. She will assess DFID’s financial and programme management capacity in the context of its Agricultural Research programme. She will assess the capacity of implementing partners and examine the delivery chain of DFID’s programmes.

Team member 3 (Independent)

She is an applied development economist. She is engaged in research projects for Durham University, the World Bank and the United Nations University – World Institute for Development Economics Research. She is also an external associate at the University of Manchester Brooks World Poverty Institute (BWPI). Her research interests include agricultural markets, trade and institutions, foreign aid and public expenditure allocation. Her research primarily relies on quantitative methods and data analysis, mainly applied to countries in Sub-Saharan Africa, South Asia and East Asia and the Pacific. She has conducted empirical investigation on the labour markets in Ethiopia.

She will be the economist and impact assessment specialist on the team. She will review completed impact evaluations by DFID, CGIAR and others. She will also review current DFID and CGIAR agricultural research impact evaluation policies and approaches. Additionally, she will contribute to the design and supervision of field research for the mission by local consultants.

Peer reviewer 1 (Independent)

He is a development economist, with over 20 years of experience in agricultural research, impact evaluation and research on poverty in Africa and Asia. In 2005, he authored a report for DFID on the rates of return to agricultural research. In 2010, he authored a report for DFID on evaluation methods for research programmes, which is now part of the standard operating procedures for DFID research programmes. In 2002 and 2009, he was a member of evaluation teams to evaluate the effectiveness of IDRC funded multi-country research programmes. He has published extensively on agriculture and on poverty. He will oversee and provide quality assurance of the impact evaluation workstream for this review.

Peer reviewer 2 (Independent)

He has 20 years’ experience as a statistician specialising in poverty and food security analysis, the design and implementation of complex household surveys and sampling strategies, monitoring and impact evaluation. He has an MSc in Tropical Agricultural Development and a PhD in Applied Statistics. He will provide a peer review and challenge function for this review, he will also provide a secondary data integrity role, interrogating DFID’s data and programme designs.

Impact evaluation team leader (TNS Global)

He has over 15 years of experience in managing research projects in Africa, including extensive experience in the agricultural sector. He has expertise in research design, quality control and delivery. He has worked in over 30 African countries. He was the Director of the Farmer First farmer segmentation study for the Bill and Melinda Gates Foundation. This was an attitudinal study of small-scale farmers to identify those likely to adopt new products and services. He has undertaken other assignments for governments, not-for-profit organisations, private firms and international donors. These have included an evaluation of mFARM, a project designed to improve the prices farmers receive, through increased transparency down the value chain. He has led social impact studies of coffee and fruit farmers in East African countries and evaluated radio programmes designed to provide information to microenterprises. He led an evaluation of milk fortification for improved nutrition in Kenya.

He will be the Project Director of the impact evaluation verification studies. He will be responsible for designing the studies, overseeing implementation, data analysis and report writing.

Impact evaluation team member 1 (TNS Global)

She has over 30 years of experience in managing agricultural and rural development programmes. She started her career as a government official in Uganda and later worked for NGOs and on international donor-funded projects. She has undertaken research assignments for a range of clients, including government agencies and NGOs. A number of her recent assignments have involved nutritional studies, using quantitative and qualitative methods, related to crop improvement programmes and horticulture projects. She will be Team Leader of the impact evaluation verification study in Uganda. She will manage the overall study and be responsible for quality control and data analysis. She will also lead one of two field research teams working in six villages, interviewing 50 farmers and leading focus group discussions.

Impact evaluation team member 2 (TNS Global)

He has extensive experience of research on African agriculture, especially with smallholders in East Arica. He specialises in socio-economic research, agricultural trade and marketing, agribusiness and monitoring and evaluation. He is proficient in the design of quantitative and qualitative data collection tools and data analysis. He also has extensive experience in field research. He is skilled in the use of Access, Excel, SPSS and STATA for quantitative work and EpiData for qualitative work. His clients include government agencies, NGOs, private sector firms and donors, including the African Development Bank, the Japan International Cooperation Agency and the United States Agricultural Development Fund.

He will be the Team Leader of the impact evaluation verification study in Kenya. He will manage and participate in the research in six villages, involving interviews with 50 farmers and focus group discussions. He will be responsible for ensuring quality control and data analysis.

Impact evaluation team member 3 (TNS Global)

She has five years of experience in conducting and managing qualitative research in East and West Africa. She led a qualitative study of small-scale dairy farmers in the Rift Valley, Kenya to develop insurance products for farmers. She worked with the Impact Evaluation Team Leader on the Farmer First farmer segmentation study for the Bill and Melinda Gates Foundation, leading qualitative research. She has also undertaken a number of studies for NGOs on women smallholder farmers, to understand their needs and priorities. She is experienced in focus group discussions, in-depth interviewing, participant observation and other ethnographic methods. She is also experienced in managing small field research teams.

She will lead one of two field research teams in Uganda working in six villages, interviewing 50 farmers and leading focus group discussions.

7. Management and reporting

7.1 We will produce a first draft report for review by the ICAI Secretariat and Commissioners by w/c 5 August 2013, with time for subsequent revision and review prior to completion and sign off in w/c 21 October 2013.

8. Expected outputs and time frame

8.1 The main deliverables will be:

9. Risks and mitigation

9.1 The following sets out the key risks and mitigating actions for this evaluation.

10. How this ICAI review will make a difference

10.1 This review will examine the impact of DFID’s agricultural research programme on smallholder farmers and poor and malnourished people in developing countries. Findings will support direct improvements in the ability of the portfolio of agricultural research projects to deliver impact for these people. We will undertake a portfolio review and examine the effectiveness of programme management, including objective-setting, the balance of risk and the use of monitoring and evaluation to feed back into future programme design. Our review will assess this cycle in detail.

10.2 Our review will scrutinise the extent to which DFID’s research agenda targets the priorities of poor farmers in developing countries and is set by them. It will focus on DFID’s investment allocation processes, including bidding and market testing and the choice of delivery channels. We will consider how these options are evaluated by DFID’s agricultural research team and how lessons learned feed into the future design of agricultural research projects. We will examine financial management and performance assessment as part of our overall assessment of the ways in which risk is managed.

10.3 We will focus in detail on seven research projects, including advanced science, adaptive research and near-market R&D. The selected projects will enable us to trace the impacts that follow from generating new technologies in the laboratory to improved livelihoods and nutrition for poor people in developing countries.

10.4 The current agricultural research programme builds on decades of earlier DFID support to agricultural research. This review will assess how learning from past evaluations has been used to inform future agricultural research choices.

10.5 This ICAI review will look explicitly at how DFID’s agricultural research programme supports and influences the CGIAR reform process. Lessons learned will be of wider interest to DFID programmes implemented through World Bank-managed multi-donor trust funds.

Footnotes

¹ Terms of Reference: Evaluation of DFID’s support to agricultural research programmes, ICAI, 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-AgResearch-ToRs-FINAL.pdf.

² Findings from a HarvestPlus Project in Mozambique and Uganda, HarvestPlus, 2010, http://content.yudu.com/Library/A1rzwf/HarvestPlusREUOFSPBr/resources/4.htm.

³ Bringing New Ideas into Practice, Experiments with agricultural innovation, P.Gildemacher and R.Mur (Eds.), 2013, KIT Publishers, Amsterdam, Chapter 4, http://researchintouse.com/resources/Learning-RIU-Africa_book2.pdf.

⁴ Project plans that track how inputs are expected to lead to desired impacts are often referred to as logical frameworks or logframes.

⁵ Data provided to ICAI by DFID.

⁶ DFID has provided core funding to CGIAR since 2002.

⁷ The current £6.2 million project with GALVmed (2012-17) is through an MoU. The earlier two grants were through accountable grants.

⁸ Data provided to ICAI by DFID.

⁹ This review is dependent on receiving sufficient and appropriate financial reports from partner organisations.