DFID’s approach to value for money in programme and portfolio management

1. Purpose, scope and rationale

Providing assurance to Parliament and the general public about the value for money of UK aid is a central feature of ICAI’s mandate. This role is acknowledged in the UK Aid Strategy, which identifies ICAI as one of the mechanisms for assuring value for money across the aid programme.¹ Value for money is a cross-cutting theme in all ICAI reviews and ICAI also undertakes reviews focusing on specific aspects of value for money.

This performance review will explore how well DFID has integrated value for money considerations into its programme and portfolio management systems and practices and whether its approach is actually delivering improvements in value for money. It will cover DFID’s bilateral aid programme (including bilateral programmes contracted to multilateral agencies, or “multi-bi aid”), and examine how DFID considers value for money in its choice of multilateral partners and influences value for money in the multilateral aid programme. The review also has a secondary purpose in generating learning for ICAI which will be used to strengthen its value for money assessments in subsequent reviews.

The review will not consider aid allocation across countries or sectors and will not review DFID’s administrative budget. It will be limited to aid spent by DFID and will consider both development aid and humanitarian assistance. It will not cover value for money in procurement, as this is the subject of another ICAI review.²

2. Background

The value for money agenda first started to emerge strongly in DFID in 2011, a year which saw the publication of DFID’s Approach to Value for Money³ as well as DFID’s Business Plan for 2011-2015.⁴ The latter included a commitment to improving both transparency and value for money. DFID also established a Quality Assurance Unit (QAU), which provides internal scrutiny of business cases and a sample of annual reviews and post-approval reviews. Following this, DFID’s Annual Report and Accounts for 2011-12 reported that value for money is “at the heart of its programme investments”.⁵ DFID launched its Smart Rules in 2014, as part of its Better Programme Delivery agenda, and a Smart Guide on value for money⁶ and other related resources followed in 2015, providing guidance on how value for money should be achieved throughout the programme management cycle. DFID’s approach is one of “empowered accountability”, where all staff have a part to play in ensuring good value for money – in particular the Senior Responsible Owner (SRO) for each programme. Over time, sector- and country-specific value for money guidance has emerged in some areas. Work is currently underway to develop a new set of sector-specific guidance documents, to be rolled out during 2017.

DFID’s overarching objective with regard to value for money is to maximise the impact of each pound spent to improve poor people’s lives in terms of “long-term transformational change”.⁷ To this end, the UK Aid Strategy notes that work has been done to put in place clear processes to drive value for money and to identify and eliminate poor-value expenditure.⁸

3. Review questions

This is a performance review built around the evaluation criteria of relevance and effectiveness.⁹ It will focus on the relevance of the value for money framework to meeting DFID’s goals and objectives in respect of the UK aid programme and its effectiveness in practice, including how DFID is learning lessons about its application of value for money approaches and how these processes are used to drive greater impact. The rationale for a performance review concerns the need to explore the adequacy of DFID’s systems, processes and capacity to assess value for money in programme and portfolio management. Value for money within specific programmes and sectors has been addressed within most ICAI reviews to date, and at a more general level in the 2015 review on DFID’s Approach to Delivering Impact.¹⁰ This review will build upon past findings on value for money and will address the questions set out in Table 1.

Table 1: Our review questions

4. Methodology

Our methodology will consist of five mutually reinforcing components, summarised in Figure 1. It will build upon the findings of previous ICAI reviews and also highlight learning for both DFID and ICAI (including potential issues to explore in future reviews).

Component 1 – Literature review: This will include a review of academic and other research reports on the topic of value for money in international aid and in other fields of public policy, as well as a review of the value for money findings of previous ICAI reports. This will help us to understand how value for money has been conceptualised and assessed in relation to international aid (including approaches used by other donors) as well as how value for money has been approached in HM Treasury guidance and by other government departments and scrutiny bodies. This understanding will provide us with important insights to frame our research in the other review components.

Component 2 – Strategic review: We will identify DFID’s overall approach to conceptualising and operationalising value for money, and assess its relevance and proportionality to the objectives and values of the aid programme. We will map the systems and processes that DFID uses for value for money measurement and analysis in portfolio and programme management (including the existence of feedback loops and learning mechanisms), and compare them to good practice from the literature and UK government guidance.

We will pay particular attention to how DFID identifies examples of poor value for money and takes corrective action. The strategic review will involve a comprehensive review of relevant policies, rules and guidance, as well as a series of key stakeholder interviews.¹¹ We will also undertake an in-depth review of DFID’s work to develop value for money metrics at sector level, including the progress made in this area and how the resulting data has been used. The findings of the strategic review will highlight key issues to be explored further in the remaining components.

To assess the effectiveness of DFID’s value for money processes, we will review a sample of centrally managed and country-led programmes.

Component 3 – Centrally managed programme reviews: We will conduct in-depth reviews of the application and impact of value for money processes in a sample of four centrally managed programmes (selecting examples of programmes where DFID is attempting to deliver results at scale; the sampling approach for all components is described in section 5 below). We will explore whether centrally managed programmes offer economies of scale or efficiencies in management and delivery. We will consider whether DFID has appropriate criteria for assessing which types of programme should be centrally managed from a value for money perspective and how these can be integrated with country portfolios in order to maximise value for money. We will examine how this approach has developed over time in response to lessons learned. This will involve a review of relevant programme documentation and management information, as well as undertaking interviews with the relevant UK-based DFID staff responsible for these programmes and representatives of those organisations undertaking their management. Where the selected programmes are implemented in the countries selected as case studies (component 4), we will also discuss these with relevant in-country staff during our visits.

Component 4 – Country case studies: We will undertake visits to a sample of four countries. The case studies will be used firstly to review the application of value for money processes in the management of the country portfolio. We will conduct in-depth interviews with country office heads and staff on their overall approach to value for money at portfolio level. We will explore how they think about value for money and how this is operationalised within the office (for example what guidance and tools are used, what capacity exists, how the approach has developed over time in response to lessons learned, etc). We will also conduct a detailed assessment of a sample of at least six programmes which will provide illustrations of how value for money has been approached and addressed in practice. The case studies will be based on a desk review of programme documents and data in advance of the country visit. This will be supplemented by interviews in-country with DFID country office staff, implementing partners and other relevant stakeholders (such as other donors, government counterparts, implementing partners or non-governmental organisations working in the selected countries and sectors).

Component 5 – Other fieldwork: In order to supplement our evidence base on how value for money has been approached in practice, we will conduct additional desk reviews of up to six outlying country-led programmes (ie examples of good and bad performance). This may include investigation of programmes in countries which are not being visited. The identification of outliers will be based on a review of DFID management information and analysis, including data on activity which has been subject to a programme improvement plan. We will also conduct additional research to explore value for money in the multilateral aid programme. ICAI’s 2015 review, ‘How DFID works with multilateral agencies to achieve impact’, analyses among other things how DFID influences its multilateral partners to maximise value for money in multilateral aid. This review is being followed up in 2017, following the publication of DFID’s Multilateral Development Review 2016.¹² Drawing on the results of that research, we will map DFID’s approach to promoting value for money within the multilateral aid programme and assess how it has developed over time in response to lessons learned. This will involve soliciting feedback from a range of DFID’s multilateral partners, using a survey instrument and/or in-depth interviews. An analytical framework for this component will be developed once our 2017 follow-up work is completed.

5. Sampling approach

We have used a purposive sampling approach to select first the country case studies and then the programmes for detailed assessment within each of those countries. Countries have been selected in order to reflect the diversity of the aid programme in terms of region, size of country office, level of fragility in the national context and the balance of development and humanitarian programming:

• Pakistan – a large programme in Asia with a range of programming and diversity of channels, Pakistan is a fragile state that has benefited from recent humanitarian assistance.
• Nigeria – a large programme in the West and Southern Africa region with a range of programming and diversity of channels, Nigeria has faced issues in respect of tackling corruption and increasing transparency.
• Uganda – a mid-size programme in the East and Central Africa region, Uganda has faced a range of development challenges and experienced significant economic growth in recent years.
• Malawi – a smaller programme in the West and Southern Africa region, Malawi faces significant development challenges and DFID is a major donor.

Together these four countries account for £790.8 million of DFID’s planned programme spend for 2016-17.¹³

It has also been noted that a country visit to Syria or neighbouring countries is likely to take place as part of a forthcoming review. Although this review is yet to be fully scoped, it may be possible for the chosen review team to undertake some investigation of value for money as part of their case study work. This evidence could then be incorporated into this value for money review. As the details have not yet been finalised, Syria has not been explicitly named as a case study country although we will keep this potential option for additional fieldwork in mind as both reviews progress.

Up to six programmes will be selected from each country for detailed review in order to explore variance in how value for money considerations have been applied and the consequences. Taking account of available data on resource allocation by sector and the availability of sector-specific value for money guidance, we will focus on the following areas of activity: disaster/humanitarian, government and civil society, education, population programme and reproductive health, other social infrastructure and services and trade, industry and business. We will also aim to cover at least one example of multi-bi arrangements in each country.

A sample of four centrally managed programmes will also be selected. In order to explore the use of such programmes to deliver results at scale, the sample will be drawn from the ten largest (by value), excluding any which have recently been the subject of an in-depth ICAI review and focusing on those which are most active in the case study countries.

6. Limitations to the methodology

This review will focus on the integration of value for money considerations into portfolio and programme management, and how effective that is at improving value for money in particular programmes and portfolios. It is not designed to reach conclusions on the intrinsic value for money of UK aid, or any particular type of aid programme. It will, however, seek to identify examples of relatively good and relatively poor value for money within portfolios.

DFID’s value for money practices are continuously developing and new guidance is likely to be launched during the review period. The review will take account of these new developments as evidence of learning, but will have limited scope to assess their effectiveness.

The methodological components have been designed to provide a sufficient level of triangulation which will deepen our understanding of both relevance and effectiveness and reduce the potential for bias which would come from over-reliance on a limited number of (internal) sources.

In respect of multilateral aid, drawing on past ICAI assessments, the review will assess whether DFID has a coherent strategy and approach to improving focus on value for money among its multilateral partners. However, it will not attempt to compare the value for money achieved by different multilateral partners, or to compare the value for money of multilateral and bilateral aid.

7. Risk management

8. Quality assurance

The review will be carried out under the guidance of the ICAI Commissioner Tina Fahm, with support from the ICAI Chief Commissioner Dr Alison Evans, ICAI Commissioner Richard Gledhill and the ICAI Secretariat. The review will be subject to quality assurance by the Service Provider consortium.

Both the methodology and the final report will be peer reviewed by Dr Catherine Shutt, a value for money expert from the Institute of Development Studies, University of Sussex, and Chris Bedford, formerly of the National Audit Office.

9. Timing and deliverables

This review will be executed over a period of nine months, beginning in February 2017.

Footnotes

1. UK aid: tackling global challenges in the national interest, HM Treasury and DFID, November 2015, link.
2. ICAI Procurement review, ICAI website, April 2017, link.
3. DFID’s approach to value for money, DFID, July 2011, link.
4. Business plan 2011-2015. Department for International Development, DFID, May 2011, link.
5. Annual report and accounts 2011-12, DFID, June 2012, link.
6. Smart Guide: DFID’s Approach to Value for Money, DFID, March 2015, not published.
7. For example as noted in the Smart Guide: DFID’s Approach to Value for Money, March 2015, not published.
8. UK aid: tackling global challenges in the national interest, HM Treasury and DFID, November 2015, link.
9. Based on the OECD DAC evaluation criteria. Principles for Evaluation of Development Assistance, OECD DAC, 1991, link.
10. DFID’s Approach to Delivering Impact, ICAI, June 2015, link.
11. Discussions with our DFID counterpart have already identified a number of people who it will be important to interview and we will use these discussions to identify further informants within DFID and elsewhere in government. We have also used our contacts from initial engagement events with civil society organisations and private sector suppliers to identify additional interview subjects outside of government and include partners/suppliers and other donors. These interviews will be loosely structured at the start and increasingly focused as the review progresses. A small number of key respondents will likely need to be interviewed more than once.
12. Raising the standard: the Multilateral Development Review 2016, DFID, December 2016, link.
13. Annual Report and Accounts 2015-16, DFID, July 2016, link.