Follow-up of ICAI Reviews from Years 1, 2 and 3

ICAI Reports Followed up in this Review

Executive Summary

I. What this report covers

All ICAI reports incorporate recommendations for action by DFID (or other relevant departments) in order to enhance the effectiveness of UK aid. In turn, DFID publishes a management response and takes action. A key component of ICAI’s mandate is to follow up on its recommendations and to assess progress made by DFID and other government departments on issues identified in our reports. Since ICAI’s inception we have undertaken two sets of follow-up work, published in our Annual Reports of 2012-13 and 2013-14, respectively.²

For 2014-15, the last year of ICAI Phase I, we have followed up on reports from across the first three years of ICAI (Years 1-3). For this review we have chosen to focus on all 13 of the reports we published in Year 3 and also on 9 selected reports from Years 1 and 2, where key issues still merit further follow-up. The reports we have covered are listed in the Contents and the Annex, where further information is also provided.

Given the scope and aim of this follow-up review, we have decided to publish it separately from the Annual Report. Together with our reports on DFID’s Approach to Delivering Impact, How DFID Works with Multilateral Agencies to Achieve Impact and Business in Development, this report is part of our final suite of reports for ICAI Phase I.

The follow-up work undertaken for this report covers, therefore, 22 of the 35 reports we published during the first three years of ICAI. This gives us a good basis from which to assess how DFID has responded to recommendations over a wide range of different reports and over time. Overall, we can see that DFID has taken action to address the great majority of our recommendations. Country offices and teams within DFID centrally have, both in their approach and in the actions taken, shown in most cases a serious intent to address the issues we have raised. We acknowledge the positive efforts that DFID teams have made in addressing our recommendations and the work which goes into it.

II. Structure of the report

After an introduction, this report first covers our follow-up findings of the four Year 1 and 2 reports which looked at education programmes. We decided to follow up all of those as a group, which allowed us to focus on the common issues across the reports, particularly how DFID can ensure that it concentrates on the quality of education provided.

This is followed by our main findings section, covering the remaining reports which were followed up individually. Where there are common issues and themes between reports we have grouped the findings together, for example, a section on reports covering issues in fragile states.

The final section of the report focusses on lesson learning and general points about how DFID has responded to our recommendations. The Annex provides more detail on the results of our follow-up work on each of the 22 reports we have covered as part of this review.

We have made a limited number of specific further recommendations, where there are significant issues that need to be addressed and we consider that it is most useful to do so. These are all included in this Executive Summary. In the body of the report we have also included, in bold italics, other issues where challenges remain to be addressed or where we have made suggestions for ICAI to focus on in the future.

III. Follow-up of education reports

Education is a key area of DFID’s work reflected in our reports. Our follow-up covered ICAI’s reviews of DFID’s Education Programme in Three East African Countries (Ethiopia, Tanzania and Rwanda); DFID’s Support for Health and Education in India; DFID’s Bilateral Aid to Pakistan; and DFID’s Education Programmes in Nigeria, all published in 2012.

Since our reports were published, DFID has shifted its focus to pupil learning, replacing enrolment as the centre of its agenda, in line with our recommendations. This is reflected, for example, in DFID’s key Education Position Paper produced in 2013. We have seen significant moves by DFID to strengthen its education cadre and its mode of operation. In all six countries there is a determined focus on improving pupil learning, with a particular emphasis on supporting teachers in the classroom. DFID has been able to use the reports, alongside other information, in working with national governments.

There are, of course, no quick fixes to improve learning and progress is likely to be slower than national ambitions. The gap between expectation and realisation will have to be carefully managed by DFID and governments. DFID should help governments to develop plans and targets which are affordable and achievable. DFID can use its experience in each country better to inform forward planning in all six countries.

Raising pupil learning will require a more equitable distribution of resources, in order that all pupils have access to a good quality education. For example, an initiative in Tanzania linking funding to reductions in high pupil:teacher ratios could provide useful lessons to deploy in other countries.

DFID has been successful in generating sound annual reviews of programmes based on good data at a national level. All six countries are strengthening school leadership, parental and community participation and local government support for schools. These developments would be stronger with improved local management information and a process of local review.

IV. How DFID learns

How DFID learns has been an important theme in ICAI’s work. In each of our reviews, we have assessed DFID’s learning and found performance to be mixed. Findings from our follow-up work reinforce this picture. For example, we cover below the issue of how the central DFID policy team on Empowerment and Accountability programming needs to guide learning across DFID’s country programmes in this area; and in a section on fragile states we look at how some DFID country offices are better at working with beneficiaries in difficult environments than others, so good practice in one should be shared. Here we focus on the follow-up of a separate report we published, in April 2014, on How DFID Learns.

We found that our report on How DFID Learns has re-energised DFID to address organisational learning and led to clear action from DFID. A Task and Finish Group on Learning was established with the objective of developing a DFID Organisational Learning Strategy and Results Framework by July 2015, to guide actions and track progress across DFID. Feedback on the report has been highly positive, particularly from within DFID, from online commentators and from the specialist media. The report is now actively referenced across the organisation, by development non-governmental organisations (NGOs) and by other UK government departments.

Significant action is now under way but culture change and incentives for learning, for example on enabling staff to provide and use challenging feedback and on encouraging learning ‘in the field’, will be essential for change to be lasting.

V. Empowerment and accountability

DFID’s internal learning was also a theme in following up on our report on DFID’s Empowerment and Accountability Programming in Ghana and Malawi. The report focussed on DFID’s promotion of social accountability through support for civil society, the media and national parliaments. We found that DFID’s work was contributing to more constructive engagement between communities and governments around the delivery of public services and development programmes. DFID also responded positively to our recommendation to improve its approach to research. DFID has developed a five year, multi-country research programme on what works in identifying and supporting social and political action for more effective empowerment and accountability, which is soon to be tendered.

We remain concerned that the central policy team is not resourced to guide learning across the portfolio. The process of converting data into practical lessons and improvements in programming must be deliberate and well managed to contribute to learning across DFID’s country offices.

VI. How DFID works with the private sector

Another current key strategic issue for DFID is its work with the private sector. DFID expects to spend £1.8 billion on economic development in 2015-16, over double that spent in 2012-13. A large share of this will be aimed at stimulating the private sector. We reviewed DFID’s Private Sector Development Work in our report published in May 2014. The report came at a time when DFID was undertaking a major restructuring of its approach to economic development, including private sector development. This resulted in the management response showing only a limited acceptance of our recommendations. We noted from our follow-up work, however, that the changes which DFID is introducing are broadly in line with our recommendations.

For example, we were keen to see clearer guidance on how to design a coherent and well-balanced country portfolio of private sector development work that matches DFID’s aim to end extreme poverty; and on defining and articulating where it can add most value. DFID has tackled these issues through the piloting and subsequent roll-out of a new inclusive growth diagnostic. Given its vital importance in DFID’s future strategy, this is an area that remains work in progress.

VII. Working in fragile states

DFID has been increasingly focussed on working in fragile states, where it has committed to spend 30% of official development assistance (ODA) by 2014-15. In February 2015, ICAI published a separate review assessing the Impact of the Scale-up of DFID’s Support to Fragile States. For this review, our follow-up work covered the following reports on DFID’s work in four fragile states: DFID’s Water, Sanitation and Hygiene Programming in Sudan, DFID’s Bilateral Support to Growth and Livelihoods in Afghanistan, DFID’s Peace and Security Programme in Nepal and DFID’s Health Programmes in Burma.

A challenge we have focussed on in our follow-up is the need, particularly in such difficult environments, to find ways of engaging with stakeholders and seeking beneficiary feedback to help ensure that programmes address needs appropriately and deliver benefits. We found that in Nepal, for example, further large-scale surveys had been undertaken to inform programming. In Afghanistan, however, while plans had been improved, actual enhanced beneficiary consultation had yet to take place. We suggest that DFID applies innovative approaches on beneficiary feedback across all its programming in fragile states.

Our follow-up findings also reinforce the importance of lesson learning within the context of fragile states, where longer term planning of development interventions can be too optimistic. Where possible, interventions should focus on addressing the chronic challenges in a manner that bridges the distinction between strictly humanitarian and development approaches.

VIII. Working across government departments

Our follow-up work included two reviews of aid programmes implemented by government departments other than DFID: the FCO and British Council Aid Responses to the Arab Spring and the Evaluation of the Inter-Departmental Conflict Pool.

Both reviews’ follow-up findings commented on the positive engagement of the organisations with ICAI’s recommendations. The institutional arrangements of these organisations were not always best suited to facilitate efficient implementation of aid programmes. For example, the FCO managed the Arab Partnership Participation Fund (APPF), which had a budget of £40 million over four years, half of which was contributed by DFID. We commented on its weak project management capacity but acknowledged the importance of responding rapidly and flexibly to a highly volatile situation that was considered a high priority for UK foreign policy.

We have also seen the FCO working to professionalise its project management systems in advance of the commencement of the new Conflict, Stability and Security Fund (CSSF) – the successor to the Conflict Pool (CP) – which began operations in April 2015. The CSSF is under the direct authority of the National Security Council (NSC) and supported by a Joint Secretariat which is housed in the FCO. It reports to the National Security Secretariat in Cabinet Office and is staffed by officials from across the NSC departments.

The FCO’s grant-making procedures are not yet fully developed, its accounting systems are not yet designed for project management and the cadre of trained project managers within its staff is not particularly strong. The FCO’s readiness to manage the increased volume of aid funds under the CSSF, therefore, remains to be demonstrated. It would be useful for DFID proactively to share its good practice tools and guidance on programme management with other departments undertaking development work.

We are also concerned about the potential risk that the way in which the CSSF is funded could generate unhelpful competition between departments and that the CSSF will become reactive in nature, due to insufficient attention to strategy and programme management.

IX. Influencing international organisations

How DFID exerts influence with key partners such as multilateral organisations was examined in two of the reports we reviewed. In the report DFID’s Support for Palestine Refugees through the United Nations Relief and Works Agency (UNRWA) we highlighted that DFID, as UNRWA’s fourth largest donor, continued to be a highly influential and valuable contributor at strategic and operational levels. DFID has responded very positively to our recommendations but achieving lasting change remains challenging in the current environment and increased funding from the international community has not materialised.

In our report on DFID’s Oversight of the EU’s Aid to Low Income Countries we were concerned about the limited assurance DFID has on the significant contributions it makes to the EU. In our follow-up work last year we found that, overall, DFID and the EU had made solid, if slow, progress. Some further progress has been made but there remains a major opportunity for DFID to work more effectively with EU delegations in-country. In our new report on How DFID works with multilateral agencies to achieve impact we recommended that DFID needs clear objectives for working with multilaterals in its country-level strategies and that working with the EU should be prioritised for this (along with the World Bank). We also note that significant levels of DFID funding, through its multilateral contributions, go into countries where there is no DFID presence. DFID funding through the EU is a significant element of this. We suggest that there is a greater focus by DFID on this area.

X. Capital projects in Montserrat

Influencing governments is also essential in many of DFID’s programmes. We followed up on our report on DFID’s Support to Capital Projects in Montserrat, where working with the government is the key focus. We found that the new DFID team and Government of Montserrat have progressed many of the points we raised, for example taking a longer term approach to financial planning and improving the maintenance of capital investments. There is both an opportunity and a challenge in achieving greater coherence between projects, to improve the overall impact of the investment portfolio.

XI. Child mortality in Kenya

Health remains an important area of focus for DFID. One aspect of DFID’s health work was covered by our report on DFID’s Contribution to the Reduction of Child Mortality in Kenya. Our follow-up work found that the DFID Kenya country office has taken substantial action in response to the report, particularly on intended beneficiary consultation, but noted that there may be negative unintended consequences of DFID’s decision to exit malaria work. An exit strategy for malaria was produced by DFID Kenya, which highlights risks stemming from the lack of capacity of other parties to help to address the funding gap. Our recommendation concerning an exit from malaria commodities was, however, conditional on a commensurate refocussing of resources on health system strengthening and on assurances of alternative commodity funding which have not yet materialised.

XII. Support to agriculture

There is an urgent need to increase agricultural productivity in developing countries and agricultural research has been an important part of DFID’s response. Our follow-up work included our report on DFID’s Support to Agricultural Research. We found that DFID has responded well to our recommendations. It has scoped out possible new development programmes and developed a geo-referenced database of research projects, which enables country offices to identify quickly what agriculture research is being undertaken in their country. The agriculture team has started to work closely with a number of DFID’s country programmes to find ways to integrate new agricultural technologies into poverty reduction programmes.

XIII. Improving nutrition

We also followed up our report on DFID’s Contribution to Improving Nutrition, where global efforts to combat the most vulnerable have been increased in recent years. Overall, DFID’s nutrition team has responded actively to our recommendations. DFID has set up an internal Nutrition for Growth Strategy group, chaired by one of the Africa directors. It will set the strategic direction for DFID’s nutrition work and lead the refresh of DFID’s approach. Since our report was only published in mid-2014, however, it is too soon to assess fully the impact of DFD’s actions.

We note that DFID has yet to respond to our recommendation that it should develop guidelines for country offices on selecting the best package and mix of interventions that would have the greatest impact on stunting, in the local context. Unless this is done and mentoring is provided to country teams, it will not maximise impacts on stunting. DFID also needs to issue guidance to staff to ensure nutrition interventions specifically address the needs of the most vulnerable.

XIV. Critical interventions

A number of our reviews tackled issues at what have proved to be critical moments in the life of the programme. Our report on DFID’s Trade Development Work in Southern Africa revealed serious deficiencies in the TradeMark Southern Africa (TMSA) programme. It was the first and only report so far to have been given a ‘red’ rating. Our work in the first quarter of 2013 prompted an Internal Audit investigation by DFID and resulted in the eventual closure of the programme in March 2014. DFID has taken our recommendations seriously. It has changed the way it approaches risk in the design of the new regional programme that will follow from TMSA, while ensuring a stronger focus on poverty. DFID is also addressing our concerns about the misreporting by TMSA on achievements against targets, through plans to appoint an independent monitoring and evaluation contractor. In learning to manage risk better, in the light of the TMSA example, however, DFID should make sure that it distinguishes between risk arising from the context of an intervention and risk of internal mismanagement.

Another review where we had significant concerns was on Girl Hub: a DFID and Nike Foundation Initiative which we reviewed in 2012. Girl Hub is a joint initiative of DFID and the Nike Foundation, which started in 2010. It aims to empower adolescent girls to reach their potential. It does this by influencing decision-makers and donors to do more to assist vulnerable adolescent girls in the areas of education, health, safety and economic opportunity. It also delivers branded communication platforms, such as radio broadcasts and magazines, to change attitudes and behaviour. Our follow-up work was particularly timely, as current DFID funding will run out by summer 2015 and consideration has been given to a Phase 2 of the project. DFID has agreed further short term funding to the Girl Hub projects in Ethiopia and Rwanda and is providing strategic advice to Girl Hub and Nike Foundation on future planning.

We recognise the importance of work with girls and the innovative nature of this partnership. We note, however, that there are ongoing issues as to governance (the Board structure and DFID’s oversight), transparency (an absence of information about Nike Foundation’s ‘in kind’ contributions) and effectiveness (from our own review and the recent DFID-commissioned evaluability study). This study concluded that Girl Hub has evidence on what kinds of outcomes have been achieved but less evidence to attribute these results directly to Girl Hub. We reported our concerns about these issues in a separate, more detailed note to DFID in March 2015, so that it could take these into account when renegotiating the arrangement.

Our Rapid Review of DFID’s Humanitarian Response to Typhoon Haiyan in the Philippines was undertaken at a critical moment in DFID’s response. The review visit took place very shortly after the typhoon hit the Philippines and the report was delivered only a few weeks after the team returned. The timing of the report was welcomed by DFID, as it enabled DFID to learn from its experience in the Philippines when it was relevant and live. We gave the report an overall ‘green’ rating, noting that DFID was well prepared to act swiftly and decisively, mobilised efficiently and met the urgent needs of the affected communities.

DFID credited our review with helping to improve accountability and focus on DFID’s Conflict, Humanitarian and Security Department’s (CHASE’s) priorities, such as scaling up stockpiles. In line with our recommendation and CHASE’s own plans, DFID increased its stockpile of non-food items in size and flexibility and has continued to deliver significant value for money through its logistics capability. This has allowed DFID to respond rapidly and efficiently in crisis situations, such as typhoon Hagupit in November 2014 and Cyclone Pam in Vanuatu in March 2015.

XV. Lessons for DFID and ICAI in relation to follow-up work

From our follow-up work overall, we have also identified issues and themes for DFID and ICAI to focus on which will improve the follow-up process and lead to more effective action by DFID.

DFID’s responses to our recommendations and the actions taken have been positive and have sometimes gone beyond the specified course of action recommended. We have also found, particularly for our recent more thematic reports, that country offices have responded better than DFID centrally. It is clear, however, that sometimes the DFID team responding has not fully understood the aim of the recommendation or the underlying issue.

DFID has a two week opportunity to comment on the facts in our reports prior to publication but not on ICAI findings or recommendations. It has three weeks to formulate a management response post-publication. We have begun to engage with DFID teams, only after publication, when they are in the process of formulating responses to our recommendations. This deeper interaction should lead to better quality responses and more effective action by DFID. It is important, however, that findings and recommendations are not negotiated in any way before publication as this could compromise our independence.

In our Annual Report last year we noted a lack of consistency in some of the management responses from DFID, particularly in the categorisation of ‘accept’, ‘partially accept’ and ‘reject’. We noted this in particular where DFID states that it is already doing work: sometimes it accepts the associated recommendations; sometimes it partially accepts them; and sometimes it rejects them. In our follow-up work for this year we have seen this lack of clarity continue. For example, several of our recommendations in our report on DFID’s Support to Capital Projects in Montserrat were rejected but we saw action being taken that should help to address the issues we raised.

From our follow-up work, we also noted that, sometimes, the way responses are presented does not reflect a thorough analysis of the position. Evidence of DFID tracking implementation is lacking and not readily available at the start of the follow-up process.

1 Introduction

Background and Purpose of the Follow-up Review

1.1 A key component of ICAI’s mandate is to follow up on our recommendations and to assess progress made by DFID and other government departments on issues identified in our reports. This informs Parliament (via the International Development Committee) and other key stakeholders as to how DFID has made progress in response to our recommendations.

1.2 For each recommendation that we publish in a report, DFID or the relevant other government department responds with ‘accept’, ‘partially accept’, or ‘reject’. It then sets out management actions to address the ICAI recommendation and gives a timeline for completion. Part of our follow-up work is, therefore, an investigation of the progress made by DFID and other departments on these management actions.

1.3 More particularly, follow-up reporting aims to:

• help Parliament to hold DFID and other government departments to account and assess progress against management actions;
• assess whether ICAI’s recommendations have had an impact on government departments’ work and, where possible, on the lives of intended beneficiaries;
• improve the impact on intended beneficiaries of the programmes we have reviewed; and
• enable learning, from both the reports and the recommendations to date, for ICAI’s second phase.

Relationship to previous ICAI follow-up work and to recent reports

1.4 Since ICAI’s inception, we have undertaken two sets of follow-up work, published in our Annual Reports of 2012-13 and 2013-14, respectively.³ These earlier follow-up exercises produced updates and analysis of DFID’s progress on the previous year’s reports. For this report we have chosen to carry out a more extensive exercise to follow up on 2013-14 (Year 3) reports as well as on selected reports from earlier years, where there are key issues remaining that merit further follow-up. This also provides an opportunity to reflect during the transition to the second phase of ICAI’s operations beginning in July 2015. Given the scope and aim of this follow-up work, we decided to publish our findings separately from our Annual Report.

1.5 This report is part of our final suite of reports for ICAI Phase I. These include, as well as our Annual Report, reviews covering DFID’s Approach to Delivering Impact,⁴ How DFID Works with Multilateral Agencies to Achieve Impact⁵ and Business in Development.⁶ The report on DFID’s approach to delivering impact also looks to summarise lessons from across our work to date, focussing on how DFID can achieve sustainable, transformative impact through its programmes and portfolio strategy. In this follow-up report we do not cover a follow-up assessment of all our multilateral reports, given that the Multilaterals review addresses DFID’s work in this area in depth.⁷

Methodology

1.6 As in previous years our approach was based on an Assessment Framework (see Annex A5). This included consideration of the trajectory of programmes and how the likely impact on beneficiaries has been influenced by DFID’s management actions, as well as modifications to account for the wider scope of the work (which addresses three years of reports). Our Year 3 follow-up work has adopted the same approach as in the past, which means that all 13 Year 3 reports are addressed within this review. For Years 1 and 2 (2011-13), however, we decided to focus our investigation on nine reports that had key issues which merited further in-depth follow-up (that is, those with major outstanding matters and risks). We made this determination based on both our original follow-up work on these reports and updates that we received in October 2014 from the relevant lead Senior Civil Servant (SCS lead).

1.7 For those Year 1 and Year 2 reports that we determined did not have key issues meriting a second round of in-depth follow-up, we restricted analysis to updates from SCS leads.

The Follow-up Review process

1.8 The first step for this follow-up work consisted of a request for an update from the DFID SCS leads for each report from Years 1 to 3. For Year 3, we also asked for:

• a demonstration of progress on each of the management actions (whether these actions are ‘off-track’, ‘on-track’, or ‘complete’);
• any contextual changes (for example, in-country) since the report;
• whether the report changed any attitudes or approaches within the office or in partner or delivery organisations; and
• whether DFID has put in place measures to record the possible impact of the management actions and, if so, any evidence collected on this impact to date.

For Years 1 and 2 we posed questions to the SCS lead that targeted key issues found in our previous follow-up work.

1.9 We then engaged the ICAI Team Leader for each report (the original leader of the review wherever possible) to analyse the SCS update alongside document reviews. This research and investigation stage also included meetings and discussions with DFID staff and other stakeholders in the UK and in-country to triangulate our findings.

1.10 From the Year 1 and 2 reports we decided to follow up, as a group, all four reports covering education. As part of this education-themed follow-up exercise, we were able to take advantage of existing planned field visits for other reviews to India, Pakistan, Ethiopia and Rwanda for face-to-face meetings with DFID and other stakeholders in-country.

1.11 For the Year 3 reports, each Team Leader’s research culminated in a meeting held with the lead ICAI Commissioner for the report, or with the Chief Commissioner, and the SCS lead (alongside other DFID staff or staff from other departments where relevant). This meeting provided a forum for follow-up on any key questions raised during the research phase of the work and was also an opportunity for DFID to respond directly to our preliminary hypotheses and findings.

1.12 The last phase of this work included a synthesis session with all Team Leaders that allowed us to share findings, analyse commonalities across DFID’s responses and determine critical factors of success (and challenge) for progress. We identified any common characteristics across DFID’s responses that have resulted in better progress against our recommendations; these are discussed in more detail in Section 4 of this report. We also considered the trajectory of impact for beneficiaries as a result of our recommendations. Lastly, we considered how both DFID and ICAI can learn from the past four years of reports and management responses.

Structure of the report

1.13 Following this introduction, the report starts with a section on the findings from our follow-up of our education themed reports. Our main findings section covers the remaining reports which were followed up individually. We have grouped the findings where there are links and common themes between individual reports. The final section of the report focusses on general points about how DFID has responded to our recommendations. The Annex provides more detail on the results of our follow-up work on each of the 22 reports we have covered as part of this review.

1.14 Throughout the report we have made a limited number of further recommendations where there are significant issues that need to be addressed and we consider it is most useful to do so. We have highlighted in bold italics where challenges remain to be addressed or where we have made suggestions for ICAI to focus on in the future.

2 Follow-up of Education Reports

2.1 This section covers the results of our follow-up of our four 2012 reports which focussed on DFID’s support to the education sector: DFID’s Education Programmes in Three East African Countries (Ethiopia, Tanzania and Rwanda) (Report 10); DFID’s Support for Health and Education in India (Report 11); Evaluation of DFID’s Bilateral Aid to Pakistan (Report 15); and DFID’s Education Programmes in Nigeria (Report 16).⁸ The four reports were followed up as a group by a single team and focussed on the key themes emerging from our previous follow-up work. These were: equity and quality; assessment; evaluation; governance; and funding. Both DFID and we consider these to be critical for the improved performance of education systems seeking to meet national goals and ambitions.⁹

Background and previous follow-up work

2.2 We have previously undertaken follow-up work on these reports. Specifically, our findings from that earlier work were as follows:¹⁰

• DFID’s Education Programme in Three East African Countries (Tanzania, Rwanda, Ethiopia):
  DFID had implemented, on a country-by-country basis, our recommendations that were designed to improve pupil learning in specific ways, such as ‘introducing a results focus into national funding for districts and schools; [and] continuing to expand support for communities to monitor and promote education’. While DFID had rejected our recommendation to revise its pilots on results-based aid, two out of the three country pilots then developed in line with our recommendation ‘to address specific challenges further down the delivery chain’.
• DFID’s Support for Health and Education in India:
  DFID’s response to our recommendations was comprehensive and had started to have a positive, practical impact on the ground. We found that the recommendation for DFID to clarify its strategy in Bihar and focus on the ‘transfer of knowledge and skills’ was implemented (also prompted by the Secretary of State’s announcement of the shift in India from financial aid to technical assistance alone by 2015).
• DFID’s Bilateral Aid to Pakistan:
  DFID responded well to our recommendation to increase its support for the low-cost private education sector (as a means of addressing Pakistan’s education access crisis as well as balancing its support through government). While its approach was still at the pilot stage, we were pleased with the strategy and innovation shown in the programming and were interested in seeing more learning across DFID on strategies for supporting low-cost, government-assisted, private education.
• DFID’s Education Programmes in Nigeria:
  We were pleased to note progress in the Education Sector Support Programme in Nigeria (ESSPIN). We had, however, recommended actions to improve the performance of UNICEF’s Girls’ Education Project (GEP3) with a review of progress after six months but little concrete progress had been made by UNICEF at the time of our original follow-up in 2013, due in part to DFID’s decision to wait until the next Annual Review later that year. The first Annual Review led to the establishment of a project improvement plan in October 2013 which was completed in April 2014. We noted that ‘both DFID and UNICEF had underestimated the scale of change needed to secure a strong capacity building capability in GEP3’.¹¹

For developments within individual country programmes since our last follow-up and key facts and figures relating to the programmes covered, see Annex A2.

DFID is now implementing a more learning-centred approach in its current programming

2.3 We are pleased to report that all six countries are following a comprehensive agenda for education, more centred on pupil learning. Since our reports, DFID has extended its programme coverage and enhanced its own internal processes. DFID has also completed the Education Position Paper (2013), an action taken partially in response to ICAI’s recommendation. In the Executive Summary of this paper, DFID notes, ‘for education to maximise its transformational potential, school youth need not only be in school but also learning’.¹² DFID has fully followed the logic of its position paper in terms of the scope and content of its current education programmes in the six countries as set out in the ‘Learning Framework’.¹³ We consider that this approach provides a sound basis for improved learning outcomes in the future.

2.4 DFID’s experience and good standing enables the technical advice it delivers to support the changes required to provide good education for all children. The scope and content of the technical advice demonstrates the extent to which DFID and the six governments are pursuing positive developments, including:¹⁴

• strategic support to governments embarking on education system transformation;
• continuing to draw more children into regular school attendance;
• assessing the competence of teachers to identify training needs;
• designing and delivering upgraded teacher training with local mentoring;
• conducting regular learning assessments to measure progress;
• generating management information for reviews and policy development;
• strengthening governance and oversight in schools and in local government;
• producing outline forward plans with funding projections; and
• conducting pilots on innovative approaches to meet the need for school places and drive performance improvement.

2.5 DFID recognised the need to strengthen its education cadre, in both numbers and capabilities. In the countries where DFID is increasing its investment in education (i.e. all but India), the number of DFID country-based staff supporting education programmes has gone up by 26% in the past three years.¹⁵ In the six countries, the total annual average expenditure on education programmes is over £350 million, covering both budget support and technical advice. The latest baseline survey, conducted for Tanzania, illustrates the qualitative progress made in its technical advice to governments. We are pleased to see this comprehensive and authoritative baseline assessment conducted for the Education Quality Improvement Programme in Tanzania (EQUIP-T). There is also a sound evaluation strategy which will:

• generate evidence on the impact of EQUIP-T on primary pupil learning outcomes, including any differential impacts for girls and boys;
• examine perceptions of the effectiveness of different EQUIP-T components;
• provide evidence on the fiscal affordability of scaling up EQUIP-T post-2018; and
• communicate evidence generated by the Impact Evaluation to policy-makers and key education stakeholders.¹⁶

2.6 As the latest baseline study, this Tanzania work sets a high standard that could be the template for subsequent baseline work in other countries.

2.7 DFID programmes now have a wider scope and content supporting the transformation of national education systems, placing huge demands on programme management.¹⁷ A key challenge is that the interventions in policy making, institution-building and service delivery in both teaching and learning have to be integrated. Through its risk analysis, DFID needed to strengthen its programme management capability. The process of nominating Senior Responsible Owners (SROs) for all major programmes began in 2014 and will continue through 2015. SROs aim to provide programme management input to complement DFID’s education expertise to secure quality and timely delivery. We welcome this development, as it recognises a potential weakness and is applying a thorough approach to its solution.¹⁸

Longer term delivery challenges

2.8 In addition to these positive findings, we recognise that governments and DFID continue to face major challenges to meet parents’ and other stakeholders’ expectations for more immediate progress in pupil learning. Many components of a successful basic education system have yet to be fully established in our review countries, which increases the challenge. Delivering results will depend particularly on getting enough properly trained teachers into classrooms.

Effective teachers

2.9 More resources are needed to produce sufficient numbers of effective teachers. Pupils learn best when they are taught by competent teachers. Ethiopia illustrates the wider scale of change now being supported by DFID. Numbers in primary schools will rise from 17 million in 2013-14 to 21.6 million in 2017-18.¹⁹ Over the same period, the DFID-funded General Education Quality Improvement Programme (GEQIP) will support 143,000 trainee teachers to complete programmes with updated curricula²⁰ to reduce pupil:teacher ratios and to ensure that 100% of teachers at all levels have been academically qualified, motivated and ethically fit.²¹ This is in line with the overall GEQIP strategy covering:

• strong government commitment and clear policy direction;
• an expanded and well managed teaching force;
• equitable provision of education services;
• a decentralised implementation structure;
• accountability of schools to communities; and
• a robust Education Management Information System (EMIS).

2.10 All six countries conducted recent teacher surveys, revealing the massive training needed to secure the competence levels for effective teaching.²² A state survey carried out for ESSPIN in 2009 found that less than 1% of primary teachers were assessed as competent.²³ Many of the other national surveys identified weaknesses in initial training, as well as demand for continuous in-service training.²⁴ DFID’s Teacher Development Programme in Nigeria directly responds to the teacher surveys. It targets improved student learning by creating more effective teachers through:

• ‘improved training of primary and junior secondary school teachers;
• more effective teacher educators (in the Colleges of Education); and
• strengthened evidence base on teachers’ effectiveness and efficiency, […] aimed at establishing effective regimes for pre- and in-service training’.²⁵

2.11 In addition to targeted training supported by local mentoring, DFID has also successfully advised governments, in line with recognised good practice, to re-design the curriculum, publish new textbooks, set understandable learning benchmarks, devise practical learning assessment processes and develop local oversight by principals, parents and locally-based education officers. All governments have now responded positively to DFID’s input and have produced comprehensive education transformation programmes which have teacher effectiveness as a central aim.

2.12 DFID has supported the creation or improvement of learning assessment systems in many countries. In India, DFID has a major role in improving the quality of assessment and its analysis for policy makers. Regular cycles for assessment are necessary to enable pupil progress to be monitored and to guide improvements to the education system. We have observed a tendency for the national learning assessments to provide results and target information to national policy makers and donors. More attention should be given to providing information and advice to classroom teachers and school principals directly to guide their efforts to upgrade pupil learning.

Realistic timescales

2.13 Improved delivery needs more time to take effect before pupil learning will meet expectations. During the design of current DFID education programmes, it became clear that the large increases in enrolment gained over the past decade will be more difficult to repeat in respect of pupil learning. Many more variables are in play and most countries have fundamental weaknesses in teacher competence and education infrastructure which have to be tackled; this requires the time to build sound foundations and consistent political support. We advise DFID to be clear in its discussions with governments on the probable rates of improvement in pupil learning in the near future, in order that realistic expectations are set. Experience in the quarterly meeting of State Education Commissioners in ESSPIN programme fully confirms the wisdom of the realistic approach.²⁶ Five years on from the initial teacher survey, pupil learning in ESSPIN supported schools is improving but as yet only at a slow rate.²⁷

Management information for decision making

2.14 At the start of a programme, the creation of a comprehensive baseline as part of the design process, as well as being the starting point for measuring progress, is recognised as good practice by DFID.²⁸ In the countries under review, we have continued to see incomplete log frames being approved and submitted to the DFID DevTracker website. We urge DFID to make greater efforts to secure baseline information at the start of a project to provide a firm foundation for work planning and monitoring.

2.15 DFID has helped governments to improve their data collection, quality control and reporting to underpin regular reviews and education policy developments. In Punjab province in Pakistan, DFID’s Education Reform Road Map produces monthly monitoring data for the Province’s Chief Minister. Problems and opportunities are identified and agreed actions implemented.²⁹ The process has linked individual schools to a committed government and is enhancing progress towards better education provision.

2.16 This has been taken one stage further in the annual Joint Review of the Education Sector (JRES) in Rwanda,³⁰ where DFID, at the invitation of the government, plays a joint chair role. This review process is based on an annual data collection process and statistical analyses. This is one of many examples³¹ where DFID’s use of data to inform programme review has contributed in a major way to service improvements.

Regulation and oversight of private sector schools needs strengthening

2.17 In Pakistan, DFID has given strong support to the expansion of ‘low cost private sector schools’ as a significant means of providing access to around 1.5 million children through the Punjab Education Foundation (PEF) in Punjab province. PEF aims approximately to double these numbers in the next three years through its expansion plan. DFID acknowledges that issues of quality, regulation, oversight and teacher pay need much more attention. We urge DFID to work with provincial governments to resolve these issues quickly, to retain the credibility of the policy. As in Pakistan, other countries with low cost private sector approaches (Nigeria and India) also need to resolve quality, regulation and oversight problems to secure positive outcomes for pupils.

2.18 DFID is funding initiatives to encourage Qu’ranic schools to provide basic literacy and numeracy teaching in order that these pupils can progress and access later opportunities such as secondary education. Experience to date shows that this development will take some time before it is fully embedded in the educational systems of these countries.

The wider case for implementing results-based funding has yet to be proven

2.19 DFID is conducting results-based funding pilots in three countries: Rwanda, Ethiopia and Tanzania. In 2015 the final evaluation of the pilots in each country will be published. There are already concerns that national governments may not be able to afford to continue with the pilots, even if they are assessed as successful.³² DFID needs to be sensitive to the constraints on government funds and take this into account when designing new initiatives.

2.20 In Tanzania there is an interesting innovation linking the release of funds to the reduction in pupil:teacher ratios, which currently range from 30:1 to over 80:1.³³ Tanzania will adopt operational strategies to reduce these ratios, which in turn will trigger the release of DFID funds. We expect that DFID will monitor this scheme closely, as it offers a means of targeting inequities. So long as the additional teachers are of good quality this should lead to better quality teaching. Other inequitable distributions of resources (for example classrooms, textbooks and physical facilities) could be targeted in a similar manner.

UNICEF Girls’ Education Project in Nigeria is beginning to operate more effectively

2.21 Our original 2012 report expressed serious concerns³⁴ about the design and plans for this programme working in Northern Nigeria. We recommended immediate action to respond to the weaknesses exhibited in the earlier phase 2,³⁵ in the GEP3 log frame, operational plan and the management structure. DFID chose to wait until the 2013 Annual Review before intervening through the establishment of a project improvement plan (PIP) from October 2013 to April 2014.

2.22 In 2014, the review of the PIP noted the UNICEF effort to recruit a new set of senior managers, their commitment to improved financial control and to improved design, planning and project implementation, including much enhanced monitoring and reporting. DFID has additionally funded the Education Data, Research and Evaluation in Nigeria (EDOREN) programme to fill a capability gap in the UNICEF/GEP3 expertise on monitoring and evaluation. The DFID 2014 Annual Review of GEP3 noted little improvement in the delivery of GEP3.³⁶ Our continuing concerns over the responses from DFID and UNICEF caused us to write to the Secretary of State in 2014.

2.23 The Secretary of State responded in September 2014, reporting that the GEP3 programme had been streamlined, senior management strengthened and more rigorous plans were being put in place.³⁷ GEP3 completed a revised log frame and operational plan by the end of 2014. Baselines still have to be established for the major components of the log frame, a task to be completed in 2015 by EDOREN. DFID decided to terminate funding for the Female Teacher Training Scholarship Scheme (FTTSS)³⁸, a key component of UNICEF’s initial strategy, due to the failure to operate effectively or produce a sound design that could overcome the problems in state administrations. DFID should have acted more resolutely in the early stages in order to secure an effective design and efficient operation of FTTSS.

2.24 We have examined a number of recent programme documents and note an improvement in the design and operations being planned for GEP3.³⁹ The new senior managers are having a positive impact on the work of the programme and GEP3 is applying some of the lessons learnt by the successful ESSPIN.⁴⁰ The extra support provided by EDOREN and the DFID Teacher Development Programme should strengthen the performance of GEP3.

2.25 We are pleased to observe that GEP3 is now on a positive track but we are concerned at the speed of DFID’s response: it took over three years from the start of the programme to strengthen it. It is important that the next phases of GEP3 are monitored closely to establish whether the programme is operating successfully. The GEP3 programme has a critical contribution to make towards meeting the educational needs of young people, especially girls, in Northern Nigeria.

Conclusions and Recommendations

2.26 In all six countries, we have seen that there is a determined focus on improving pupil learning, with a particular emphasis on supporting teachers in the classroom. There are, however, no quick fixes to improve learning and progress is likely to be slower than national ambitions. DFID is well aware of the scale of the challenge to improve pupil learning. The gap between expectation and realisation will have to be carefully managed by governments and DFID. It is sharing good practice and offering the best advice to governments who continue to value DFID expertise and support. DFID should help governments to secure plans and targets which are affordable and achievable. DFID can use its experience in each country better to inform forward planning in all six countries.

2.27 In Pakistan⁴¹ and Tanzania⁴² concerns have been raised about those areas with lower shares of education resources and the consequential impact on pupil learning. Raising standards of pupil learning will require a more equitable distribution of resources in order that all pupils have access to quality education. Whilst the quality of teachers will always be key, an initiative in Tanzania linking DFID funding to reductions in high pupil:teacher ratios could provide some very useful lessons which could be deployed in other countries to remove extreme inequalities in the distribution of resources, including, for example, both very high and very low pupil:teacher ratios.

2.28 DFID has been successful in generating sound annual reviews of education programmes based on good data at a national level. All countries reviewed are strengthening school leadership, parental and community participation and local government support for schools. These developments would be even stronger with improved local management information and a process of local review. DFID could usefully encourage this local process, which could lead to action plans designed to improve performance at this critical level in the education system. Where attention has been focussed on pupil and teacher attendance it has proved to be effective.

3 Main Follow-up Findings

3.1 In this section we have summarised the key findings from our main follow-up work. Each report has been followed up individually. Where there are common issues and themes between reports we have grouped the findings together. The detailed findings for each of the 22 reports we have followed up can be found in the Annex.

How DFID learns

3.2 From 2011-15 DFID has allocated at least £1.2 billion for research, evaluation and personnel development. This has resulted in considerable volumes of research and information, much of which is publicly available. We refer to learning as the extent to which DFID uses information and experience to influence its decisions. DFID’s learning from its experience and the research it has generated does not appear to be consistent. In each of our reviews, we have assessed DFID’s learning and found mixed results.

3.3 The findings from our follow-up work reinforce this picture. For example, we cover below the issue of how the central DFID policy team on Empowerment and Accountability needs to guide learning better on this type of programming across DFID’s country programmes. There are also important learning points arising from our follow-up of reports of DFID’s work in fragile states. For example, we note that some DFID country offices are better at working with beneficiaries in difficult environments than others, so good practices in one should be shared.

3.4 In this section, we focus on a separate report we published in April 2014, How DFID Learns (Report 34), and follow up our recommendations as part of this report.⁴³ The report has re-energised DFID to address organisational learning. Although the formal management response committed only to limited action, DFID has taken the report very seriously. A Task and Finish Group on Learning was established with the objective of developing a DFID Organisational Learning Strategy and Results Framework, by July 2015, to guide actions and track progress across DFID.

3.5 Feedback on the report has been highly positive, particularly from within DFID, from online commentators and from the specialist media. The report is now actively referenced across the organisation, by development NGOs and by other UK government departments.

3.6 DFID took seriously our recommendation to focus on consistent and continuous organisational learning during the implementation phase of its activities. DFID’s response includes seeking to create a better system for collating and using the lessons learnt from annual reviews, project completion reports and evaluations. If successful, this has the potential to reduce significantly the considerable challenge of staff simply having too much information to take in. DFID is also undertaking a multi-country pilot to test external feedback tools. A learning champions network has been established to work together to embed a more co-ordinated approach to organisational learning; furthermore, Professional Development Conferences for DFID advisors now all incorporate reflective sessions. The Permanent Secretary has agreed that DFID’s Alumni Network will help to launch a scheme for alumni to act as mentors for DFID staff. DFID is also seeking to develop key performance indicators that help to define expected behaviours and practices on learning.

3.7 Our report identified a large variation across DFID as to whether (and how much) time was given to staff to learn. We recommended that allocating time to learning was an investment in better delivery and should be integral to it. DFID has stated that it will consider how best to ensure that its staff spend enough time in the field engaging with beneficiaries and delivery partners within their own country; we remain concerned, however, about whether this is taken seriously by senior staff and will be applied consistently among offices.

3.8 We note that there has been a clear and positive attitudinal change in the organisation since our report, coupled with the roll-out of DFID’s Better Programme Delivery process.⁴⁴ Staff feel more able to be open, in particular to discuss failure. Scepticism remains, however, as to whether this change is permanent. We will be interested to see what specific incentives will be put in place in the future to reward managers for enabling their staff to be consistently honest, raising and addressing issues that enable the work of DFID to be more effective.

Empowerment and accountability

3.9 DFID’s internal learning was also a key theme in following up on our report on DFID’s Empowerment and Accountability Programming in Ghana and Malawi (Report 28).⁴⁵ The report focussed on DFID’s promotion of social accountability through support for civil society, the media and national parliaments. We found that DFID’s work was contributing to more constructive engagement between communities and governments around the delivery of public services and development programmes. DFID also responded positively to our recommendation to improve its approach to research.

3.10 Partially in response to our review, DFID adopted the notion of ‘strategic interventions’, which combines three types of interventions:

• improving the enabling environment for collective actions (for example, improving the legal environment for the development of civil society organisations or for promoting freedom of information);
• scaling up citizen engagement with government; and
• bolstering the government’s capacity to respond to citizens’ concerns.

3.11 We welcome the commitment to work simultaneously on the supply and demand sides of social accountability, rather than supporting adversarial models of social accountability.

3.12 Two of our recommendations were only partially accepted. In the first, we suggested that DFID should promote constructive community engagement with government around the delivery of public services and development programmes with closer links to DFID’s sector programmes. DFID did not want to make public service delivery the principal focus of its programming at the expense of programmes directed to achieve broader social and political change.

3.13 DFID also only partially accepted the recommendation to move away from large competitive grant-making funds towards more targeted or managed support with smaller portfolios and more tailored capacity building. While DFID agreed that such a model would be useful in some country contexts, it wanted to retain the flexibility to experiment with different funding models in different contexts.

3.14 Our third recommendation called for a more structured but flexible approach to learning. DFID accepted this recommendation and developed a five year, multi-country research programme on what works in identifying and supporting social and political action for more effective empowerment and accountability, which is soon to be tendered.

3.15 We remain concerned that the central policy team is not resourced to guide learning across the portfolio. In particular, the process of converting data into practical lessons and improvements in programming must be deliberate and well-managed to contribute to learning across DFID’s country offices.

3.16 DFID needs to decide whether these accountability programmes are best approached as discrete activities or integrated with others. This was an issue we saw in Nigeria in our report on DFID’s Approach to Anti-Corruption and its Impact on the Poor⁴⁶ in programmes that involved integrating accountability interventions with service delivery programmes.

How DFID works with the private sector

3.17 Another current key strategic issue for DFID is its work with the private sector. DFID expects to spend £1.8 billion on economic development in 2015-16, over double that spent in 2012-13. A large share of this will be aimed at stimulating the private sector, including direct support to enterprises and working with governments to create a more conducive investment environment in developing countries. We reviewed DFID’s Private Sector Development (PSD) Work (Report 35)⁴⁷ in our report published in May 2014. The report came at a time when DFID was undertaking a major restructuring in its approach to economic development, including private sector development. This may account for the management response’s limited acceptance of the initial recommendations. We noted from our follow-up work, however, that the changes that DFID is introducing are often in line with our recommendations.

3.18 We recommended that DFID should provide clearer guidance to its staff on how to design a coherent and well-balanced PSD country portfolio that matches DFID’s aim to end extreme poverty through economic development and transformational change.

3.19 Our follow-up work showed that DFID has taken a number of steps to translate its high level objectives into practical implementation. These include the pilot and subsequent roll-out of a new inclusive growth diagnostic. The diagnostic is a tool to help identify current patterns of growth, together with an analysis of the opportunities for and the constraints to economic development. Options are then developed to address the constraints. We expect the growth diagnostics to allow for greater co-ordination and guidance from the centre without being overly prescriptive and we believe the guidance on the diagnostic should be well disseminated among and used by DFID’s private sector advisors.

3.20 We also recommended that DFID should clearly define and articulate where it can add most value in PSD relative to other donors and stakeholders and be more realistic in its ambitions and the impact it is trying to achieve. The roll-out of the economic growth diagnostic requires DFID country offices to consider these issues. The options for interventions addressing the constraints are to be developed in the light of DFID’s comparative advantage. The related guidance also sets out the limitations that developing different sectors can have in generating inclusive and systemic development. DFID appears to be heading in the right direction; it still needs, however, to refine its approach to building a coherent portfolio at the country level, including the particular challenges for the private sector in fragile states.

3.21 DFID has undertaken a number of initiatives centrally to understand better the needs of and to work more closely with the private sector. These include the development of a corporate relationship management system, an advisory panel with representation from the private sector and a number of private sector roundtables focussing on specific sector issues. In-country relationships with the private sector still, however, depend heavily on individual relationships of advisors in post. These issues are covered in more depth in our report on Business in Development.⁴⁸

3.22 Despite the progress DFID is making on redefining its approach to economic development and the private sector we still consider that DFID could do more to (a) determine its global comparative advantage in delivering PSD projects; (b) move faster to adapt its processes to be more responsive when working with and when seeking to influence the private sector; and (c) ensure that the programmes developed reflect the realistic opportunity of countries to grow a private sector. Given its vital importance in DFID’s future strategy, this is an area that ICAI may choose to revisit in the future.

Working in fragile states

3.23 DFID has been increasingly focussing its activities on fragile states, where it has committed to spend 30% of ODA by 2014-15. This amounts to almost doubling the value of the UK’s bilateral support to fragile states compared to 2011-12. This figure excludes multilateral expenditure and expenditure through centrally-managed programmes. In February 2015, ICAI published a separate review assessing the Impact of the Scale-up of DFID’s Support to Fragile States (which is outside the remit of our follow-up work).⁴⁹

3.24 For this follow-up review ICAI considered four previous reviews of DFID programmes in fragile states: DFID’s Water, Sanitation and Hygiene Programming in Sudan (Report 19),⁵⁰ DFID’s Peace and Security Programme in Nepal (Report 20)⁵¹, DFID’s Health Programmes in Burma (Report 25)⁵² and DFID’s Bilateral Support to Growth and Livelihoods in Afghanistan (Report 31).⁵³ Broadly, these reviews were consistent with the conclusions of ICAI’s 2015 fragile states review, that DFID does not yet have ‘a convincing positive trajectory towards achievement of the UK Government’s stated goal of tackling conflict and fragility’,⁵⁴ even if individual programmes are delivering basic services on the ground.

3.25 The fragile states reviewed which are currently in conflict – Afghanistan and Sudan – illustrate the particular difficulties of planning ahead in fragile state contexts. DFID has been carrying out one of its largest ever humanitarian operations in Sudan in response to the crisis in Darfur, which has been in conflict since 2003. Previous ICAI follow-up work recommended that DFID move away from short term emergency channels as they were creating dependency. DFID had agreed with the recommendation and was planning to focus on longer-term development and resilience programmes. The context in Sudan, however, has significantly deteriorated since the fieldwork was done for the initial ICAI review in 2012. The separation of South Sudan, failing peace process efforts and increasing numbers of displaced people have meant that DFID Sudan has not been able to focus on resilience as it intended, due to the necessity of responding to more pressing humanitarian needs.

3.26 Both the Afghanistan and Sudan reviews reinforced the importance of lesson learning within the context of fragile states where DFID does not have significant time for programme design. Consequently, long term planning of development interventions can be difficult. DFID should, where possible, address both short-term needs and longer-term goals simultaneously. This could be done by focussing interventions on addressing the chronic challenges in a manner that bridges the distinction between strictly humanitarian and development approaches.

3.27 Good examples of such a way of working can be seen in DFID Sudan’s programmes. DFID Sudan has attempted to introduce longer-term approaches to emergency measures in its programming, for example by exploring options to introduce payments for services such as water for internally displaced people (IDPs). This contributes to sustainable financing of service provision but also reduces tensions between the local populations and IDPs. DFID Sudan’s investment in the World Food Programme’s (WFP) electronic food voucher programme (instead of direct food provision) further illustrates the organisation’s shift towards a more sustainable emergency assistance delivery model.

3.28 In our Sudan review we had recommended that, at corporate level, DFID should ensure that its WASH policy framework prioritises early planning for transition from emergency assistance through early recovery to development programming in the context of protracted and chronic crises. DFID’s management response stated that it would, by September 2013, ‘produce guidance on multi-year approaches to humanitarian planning to improve the effectiveness of its work in protracted crises’.⁵⁵ DFID produced such guidance in 2013. The guidance, however, has not had the desired impact as the humanitarian advisors in charge of planning and programming were not aware of the document. In February 2015 DFID Sudan was provided with a review of best practice and advantages of multi-year humanitarian business cases. It remains to be seen whether this document will have a more significant impact than the 2013 guidance.

3.29 There are particular difficulties working with governments in fragile states. These, however, should not constrain all DFID activities, which has happened in the past. For example, in Afghanistan delays in appointing senior government positions meant that progress in some of DFID’s programmes was slowed. As a possible way around this difficulty, DFID should collaborate with other stakeholders, including local government and civil society organisations, to ensure continuity of its programmes when working with the central government is challenging. DFID Burma, in its health programming, provides a good example of effective working with government at different levels, as well as with other stakeholders.

3.30 In response to ICAI’s recommendations, DFID Afghanistan has contracted statistics and evaluation advisors, who provide training and are responsible for ensuring that suppliers are able to meet DFID’s requirements on beneficiary and independent monitoring and evaluation. It has also clearly outlined its approach to evidence, analysis and learning.

3.31 We previously made a specific recommendation on Afghanistan that DFID should do more to engage with local government offices, NGOs and other civil society organisations. Although it was not a formal recommendation, a similar concern was raised about DFID Nepal, which DFID Nepal has addressed. Security constraints in Afghanistan, however, mean that DFID staff are able to visit a very limited number of provinces, which makes conducting beneficiary consultations and monitoring and evaluation work particularly difficult. In our original review of DFID’s Support to Growth and Livelihoods in Afghanistan, we recommended that DFID Afghanistan should develop a list of Afghan research and development organisations with the expertise to undertake beneficiary consultation. DFID has only partially responded to this recommendation, having undertaken a number of scoping activities under individual programmes to explore the supplier market in Afghanistan for beneficiary monitoring work. DFID Afghanistan has not, however, to date produced a comprehensive list of Afghan research and development organisations that could be called upon to conduct beneficiary monitoring. This list would be especially useful when DFID country staff rotate.

3.32 DFID Nepal, partially in response to ICAI’s recommendations, increased its beneficiary monitoring activities. It co-funded, with the United Nations Development Programme (UNDP), a survey on peace and security across 13 of the most unstable districts in Nepal covering 3,000 respondents. The survey findings are informing, in particular, the reformulation of DFID Nepal’s work under the new Peace Support programme and the Security and Justice for the Poor programme. DFID Nepal also strengthened the methodology and approach of a Government-led ‘perception survey’ as part of its formulation of the 2014-2017 Nepal Peace Trust Fund (NPTF) Strategy. The survey results were distributed widely within the Nepali government, NGOs and international bodies involved in the peace process. We welcome this type of beneficiary consultation. We suggest that DFID adopt this, or more innovative approaches, on beneficiary feedback across all its programmes in fragile states. This was an issue that we have also covered in our reports on the Scale-up of DFID’s Support to Fragile States and DFID’s Approach to Anti-Corruption and its Impact on the Poor.

3.33 DFID Nepal has developed a comprehensive peace and security strategy, which combines peace and security-related outputs, indicators and baselines of DFID Nepal’s programmes and covers directly and indirectly attributable results.

3.34 Our review of DFID’s Health Programmes in Burma awarded an overall ‘green’ rating, as we found that DFID had designed and delivered an appropriate health aid programme in a country with significant capacity and access constraints. DFID had demonstrated clear leadership and co-operation with beneficiaries, the local government and other donors.

3.35 Since the initial ICAI review, the DFID health programme is now delivered mainly through the Three Millennium Development Goals (3MDG) Fund.⁵⁶ The Fund focusses on maternal, newborn and child health; HIV/AIDS, tuberculosis and malaria; and health systems strengthening. Despite positive results, such as an increased number of midwives trained and the reviews of prison healthcare and the health system, some issues remain. The progress in developing the 3MDG Fund has been slower than expected but the recent involvement of Burma’s Ministry of Health in the 3MDG fund board should improve the fund’s transparency and relationship with the government.

3.36 DFID has been playing a key role in managing the challenging relationship between the Government of Burma and the 3MDG Fund. DFID should continue being active in ensuring that the Fund works effectively with the national government and other organisations that operate in the health sector in Burma.

3.37 The lack of reliable data on the sector makes it difficult to assess accurately the added value of DFID’s various contributions and of the work of the 3MDG Fund. The National Household Survey on Demography and Health, which is to be published in March 2016, should help to address this concern.

3.38 As part of our follow-up work, we found examples of well-managed transitions in key DFID personnel. To help to enable a successful transition, successors were identified early, hand-over included up to a month of joint working in Burma and after handover regular contact was maintained with predecessors. This approach has had clear benefits that can be replicated elsewhere where similar changes in key DFID personnel occur.

Working across government departments

3.39 We followed up on reviews of aid programmes implemented by government departments other than DFID: the Foreign and Commonwealth Office’s (FCO) and British Council’s use of aid in response to the Arab Spring (Report 24)⁵⁷ and the Evaluation of the Inter-Departmental Conflict Pool (Report 12).⁵⁸

3.40 Both reviews’ previous follow-up findings commented on the positive engagement of the organisations with ICAI’s recommendations. We have seen that the institutional arrangements of these organisations were not always best suited to facilitate efficient implementation of aid programmes. For example, the FCO managed the Arab Partnership Participation Fund (APPF), which had a budget of £40 million over four years, half of which was contributed by DFID. We commented on its weak project management capacity but acknowledged the impact of the requirement to respond rapidly and flexibly to a highly volatile situation that was considered a high priority for UK foreign policy.

3.41 We have noted that other government departments do not have DFID’s level of project management capacity and culture of managing for results. The FCO responded well to our recommendation of incorporating theories of change into its 2014-15 country strategies and bringing in DFID experts to conduct training sessions for its staff on theory of change methodology. The FCO is also working with its partners to set baselines and generate the data needed to measure the impact and progress of its funded activities. It may take several programme cycles for theory of change thinking to be fully embedded into FCO strategy setting.

3.42 We had also recommended that the APPF should strengthen its knowledge management processes and share experiences and lessons learned across its country programmes and partners. The FCO has responded by bringing together its country and London based staff in conferences and thematic learning events; as well as by increasing the numbers of meetings that are held with local partners and other donors in each country. The FCO has developed an intranet for the APPF, in order to enable the sharing of documents and the referral of questions across the network. We welcome the strengthening changes that the FCO is making to create a network for sharing knowledge and envisage that they will contribute towards a more coherent portfolio.

3.43 Institutional constraints have made it more difficult for the FCO to respond to our recommendations on project management. The FCO has a one-size-fits-all approach to grant-making with complex and rather inflexible procedures, which do not distinguish between organisations with different levels of delivery and financial management capacity. This makes it difficult for the FCO to partner with local civil society organisations (CSOs). Nevertheless, the FCO has been able to fund CSOs via international partners, such as CARE, the British Council, BBC Media Action and the National Council for Voluntary Organisations. Furthermore, the FCO’s financial accounting system is not suitable for project management. The FCO agreed that the system is inadequate. It has not, however, yet been able to change it. In the absence of a new system, the FCO has begun to prepare other software tools that can help to work around the problem.

3.44 We have seen the FCO working to professionalise its project management systems in advance of the commencement of the new Conflict, Stability and Security Fund (CSSF) – the successor to the Conflict Pool (CP) – which began operations in April 2015. The CSSF is under the direct authority of the National Security Council (NSC) and supported by a joint secretariat which is housed in the FCO. It reports to the National Security Secretariat in the Cabinet Office and is staffed by officials from across the NSC departments. The FCO’s grant-making procedures are not yet fully developed, its accounting systems are not yet designed for project management and the cadre of trained project managers within its staff is not particularly strong. We are concerned about the FCO’s readiness to manage the increased volume of aid funds under CSSF: CSSF combines the CP portfolio with the Peacekeeping Budget and the Deployed Military Activity Pool and has a total budget of £1 billion a year.⁵⁹

3.45 Our 2012 review of the Conflict Pool found that it was delivering useful, if localised, results. It lacked a clear strategic framework, however, had cumbersome governance and management arrangements and had little capacity for measuring results. This year we followed up on whether our recommendations regarding the CP informed the design of CSSF. Our preliminary review suggests that there is a stronger focus on strategic and programme management in the design of the CSSF. The programme standards are still under development but the current guidelines mandate stronger country strategies based on explicit theories of change, with higher-level outcomes and indicators.

3.46 These new strategies and guidelines will need to be tested through implementation. DFID should proactively share its good practice tools and guidance on programme management with other departments undertaking development work.

3.47 We are concerned about the potential risk that the way in which the CSSF is funded will generate unhelpful competition between departments. We are also concerned that it may become reactive in nature, due to excessive demands on the NSC’s time and consequently insufficient attention paid to strategy and programme management. Both ICAI and the IDC may choose to keep the CSSF under scrutiny in the coming years.

Influencing international organisations

3.48 Almost two thirds (£6.32 billion or 62.9%)⁶⁰ of DFID’s £10.1 billion total expenditure was spent through multilateral agencies in 2013-14. This includes where DFID country offices channel funding through multilaterals at a local level as delivery partners. As part of this review, we followed up on DFID’s Support for Palestine Refugees through the United Nations Relief and Works Agency (UNRWA) (Report 27)⁶¹ and DFID’s Oversight of the EU’s Aid to Low Income Countries (Report 17).⁶² DFID responded positively to the original recommendations of both reports but challenges remain about engaging with such a range of international organisations. The former review highlighted that DFID, as UNRWA’s fourth largest donor, contributing over £170 million in support from 2008-2012, continued to be a highly influential and valuable contributor at strategic and operational levels.

3.49 DFID seconded a planning officer to provide technical support in drafting UNRWA’s medium term strategy document and to work with hosts, donors and UNRWA to identify strategic priorities to shape UNRWA’s planning process. DFID has been advocating for reform within UNRWA, particularly in relation to UNRWA’s provision of vocational training, education and health, as well as encouraging UNRWA to move away from food to cash assistance. It is important that DFID considers using short term contracted technical assistance to provide the input needed to push forward reforms (including those to ensure that UNRWA is financially sustainable) at the operational level and to ensure that the field offices are fully engaged in the reform process.

3.50 Influencing partners and working across Whitehall are also vital in DFID’s contributions to the European Union (EU). In our report on DFID’s Oversight of the EU’s Aid to Low-Income Countries, we were concerned about the limited assurance DFID has on the significant contributions it makes to the EU, especially given DFID’s restricted discretion to vary those contributions. In our follow-up work last year we found that, overall, DFID and the EU had made solid, if slow, progress. Some further progress has been made since then: DFID has provided support to the development of the EU’s results framework and has helped to ensure that key development challenges (such as delivering better results for girls and women) are prioritised by the EU. Following requests by DFID and others, the new Development Commissioner (Neven Mimica) is now calling this issue one of his top priorities.

3.51 There remains a major opportunity for DFID to work more effectively with EU delegations in-country. In our report on How DFID works with multilateral agencies to achieve impact⁶³ we recommended that DFID needs clear objectives for working with multilaterals in its country-level strategies and that working with the EU should be prioritised for this (along with the World Bank). We also note that significant levels of DFID funding, through its multilateral contributions, go into countries where there is no DFID presence. DFID funding through the EU is a significant element of this. In our Multilaterals report we state that we would like to see a greater focus by DFID on this area.

Capital projects in Montserrat

3.52 Influencing governments is also essential in many DFID programmes. We followed up our report on DFID’s Support to Capital Projects in Montserrat (Report 26),⁶⁴ where working with the government is a key focus. The UK spent £114.8 million on support to Montserrat between 2009 and 2013, which amounts to approximately £23,000 for each of the 5,000 people on the island. DFID’s contribution to Montserrat’s budget amounts to 60%.

3.53 We examined DFID’s support to capital investments in areas such as transport and power and recommended that DFID develop a long term coherent plan for the island that better integrated the programme of projects as a whole. This recommendation was not accepted. Our follow-up work, however, found that DFID has been working with the new Government of Montserrat (GoM) to develop a longer term, more strategic approach to financial planning, as well as practical ways to integrate better capital and revenue planning. We therefore repeat our recommendation that in developing a clear and concise strategic plan with the GoM over the next 18 months, DFID should ensure that it shows how the different projects add up to a coherent impact on livelihoods and economic progress. It should set out the level and composition of financial and technical assistance and a projection of capital costs and necessary revenue support over a minimum five year period. DFID should also ensure that this plan is widely publicised among the population on the island.

Child Mortality in Kenya

3.54 Our report on DFID’s Contribution to the Reduction of Child Mortality in Kenya (Report 33)⁶⁵ had a significant impact, contributing to DFID’s decision to exit malaria work in Kenya. We recommended that DFID should develop a clear exit strategy for funding basic health supplies in Kenya (such as bed nets) and rebalance support to the Government of Kenya to strengthen its health system and focus on achieving a long-term and co-ordinated approach amongst development and financing agencies, including the Global Fund to Fight AIDS, Tuberculosis and Malaria and the Gavi Alliance, to strengthen the health system.

3.55 The decision to exit from malaria work, which DFID is scheduled to do by December 2015, was not welcomed by other donors. The USAID health sector lead in Kenya stated that there will be grave consequences, as USAID and the Global Fund will not be able to fill the funding gap. Local DFID staff noted that their influence in the sector could also be reduced. We consider that DFID is moving strategically in the right direction as an exit from funding basic supplies will permit DFID to focus on a broader approach to strengthening Kenya’s healthcare system. We are, however, concerned about how the recommendation is being implemented, namely the speed of DFID’s exit from funding basic supplies. It is also not clear whether the money saved from DFID’s exit from work on malaria will be invested in a co-ordinated approach to the strengthening of Kenya’s health system, as proposed in our report.

3.56 We were also concerned about the devolution of health services in Kenya, where power and financial authority are being delegated to 47 counties. DFID is responding well to the challenge and continues to prioritise remote and hard-to-reach regions, as evidenced by its new maternal health programme. DFID policy division is producing guidance on value for money assessments in malaria programmes that includes consideration of equity. The guidance will be published in 2015. These actions, whilst welcome, are insufficient. Our original recommendation was that DFID should centrally specify its policy on equity and reaching remote populations: at the moment there is no corporate equity guidance on how to make trade-offs between reaching remote populations and the increased costs associated with doing so.

3.57 We were encouraged that the issues of abuse and poor quality of care that were highlighted to us by beneficiaries have been taken up with the Government of Kenya and are currently under active consideration. If these issues are addressed, the quality of care for mothers should significantly improve.

3.58 Our research for the initial review revealed that local DFID Kenya staff were not aware of all the DFID projects operating in Kenya. We had suggested that DFID code all its projects on an information system in order to allow country staff immediately to access information about the centrally-managed programmes that operate in their countries. We have seen some progress, as DFID has started mapping its current activities globally and DFID Kenya has organised sectoral meetings to draw in centrally-funded projects. We have not, however, been given a date as to when our recommendation of having an easily accessible database of DFID projects operating in each country will be available. While information sharing about central DFID programmes is improving and some progress has been made on the real integration and leverage of them in country portfolios, there is still more to do.

Support to agriculture

3.59 Our review of DFID’s Support to Agricultural Research (Report 29)⁶⁶ considered DFID’s responses to the estimates that food demand is expected to increase by 70% by 2050 as a result of population growth and changing food habits. DFID has been funding agricultural research in developing countries for the past 50 years; its annual expenditure in 2011-2013 amounted to £75 million. DFID’s research is focussed on three main areas:

• developing new agricultural products (crops, livestock breeds, resilient farming systems);
• testing interventions and delivery models for rapidly scaling up the use of new technology by farmers; and
• increasing the understanding of the complex political, social and economic context that determines the success of agricultural investments.

3.60 We found DFID’s agricultural research programme to be effective and innovative. In order for the research to have the intended effect, however, we stressed that DFID’s research and development programmes should collaborate better. DFID responded well to our recommendation, scoped out possible new development programmes and developed a geo-referenced database of research projects, which enables country offices to identify quickly what agriculture research is being undertaken in their country. The agriculture team has started to work closely with a number of DFID’s country programmes to find ways to integrate new agricultural technologies into poverty reduction programmes. This has the potential to improve significantly the impact for beneficiaries.

3.61 We remain concerned that DFID has not done enough to scope the needs of different classes of farmers and women farmers, which could result in the needs of the most vulnerable and marginalised farmers being overlooked in favour of those better off. DFID has also not addressed the issue of environmental sustainability in a satisfactory manner – it should place more emphasis on the potential adverse environmental impacts of conventional farming systems, such as land degradation, and also focus more on low-input, ecologically-focussed agricultural technologies.

Improving nutrition

3.62 Our review of DFID’s Contribution to Improving Nutrition (Report 36)⁶⁷ found that DFID has played a leading role in mobilising the global development community to combat undernutrition and in setting the global agenda. DFID also responded to the challenge of global undernutrition by significantly increasing its investments in nutrition. It has committed £3.3 billion to nutrition in the period 2013-20. DFID’s programmes concentrate on interventions during the first two years of a child’s life, as interventions over this period have been shown to have the greatest impact on reducing stunting and improving cognitive development.

3.63 Overall, DFID responded positively to our recommendations. It is too soon to assess fully the impact of our recommendations as our report was only published in mid-2014 but we have seen significant progress. Our report helped initiate an internal Nutrition for Growth Strategy Group, which is chaired by one of the Africa directors in DFID. It will set the strategic direction for DFID’s nutrition work, horizon scan and lead the refresh of DFID’s approach. There is evidence of increasing engagement from some country offices, such as those in Somalia and Nigeria, although in Pakistan there was evidence that the office was not drawing enough on the guidance and best practice available in its design of new programmes in nutrition. In addition, in January 2015, DFID completed a draft Nutrition and Economic Development Action Plan. The plan could benefit from a stronger theory of change with clearly identified priority actions for work with the private sector.

3.64 DFID has yet to respond to our recommendation that it should develop guidelines for country offices on selecting the best package and mix of interventions that would have the greatest impact on stunting, in the local context. Unless this is done and mentoring is provided to country teams, it will not maximise impacts on stunting. DFID also should issue guidance to staff to ensure that nutrition interventions specifically address the needs of the most vulnerable.

Critical interventions

3.65 A number of our reviews tackled issues at what have proved to be critical moments in the life of the programme. In the follow-up of education reports earlier in this review, we noted our concerns about the UNICEF Girls’ Education Project (GEP3) in Nigeria and how, although immediate action was needed to respond to weaknesses we identified, it took over three years from the start of the programme to strengthen it. This section covers the follow-up of three reviews undertaken at critical times in the programme cycle providing DFID with the opportunity for timely course correction or lesson learning.

3.66 Our report on DFID’s Trade Development Work in Southern Africa (Report 30)⁶⁸ (TMSA) revealed serious deficiencies in the TMSA programme. It was the first and only report so far to have been given a ‘red’ rating. Our work in the first quarter of 2013 prompted an Internal Audit investigation by DFID and resulted in the eventual closure of the programme in March 2014. Mechanisms were then put in place to transfer TMSA activities to more effective DFID partners. DFID has taken action to ensure more effective management of the programme during the closure period, including financial oversight, procurement, target setting and monitoring; it has since returned to its more traditional aid delivery in the form of technical assistance rather than infrastructure.

3.67 DFID has responded to ICAI’s recommendations by fundamentally changing its risk management processes for this programming without reducing its commitment to trade work. For example, the planned replacement for the regional trade programme will have a stronger focus on poverty, a focus on small and informal traders and an improved regional/national balance. This has the potential to lead to improved development outcomes. We continue to encourage innovation and managed risk-taking in the design of interventions. It is also key that design should consider and address the risk of fraud, corruption and inadequate governance. In light of the TMSA example, DFID should make sure that, for future risk management, it distinguishes between risk arising from the context of an intervention and risk of internal mismanagement.

3.68 One of our previous concerns about the programme was the misreporting to DFID of TMSA’s achievements against targets. DFID has addressed this in the new programme which is currently being designed, by planning to appoint an independent monitoring and evaluation contractor to work independently alongside the main management agent. We welcome the shift towards continuous independent monitoring by other suppliers. DFID has adopted this approach for other programmes, including TradeMark East Africa.

3.69 We recommended that DFID should undertake comprehensive analysis of the impact on trade and the poor, both at the outset and throughout implementation, when developing trade programmes. Our previous follow-up work noted that, despite the good work done by DFID on trade and poverty methodologies, neither TMSA nor DFID Southern Africa had used or referred to it. This remained the case despite DFID’s own stock-take, which showed that, after six months, most staff responsible for the management and design of trade programmes within DFID were aware of the new guidelines. We remain concerned that there is no mechanism to ensure that the review and the previous guidance produced are used in the future.

3.70 DFID has been engaging in more research activities, such as supporting studies and seminars to analyse the positive and negative effects of the Tripartite Free Trade Agreement (TFTA) on country and regional levels. It has not yet followed up to see if its actions have stimulated policy responses in the countries concerned.

3.71 In the course of our work following up on DFID’s implementation of the recommendations of our report on DFID’s Trade Development Work in Southern Africa, we examined how our recommendations had affected the management of other regional trade programmes. In so doing, we learnt that payment in advance of need, an issue we found on the TMSA report, may also have occurred in the case of the Support to West Africa Regional Integration Programme (SWARIP), a programme similar to TMSA in that it aims to help reduce the costs of doing business in West Africa through better regional integration.

3.72 During our follow-up work on TMSA, DFID informed us that, as part of the SWARIP programme, £1 million was paid to the Economic Community of West African States (ECOWAS) Secretariat in 2010-11. This was paid into a pooled donor fund in which DFID funds were mixed with those of other donors. It was therefore not dedicated as such to specific activities which DFID was alone in funding.

3.73 A DFID Internal Audit Department report into its funding to ECOWAS in 2012 found that support for the transfer of £1m in December 2010 in one transaction was unnecessary, especially as the 2010 audited statements showed a surplus of nearly $1.5m at the year end. The report also concluded that this funding may have been in advance of need. By this point DFID’s funding to the multi-donor fund had been suspended, and has not been resumed since then. DFID have requested that ECOWAS provides details on disbursement of this money but as of May 2015, and despite DFID following up on this, ECOWAS has not provided any details of how this money has been spent.

3.74 DFID has informed us that, despite concerns that funding has been in advance of need, it does not have evidence that this has either been spent or has been spent on inappropriate items, and it is continuing to push ECOWAS for clarity on the use of this funding and has demanded the return of funds that have not been properly accounted for. In 2015, we will be following up on this to ensure that DFID has gained full clarity on the use of this funding and, if these funds have been disbursed inappropriately, is taking action to secure their return.

3.75 Another report where we had significant concerns was Girl Hub: a DFID and Nike Foundation Initiative (Report 5).⁶⁹ Girl Hub is a joint initiative of DFID and the Nike Foundation which started in 2010, so our review in 2012 was relatively early in the life cycle of the programme. Girl Hub aims to empower adolescent girls to reach their potential. It does this by influencing decision-makers and donors to do more to assist vulnerable adolescent girls in the areas of education, health, safety and economic opportunity. It also delivers branded communication platforms, such as radio broadcasts and magazines to change attitudes and behaviour.

3.76 Our original report raised concerns about governance, how DFID worked with Nike Foundation and a lack of clarity on how the programme was intended to effect change and how this would be measured. Our 2013 follow-up work on Girl Hub noted that DFID had made progress on many of the earlier ICAI recommendations, fully or partly. The further follow-up work undertaken for this report was particularly timely as current DFID funding will end by summer 2015 and consideration is being given to a Phase 2 of the project. DFID has agreed further short term funding to the Girl Hubs in Ethiopia and Rwanda and is providing strategic advice to Girl Hub and Nike Foundation on future planning.

3.77 We recognise the importance of work with girls and the innovative nature of this partnership. We remain concerned, however, about aspects of governance, such as the lack of clarity over Nike Foundation’s role and the degree of DFID oversight. Girl Hub is located in Nike Inc’s offices. Nike Foundation has members on the board (including its CEO and President, who chairs the Girl Hub board) but DFID does not have any members nor does it attend the Girl Hub board meetings or receive the minutes of the meetings. DFID communicates directly with the Nike Foundation on all matters concerning Girl Hub. It meets Nike Foundation senior management every six months in a Senior Strategic Collaboration Group and at other times on the management of its accountable grant. It is not clear how effective this group is at addressing strategic issues.

3.78 The total value of Nike Foundation’s ‘in-kind’ support is £16.7 million. A breakdown of Nike Foundation’s in-kind support is not available to DFID or ICAI. DFID needs to understand the detailed contribution of Nike Foundation to assess properly the overall value for money of the programme. There are no other funders.

3.79 A recent evaluability review commissioned by DFID concluded that Girl Hub has evidence on what kinds of outcomes have been achieved but less evidence to attribute these results directly to Girl Hub. In our view, DFID currently lacks independent evidence on the effectiveness of the programme on which to base its Phase 2 funding decision.⁷⁰ We reported our concerns about these issues in a separate, more detailed note to DFID in March 2015 so that they could take these into account when renegotiating the arrangement.

3.80 Our Rapid Review of DFID’s Humanitarian Response to Typhoon Haiyan in the Philippines (Report 32)⁷¹ was undertaken at a critical moment in DFID’s response. The review visit took place very shortly after the typhoon hit the Philippines and the report was delivered only a few weeks after the team returned. We gave the report an overall ‘green’ rating, noting that DFID was well prepared to act swiftly and decisively, mobilised efficiently and met the urgent needs of the affected communities.

3.81 Our rapid review enabled us to see humanitarian work at the point of delivery in the aftermath of a natural disaster and to provide timely feedback on its effectiveness. The timing of the report was welcomed by DFID, as it permitted DFID to learn from its experience in the Philippines when it was relevant and live and so contribute to the organisation’s transparency and learning objectives.

3.82 DFID credited our review with helping to improve accountability and focus on DFID’s Conflict, Humanitarian and Security Department’s (CHASE) priorities, such as scaling up stockpiles. In line with our recommendation and CHASE’s own plans, DFID increased its stockpile of non-food items in size and flexibility and has continued to deliver significant value for money through its logistics capability. This has allowed DFID to respond rapidly and efficiently in crisis situations, such as Typhoon Hagupit in November 2014 and Cyclone Pam in Vanuatu in March 2015.

3.83 In line with our recommendations, DFID has allocated £5 million to a trust fund managed by the Asian Development Bank (ADB), to support the Government of Philippines (GoP) to ‘build back better’, as set out in its Yolanda Comprehensive Recovery and Reconstruction Plan.⁷² The fund is directed by a government-managed steering committee. Contributions to the ADB fund, however, have been disappointing. The only other contribution to date is approximately $400,000 from Finland. UNDP had set up a separate fund prior to the ADB and the majority of donors that DFID was hoping would join the ADB fund opted for the other alternative. DFID continues to support the ADB fund, as it is more closely aligned with the GoP’s priorities. We are concerned about how DFID will ensure value for money for its investment in the ADB fund and urge DFID to keep the ADB fund under close scrutiny, collaborating with the FCO, which currently represents DFID’s interests on the ADB fund’s steering committee.

3.84 There are still challenges on DFID’s ability to influence ‘building back better’ and longer-term climate resilience efforts. We recommended that DFID should set out clearly how it would respond to humanitarian emergencies where it has no in-country presence, incorporating lessons from the Haiyan response. While DFID has not yet completed this action, it is work in progress and will incorporate lessons learnt from DFID’s responses in Haiti, Syria and Iraq as well as for the Ebola crisis, which tested DFID’s ability to set up and start operating rapidly from makeshift facilities. There may also be useful lessons from looking more generally at DFID’s approach to working with multilateral organisations in countries where it does not have an office. This is something we recommend DFID focus on more in our report on How DFID Works with Multilateral Agencies to Achieve Impact.

4 How DFID has Responded

4.1 Having analysed the progress DFID has made in responding to recommendations in individual reports, in this section we look at the way DFID has responded to ICAI’s recommendations overall. We consider some of the challenges we have seen for DFID in responding clearly and effectively and where improvements can be made to how responses to recommendations are developed. We also identify what lessons there are in our follow-up work for future follow-up work by ICAI.

Overall DFID has taken action on the great majority of ICAI’s recommendations

4.2 The follow-up work undertaken for this report covers 22 of the 35 reports we published during our first three years: 13 from 2013-14 and 9 from earlier years. This gives us a good basis from which to assess how DFID has responded to recommendations over a wide range of different reports and over time. Overall, we can see that DFID has taken action to address the great majority of ICAI’s recommendations. DFID country offices and teams within DFID centrally have mostly shown, both in their approach and in the actions taken, a very serious intent to address issues that we have raised.

4.3 From our follow-up work we can see that, in many cases, the clear action taken by DFID to address recommendations recognises that we are raising issues that help DFID teams to take forward challenges, both internally and with the partners they work with, in implementing programmes. This was the case, for example, in DFID’s Support for Palestine Refugees through UNRWA. Another example is that some of the DFID country offices covered by our education-focussed reports have told us that they found the analysis a useful basis for dialogue with government and other stakeholders.

4.4 In other cases, our recommendations support the strong learning agenda of the responsible DFID team. For example, our Rapid Review of DFID’s Humanitarian Response to Typhoon Haiyan in the Philippines was ‘fundamental’ to DFID’s learning from the Philippines response.⁷³ We recognise and commend the positive efforts DFID teams have made in addressing our recommendations.

Ownership from the right people is key to effective responses

4.5 We have noticed that, as we have moved to producing more thematic reports, the challenge of responding effectively has increased, because a larger number of people (cutting across teams in DFID) are required to come together to make the changes needed. For example, in DFID’s Contribution to the Reduction of Child Mortality in Kenya we found that there was a strong response to the report from the country office in Kenya but that the issues that were more corporate (for example providing clearer central guidance on how to make trade-offs between reaching remote populations and the increased costs that this involves) have not been responded to as effectively.

4.6 Ownership by key people in DFID is essential if recommendations are to be implemented thoroughly and benefits realised. This is more complex for thematic reports. We saw, however, in the response to our How DFID Learns report that complex cross-cutting issues can be taken up and worked on across DFID teams in a joined-up way. DFID needs to ensure that, in such cross-cutting cases, the individuals leading the responses are able to address issues across DFID, whether at the centre, in-country or both.

Where other departments are involved, there are specific challenges

4.7 Where other government departments are involved, there is less development expertise and a greater need for cross-departmental co-ordination. In the follow-up to our report on the Foreign and Commonwealth Office’s and the British Council’s use of aid in response to the Arab Spring, for example, we noted that, although the FCO had begun to work more seriously on results management, this requires a significant culture change that will take time to get right.

4.8 Our follow-up to the Evaluation of the Inter-Departmental Conflict Pool focussed on the new CSSF. As we noted earlier, the CSSF will be under the direct authority of the National Security Council (NSC). Cross-government working is inherently difficult but, added to this, the departments in question have limited project management capacity and no embedded culture of managing for results.

4.9 Some of the same issues with cross-Whitehall working were also apparent in our work on DFID’s Support to Capital Projects in Montserrat and Oversight of the EU’s Aid to Low-Income Countries. In Montserrat, several government departments need to combine for an effective overall approach and our follow-up work showed that this is now improving. In its work with the EU, DFID needs to work more closely with the Foreign Office in countries where there is no DFID country office and development work needs to be led through the Foreign Office post.

4.10 More of what DFID is planning to do will be under a cross-departmental approach in the future. We note, however, that an understanding of development approaches is typically less embedded when a cross-departmental approach is used. The tools used to plan interventions and assess progress, such as business cases and structured monitoring and evaluation, are not so well understood or used. They are areas where wider UK objectives and national interest also play a role in shaping the nature of the interventions, while still fully meeting the definitions of overseas development assistance. Yet the process of signing off a country programme, which in a DFID-focussed country is where most of the UK contribution will be, is still a very DFID-centric one.

4.11 There is an an opportunity to address these barriers and ensure more effective joined-up interventions in the future. We consider that the NSC should play a more active role in signing off the overall strategic response in different countries and regions. For example, in Pakistan, where DFID is a major investor, there should be both a greater recognition of the political dimension and its impact on what makes for effective interventions; and a greater understanding of the development agenda and the tools required to deliver it. This is an important area for ICAI, which may choose to revisit this in the coming years.

Improvements can be made to the way responses to recommendations are developed

Understanding the recommendation

4.12 Whilst DFID’s response to recommendations and the actions taken have been positive, it is clear that sometimes the DFID team responding has not fully understood the aim of the recommendation or the underlying issue. It is notable that, in follow-up meetings with the senior DFID officials responsible for responding to reports, we are often told that they wish that they had had these meetings earlier to understand better the Commissioners’ intent. It appears to us, from our research and meetings with senior DFID staff, that the formal responses have not always been based on a thorough look at the Conclusions and Recommendations section of our reports, where more context and detail about the recommendations is provided than in the Executive Summary.

4.13 We have worked on providing fewer, more succinct recommendations. We have also begun to engage more with DFID teams in the process of formulating responses, for example on our report on Assessing the Impact of the Scale-up of DFID’s Support to Fragile States. We have done this by meeting with DFID staff, following publication of the report, to clarify their understanding of our key messages and to discuss ideas related to their management responses. This deeper interaction should mean better quality responses, leading to more effective action by DFID. It is important, however, that findings and recommendations are not negotiated in any way before publication as this could compromise ICAI’s independence.

Consistency of the formal management response to recommendations

4.14 In our Annual Report last year we noticed a lack of consistency in some of the management responses from DFID: particularly in the categorisation of ‘accept’, ‘partially accept’ and ‘reject’. We noted this in particular where DFID states that it is already doing work: DFID sometimes accepts the associated recommendations; sometimes partially accepts them; and sometimes rejects them. In our follow-up work this year we have seen this lack of clarity continue for a number of reports.

4.15 For example, several of our recommendations in our report on DFID’s Support to Capital Projects in Montserrat were rejected but we then saw evidence of action being taken that should address some of the issues we raised (for instance, on improving the maintenance of capital investments and taking a longer term approach to financial planning). In the case of our report on How DFID Learns, the formal response to our recommendations did not reflect the significant action that DFID was, in practice, taking.

4.16 How ‘partially accept’ is interpreted is a particular problem. We note that there has been an increase in the proportion of recommendations that were partially accepted for the Year 3 reports – from about 10% in Years 1 and 2, to over 30% in Year 3. The rationale for why recommendations were partially accepted is usually not clear in the responses and, as reported above, action often has, in fact, been taken to address them.

4.17 One case where there was a good explanation for the partial acceptance of recommendations was in the response to our report on DFID’s Empowerment and Accountability Programming in Ghana and Malawi. DFID provided a more detailed breakdown of what it did and did not agree with in the recommendations, using the ‘management actions already taken’ section of the response to do so. Our sense is that responses vary in tone depending on the underlying attitude of the responding team – some see recommendations as a constructive vehicle for change but some as a critique to be defended.

4.18 Evidence of DFID tracking implementation is lacking and not readily available at the start of the follow-up process. For example, the management response to our recommendations in our report on DFID’s Health programmes in Burma did not fully reflect some of the challenges in implementation, such as the proposed baseline survey. This was modified at the request of the Government of Burma to be combined with the forthcoming National Household Survey on Demographic and Health.

Lessons for ICAI from follow-up work

4.19 From our experience of undertaking follow-up work over the last three years – and particularly for this report – there are a number of lessons for the future operation of ICAI. We recognise the work and effort that the recommendations we have made require of DFID. We have sought to raise only the most vital issues emerging from our work in the simplest, most direct and comprehensible way. Generally, of course, more work is required by DFID where the report raises more significant concerns or where a more thematic cross-DFID issue needs addressing.

4.20 An important issue is the timing of some ICAI reviews. From our follow-up work it is clear that sometimes the most important findings have come from reviews where it was not initially seen as the right time to review an area. For example, it was seen as too early to investigate when models were emerging or where processes were already under review. This applied in our reviews of Girl Hub and Private Sector Development for instance.

4.21 In these circumstances, it can be tempting to wait for a programme or stage to complete before assessing it. It is important, however, not to be put off conducting an examination at a difficult time. Rigorous monitoring in real time allows valuable course correction. For example, on DFID’s Trade Development Work in Southern Africa, if the problems we identified had been left until further time had been spent on the programme, more taxpayers’ funds would have been wasted. Similarly, if the challenges that we identified on the GEP education programme in Nigeria had been responded to more quickly, then the needs of school children in Northern Nigeria could have been met sooner.

Footnotes

1. 1. 2. Annual Report to the House of Commons International Development Committee 2013-14, ICAI, June 2014, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Annual-Report-13-14-FINAL.pdf; Annual Report to the House of Commons International Development Committee 2012-13, ICAI, June 2013, http://icai.independent.gov.uk/wp-content/uploads/2011/11/ICAI-Annual-Report-2012-13.pdf.