DFID’s Education Programmes in three East African Countries

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We wish to assess the education programmes funded by the Department for International Development (DFID) in three East African countries. These Terms of Reference outline the purpose and nature of the review and identify the main themes that it will investigate. A detailed methodology for the study will be further developed during an inception phase.

2. Background

2.1 The 2000 World Education Forum concluded that education ‘is the key to sustainable development and peace and stability within and among countries, and thus an indispensable means for effective participation in the societies and economies of the twenty-first century’.¹

2.2 By 2014 across Africa and Asia, DFID is committed to supporting 9 million children in primary school and 2 million in secondary school (700,000 of whom will be girls); and to train more than 190,000 teachers to improve the quality of education and children’s learning.² This study will evaluate DFID’s current activities in the East Africa region, comparing the education programmes in three countries: Tanzania, Rwanda and Ethiopia.

2.3 Measuring the value for money and effectiveness of education involves looking at not only how well the current education system works (e.g. levels of enrolment and retention in primary and secondary education) but also longer-term economic and social outcomes such as literacy levels and ultimately employability. Ultimately, what matters is the positive, lasting impact of education on the lives of intended beneficiaries.

Enrolment

2.4 Working to increase levels of enrolment in primary and secondary education is an important first step in delivering this impact. According to the United Nations Educational, Scientific and Cultural Organisation (UNESCO): ‘Nearly 29 million children of primary school age in sub-Saharan Africa – 54% of them girls – were not enrolled in school in 2008’.³ DFID has prioritised increasing coverage of education through to 2015 as part of its commitment to meet the Millennium Development Goals. This review will assess whether different programmes have met their enrolment goals.

2.5 In developing countries, poverty hinders the education of girls, who are less likely to be in school, especially those living in rural areas. Studies have shown, however, that an educated woman is more likely to invest in her family and the education of her children, providing a multiplier effect for her community: ‘When a girl in the developing world receives seven or more years of education, she marries four years later and has two fewer children.’⁴ She is also better equipped to find a job and often more empowered in her family and more active in her community and government.⁵ The education of girls is directly linked to decreased risk of maternal death and better health outcomes overall.

Approaches to investment in education

2.6 One approach to investment in education which has gained in popularity is the use of results-based incentives for teachers and students who achieve learning outcomes. DFID plans to include this approach in its future investment in education.

2.7 Such programmes have been effective in Bangladesh and Mexico. In Bangladesh, the number of girls enrolled in secondary education increased to 3.9 million in 2005 from 1.1 million in 1991, through a stipend programme. 33,000 teachers were trained for 7,000 participating schools and 25 new secondary schools were built in remote areas to reach disadvantaged communities. Secondary School Certificate pass rates for girls in the project increased from 39% in 2001 to 63% in 2008.⁶

2.8 In Mexico, the Oportunidades incentive programme offers grants for school attendance. The grants increase as children progress to higher grades and, beginning at the secondary level, are slightly higher for girls than for boys due to the higher drop-out rates among girls. The programme had the largest impact on secondary school enrolments, which increased by over 20% for girls and 10% for boys.⁷

2.9 In this review, we will explore the effectiveness of both traditional education financing and results-based programmes as ways of improving education outcomes.

Case studies for this review

2.10 Case study countries for this review will be Ethiopia, Tanzania and Rwanda. The three countries each have a different approach to their education strategy. Each is a recipient of DFID financial support to major government programmes, either as general budget support, sector budget support or support to a national initiative. DFID also provides funds for specific education projects and advisors.

3. Purpose

3.1 To assess and compare the value for money and effectiveness of DFID’s education programmes in Tanzania, Rwanda and Ethiopia and to learn lessons for education programmes more generally if possible.

4. Relationships to other evaluations/studies

4.1 In 2002, DFID published an evaluation of assistance to primary education from 1998 to 2001, which described and commented on the change from projects to sector-wide approaches that had taken place during the period.⁸ One of the conclusions of this evaluation was that:

‘Evaluation has been insufficiently integrated into the design of education projects and programmes. Although DFID has been sensitive to the mistakes that appear to have been made in design and implementation, their identification has depended too much upon serendipity, and not enough upon careful analysis of relevant evidence…

The mistakes and misallocations experienced with project aid in the education sector could have been reduced, not only by ensuring more effective evaluation, but also by paying closer attention to the evidence on the efficacy, or otherwise, of favoured innovation.’

4.2 In 2005, as part of a broader study looking at gender equality and women’s empowerment, DFID undertook an evaluation into education, focussing on girls.⁹ The report noted that:

‘…the main focus of attention in DFID’s work on education has been on primary schooling. This is insufficient to ensure gender equality and women’s empowerment is achieved. Although the Gender MDG targets gender parity at the post-primary level, DFID has fewer programmers at the post-primary level, and these are less likely to identify gender as an area of concern compared with those at the primary level.

Considerable knowledge has been generated, including within DFID, with regard to constraints to achieving gender equality in education, and strategies that work to support this goal. However, it is not always clear how this knowledge is being disseminated to, and used by, country programmers.’

4.3 More specifically for East Africa, the Strengthening Primary Education Programme in Kenya was evaluated in 1998, covering the period 1991-96.¹⁰ Among the evaluation’s findings was that the design had shortcomings, such as an over-dependence on UK rather than local experience, a lack of involvement of key local individuals and general over-ambition. Whilst the project evaluation noted that impact was observed in teacher training and system strengthening, ‘the project has had no direct impact on access, equity or wastage’. Notably, it found that ‘cost recovery [from parents and children] was out of control, that schools encouraged wastage and that there was resistance to new practices’.

4.4 The National Audit Office undertook a study of bilateral support to primary education in 2010,¹¹ which looked at DFID programmes in Ghana, Kenya, Ethiopia and India. At the subsequent Public Accounts Committee hearing, DFID was criticised for not placing enough emphasis on value for money in deciding where and how to spend and not going beyond enrolment to look at attainment and the outcome experiences of children.¹²

4.5 UWEZO, an East African organisation, produced a report in July 2011 entitled Are our children learning? Numeracy and Literacy across East Africa.¹³ This looked at educational outcomes in Tanzania, Kenya and Uganda by carrying out a survey of children’s literacy and numeracy. The report found that, although levels of enrolment have increased in each of the three countries, literacy and numeracy levels remain poor.

4.6 This review will take into account the reports discussed above as well as other learning to date.

5. Analytical Approach

5.1 This study will seek to draw conclusions on the value for money and effectiveness of UK education funding in the East Africa region. It will focus on outcomes beyond enrolment – through the stages of attendance, progression, completion and attainment – to the wider impact, as far as it is possible to assess.

5.2 The study will involve delivery and impact evaluations of elements of the education programmes. The study will take particular account of the impact of the context on the delivery and performance of the UK’s aid. It will draw conclusions and determine lessons both specifically for the country programmes and on a comparative basis as well as learning wider lessons for education programmes generally if possible.

5.3. Due to constraints of time, the study should primarily draw evidence from currently available data, particularly impact monitoring and evaluation information provided as part of programme reporting. It will, wherever possible, seek to validate such monitoring evidence using third party assessments.

5.4 The study should systematically identify funding, inputs, activity, outputs and (where they are measurable) outcomes for each programme. For example, we will seek to examine the following aspects of education programmes:

6. Indicative Questions

6.1 This review will use as its basis the standard ICAI guiding criteria and evaluation framework, which are focussed on four areas: objectives, delivery, impact and learning. The questions outlined below comprise those questions in our standard evaluation framework which are of particular interest in this review, as well as other pertinent questions we want to investigate. The full, finalised list of questions that we will consider in this review will be set out in the inception report.

6.2 As this is a comparative study, all questions will need to be explored in each case study country in order that variations in outcomes between countries and programmes can be explained as far as possible.

6.3 Objectives

6.3.1 Is there a clear and convincing plan, with evidence and assumptions, to show how the programme will work?

6.3.2 Is the assistance based on a strategic, realistic and well-evidenced assessment of options?

6.3.3 Does the programme complement the efforts of government and other aid providers and avoid duplication?

6.4 Delivery

6.4.1 Is the choice of funding and delivery options appropriate?

6.4.2 Is general budget support an adequate aid delivery mechanism for promoting the necessary improvements in education quality?

6.4.3 Are resources being leveraged so as to maximise impact and provide value for money?

6.4.4 Does this programme take into account the needs of the intended beneficiaries, using available knowledge and appropriate methods?

6.5 Impact

6.5.1 What does the available evidence tell us about achievements against the output and outcome indicators?

6.5.2 What is the long-term and sustainable impact of the programme? What are the prospects for improvement, including assuring financial sustainability and local ownership?

6.5.3 Is there transparency and accountability to intended beneficiaries, donors and UK taxpayers?

6.6 Learning

6.6.1 Are there appropriate arrangements for monitoring inputs, processes, outputs, results and impact?

6.6.2 Is there evidence of innovation and use of global best practice?

6.6.3 Is there anything currently not being done in respect of the programme that should be undertaken?

6.6.4 Have lessons about the design and delivery of the programme been learned and shared effectively?

7. Outline Methodology

7.1 The study will have a number of elements:

• a contextual analysis on East Africa as a region, including using available literature and expert evidence, focussing on the three case study countries;
• a review of DFID’s documentation and files on their education programmes in the three case study countries, focussing on delivery mechanisms and evidence of performance;
• a review of independent and third-party assessments of education impact data in the case study countries;
• a detailed analysis of costs, benefits and rates of return at each stage of the delivery chains within programmes, including involvement of the private sector (where relevant);
• meetings with intended beneficiaries and their communities in case study countries (both direct and representative, both urban and rural);
• interviews with country-based respondents from the government, DFID, civil society and peer organisations;
• direct observation of delivery;
• work by region-experienced staff from the Center of Evaluation for Global Action (CEGA) to draw on CEGA’s own impact evidence and to assess the work of others;
• cross-referencing to other relevant ICAI reviews; and
• follow-up of any relevant International Development Committee recommendations.

8. Timing and Deliverables

8.1 The review will be overseen by Commissioners and implemented by a small team from ICAI’s consortium. The review will begin in November 2011 with a final report available in the first quarter of 2012.

ANNEX

Education spending in case study countries

The following table sets out data on education spending in our case study countries, from the 2011 Education for All Global Monitoring Report.¹⁴ This includes both public expenditure and total aid spending.

1. Direct aid is aid reported as direct allocations to the education sector.
2. Total aid to education is calculated as ‘direct aid to education plus 20% of general budget support (aid provided to governments without being earmarked for specific projects or sectors), the latter representing the estimated 15% to 25% of budget support that typically benefits education’.

DFID education spending in case study countries¹⁶

Footnotes

1. ↑ Education For All: Meeting Our Collective Commitments, text adopted by the World Education Forum Dakar, Senegal, 26-28 April 2000, www.unesco.org/education/efa/fr/ed_for_all/dakfram_eng.shtml.
2. ↑ UK Aid; Changing Lives, delivering results, DFID, 2011, www.dfid.gov.uk/Documents/publications1/mar/BAR-MAR-summary-document-web.pdf.
3. ↑ Education for All Global Monitoring Report 2011: Sub-Saharan Africa Regional Overview, UNESCO, 2011, http://unesdoc.unesco.org/images/0019/001913/191393e.pdf.
4. ↑ The Girl Effect Data: Why should we pay attention to girls?, The Girl Effect, www.girleffect.org/uploads/documents/1/Girl_Effect_Fact_Sheet.pdf.
5. ↑ Africa’s Future is Female, ONE, http://one.org/international/documents/africasfuture/africasfuture_en.pdf.
6. ↑ IDA at Work: Bangladesh, Stipends Triple Girls Access to Schools, International Development Association, http://web.worldbank.org/WBSITE/EXTERNAL/EXTABOUTUS/IDA/0,,contentMDK:21227882~menuPK:3266877~pagePK:51236175~piPK:437394~theSitePK:73154,00.html.
7. ↑ Shanghai Poverty Conference: Case Study Summary, World Bank, http://info.worldbank.org/etools/docs/reducingpoverty/case/119/summary/Mexico-Oportunidades%20Summary.pdf.
8. ↑ From Projects to SWAPs: An Evaluation of British Aid to Primary Schooling 1988-2001, DFID, EV639, September 2002, www.oecd.org/dataoecd/49/26/35078940.pdf.
9. ↑ Evaluation of DFID Development Assistance: Gender Equality and Women’s Empowerment, Phase II Thematic Evaluation: Education, DFID, Working Paper 11, March 2005, www.dfid.gov.uk/Documents/publications1/evaluation/wp11.pdf.
10. ↑ Strengthening Primary Education in Kenya, DFID, December 1998, www.dfid.gov.uk/Documents/publications1/evaluation/ev627.pdf.
11. ↑ Bilateral Support to Primary Education, National Audit Office, June 2010, www.nao.org.uk/publications/1011/dfid_support_to_education.aspx.
12. ↑ Examination of Witnesses: the Department for International Development’s bilateral support to primary education, Public Accounts Committee, December 2010, www.publications.parliament.uk/pa/cm201011/cmselect/cmpubacc/594/10111002.htm.
13. ↑ Are Our Children Learning? Numeracy and Literacy across East Africa, UWEZO, 2011, www.uwezo.net/index.php?i=214.
14. ↑ The hidden crisis: armed conflict and education, 2011 Education for All Global Monitoring Report, UNESCO, 2011, http://unesdoc.unesco.org/images/0019/001907/190743e.pdf.
15. Purchasing power parity is an exchange rate adjustment that accounts for price differences between countries, allowing international comparisons of real output and income.
16. ↑ Information from: DFID Ethiopia Operational Plan 2011-15, DFID, April 2011, www.dfid.gov.uk/Documents/publications1/op/ethiopia-2011.pdf;
    DFID Rwanda Operational Plan 2011-15, DFID, May 2011, www.dfid.gov.uk/Documents/publications1/op/rwanda-2011.pdf; and
    DFID Tanzania Operational Plan 2011-15, DFID, May 2011, www.dfid.gov.uk/Documents/publications1/op/tanzania-2011.pdf.