DFID’s approach to empowerment and accountability

1. Introduction

1.1 The Independent Commission for Aid Impact (ICAI) is the independent body responsible for scrutinising UK aid. We focus on maximising the effectiveness of the UK aid budget for intended beneficiaries and on delivering value for money for UK taxpayers. We carry out independent reviews of aid programmes and of issues affecting the delivery of UK aid. We publish transparent, impartial and objective reports to provide evidence and clear recommendations to support UK Government decision-making and to strengthen the accountability of the aid programme. Our reports are written to be accessible to a general readership and we use a simple ‘traffic light’ system to report our judgement on each programme or topic we review.

1.2 We have decided to conduct an evaluation of DFID’s empowerment and accountability work in Africa. In recent years, the goal of empowering citizens to hold their states to account has become an important element of DFID’s approach to improving government performance and service delivery in developing countries. Within DFID, empowerment and accountability refers both to a strand of governance programming and to a cross-cutting theme increasingly incorporated into programming in other sectors.

1.3 These Terms of Reference outline the purpose and nature of the evaluation and the main themes that it will investigate. A detailed methodology will be developed during an inception phase.

2. Background

2.1 While the idea of empowering citizens to hold their states to account has been part of DFID’s development philosophy for many years, empowerment and accountability has been given a more prominent place in its governance programming since DFID’s 2011 Bilateral Aid Review.¹

2.2 Over the past decade, DFID and other donors have conceptualised governance programming as having ‘supply’ and ‘demand’ sides. Supply-side programmes work on building state capacity through public sector reforms. The shortcoming of such programmes is that they tend not to work without political support. Demand-side governance programming was conceived as a way of building political support. It refers to initiatives designed to increase public pressure on governments to improve their performance.

2.3 While empowerment and accountability is not a sector with clearly defined boundaries, relevant DFID programming includes:

• support: for civil society organisations (CSOs), including non-governmental organisations (NGOs), community-based organisations, business associations and trade unions; for research, advocacy, campaigning, monitoring, network building and social mobilisation;
• support for formal accountability institutions such as parliamentary committees, public auditors, ombudsmen and anti-corruption commissions;
• investments in institutional reforms designed to improve the climate for empowerment and accountability, such as free media, civil rights and government transparency; and
• the development of social accountability mechanisms, such as citizen report cards and social audits, often as part of sectoral programmes.

2.4 ‘Social accountability’ can be defined as: ‘an approach towards building accountability that relies on civic engagement, i.e. in which it is ordinary citizens and/or civil society organizations that participate directly or indirectly in exacting accountability’.² It can be portrayed as a four-stage cycle:

• Information: the sharing of information on rights, resource allocation and government performance to increase citizen understanding;
• Feedback: the ability of communities to communicate their feedback or grievances to government;
• Response: the capacity and willingness of government to respond to that feedback, which may involve changes to political processes; and
• Redress: improvements in government performance in response to citizen demand.

If these steps all work, they should lead to better development outcomes. Empowerment and accountability programmes help to strengthen one or more aspects of this accountability loop, either for society in general or in respect of specific issues.

2.5 The 2011 Bilateral Aid Review included a commitment to scaling up DFID’s investment in this area.³ DFID’s Business Plan 2011-2015 states that DFID will ‘support 40 million people to have choice and control over their own development and to hold decision-makers to account’⁴ (see Figure 2 on page 3). From 2011, DFID rules stipulated that, in countries receiving general budget support, an amount equivalent to 5% of budget support expenditure must be spent on strengthening national accountability.⁵

2.6 To support these commitments, DFID has been assessing the evidence base on what works in empowerment and accountability programming.⁶ It is developing an online facility for country offices to share ideas and experiences and seek advice from colleagues and external experts. It plans to commission a series of impact evaluations of empowerment and accountability programmes. The Governance, Open Societies and Anti-Corruption Department (GOSAC) within Policy Division acts as a central hub for this work, while the Africa Regional Division has a specialist adviser in this area.⁷

2.7 DFID also supports a series of long-term research projects that are closely related to its empowerment and accountability work. It supported the Development Research Centre on Citizenship, Participation and Accountability (‘Citizenship DRC’), a research consortium bringing together researchers from donor and developing countries and supported by a co- ordinating team based at the Institute of Development Studies. The consortium has produced various publications assessing international experience in this area.⁸ DFID also supports the African Power and Politics Programme (APPP), which brings together research centres and think tanks in the UK, France, Ghana, Niger, Uganda and the USA to work on a range of issues, including developing ‘a realistic take on elections, citizen empowerment and public goods’.⁹

2.8 There are at present no firm data available on DFID expenditure on empowerment and accountability. This is because empowerment and accountability is often included as a component of wider programmes on governance or other sectors and is not accounted for separately in DFID’s financial reporting. Given DFID’s commitment to spending an amount of equivalent to 5% of general budget support on accountability programmes, the total sum should be not less than £11 million and is likely to be considerably more.¹⁰

3. Purpose of this review

3.1 To assess whether DFID’s programming on empowerment and accountability is designed and delivered effectively and is likely to achieve meaningful results for its intended beneficiaries.

4. Relationship to other reviews

4.1 There is limited evidence available on the impact of empowerment and accountability programming, in large part because of the technical difficulties involved in assessing impact. Much of the current programming is based on theory, rather than solid evidence of what works.

4.2 To address the evidence gaps, DFID’s Research and Evidence Division (RED) commissioned a ‘preliminary mapping’ of the evidence base, which was published in April 2011.¹¹ It reviewed existing studies to assess the impact of donor empowerment and accountability programming in four areas: service delivery; governance outcomes; political transformation; and confidence, capability and access to assets. It found the evidence to be fragmentary and incomplete. The academic literature also contains a number of other syntheses of impact evaluations in related areas.¹²

4.3 DFID now proposes to conduct a ‘macro-evaluation’ to collect and synthesise the evidence available across the empowerment and accountability portfolio in 28 countries. To this end, it commissioned an initial assessment in order to identify possible evaluation questions and determine whether evidence would be available to answer them. It found major gaps in the evidence and recommended a staged process, including identifying and describing relevant DFID projects, producing a synthesis of existing reviews and evaluations and finally developing an updated theory of change for this policy objective. This process is due to begin in June 2013 and will be completed in 2016. The first phase will produce a mapping of DFID programming on empowerment and accountability and will take four to six months.

4.4 In view of this work, our evaluation will not attempt to map DFID’s empowerment and accountability portfolio. Instead, we will select some specific empowerment and accountability activities to examine in depth, with a view to assessing their results to date and the factors that determine their prospects for success.

5. Analytical approach

5.1 To complement DFID’s planned overview work, our evaluation will focus on programming in two countries and explore specific links in the results chain. These links may include:

• how effectively the programmes increase citizen awareness of government spending and performance; and
• how effectively the programmes enable citizens to provide feedback on government performance.

5.2 We have also consulted with the International Development Committee on the scoping and focus area for this evaluation. Committee Members expressed an interest in particular in examining how accessible information on aid projects and government performance is for intended beneficiaries, particularly in areas with limited access to electronic communications or social media.

5.3 Our review will focus on social accountability programmes of the type summarised in paragraph 2.4. It will cover both programmes that are solely designed to promote social accountability and elements of social accountability included within major sectoral programmes, such as health, education and rural development. We will not examine electoral assistance (which was the subject of a previous ICAI evaluation¹³) or economic empowerment initiatives such as cash transfer programmes (subject of a recent National Audit Office review¹⁴).

5.4 The regional focus will be Africa, with Ghana and Malawi as the two case study countries. These have been selected based on a number of criteria, in particular the scale and maturity of their empowerment and accountability programmes. They also illustrate different country contexts. Ghana has a relatively well established democratic system, with two peaceful changes of government through the electoral process. It has a strong civil society sector, which has received donor support for advocacy and accountability-related activities over many years. By comparison, Malawi’s civil society sector is not as developed and its political environment is apparently less conducive to citizen engagement and influence.¹⁵

5.5 DFID’s empowerment and accountability portfolio is relatively new. Most of its programmes have been launched since the 2011 Bilateral Aid Review, based on ’emerging guidance’.¹⁶ While DFID has provided support to civil society organisations for many years in some African countries, these programmes were usually not designed to facilitate impact evaluation. Given the immaturity of the programmes, our approach will be to focus our assessment on the initial phases in the results chain – namely, the transmission of information on government spending and performance to citizens and the development of channels through which citizens provide feedback on government performance. By reviewing the available results on information and feedback loops, we will be able to make an assessment as to whether the programmes are delivering the results expected at this stage of their implementation. While we may not be able to test the theory of change through to improvements in government performance, we will be able to make an assessment as to whether the programmes are on track to deliver results to the intended beneficiaries.

5.6 In each of the case study countries, we will make a selection of programmes to review in detail. The selection will include programmes that are dedicated to supporting social accountability and sectoral programmes (e.g. in health, education or agriculture) that include elements of social accountability. The sample is likely to include:

• Strengthening Transparency, Accountability and Responsiveness in Ghana (STAR¹⁷): a joint donor project supported by DFID, the United States, the European Union and Denmark, with a DFID contribution of £12.5 million or 48% of the total from 2010 to 2015.¹⁸ STAR contains a range of initiatives to improve the accountability and responsiveness of the Government of Ghana, chiefly by working with civil society organisations and traditional authorities;¹⁹ and
• the Building Empowerment and Accountability in Malawi (BEAM) programme: a £19.7 million portfolio programme supporting work with civil society and citizens as well as more formal institutions and initiatives such as the Anti-Corruption Bureau and public financial management reform, to support empowerment and accountability. The review will look in particular at two components of BEAM:
  • the Community Based Monitoring Programme (in its current second phase known as ‘Kalondolondo’). Kalondolondo is a three-year, £2.5 million programme to support the use of scorecards to strengthen social accountability around local services together with advocacy at the national level on policy issues; and
  • the Tilitonse multi-donor civil society governance fund. Tilitonse is a four-year, £15 million programme that aims to provide funding to local organisations and coalitions whose work improves levels of information, monitoring and influencing on government policies and programmes to increase citizen voice and promote increasingly inclusive, accountable and responsive governance. Tilitonse promotes the use of political economy and broad-based coalitions to achieve its goals.

5.7 For each programme, we will assess:

• whether the design is sound, based on current theories and evidence, including whether it is developed with the participation of intended beneficiaries, whether it makes effective use of DFID and other research and guidance material and whether it is informed by sound political analysis;
• whether it is being delivered effectively; and
• what results are being delivered currently and are likely to be achieved in the future.

6. Indicative evaluation questions

6.1 During the inception phase, we will develop an evaluation framework setting out the evaluation questions and the methods to be used for answering them. Likely evaluation questions will include:

6.2 Objectives

6.2.1 Does DFID have sound strategies for promoting empowerment and accountability, with clear objectives and logical ‘theories of change’?²⁰

6.2.2 Are DFID empowerment and accountability programmes based on sound technical guidance and international experience?

6.2.3 Are empowerment and accountability programmes relevant to the country context, developed in consultation with the intended beneficiaries and based on sound political analysis?

6.3 Delivery

6.3.1 Are the programmes effectively delivering their agreed activities and outputs?

6.3.2 Are intended beneficiaries and national stakeholders effectively engaged?

6.3.3 Are the programmes flexible and responsive to changes in circumstances?

6.3.4 Are risks to the achievement of programme objectives identified and managed effectively?

6.4 Impact

6.4.1 Are the programmes delivering the results expected at this stage of their implementation?

6.4.2 Are the programmes likely to deliver clear and significant benefits for the intended beneficiaries?

6.4.3 Are the results likely to be sustainable?

6.5 Learning

6.5.1 Are the programmes drawing on international experience and evidence as to what works?

6.5.2 Are there appropriate arrangements for monitoring inputs, processes, outputs, results and impact?

6.5.3 Does DFID manage and share knowledge effectively in this area of programming?

7. Methodology

7.1 A detailed evaluation methodology will be developed during the inception phase. It is likely to include the following elements across two phases.

7.1.1 Advance preparation is likely to include:

• a literature review of current theories and evidence underlying donor empowerment and accountability programming, making use of existing syntheses as far as possible;
• a review of current approaches to measuring the impact of empowerment and accountability programming, including consultations with DFID and UK development CSOs;
• a review of DFID’s guidance and technical support for empowerment and accountability programming, including a review of DFID documentation and interviews with the relevant DFID policy teams. This may include a survey of DFID governance advisers; and
• choosing a sample of programmes to review, including dedicated empowerment and accountability programmes and elements of empowerment and accountability within major sectoral programmes.

7.1.2 Field research in Ghana and Malawi is likely to include:

• consultations with DFID staff, implementing partners, national counterparts and other informed observers on the programmes and the contexts in which they are being delivered;
• interviews with intended beneficiaries and targeted government service providers; and
• detailed examination of the selected programmes, including assessment of implementation arrangements, delivery of activities, outputs and, to the extent possible, outcomes.

7.2 During the first phase, we will select the sample of projects and make an assessment of the availability of data on emerging impacts. We will determine the best method of obtaining feedback from the intended beneficiaries on the interventions. This will include additional field research among the target population, such as through surveys or focus groups, to supplement information available from regular project monitoring. These methods may be used to test whether there has been any change in knowledge of and attitudes towards government performance within the target population. This will be supplemented by interviews with intended beneficiaries and other stakeholders to determine whether there has been any development of mechanisms through which citizens provide feedback to government on its performance.

7.3 There are a number of well-known methodological challenges to evaluating empowerment and accountability programmes. First, there is the inherent difficulty of measuring impact. These programmes seek to influence complex political interactions between states and citizens. Any changes they bring about are difficult to quantify. Furthermore, political processes evolve rapidly and are influenced by many factors beyond the scope of the programme, making attribution difficult. Programmes are often set up in a flexible manner to respond to opportunities as they arise, with the result that objectives and approaches change too rapidly to allow for longitudinal analysis (i.e. tracking changes in set variables over time). During the inception phase, we will assess how these challenges apply to the programmes we are reviewing and will ensure that our methodology is as robust as it can be.

7.4 In all our work, we place particular emphasis on the importance of beneficiary consultations. For empowerment and accountability programmes, the challenge is that, while the direct recipients of DFID support may be intermediary organisations such as CSOs, the intended beneficiaries are the public at large. As CSO grantees have an interest in the continuance of the support, they cannot be treated as representatives of or proxies for the public. Our methodology may, therefore, need to capture changes in public knowledge or attitudes through surveys or focus groups.

8. Timing and deliverables

8.1 The evaluation will be undertaken by a small team from ICAI’s consortium and overseen by Commissioners. The lead Commissioner will be Diana Good. The review will commence in April 2013, with a final report available during the third quarter of 2013.

Footnotes

1. Bilateral Aid Review: Technical Report, DFID, March 2011, http://www.dfid.gov.uk/Documents/MAR/FINAL_BAR%20TECHNICAL%20REPORT.pdf.
2. Social Accountability: An Introduction to the Concept and Emerging Practice, World Bank, December 2004, http://siteresources.worldbank.org/INTPCENG/214578-1116499844371/20524122/310420PAPER0So1ity0SDP0Civic0no1076.pdf.
3. Bilateral Aid Review: Technical Report, DFID, March 2011, http://www.dfid.gov.uk/Documents/MAR/FINAL_BAR%20TECHNICAL%20REPORT.pdf.
4. Department for International Development Business Plan 2011-2015, DFID, May 2011, http://www.dfid.gov.uk/Documents/DFID-business-plan.pdf.
5. Although the formal commitment states that DFID will spend an amount equivalent to ‘up to 5%’ of its budget support on accountability programming, in practice it has treated the commitment as spending ‘at least’ 5% of its budget support. Strengthening accountability in budget support countries: briefing note for DFID country offices, DFID, 2012.
6. A preliminary mapping of the evidence base for empowerment and accountability, DFID, April 2011.
7. Governance, Open Societies and Anti-Corruption Department (GOSAC) Operational Plan 2011-2015, DFID, May 2012, http://www.dfid.gov.uk/Documents/publications1/op/gosac-2011.pdf.
8. For example, Blurring the Boundaries: Citizen Action Across State and Societies, Citizenship DRC, 2011, http://www.drc-citizenship.org/system/assets/1052734700/original/1052734700-cdrc.2011-blurring.pdf?1302515701.
9. About APP, http://www.institutions-africa.org/page/about-us.
10. DFID’s total spending on general budget support in 2011-12 was £225 million. Department for International Development Annual report and Accounts 2011-12, DFID, July 2012, page 90, http://www.dfid.gov.uk/Documents/publications1/departmental-report/2012/Annual-report-accounts-2011-12.pdf.
11. A preliminary mapping of the evidence base for empowerment and accountability, DFID, April 2011.
12. For example, Blurring the Boundaries: Citizen Action Across State and Societies, Citizenship DRC, 2011, http://www.drc-citizenship.org/system/assets/1052734700/original/1052734700-cdrc.2011-blurring.pdf?1302515701.
13. Evaluation of DFID’s Electoral Support through UNDP, ICAI, April 2012, http://icai.independent.gov.uk/wp-content/uploads/2010/11/UNDP-report-FINAL.pdf.
14. Department for International Development: Transferring cash and assets to the poor, National Audit Office, November 2011, http://www.nao.org.uk/wp-content/uploads/2011/11/10121587.pdf.
15. In 2011, DFID chose to discontinue general budget support to Malawi due, among other things, to concerns over a deteriorating human rights record and problems around freedom of the press and democratic space. See The Management of UK Budget Support Operations, Independent Commission for Aid Impact, May 2012, page 8, http://icai.independent.gov.uk/wp-content/uploads/2010/11/ICAI-Budget-Support-Final-Report-3.pdf.
16. Strengthening empowerment and accountability in international development: emerging guidance, DFID, 2011.
17. See: www.starghana.org/.
18. The Danish Agency for International Development (DANIDA) contributes £6.5 million (25%), the European Commission contributes £3.8 million (15%) and the United States Agency for International Development (USAID) contributes £3.1 million (12%). STAR-Ghana Annual Financial Report – Year 1, Coffey International Development, May 2012, page 4.
19. Under the Ghanaian Constitution, traditional chiefs are recognised and play a role in local governance, while ‘Houses of Chiefs’ exercise some influence at national and regional levels.
20. A ‘theory of change’ is a conceptual model of how a planned intervention will produce the intended impact. Programme evaluations are often designed to test whether this theory of change has proved to be valid.