Rapid Review of DFID’s Smart Rules

Executive Summary

DFID has recently reformed its procedures on programme management. It has called these new procedures Smart Rules: Better Programme Delivery.1 The first edition of the new rules was put in place in June 2014. An updated version was issued in October 2014.

This note provides a concise commentary from ICAI on the fitness for purpose and effectiveness of DFID’s new Smart Rules. It is designed to apply lessons learned by ICAI in its reports to date in order to help DFID to refine and enhance the Smart Rules’ impact. It will also provide members of the International Development Committee with an independent commentary on the Smart Rules.

This is a briefing note (not a normal ICAI report) and, therefore, has not followed the normal ICAI approach. We have undertaken desk-based analysis of the Smart Rules and of our ICAI reports to date, as well as conducting interviews with some key stakeholders within DFID. We assessed whether the Smart Rules contribute to strengthening DFID’s approach to programme management; we did not interrogate procedural detail.

We have looked at the Smart Rules through three specific lenses, the extent to which they:

  1. simplify and streamline project lifecycle management and programme management;
  2. address the key challenges and recommendations identified in the ICAI work to date; and
  3. act as an effective component of the wider transformation of DFID, the organisation’s learning and its readiness for the challenges of aid delivery over the period ahead.

Findings

We applaud the department in undertaking its End to End Review and the subsequent publication of the Smart Rules. There has been a serious attempt to grapple with some of the worst examples of procedural inefficiency. We recognise the positive underlying set of philosophies at work. In procedural simplification they are a positive step forwards. There are some further, relatively basic enhancements, however, that need to be made to ensure that the rules are truly usable and useful.

Some of the rules respond to critical challenges from our ICAI reports. Major issues require further work, though, and enhanced guidance is needed to help DFID achieve greater impact and value for money.

To be truly transformative, the Smart Rules will need to be supported by a change process. DFID could be doing more to drive change across the organisation. Senior staff could be doing more to set out a clear direction for the department that puts in place a process of change that is consistent with the vision of the Smart Rules.

Recommendations

Recommendation 1: DFID needs to continue to refine the Smart Rules to facilitate ease of use by teams in the field, with a particular focus on clearer principles, focussed technical guidance and examples of where discretion can be applied.

Recommendation 2: The Smart Rules need to be enhanced in key areas to meet critical challenges identified by ICAI. They need to:

  • be more explicit about intended beneficiary involvement;
  • clarify the relationship between value for money, sustainability and impact;
  • ensure a consistent approach to risk, adaptation and learning;
  • enable mobilisation and increased realism in planning; and
  • make senior staff (such as heads of office) specifically accountable for continuous learning.

Recommendation 3: DFID needs to maintain the momentum of the change programme which has arisen from the End to End Review and continue to engage all departments in a dynamic transformation focussed on improving the impact of UK aid on the poor.

Recommendation 4: DFID leadership needs to define a compelling vision and mission for the organisation for the next decade, focussed on reducing poverty for beneficiaries; and use this to establish some explicit principles for the overall transformation ahead.

Recommendation 5: DFID needs to continue to leverage its leadership role in the global aid community to streamline overall system effectiveness and improve collaboration between partners.

1 Introduction

1.1 On 29 April 2013, DFID launched a five month-long ‘End to End Review’ process in response to growing concerns about the burden, complexity and fitness for purpose of its processes and programme management cycle. The review came to four key conclusions:2

  • DFID’s approach to programme management needed to change. Processes needed to be pared back, paperwork reduced and DFID needed to have a clearer focus on delivery;
  • programmes needed to be flexible and responsive to changing political realities and conflict dynamics on the ground. In order to do this, DFID needed to commission and manage adaptive, flexible programmes;
  • collective responsibility and clearer accountabilities were required if DFID were going to deliver effectively. Staff needed to make decisions in a way that is contextually aware. They needed to be confident in their professional judgement to ensure proportionality, as well as rigorous in ensuring a clear audit trail; and
  • changing processes needed to be matched with cultural and behavioural changes. This required new incentives and collective leadership across the organisation and at all levels.

1.2 It was decided that the Smart Rules should be the first concrete output from the End to End Review; the new rules were published in July 2014. From the beginning it was recognised that they would need to be part of a wider response to the challenges identified by the End to End Review. The Smart Rules are characterized as providing ‘the operating framework for the Department for International Development’s (DFID’s) programmes’.3 The new document replaced its previous corporate rules on MoneySight and the programme management elements of ‘the Blue Book’, which had been in place for more than a decade.

1.3 The first edition of the Smart Rules was reviewed after three months and expired on 30 September 2014. The current version will expire on 1 February 2015.

Our methodology

1.4 The heart of our approach was an analysis of the Smart Rules (the version published in October 2014) against ICAI findings and recommendations arising in the three and a half years since our inception. This identified whether the new Smart Rules reflected and took into account the findings and recommendations of ICAI’s individual review reports and synthesis reports (see section from paragraph 2.20 on page 6). We also undertook a focussed set of interviews with members of the Better Delivery Taskforce and other stakeholders within DFID to gain an insight into the drivers, intent and focus of the new approach. DFID’s Head of Programme Delivery presented the rules to an ICAI Commissioners Board Meeting; we conducted a workshop with ICAI Commissioners and members of the Contractor team to identify key strengths and areas for further focus.

1.5 This review was undertaken rapidly in order to inform the early take-up and effectiveness of the Smart Rules. It focussed only on the comparison of the Smart Rules with ICAI’s analysis and recommendations. It did not compare the Smart Rules with other organisations’ programme management guidance or wider benchmarks of global good practice in this area.

2 Findings

2.1 We have looked at the Smart Rules through three specific lenses, the extent to which they:

  • meet DFID’s stated core objectives with regard to the simplification and streamlining of project lifecycle management and programme management;
  • address the key challenges and recommendations identified in ICAI’s work to date; and
  • will act as an effective component of the wider transformation of DFID, the organisation’s learning and its readiness for the challenges of aid delivery over the period ahead.

Do the Smart Rules achieve their stated objectives?

2.2 The Smart Rules’ stated aim is that they will:

  • encourage DFID teams to focus more on the ‘what and how of delivery and less on the why and rationale’;
  • introduce leaner documentation and processes that encourage a proportionate approach, to help people spend their time on the right things to deliver results and effectively manage risk; and
  • ‘bring together all the information [the department needs] to comply with DFID/HMG rules in one place, which saves time and increases compliance’.4

2.3 DFID staff report that four drivers underpin the Smart Rules. These are to:

  • move from a rules-based to a more principles-based approach to programme management, creating deeper ownership and engagement;
  • simplify and clarify mandatory rules;
  • direct DFID’s effort proportionately on what matters most (i.e. remove generic mandatory compliance tasks, creating leaner documentation and processes); and
  • demonstrate the space for discretion where frontline staff are trusted to innovate, take risks and adapt to realities on the ground. This is intended to encourage teams to focus more on the what and how of delivery and less on the why and rationale.5

Moving from rules to a more principles-based approach

2.4 We are very supportive of DFID’s intention to move towards a more principles-based approach. We recognise the inherent tension, however, in starting off such an approach with a set of rules. We were surprised to find that the 10 principles set out at the front of the Smart Rules do not mention specifically achieving aid impact, the beneficiaries of UK aid or learning.6 At present they are clearly a set of principles for a rationalisation of the programme lifecycle rather than those that will underpin the transformation of the department. Our view is that the principles need to make explicit the need to focus on poverty reduction and impact on the beneficiaries. So there needs to be a statement of the ‘why and the rationale’ to guide the ‘what and the how’ which is the focus of the rules. We believe that setting out the rationale will inspire DFID staff, indicating what the rules are ultimately aiming to achieve. It will avoid the risk that sight of the guiding principle might be lost through leaving this central DFID remit implicit.

2.5 We think that if the principles were better clarified and spoke to the main drivers of DFID’s remit, the rules would be further rationalised and aligned with the principles. The clear articulation of a limited set of primary principles would provide a useful context for determining whether proposed new rules are appropriate.

2.6 The current Smart Rules document does contain what we think are some excellent principles, though they are not identified as such. We think the Operating Standards which currently appear only at page 15 (outlined in section 3.1 on technical quality7) could make better primary principles for the rules as a whole than those in place now. In summary (see Annex 2 for complete content), they are to:

  • understand the political and economic context and how DFID’s interventions will affect and/or be affected by them;
  • consider how each intervention will contribute to poverty reduction, addressing the underlying causes of poverty and the impact on different social and economic groups;
  • ensure sustainability and resilience;
  • avoid doing harm; and
  • ensure that the views and experiences of citizens and beneficiaries inform the design and delivery of DFID’s programmes.8

As a minimum, these important ideas need to be more front and central to the Rules than they appear to be as Operating Standards.

Stream-lining and reducing bureaucracy

2.7 We find that the Smart Rules are a good beginning to the streamlining process. The 200 compliance steps previously in place have been distilled to 37 rules. We have already heard in country offices that they are pleased with the greater flexibility and simplicity of the approach compared with the prior bureaucracy and lack of discretion.

2.8 As a first step to bringing together all the information in one place, the 105-page Smart Rules also appear to be going in the right direction. The Smart Rules do not, however, stand on their own. They refer to over 25 different sources of guidance (ranging from UK aid branding advice to a cost extension decision-making guide), at least six templates and a growing number of smart guides. Nevertheless, this still contrasts favourably with a reported 120 different documents and thousands of pages that DFID staff had to consult previously when drafting a business case.9,10 Because the rules do not stand alone, however, this raises a potential challenge to ensure that all linked documents remain up-to-date. There is also the risk that the number of guides and rules creep back up over time.

2.9 The rules need to be as lean as possible and could be pared down further. There could also be further streamlining by moving some content into annexes. There is some valuable headline advice and checklists which is more useful than the procedural detail and better achieves the goal of empowering staff to use their initiative in deploying the Smart Rules.

2.10 A Programme Cycle Committee (PCC) has been established and considers any proposals for new rules or changes to the rules. We understand that it will seek to protect against more rules being added. It will be equally important that it focusses on the quality assurance and usability of the current rules. The process for the day-to-day management of the Smart Rules has been set out to the Department although it is not clear whether the PCC will have the ongoing authority to facilitate the possible inputs. We think that DFID’s Investment Committee and PCC should be fully interlinked to maintain senior management oversight of the rules approval process. It is also critical that the development of the Smart Rules remains in lockstep with any wider transformation of the department.

Focussing on implementation

2.11 There has been an effort to reinforce the focus on delivery and the introduction of the Delivery Plan as a key document is a major step forward. We are concerned though that the Smart Rules are still not focussed sufficiently on the implementation stage of the programme cycle. We find little in the rules that provides clarity on how staff should oversee day-to-day implementation tasks. It is notable that, of the 37 rules, only 3 sit under the ‘delivery’ sub heading.11 Pages 47 to 105 of the Smart Rules deal with Programme Design and Delivery. While covered in the Operating Standards, we can find five sections that indicate how DFID should relate to the partners through which UK aid is spent on a day-to-day basis. These references to partner relations need to be better signposted. The rest of the rules in that section focus on what documentation is required, such as the business case, fiduciary risk assessments and annual reviews. Additional ‘bottom up’ guidance that helps staff have more clarity on how they are expected to implement projects is required. This exists in accompanying ‘smart guides’, however we think that the rules need to be further rebalanced.

Partnership Principles as a guiding concept

2.12 We agree that the focus on Partnership Principles12 is important, particularly for fragile states. We think that the Partnership Principles need more emphasis in the rules as a basis for the context for decision-making; e.g. the establishment of the right risk management approaches, governance standards and interactions with proposed beneficiaries. This idea could act as a stronger link between the analysis in the Country Poverty Reduction Diagnostics13 and the critical elements required in programme design and the Delivery Plans.

The Senior Responsible Owner

2.13 The rules focus on the role of particular DFID staff, identified as Senior Responsible Owners (SROs); 30 of 37 rules mention the SRO specifically. We are positive about the introduction of the concept of SROs to DFID and recognise the value that such roles have brought to other Government departments. We agree that identifying clearer accountability is important and the benefit of knowing who can make the key judgements and decisions at a particular point in the project lifecycle is a step forward. There is some risk that, in a complex multi-dimensional programme, the SRO role will over-simplify apparent accountability without recognising interdependencies. We think the relationship between the accountability of SROs and others for delivery and risk, particularly heads of department, needs to be further spelled out in the document. Also, these roles will only meet the planned objectives if they are operating within the right overall environment of risk appetite and supporting culture. If they simply represent the focus of potential blame for failure, they will not be attractive or effective.

Whole programme life accountability

2.14 One of the key challenges that we have witnessed in our work has been the churn of DFID staff in post, both at the centre and especially in the field, which results in loss of institutional memory. We have seen no indication that DFID’s high staff turnover rate (particularly in more fragile countries) is changing. We note, therefore, that several staff are likely to be SRO over a project lifecycle. DFID staff often move to new positions and countries with incomplete handovers.14 We think that there might be scope for DFID to consider whether and how staff remain accountable for their actions when SRO of a project after they have moved to new jobs (as happens in other organisations). As a minimum, we believe that the SRO duties will need to be updated to reinforce the handover obligations and that a formal record of accountability should be established which can access past SROs for programme review and adaptation.

2.15 The introduction of the Delivery Plan approach15 could contribute to building continuity of oversight and learning. Maintained plans could become living documents that, like a diary, identify actions, justify decision-making processes and act as a foundation for more effective handover.

Creating space for discretion

2.16 Enabling effective discretion, where the judgement of frontline staff is trusted, is important and key to the success of adaptive programming and continuous learning. We note that the operating standards in the Smart Rules are intended to provide the space for discretion and professional judgement according to the individual context.16 There is little comment, however, as to where these ‘smart spaces’ are. More clarity is needed to help staff to understand how this discretion is to operate in practice. We think that it would be useful to refer to specific examples where the judgement of frontline staff has been trusted so that they can innovate, take risks and adapt to the realities on the ground. There is a need for a much more explicit set of connections between: a) the ideas of risk management and risk appetite, which are alluded to in the document; and b) the concept of discretion and accountability.

Clarity of message

2.17 There has been an admirable effort to organise much disparate material into a coherent document but aspects of the Smart Rules structure are still not intuitive. The content can be difficult to follow and it is a challenge for the reader to prioritise what matters most and when to apply components in the process of establishing, running and evaluating a programme.17 Some of the document is confusing and still has a flavour of the multiple sources from which it has been drawn.18 The process of consolidating and integrating previous material has created a mixture of styles, content and examples. It would be useful to have more cross-referencing within the document. There has been less internal streamlining of specific rules than we would like to see.19

2.18 The idea of 10 principles and 37 rules is a compelling and effective concept. We agree with DFID’s vision to move to a more principles based approach. Our view, however, is that the principles are a mixed bag of concepts and do not sit under an anchoring vision or clear mission. A stronger and clearer link to the structure of the project life cycle would be helpful.

2.19 The distinction between what are mandatory procedures, what are required standards and what are expected behaviours is not made sufficiently explicit. Further clarity on what is mandatory and what is guidance is needed; either a statement is a rule or some form of guidance. There should be no ‘rules by stealth’. We note the document still contains many ‘musts’ beyond the 37 rules. For example, ‘Such items cannot be charged against DFID’s budget as normal expenditure in delivering international development. They must, however, still appear in DFID’s accounts’;20 or, ‘All programmes must be approved by the ICF Board (chaired by DFID)’.21

Do the Smart Rules address the challenges and recommendations identified by ICAI?

2.20 We recognise that the current version of the Smart Rules will address some of the concerns we have identified in our various ICAI assessments. Some key themes that recur in our findings are that:

  • accountability needs to be clearly defined;22
  • DFID’s approach to learning has to be adaptive;23 and
  • systems and processes can get in the way of effective delivery.24,25,26,27

2.21 We also recognise that the Smart Rules are not meant to be the only vehicle through which DFID intends to make improvements to its impact and effectiveness.

2.22 There are, however, some gaps in the current version, which means that a number of our key findings and recommendations will not be addressed. We believe that these additional areas need to be incorporated more fully in future versions of the Smart Rules and the wider transformation plans for the department, for instance:

  • the need to focus on beneficiary participation and consultation;28,29,30,31
  • the need to always keep in sight the priorities of reducing poverty, ensuring value for money and impact;32,33,34,35
  • that design and implementation needs, from the first, to be planning for sustainable solutions and exit strategies;36,37,38
  • that DFID needs to work equally effectively with all its delivery channels;39,40,41,42,43,44
  • that DFID needs to manage risk effectively;45,46,47,48
  • the need to ensure that the mobilisation of resources (partners/contractors) is timely;49
  • that monitoring (especially short term monitoring to enable adaptive working) needs to be effective;50,51,52
  • an emphasis on lessons learned and shared (knowledge and know-how);53,54,55,56,57,58,59 and
  • that knowledge needs to be maintained, particularly through handover.60,61,62

Beneficiary participation and engagement

2.23 Many of our reports have identified the need for greater beneficiary engagement as a critical success factor. The Smart Rules contain reminders to ensure that the views and experiences of citizens and beneficiaries inform design and delivery.63 We note, however, that these are principles for ‘consideration’ to ‘guide’ design of programmes.64 The Smart Rules do not make it clear to staff what the minimum standards for beneficiary participation/consultation are. We think they should. There is a need for a clearer principle on this, as well as more explicit guidance as to what good beneficiary involvement activities and governance approaches need to be put in place in programme design, mobilisation, execution and exit. Without this vital component, DFID will be missing a key element of their ability to know that the programmes are working in the way it intends, to make timely course corrections and to learn from what is working and what is not in the eyes of the intended recipients.

2.24 We note that there is no specific mention of beneficiaries’ role in the monitoring and evaluation sections of the Smart Rules beyond some general consideration. We also find the relative lack of guidance on the appropriate levels of direct supervision by DFID staff and field visits to validate the beneficiary experience throughout the programme lifecycle to be a deficit. This lack of real time access, knowledge and insight by DFID teams has lain behind many of the challenges we have seen with programmes, especially those delivered through contractors, NGOs or multilateral partners. This is exacerbated by the fact that DFID staff do not implement programmes directly and commission programmes through a range of partners.

2.25 The Smart Rules set out that the SRO now has accountability for ensuring compliance of programmes and projects with the Gender Equality Act.65 While there is (rightly) a high degree of focus on the issue of gender in the rules, we are concerned that wider issues of vulnerable groups in general, such as poverty, disability or lack of access, are less well represented. We believe that the guidance for teams to help do no harm could also usefully be bolstered.

Defining value for money and impact

2.26 Value for money is mentioned throughout the Smart Rules document. We are pleased to see that there is a clear focus on the concept. We have identified in a number of our reports that value for money is sometimes narrowly aligned with the relative costs of interventions. ICAI believes value for money must be seen in the context of long-term sustainability and impact. There seems to be an implicit assumption in the Smart Rules that programmes providing value for money will deliver on poverty reduction. We would expect that the Smart Rules spell out more clearly the causal relationship between value for money considerations, sustainability and impact.66 There is also an opportunity to help teams to look more at the relative burdens of different administrative models and their impact on value for money.

Working effectively through all delivery channels

2.27 In our report on DFID’s use of contractors,67 we noted that procurement processes for sophisticated programmes should be improved. DFID has done much work to change its approach to procurement, much of which is reflected in the Smart Rules. For instance, closer early engagement with contractors is suggested.68

2.28 The International Development Committee noted recently that: ‘Despite a small decline in 2012-13, spending through multilateral organisations remains the largest element of DFID’s bilateral programme at £1,075 million.69 The total funding through multilaterals is close to 70% of DFID’s total expenditure. The Smart Rules do not provide mandatory rules for engagement with multilateral organisations. We understand that a separate multilaterals rule book was considered. While we agree that having multiple rule books would be counterproductive, we think the Smart Rules appear to assume that bilateral aid is the standard DFID approach. More is needed to help staff to manage how DFID engages with multilaterals through the programme cycle given the differences in their mandate, structure and delivery mechanisms from other channels of UK aid.70

2.29 The Smart Rules state that the Multilateral Aid Review (MAR) should be used as a starting point for the UK’s engagement.71 The standards call for further analysis of a multilateral’s country-level expertise/fitness for purpose when deciding on the delivery partner (which ICAI has recommended)72,73 but these need to be better signposted. We note that DFID practice currently varies as to whether there should be separate risk assessments and due diligence for individual projects funded through multilaterals via the bilateral programme. Clearer guidance is necessary, given that the majority of DFID’s expenditure is through multilateral organisations.

2.30 We have noted previously74,75 how the knowledge and learning collected by DFID’s partners and contractors is usually not retained by DFID. The Smart Rules address this by ‘prioritising learning from contractors, stakeholders and beneficiaries at all stages in the delivery chain and having the confidence to learn from and communicate what is working well and what is not’.76 We applaud the intention but it is not clear to us from the Smart Rules how it will be achieved in a way that will preserve institutional memory (see also paragraphs 2.43-2.46 on page 11).

Defining risk appetite and managing risk

2.31 Risk management has been a consistent theme of our ICAI reviews and we are pleased to see that the rules reference this concept at multiple points. The rules contain a mix of generalised procedures (e.g. Treasury Orange Book) and standard mechanisms.

2.32 The complexity of the risk management challenge in much of the work that DFID undertakes is not fully reflected in the Smart Rules. In particular, the Smart Rules follow a rather narrower approach to managing the risk of fraud and corruption than ICAI recommended the department to take in our 2011 Anti-Corruption Report.77 The Smart Rules describe the process for identifying and developing strategies to manage risk at the programme level.78 We recommended country level anti-corruption strategies for countries that were highly at risk.79 These were to set out an integrated programme of activities and dialogue processes, including strengthening ties with local law enforcement bodies and establishing a whistle-blower hotline. This is not mentioned in the rules, albeit the broad concept is included within the Partnership Principles assessment and accountabilities for head of office.

2.33 There is a clear process for reporting fraud to a specific unit.80 There is a clear statement on zero tolerance of fraud or corruption.81 There is no clarity, however, about how processes should differ for managing different corruption risks with different aid types.

2.34 The Smart Rules do not make it fully clear how the department calibrates what are or are not acceptable and appropriate risks. Section 5 on ‘Accountabilities’ states that the Head of Spending Team is responsible for defining an appropriate risk appetite and the SRO for articulating and summarising that risk.82 We note that three examples are given of different risk appetites for different contexts in the rules.83 That is helpful. We understand that DFID’s appetite for risk will vary from place to place. DFID could consider further tightening its guidance to staff on DFID’s risk appetite and what that means for acceptable levels of failure or wastage.

Ensuring effective mobilisation

2.35 We have observed that, during the initial stage of the project cycle, there are consistent delays in the timely mobilisation of resources.84 Mobilising contractors can be a lengthy process, initial interactions with partners can be sporadic, partner governance shaky or early team leadership inadequate. The Smart Rules section on project mobilisation85 does not refer to (or address) such delays. We think that DFID needs to do more to ensure that projects begin at the time planned, or amend its timetabling and guidance accordingly. The rules need to be clear on the department’s expectations, what action can be taken to speed up mobilisation and the need for realism in planning. Strengthened guidance for DFID teams would be a significant help.

Guidance on sustainable solutions and exit strategies

2.36 Many of our recommendations have focussed on the issue of sustainability of programmes after the exit by DFID and implementing partners.86,87,88,89,90 The Smart Rules mention planning for long-term sustainability and resilience in the technical standards91 and prompt DFID staff to consider how the benefits of a project or programme might be sustained beyond the period of UK support.92 The document does not, however, indicate procedures, standards or expected behaviours for how DFID should integrate planning for handover to beneficiaries throughout the lifecycle in its work.93 The smart rules focus on the financial controls for DFID exiting the projects rather than the real steps to create proper local governance which will enable sustainability after exit. This aspect needs to be made explicit.

Clearer architecture and linkage for monitoring

2.37 ICAI reports often note the challenges of effective monitoring of programmes. We have identified issues with the metrics being used, the quality of data and the overall approaches to assessing progress and impact.94,95,96,97 Monitoring through the programme delivery lifecycle is an important element in the Smart Rules. Any mention, however, is almost always guidance. The rules could do more to outline explicitly the connections between ongoing adaptive learning (with monitoring being a key element) and successfully providing impact and value for money. More clarity is needed about the overall architecture of monitoring and control and how that sits over the programmes. The Smart Rules have focussed on new models for the Business Case, Delivery Plan and Annual Review but have not drawn the lines of sight between these documents and a fit-for-purpose monitoring regime which practitioners could use or flex.

Honest assessment and continuous learning

2.38 ICAI’s review of How DFID Learns made some major recommendations with regard to the value of continuous learning and institutional learning across the organisation. Section 5.8 (‘Learn and Adapt’) of the Smart Rules98 provides a good brief overview of continuous learning necessary for DFID to succeed in programme delivery. Our view, however, is that this section should appear at the beginning of the cycle and not only be referred to on the last page. It will need considerable support from senior management to be achieved (see paragraph 3.16 on page 15). We also note that continuous institutional learning could have greater prominence in the Smart Rules (see recommendation 1 of our review of How DFID Learns).99

2.39 While DFID is improving its acknowledgement of and dealing with failure, further changes are needed to ensure that there is a consistent approach to risk, adaptation and learning. DFID staff have told us that some senior managers have declared a ‘zero risk appetite’. We note that the Smart Rules appear to omit making senior staff (such as heads of office) specifically accountable for continuous learning and ongoing, adaptive programming. This is required for consistent messaging. The role of the SROs with regard to learning should be further clarified.

2.40 We have noted in many of our reports that the monitoring processes encourage overly positive evaluations of progress and outcomes. They can lack integrity and the evaluations of implementing partners are too often taken at face value without validation. The tenth principle for the rules says: ‘Honesty: We proactively escalate concerns and risks through the management chain so that there are no surprises.’ Our How DFID Learns report indicates that some staff report that managers continue to pressure teams to inflate scores.100 While the Smart Rules do refer to avoiding optimism bias in the context of programme progress,101 our finding that senior staff continue to pressure teams to inflate project scores does not seem to be sufficiently responded to in the Smart Rules.102 We would expect this to be addressed specifically since, among other reasons, it is not consistent with the Civil Service Code (mentioned in the Smart Rules principles). This is an area where the wider culture change will be vital and the management team from the top down will need to be developed to send the right signal with regard to risk and space for failure as well as setting realistic targets.

2.41 There does not appear to be any reference in the rules to the need for smooth handovers between staff with programme responsibilities. We consistently see this a deficit of DFID’s delivery.103 Maintaining continuity and institutional memory is key. The rules need to spell out clearly what DFID’s expectations are for this and, in particular, the role of the SROs.

2.42 Recommendation 4 in our report on How DFID Learns says that staff need to be given more time to share lessons. Ensuring learning is shared needs to be clearly made a responsibility of heads of department. The report also notes the importance of experience gained in the field and the lessons shared.104 We note that the Smart Rules do not provide for more time to be spent in the field listening to beneficiaries.

2.43 As we also noted in our How DFID Learns report,105 using the best available evidence106 does not necessarily mean that it is easily accessible and relevant.107 We believe that DFID should continue to reinforce the extent to which evidence of all types is sought and used to direct programmes. Commissioners are also of the opinion that beneficiaries are an important source of relevant knowledge that can contribute to learning and programme development.

2.44 While the use of learning/evidence is mentioned numerous times in the Smart Rules, they do not address how these resources will be managed and accessed. While this may not be the role of the Smart Rules, we note that how such resources are managed will have a significant impact on whether better programme delivery is achieved.

2.45 We note DFID’s associated activities, such as the putting in place of a new management information system (the Aid Management Platform) and the planned replacement of its electronic document management system (Quest). We will be interested to see how a focus on continuous learning and adaptive programming can be integrated into such systems and processes.

2.46 The use of practical case studies,108 which illustrate what is possible under the Smart Rules, is a good step forward. This could be particularly effective if the wiki-type approach allows staff to share their experiences and know-how. Quality and accessibility of the studies will be important.

2.47 We note that DFID outlines specific requirements of the business case in its response to our recommendations in our report DFID’s Peace and security programme in Nepal. It said that there is a planned security and justice business case which will be used as a model for all future business cases, as well as for informing existing programming.109 It is not clear how many models there are and how this approach fits with the flexible business case structure outlined in the Smart Rules.

Smart Rules as a tool for transformation

2.48 In our conversations with DFID leadership, the Smart Rules have been presented as being in the vanguard for transformational change within DFID. It is clear that the End to End Review raised wider concerns than just the systems and processes. It is also clear that the broader agenda for change within the world of aid is going to drive the need for real transformation in the way that DFID works over the period ahead. We have also, therefore, looked at the Smart Rules as one foundation for this development and considered what else will be required for DFID to be fully fit for purpose for the challenges ahead.

2.49 As referenced earlier, our view is that the changes to the rules are necessary but not sufficient. It is unusual to begin a change process with redrafting rules, rather than having rule change as one part of a broader endeavour. The rules are an important enabling foundation but they need to be part of a stronger vision and strategy brought to life by a clearly defined change process. This must be supported and resourced by an inspired and inspiring DFID senior management team, which is capable of exciting the organisation itself and positioning it within the wider global aid community.

Maintaining momentum

2.50 We find that DFID’s work on the Smart Rules has been a good start. It is too early to declare that the desired change has been achieved. DFID’s internal risk assessment (which acknowledges that this is a long-term organisational change process), notes: ‘Declaring success too early taking pressure off wider leadership and culture change so staff do not feel a significant difference and disengage.’ We agree wholeheartedly. Transformation will take time and investment. It should be seen as a process that will take several years to achieve. The first basic test, in fact, will be whether the Smart Rules themselves are being applied and used.

2.51 To achieve the necessary change, new incentives and effective collective leadership across DFID are necessary. We are aware that the first steps have been taken to show both staff and partners that the leadership team is fully committed. Targeted messages have been given to staff from the Secretary of State and members of DFID’s Executive Management Committee. Messaging (and model behaviour) needs to be consistent from DFID’s senior staff.

Defining the vision

2.52 As noted in the Smart Rules: ‘Better programme delivery is not an end in itself. We want operational plans and programmes that provide benefits to the poor and that tackle the underlying causes of poverty and conflict.110 If the necessary transformation is to be achieved, the department needs to be inspired.

2.53 The principles underlying the Smart Rules should frame the philosophy and key objectives of DFID not just for the Smart Rules. They should be principles that the transformation of the department should seek to enable. They need to focus on the department’s core role and tasks; at present they do not mention impact, beneficiaries or learning.

Being consistent

2.54 We are aware that these messages are not yet being taken up by all staff. As noted in our report on How DFID Learns, the distributed organisation creates the opportunity for multiple organisational cultures to emerge in DFID. DFID needs to maintain its messaging on what change is expected (particularly over the next 12 months into the new parliamentary cycle), across all parts of the organisation.

Creating a single culture

2.55 It remains unclear whether the DFID culture can support the trust and partnership required to share lessons and improve programme delivery. By the nature of rules there is a tendency for this activity to feel like a ‘push’ from the centre, as opposed to a ‘pull’ from the field. The way that bottom-up engagement with the principles and wider vision are created will be as important as a good standard set of simplified documents and processes. The Smart Rules require continuity, coordination and cooperation between country offices and with central UK departments. It will be important to build and share a common purpose and the necessary knowledge and know-how.

Investing in the change

2.56 The Smart Rules alone require some significant investments in capacity in order to be effective and achieving the wider transformation is even more demanding. Work is underway to increase the capacity and capability of staff, with a focus on programme leadership and management. DFID has reported to us that over £1 million has been allocated for ‘capability investments’ in the period 2014-16. The focus of these investments is project leadership training. While positive, we consider that the wider cultural transformation needs to be fully resourced and budget provided (throughout DFID) to support all staff. It will be important that the relevant parts of programmes are resourced in line with the intent, as well as explicit new funding at the centre and in country offices. Our view is that this is a legitimate investment that will improve effectiveness and efficiency. There also needs to be a recognition that some of the recent investments in capability (e.g. in procurement, commercial advisors and monitoring and evaluation) may need to be redirected to ensure that the tone and involvement of these new skills is in line with the empowerment agenda underpinning this intent.

Working with and leading the international community

2.57 DFID does not operate in a vacuum. It is an important and, indeed, leading part of the international development community. Any transformation that DFID undertakes needs to be responsive to the shifting priorities and philosophies of global organisations and initiatives.

2.58 There is also a real opportunity for DFID, through its relative scale, reach and experience, to influence and shape these trends. As an example, the whole area of operating in fragile states has been one where DFID has taken a lead, as well as in areas such as the Busan New Deal thinking. There will be new global development imperatives emerging from the post-2015 Sustainable Development Goals process and, aside from contributing to this process, DFID will need to align its organisation and priorities with this agenda over the period ahead. This will require further reshaping of the operating model, especially the balance between global programmes and country models; and will involve new ways of working with multilateral and bilateral partners. The Smart Rules and the broader transformation will need to adapt to this reality and DFID will need to engage proactively with the other players and host governments to be effective in the period ahead.

3 Conclusions and Recommendations

Conclusions

3.1 Overall, we applaud the department in undertaking its End to End Review and the subsequent publication of the Smart Rules. There has been a serious attempt to grapple with some of the worst examples of the accumulating procedural inefficiency that had built up over the previous decade. We recognise the positive underlying set of philosophies at work.

3.2 As an exercise in procedural simplification, the Smart Rules are a positive step forwards and the working of the various stakeholders engaged in these processes should be improved as a result. There are some further, relatively basic enhancements, however, that need to be made to ensure that the rules are truly usable and useful.

3.3 In terms of responding to the critical challenges that we have observed in our ICAI reports, there is evidence that some aspects of poor design and delivery have been tackled. Some key issues, however, require further work and enhanced guidance really to help the teams across DFID to achieve greater impact and value for money.

3.4 The Smart Rules have been the first part of the wider transformation agenda. Beginning with rule changes is not ideal, albeit we hope it will prove a pragmatic choice that will bear fruits. It would have been better, from the start, to have the Smart Rules change as one element of a wider programme of cultural transformation of the department. This would have begun by setting a new vision, redefining leadership and accountability and building capabilities for key elements of the change. Some of these elements are now being put in place.

3.5 It is, however, vital that the Smart Rules are now put into the context of a more holistic transformation agenda; this should be driven by the external factors shaping the work of the department and the learning of the past few years about what works in development (including insights from ICAI reports).

3.6 The importance of the role of the leadership in DFID as role models and explicit champions of the change cannot be overestimated. It is also critical that the momentum of this activity be sustained, especially across the upcoming UK electoral cycle. There is a need to develop a stronger transformation brand, vehicle and operating model to ensure that this happens.

Recommendations

Recommendation 1: DFID needs to continue to refine the Smart Rules to facilitate ease of use by teams in the field, with a particular focus on clearer principles, focussed technical guidance and examples of where discretion can be applied.

3.7 We recommend that an adaptation of the Operating Standards outlined in Section 3.1 of the Smart Rules (‘Technical quality’, see Annex 2) would make better primary principles for the rules as a whole than those in place now. They should appear at the outset as a cultural framework for the rules.

3.8 The principles need to inspire and make explicit the main drivers of DFID (e.g. reducing poverty and value for money). These primary principles can then be used to rationalise and align the rules.

3.9 More emphasis is required on the headline advice, checklists and ‘bottom up’ guidance to empower frontline staff in using their initiative when deploying the Smart Rules. The rules need to be further rebalanced away from the procedural detail. There should be no ‘rules by stealth’.

3.10 The SRO and head of department duties need to be spelled out to reinforce handover obligations and accountability for delivery and risk – particularly making the link between risk management / appetite and the need for discretion and accountability.

3.11 DFID’s Investment Committee and PCC should be fully interlinked to maintain ongoing senior management oversight and accountability of the Smart Rules and the rules approval process. It is critical that the ongoing development of the Smart Rules remains in lockstep with any wider transformation of the department.

Recommendation 2: The Smart Rules need to be enhanced in key areas to meet critical challenges identified by ICAI. They need to:

  • be more explicit about intended beneficiary involvement;
  • clarify the relationship between value for money, sustainability and impact;
  • ensure a consistent approach to risk, adaptation and learning;
  • enable mobilisation and increased realism in planning; and
  • make senior staff (such as heads of office) specifically accountable for continuous learning.

3.12 The Smart Rules should make it clear to staff the minimum standards for beneficiary participation/consultation. This requires more explicit guidance as to good beneficiary involvement activities and governance approaches in programme design, mobilisation, execution and exit.

3.13 The relationship between value for money, sustainability and impact needs to be spelled out more clearly. Unless this is made clear, there is a risk that the emphasis both within DFID and its implementing partners will be on cost rather than value. The rules could do more explicitly to outline the connections between ongoing adaptive learning (with monitoring being a key element) and successfully providing impact and value for money.

3.14 Strengthened guidance is required as to what action can be taken to speed up mobilisation and increase realism in planning. We would like to see more clarity about the overall architecture of monitoring and control and how that sits over the programmes.

3.15 The emphasis and guidance on delivery plans should be strengthened to act as a foundation for more effective handover and ensuring better institutional memory.

3.16 The continuous institutional learning that is necessary for DFID to succeed in programme delivery will require considerable support from senior management. Senior staff (such as heads of office) should be made specifically accountable for continuous learning and ongoing, adaptive programming to ensure consistent messaging.

Recommendation 3: DFID needs to maintain the momentum of the change programme which has arisen from the End to End Review and continue to engage all departments in a dynamic transformation focussed on improving the impact of UK aid on the poor.

3.17 DFID needs to maintain its messaging on what change is expected (particularly over the next 12 months into the new parliamentary cycle), across all parts of the organisation.

3.18 The wider cultural transformation needs to be fully resourced and budget provided (throughout DFID) to achieve the required result. This includes recent investments in capability that may need to be redirected to ensure that the tone and involvement of these new skills is in line with the empowerment agenda underpinning this transformation.

Recommendation 4: DFID leadership needs to define a compelling vision and mission for the organisation for the next decade, focussed on reducing poverty for beneficiaries; and use this to establish some explicit principles for the overall transformation ahead.

3.19 It will be important to build and share a common purpose and the necessary knowledge and know-how between country offices and the central UK departments. The Smart Rules can support this by continuing to develop, coordinate and adapt a good standard set of simplified documents and processes.

Recommendation 5: DFID needs to continue to leverage its leadership role in the global aid community to streamline overall system effectiveness and improve collaboration between partners.

3.20 This will require further reshaping of the operating model, especially the balance between global programmes and country models; and will involve new ways of working with multilateral and bilateral partners.

3.21 DFID will need to engage proactively with other players and host governments to be effective in the period ahead.

Annex

This annex sets out:

  • the principles that currently appear at the beginning of DFID’s Smart Rules; and
  • the list of standards provided in Section 3.1 of the Smart Rules on technical quality.

The Smart Rule Principles111

In applying the Smart Rules, we are:

  1. Professional
    We lead the UK Government’s fight to end poverty and address its underlying causes. We follow the Civil Service Code and HM Treasury’s guidance on Managing Public Money.
  2. Transparent
    British taxpayers and beneficiaries have a right to know what we’re doing, why we’re doing it, how we’re doing it, how much it will cost and what it will achieve.
  3. Innovative
    We are prepared to do things differently to deliver better outcomes. We are creative and try new and original ways to deliver government policy.
  4. Ambitious
    We are ready to propose difficult, transformational programmes in high-risk environments and discuss the risks with ministers and colleagues.
  5. Context-specific
    We strive to understand the local political and operational environment within which we work and ensure that our programmes and aid instruments suit and influence the political context.
  6. Evidence-based
    We learn from evidence, share lessons and change course as the context – or our understanding of the context – changes. We make an open and genuine commitment to identify mistakes, learn from them and share lessons.
  7. Responsible and accountable
    We are responsible for delivering the results we have committed to, with a clear understanding of our role and the role of others in delivering the government’s policy on poverty reduction. We are accountable for effectively managing programme risk and performance.
  8. Proportionate and balanced
    We use common sense and judgement to present reasoned, evidence- and risk- based proposals that are appropriate for the situation, the information available and level of urgency.
  9. Collaborative
    We work together and help and support each other right across the organisation.
  10. Honest
    We proactively escalate concerns and risks through the management chain so that there are no surprises.

Proposed Primary Principles

The following list of standards (1 to 5) appears at page 15 in the Smart Rules to provide guidance at each point of the programme cycle. We see these as fundamental to the successful operation of the programme cycle and therefore ideal as the primary guiding principles for the Smart Rules as a whole.

As noted, the primary principles will provide the context and guidance for determining the necessary rules. They make explicit the DFID remit of poverty reduction and the focus on achieving impact for the poor. They are inspiring and will also provide the rationale for decision making and approaches taken.

Part 1: Principles, Rules and Standards; Section 3: The Operating Standards; 3.1: Technical Quality

Better programme delivery is not an end in itself. We want operational plans and programmes that provide benefits to the poor and that tackle the underlying causes of poverty and conflict.

There are a number of technical considerations to guide the design and delivery of adaptive operational plans and programmes. These include, but are not limited to: the political economy; conflict and fragility; institutional environment; climate change, resource scarcity and environmental vulnerability; gender equality; social and poverty impact; and human rights.

  1. Understand the political and economic context and how DFID’s interventions will affect and/or be affected by them. Is the intervention realistic and feasible given the underlying political-economy dynamics? Do you understand which coalitions will or could support or prevent change? What are the key formal and informal institutions which will affect your intervention?
  2. Consider how each intervention will contribute to poverty reduction, addressing the underlying causes of poverty, and the impact on different social and economic groups, including analysis of different needs, rights and patterns of discrimination due to location, gender, disability, ethnicity, religion, sexuality, age and so on. Also, consider the contribution to broader social and political participation.
  3. Ensure sustainability and resilience. How will you generate lasting benefits for citizens in the face of possible future shocks (e.g. political, economic, security, environmental, social, climatic)? How do you support resilient households, firms, institutions, societies and environments capable of coping with uncertain futures? This could include: supporting opportunities to deliver climate and environmental benefits; fostering positive change in the political settlement; contributing to peace and stability; providing a stable and good investment climate in which firms can operate and create jobs; promoting rights and choice; and/or addressing underlying gender barriers.
  4. Avoid doing harm by ensuring that our interventions do not sustain unequal power relations; reinforce social exclusion and predatory institutions; exacerbate conflict; contribute to human rights risks; create or exacerbate resource scarcity, climate change and/or environmental damage; and/or increase communities’ vulnerabilities to shocks and trends. Ensure that our interventions do not displace/undermine local capacity or impose long-term financial burdens on partner governments.
  5. Ensure that the views and experiences of citizens and beneficiaries inform the design and delivery of our programmes. How do programme designs reflect local and national preferences (for instance by promoting collective accountability through democratic institutions such as community groups, local councils and national parliaments)? How do programmes capture feedback from partners, communities, individuals we work with (for instance using mobile technology, public audits, third party monitoring, field visits)?

Footnotes

  1. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/361330/Smart_Rules-oct14.pdf
  2. Pete Vowles, Adaptive programming, October 2013, https://dfid.blog.gov.uk/2013/10/21/adaptive-programming/
  3. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 4.
  4. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 4.
  5. Pete Vowles and Tom Wingfield, DFID is changing its approach to better address the underlying causes of poverty and conflict – can it work?, 9 October 2014, http://oxfamblogs.org/fp2p/dfid-is-changing-its-approach-to-better-address-the-underlying-causes-of-poverty-and-conflict-can-it-work-guest-post-from-tom-wingfield-and-pete-vowles/
  6. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 10.
  7. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 15.
  8. We suggest also that these could be expanded to include clearer concern for value for money and continuous learning.
  9. Staff reported to us that this had become ‘dysfunctional’.
  10. See also ICAI’s comment on this in ICAI, How DFID Learns, April 2014
  11. The 37 rules are classified into the following sections: Operating framework (5 rules), Design (9), Mobilisation (3), Procurement and competitive tendering (7), Delivery (3), Financial management (8) and Extension and closure (2).
  12. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 20.
  13. The Country Poverty Reduction Diagnostic tool seeks to identify the underlying barriers to poverty reduction, the space for UK action and identify investments that will have the most transformational impact. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 45, 63.
  14. ICAI, How DFID Learns, April 2014, page 16, paragraph 2.55.
  15. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 90.
  16. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 7.
  17. For instance, annual reviews are described as ‘providing an assessment of performance, ongoing relevance, value for money and any remedial action required’ (page 40), and there is guidance on what an annual review should contain (see page 94). Page 50 asks the questions ‘Are we tracking recommendations from annual reviews and performance improvement measures?’; the Delivery Plan, (page 90) is indicated as the means of tracking such changes. The importance of the delivery plan as a living, operational document appears to be underplayed in contrast to the annual review. Our view is that the delivery plan could be a very powerful tool for ensuring that continuous adaptation and learning take place.
  18. For instance, section 3.9 on Due Diligence Assessment,
  19. For instance, mandatory approval requirements for business cases appear on pages 34, 37, 38, 51, 52, 67 etc.
  20. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 100.
  21. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 26.
  22. ICAI, The Management of UK Budget Support Operations, May 2012.
  23. ICAI, How DFID Learns, April 2014.
  24. ICAI, DFID’s Peace and Security Programme in Nepal, February 2013.
  25. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013.
  26. ICAI, DFID’s Climate Change Programme in Bangladesh, November 2011.
  27. ICAI, How DFID Learns, April 2014.
  28. ICAI, DFID: Programme Controls and Assurance in Afghanistan, March 2012; ICAI, DFID’s Education Programmes in Three East African Countries, May 2012; ICAI, DFID’s Approach to Anti-Corruption, November 2011; ICAI, How DFID Learns, April 2014; ICAI, DFID’s Contribution to the Reduction of Child Mortality in Kenya, March 2014.
  29. ICAI, DFID’s Support to Capital Projects in Montserrat, July 2013.
  30. ICAI, DFID’s Support for Palestine Refugees through UNRWA, September 2013.
  31. ICAI, DFID’s Water, Sanitation and Hygiene Programming in Sudan, February 2013.
  32. ICAI, DFID’s Peace and Security Programme in Nepal, February 2013.
  33. ICAI, DFID’s Support to the Health Sector in Zimbabwe, November 2011.
  34. ICAI, DFID’s Education Programmes in Three East African Countries, May 2012.
  35. ICAI, Evaluation of DFID’s Support for Health and Education in India, May 2012
  36. ICAI, Evaluation of DFID’s Support for Health and Education in India, May 2012
  37. ICAI, DFID’s Climate Change Programme in Bangladesh, November 2011.
  38. ICAI, DFID’s Livelihoods Work in Western Odisha, February 2013.
  39. ICAI, DFID’s work with UNICEF, March 2013.
  40. ICAI, The Effectiveness of DFID’s Engagement with the World Bank, March 2012.
  41. ICAI, Evaluation of DFID’s Electoral Support through UNDP, April 2012.
  42. ICAI, DFID’s Support for Palestine Refugees through UNRWA, September 2013.
  43. ICAI, DFID’s Oversight of the EU’s Aid to Low-Income Countries, December 2012.
  44. ICAI, The Effectiveness of DFID’s Engagement with the Asian Development Bank, July 2012.
  45. ICAI, DFID’s Health Programmes in Burma, July 2013.
  46. ICAI, Girl Hub: a DFID and Nike Foundation Initiative, March 2012.
  47. ICAI, DFID’s Support to the Health Sector in Zimbabwe, November 2011.
  48. ICAI, Evaluation of the Inter-Departmental Conflict Pool, July 2012.
  49. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013.
  50. ICAI, How DFID Learns, April 2014.
  51. ICAI, Evaluation of the Inter-Departmental Conflict Pool, July 2012.
  52. ICAI, Evaluation of DFID’s Support for Health and Education in India, May 2012.
  53. ICAI, How DFID Learns, April 2014.
  54. ICAI, DFID’s Empowerment and Accountability Programming in Ghana and Malawi, October 2013.
  55. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013.
  56. ICAI, DFID’s Contribution to Improving Nutrition, July 2014.
  57. ICAI, The Management of UK Budget Support Operations, May 2012.
  58. ICAI, DFID’s Oversight of the EU’s Aid to Low-Income Countries, December 2012.
  59. ICAI, DFID’s Support to Capital Projects in Montserrat, July 2013.
  60. ICAI, How DFID Learns, April 2014.
  61. ICAI, DFID’s Water, Sanitation and Hygiene Programming in Sudan, February 2013.
  62. ICAI, DFID’s Contribution to the Reduction of Child Mortality in Kenya, March 2014.
  63. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), pages 16, 20, 50, 54, 104.
  64. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 16.
  65. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 51.
  66. For instance, the discussion of value for money on page 19 does not refer to either impact or sustainability. It talks about the ‘best feasible programme’ (which is a good concept) and the ‘impact of each pound’ but does not set out the department’s vision of what is real value in terms of achieving lasting impact.
  67. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013, page 1, Delivery paragraph.
  68. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 53.
  69. International Development Committee, Department for International Development’s Performance in 2012-2013: the Departmental Annual Report 2012-13, 5 March 2014, paragraphs 28-36, http://www.publications.parliament.uk/pa/cm201314/cmselect/cmintdev/693/693.pdf.
  70. ICAI, DFID’s work with UNICEF, March 2013.
  71. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 16.
  72. ICAI, DFID’s work with UNICEF, March 2013.
  73. ICAI, DFID’s work with UNICEF, March 2013.
  74. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013.
  75. ICAI, How DFID Learns, April 2014.
  76. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 104.
  77. ICAI, DFID’s Approach to Anti-Corruption, November 2011, page 15, paragraph 4.5.
  78. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 17.
  79. ICAI, DFID’s Approach to Anti-Corruption, November 2011, page 16, paragraph 4.11.
  80. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 15.
  81. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 18.
  82. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 31.
  83. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 29.
  84. ICAI, DFID’s Use of Contractors to Deliver Aid Programmes, May 2013, page 11, paragraph 2.37.
  85. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 67.
  86. ICAI, DFID’s Water, Sanitation and Hygiene Programming in Sudan, February 2013.
  87. ICAI, DFID’s Contribution to the Reduction of Child Mortality in Kenya, March 2014.
  88. ICAI, DFID’s Livelihoods Work in Western Odisha, February 2013.
  89. ICAI, Evaluation of DFID’s Support for Health and Education in India, May 2012.
  90. ICAI, DFID’s Climate Change Programme in Bangladesh, November 2011.
  91. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 15.
  92. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 55.
  93. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 101.
  94. ICAI, DFID’s Support for Civil Society Organisations through Programme Partnership Arrangements, May 2013
  95. ICAI, How DFID Learns, April 2014.
  96. ICAI, Evaluation of the Inter-Departmental Conflict Pool, July 2012.
  97. ICAI, Evaluation of DFID’s Support for Health and Education in India, May 2012.
  98. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), pages 104, 105.
  99. ICAI, How DFID Learns, April 2014, page 27, recommendation 1 and paragraph 3.20 et seq.
  100. ICAI, How DFID Learns, April 2014, page 20, paragraph 2.74.
  101. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 65.
  102. ICAI, How DFID Learns, April 2014, page 20, paragraph 2.74.
  103. ICAI, How DFID Learns, April 2014, page 26, paragraph 3.6.
  104. ICAI, How DFID Learns, April 2014, page 29, Recommendation 4.
  105. ICAI, How DFID Learns, April 2014.
  106. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 18, ‘Using the best available evidence (research, evaluation results and statistical data) is important for delivering development programmes that give us the best value for money.’
  107. ICAI, How DFID Learns, April 2014, page 14, paragraph 2.42.
  108. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 6.
  109. DFID, Management Response to ICAI’s report on DFID’s Peace and Security Programme in Nepal, February 2013, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/142896/ICAI-man-resp-effectiveness-peace-security-program-Nepal.pdf.
  110. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 15.
  111. DFID, Smart Rules; Better Programme Delivery, October 2014 (version expires on 1 February 2015), page 10.