(c) the Commission has made a creditable and credible stab at the exercise. Member States should certainly welcome the EU Results Framework. If they do not already have one of their own, they should start at once to prepare one.
There are, however, some issues on which it would be appropriate for Member States to ask for further work.The outstanding items are:
- Thinking through the problem of tying the Results Framework to the SDGs, which are themselves illogical and overly complex.
- Broadening the scope. The results framework is presented as relating to ‘International Cooperation and Development’, which sounds all-encompassing, but in fact relates only to the work of the Development Commissioner, and within his remit, mainly to aid-financed activities. Thus, the Results Framework does not cover humanitarian aid, but more important has nothing yet to say about trade, climate change, foreign and security policy, migration, or any other aspects of the policy coherence agenda. Surely, Member States should ask that the Commission have a Results Framework covering all its activity? Perhaps the new Security Strategy will provide a template in that respect.
- The EU Framework is presented as ‘sectors’ rather than ‘goal areas’. This probably reflects the spending orientation of the document, but also seems to mix ends and means in the same way as the SDGs, and to be incomplete.
- When we come to the indicators, it is striking that both the Level 1 and Level 2 indicators are selective. In addition, some of the indicators are oddly placed; and some EU sectors have national indicators, but no indicators of EU performance. The main problem is that the EU has not articulated (in this document) a theory of change which links inputs to outputs to outcomes.
- Finally, it goes without saying that a final version of the Results Framework will need to list results, rather than just result areas or indicators.
Why a Results Framework is useful
A results framework has obvious value, in a world in which the development sector has become preoccupied by impact. Just as an example, DFID published a paper on its results framework in September 2014. It summarises the purpose as follows:DFID’s Results Framework is a tool that we will use to monitor and manage progress and report publicly on delivery. It sets out the development outcomes DFID is seeking to contribute to, the actual results we will deliver, and the metrics we will use to measure our organisational effectiveness and efficiency. This is the first results framework in which we have set out actual development outputs that DFID will deliver against for which we can be held accountable. This marks a significant step forward. By measuring results we get a much better idea of what works and what does not so we can refine our programmes accordingly. We are also able to manage our resources to deliver these results.The DFID Results Framework, applicable to bilateral aid only, is available here (see Table 2a). There is a separate table for the multilateral programme. This all seems sensible, though some of the points I want to make about the EU Framework also apply to DFID. There are two generic issues that are perhaps worth pointing to straight away. First, it is important for a Results Framework to be clear about the scope. In the case of the DFID Framework, and also the EU Framework, as we shall see, the scope is limited to aid. This does not seem complete, in a world where development cooperation is increasingly seen to be about policy as well as aid – policy coherence, or Beyond Aid. The UK parliament’s International Development Select Committee criticised DFID for not reporting better on policy issues. It said that
We have made the point that action related to Policy Coherence for Development has not been well-reported in the past. There is a section in the latest Departmental Report which covers PCD, but the key themes are not visible in DIFD’s overall objectives or in its results frameworks, which focus too much on spending. Furthermore, DFID only offers a partial lens on the UK Government’s policy and activities. A number of witnesses made the point to us that better reporting was required. We have demonstrated that many other countries offer comprehensive PCD reporting.In looking at solutions, the Committee received several practical suggestions that could be thought of as extended Results Frameworks. Thus,
Anumber of witnesses, including DFID itself, noted the importance of cross-Government strategy documents (like the cross-Whitehall ‘Health is Global’ strategy), Cabinet Committees, jointly-owned cross-Government funds (like the International Climate Fund), joint ministerial appointments, and so on. There were many practical suggestions. For example, ODI proposed the “re-introduction of Public Service Agreements (or equivalent) (and) a reporting requirement to Parliament on PCD.” The British Overseas Aid Group made a different suggestion: the introduction of a “shared, coherent strategic policy framework to ensure government interventions across the board promote complementary policies.” UKAN interpreted this putative framework as a “Cross-Whitehall PCD strategy/action plan”, linked to a cross-departmental task force to track and monitor implementation.The second issue, of course, is that the whole results agenda is contested, or if not contested, then recognised to be replete with attribution and measurement problems. I have written about this before, most fully in 2011. Others also have much to say on the subject of reductionism, the bias to linear thinking, and the lack of attention to complexity and learning. See, for example, my review of Ben Ramalingam’s book on complexity, or Robert Chambers’ blistering attack on results-based aid. I was reminded the other day that Albert O. Hirschmann pioneered much of this thinking with his work on the ‘hiding hand’ in Development Projects Observed, first published in 1967. Does this mean that results frameworks are useless? No, but they do need to be approached with common sense. As professor Adrian Wood observed about the MDGs, ‘take them seriously, but not literally’. I wrote back in 2011 about what I called a post-Fordist approach to results, or Results 2.0, concluding that it was important to avoid ‘bean-counting’ (Andrew Mitchell’s phrase), and be sensitive to qualitative measures, as well as indirect effects.
Introducing the Results Framework
Let’s turn to the EU initiative. The Staff Working Paper explains the purpose of the exercise, contextualises the framework in relation to the ongoing SDG discussions, and summarises how data will be collected and used. The actual framework is in a series of appendices. The overall structure is as in Figure 2 below. There are three levels, dealing respectively with development progress in EU partner countries, EU contributions to the above, and organisational performance. These notes concern only Levels 1 and 2.
Discussion
First, it is worth reminding ourselves that the SDGs are work in progress and may change by September 2015, when they are to be agreed in New York. Actually, this looks increasingly unlikely, but I for one will be pleased if they do – see earlier pieces here and here for the reasons why. Among the many reasons to be irritated by the current draft is the failure to distinguish properly between ends and means. For example, growth is not an end in itself, rather a means to other ends, like jobs, income and security. Indeed, jobs are also a means to an end. The structure of the SDGs reflects a lack of literacy as to how logical frameworks and results matrices are supposed to work. A question, then for the EU: is it sensible to tie the results framework to a patently inadequate set of SDGs? Second, the results framework is presented as relating to ‘International Cooperation and Development’, which sounds all-encompassing, but in fact relates only to the work of the Development Commissioner, and within his remit, mainly to aid-financed activities. It just happens that the relevant service was renamed at the beginning of the Juncker Commission, from ‘Development’ to ‘International Cooperation and Development’. Thus, the Results Framework does not cover humanitarian aid, but more important has nothing to say about trade, climate change, foreign and security policy, migration, or any other aspects of the policy coherence agenda. This is surely a problem. Should not Member States insist that the Commission have a Results Framework covering all its activity? Perhaps the new Security Strategy will provide a template in that respect. Third, it is interesting that the EU Framework is presented as ‘sectors’ rather than ‘goal areas’. This probably reflects the spending bias of the document, but also seems to mix ends and means in the same way as the SDGs, and to be incomplete. On the latter point, why, for example, is nutrition defined as agriculture only, and not health (and we will come shortly to the importance of transitory as well as chronic food insecurity)? And why are food shocks the only kinds of shocks considered? Don’t financial or commodity price shocks count? Or extreme weather events? Fourth, we then come to the indicators, where it is striking that both the Level 1 and Level 2 indicators are incomplete. In addition, some of the indicators are oddly placed; and some EU sectors have national indicators, but no indicators of EU performance. For example (and selectively, because similar points could be made about most of the sectors):- The poverty ‘sector’ has a poverty headcount indicator, but not a poverty gap indicator, and has no EU-relevant targets at all. There are no non-income measures of poverty included. There is an inequality target which does not belong here, and also has no EU-relevant targets.
- The nutrition indicator is for stunting, with no indicator for wasting. The EU-level indicator is for only one possible ‘driver’ of changes in nutrition status, viz nutrition programmes, with no recognition of other drivers, like growth, health care, education programmes and so on.
- The Sustainable and Inclusive Agriculture sector has only one national-level indicator, added-value per hectare, which may (but may not) measure productivity increases, but certainly does not measure either sustainability or inclusiveness. The EU-level indicators are partial.
- The employment and social protection sector has two Level 1 indicators, viz the proportion of employed people living below $US1/day and the share of older people receiving pensions. That is very limited on both counts. Where, for example, is the number of ’jobs’ or livelihoods, including those in the informal sector? The Level 2 indicator, there is only one, if for the number of people who have benefited from EU-supported active labour market programmes, so no mention of social protection.
What would be better?
What would be better? I started out by saying that this exercise was not straightforward. At the same time, the EU would be well-served by further work on the high-level objectives, supported by a full theory of change to generate better indicators and areas of activity.Member States should certainly welcome the EU Results Framework. If they do not already have one of their own, they should start at once to prepare one. They should also engage with Results Frameworks produced by developing countries, and should wherever possible align behind them. If everyone did this, the quality of joint programming would benefit hugely.
Table 1 – EU Sectors (Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf)
Table 2 (Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf)
Table 3 (Source: http://data.consilium.europa.eu/doc/document/ST-7604-2015-INIT/en/pdf)