The DAC criteria, Part 5. Non-negotiable criteria

It is ironic that evaluation has been unable to prevent - or even just sufficiently critique - some of the most damaging policies and strategies that have originated in both economically rich and poor countries. What should evaluators have done to prevent at least some of the destruction wreaked upon countries in the 1980s and 1990s by the World Bank and IMF-driven Structural Adjustment Programmes (SAPs)? Would we today have done better than the largely ineffective monitoring systems that failed to provide analyses and syntheses that reflected reality? Would we have been able to foresee the dire negative consequences of the Green Revolution (see here, here, here and here) in Asia? Are we about to repeat them? If evaluators are unable to prevent such disasters from happening, what good is the work that we do? How can similar situations be prevented in future?

These are some of the questions that should keep us awake at night.

Much depends on the evaluands. Robert Picciotto and others have been arguing for years that we should be in a position to evaluate the effects on development of macro issues such as trade policies, financial architectures and flows, foreign direct investment, global value chains, and so on - the "transmission belts of globalisation". We have little power in these domains. There is too often a lack of resources or political will to attend to those issues that have a major influence on the development trajectory of a country.

But much also depends on our evaluation questions and criteria, and on the flexibility and nuance with which we are allowed to apply them. So, what should change if we want our criteria to support development more effectively?

Our criteria depend on what we consider to be 'Development' in the SDG era

The basic tenet of my argument around a revised set of DAC criteria is that they must ENSURE that we evaluate FOR development that can sustain, rather than just help us to 'evaluate development'. See my previous post in this regard, as well as my article in Evaluation Matters of the African Development Bank, where I first wrote about the DAC criteria. So our evaluation criteria should compel us to attend to those critical aspects that are supposed to enable or support positive development trajectories at national or regional level.

This means that we should at the very least consider the following:

The values, ideologies and/or models within which we evaluate contributions to 'development'. Development is a highly contested concept, with vastly different ideas about how it can be achieved. Understandably, evaluators usually do not get entangled in evaluating a specific development model or ideology (examples can be found in a 2013 book chapter by AK Shiva Kumar and myself where we refer to the human development, human rights and human security approaches to development). But when we evaluate (portfolios of) projects, programmes or single sector policies, we almost always fail to make explicit the values, ideologies and models about development that frame what we evaluate, how we assess trade-offs, and arrive at summative judgements.

'Development effectiveness', not only 'intervention effectiveness'. Development effectiveness is usually assessed at country or regional level. The proliferation of global development related indices (and now also the SDG reporting modalities) compels comparison and competition between countries and regions. But not every investment labelled a 'development intervention' automatically contributes to development from a national perspective, even if successfully executed - especially when unrealistic assumptions are made at the 'higher levels' of theories of change, with vague statements about the linkages between the intervention objectives and societal or ecosystem impacts. As a result, we seldom focus on the development trajectory of a country or region.

The risk of 'ersatz' development. Eminent economist Ha-Joon Chang refers to 'ersatz' development as development based upon uncoordinated interventions that do not build on synergies or enable system coherence that facilitates change at a macro level. Such interventions tend to attend to the level of 'community' or 'pilots', without scaling, and strengthen the illusion that "every bit helps". This very common problem can be propagated by aid or philanthropic agencies, the private sector or a government. It is rife where governments are dysfunctional, fragile or lacking in confidence or expertise, or where a funder has little interest in engaging with integrated development policies and plans. A 'bottom-up', micro perspective of development tends to prevail, and so do debates about the much-lamented and much-debated micro-macro disconnect, see here and here, for example.

The implications for our criteria of how we define 'Development'

Cognisant of the four issues noted above, our evaluation questions and criteria should force us to

• focus on summative judgments about contributions to development effectiveness, not only on intervention effectiveness;

• ensure that we assess the significance of the evaluand at a particular time with reference to the development trajectory of a country or region;

• consider issues of synergy, complementarity and coherence (or alignment and harmonisation), and evaluate for scaling, where appropriate, in order to help ensure that any micro-macro disconnect is minimised;

• have a strong focus on the sustainability of positive development impacts, and the role of dominant resource flows in enabling this (or not).

Our criteria also depend on Development as Complex Adaptive System (CAS)

When we consider the implications of viewing and evaluating development from a complex adaptive systems perspective, we need to consider the implications for our profession and practice of the following defining concepts of complexity:

Interconnectedness leads to interdependence between the elements and dimensions of a system, and gives rise to dynamic, complex behaviours. Change is discontinuous (periods of stability alternated with periods of change) - influenced by the agents in the system, their 'rules of behavior' and the types of relationships between them.

Emergence: The dynamic networks of interactions between components, many of which cannot be predicted, can lead to a whole (system) that emerges at a 'higher order' that is greater than the sum of the parts. The order is emergent, not predetermined or controlled.

Self-organisation, adaptation: The system is never static. Individual and collective behaviours self-organise as things change, adapting to the changing environment influenced by history and feedback in order to increase survivability and find the 'best fit' with the environment. Some overall order arises from local interactions between parts of an initially disordered system; macro-scale patterns of behaviour can therefore arise spontaneously, amplified by positive feedback.

Non-linearity: The relations between the system and its environment is non-linear, and has multiple feedback loops. It is highly sensitive to initial conditions; and small changes in inputs, physical interactions or stimuli can cause very significant changes; vice-versa, a huge upset to the system may not change it in significant ways. Reinforcing feedback loops can lead to tipping points (when thresholds are overcome) and transformational change (when the system moves from one stable state to another).

Path-dependence: A set of decisions for any given circumstances is limited by the decisions made in the past, even though past circumstances might no longer be relevant. Initial conditions can also leave a persistent mark (imprint) on a society or organisation, shaping behaviours and outcomes in the long run even when contexts change.

Co-evolution: Dynamic systems adapt and evolve with a changing environment. They co-evolve with other related systems in an ecosystem (and should not be seen as always adapting to a 'changing environment'), and across multiple levels or scales.

The implications for our criteria of viewing Development as CAS

First, an insufficiently integrated perspective will inevitably lead to oversimplification of situations and changes, and incorrect assumptions about how transformational change and development are likely to happen, or development trajectories will evolve. Development interventions cannot be considered in isolation of one another and of the ecosystem around them. Economic, environmental, socio-cultural and political aspects can also not be treated as separate entities, as the philosophy underlying the SDGs clearly confirms (although the 2030 Agenda underplays the importance of culture).

Second, development dynamics and outcomes are often but not always unpredictable. Historical contexts and experiences, and societal cultures that have co-evolved over time can and should provide insights into what might happen, as such co-evolution shapes the psyche of a society. But demanding predictability through rigidly kept logframes and results-based management (RBM) as currently practiced will stymie development.

Third, understanding at least to some extent what might be essential and/or sufficient initial conditions or preconditions towards desired changes will assist with development planning and implementation, but cannot be taken as a certainty. Adaptive management and agile responses to co-evolving systems are essential, demanding the freedom and incentives to innovate and improvise, and embeddedness in local understandings.

Fourth, desired changes in a certain direction will not happen automatically through interventions, and possibly not at all, unless appropriate, culture-sensitive new ideas and incentives are in place, and obstacles to progress at local level are removed. Furthermore, if development is to be effective and accelerated, concepts such as feedback loops, synergistic effects, harmonisation, catalytic change, tipping points, sustainability and transformational change should be understood from both theoretical and practical perspectives, and designs and implementation strategies, as well as their evaluations, done with these in mind.

A non-negotiable set of evaluation criteria for development

All these implications direct us to a set of criteria that should be seen as non-negotiable during the SDG era - including the nuances in their descriptions - if we are serious about evaluating for development that will sustain. Even though assessment based on some of these criteria might be challenging, they have to be considered.

Alternatively, if we find that within the normal resources and operations of evaluators it is impossible to attend to one or more of these, we have to consider the implications for our profession and practice - in particular, for those of us who purport to fund, commission and do evaluations in support of 'development'.

COHERENCE (& INTEGRATION). Its description and rubrics will be nuanced to focus assessments on the extent of (i) integration during planning and execution of development efforts (of whatever is deemed important, such as disciplinary approaches, models, and in particular given the need for sustainable development, also relevant environmental aspects; (ii) complementarity and synergy - whether synergistic effects have been planned for or achieved - this is where for example policies, interventions and/or cross-cutting principles work together to ensure acceleration or amplification of development impacts; (iii) harmonisation (or alignment) - whether relevant policies, interventions, principles, etc. have been aligned or at least do not oppose or obstruct one another towards development objectives at national or regional level; and (iv) coordination between efforts, wherever relevant. This will demand intensive engagement with influencing factors. Two excellent recent ICSU reports here and here highlight ways to work with these concepts, with interesting examples relating to the SDGs. This approach can be expanded to cover other types of work.

SIGNIFICANCE. Its description and rubrics will be nuanced to focus assessments beyond 'Relevance', on what is actually 'Significant' from a development perspective. Rubrics will make explicit the values and reasoning around the definition of "Significance', but will include issues such as (i) the scope (breadth, depth, coverage) of the effort; (ii) its relevance and timeliness at a particular point or over a specified period of the evolution of the effort given the development trajectory of the country or region; and (iii) the extent to which deeper layers of causes (what we used to call "root causes"), catalytic action, bottlenecks, tipping points and/or transformational change have been considered and/or addressed during planning, or have been achieved - and contributions made to that - during or after execution.

RESPONSIVENESS & IMPROVISATION (can also be defined as ADAPTABILITY & ADAPTATION). Its description and rubrics will be nuanced to focus attention on the need to be flexible and agile, as well as innovative and able to improvise in (i) responding to context and changing circumstances, including the co-evolution of context and culture; and in (ii) anticipation of risk, including dealing with destructive power asymmetries and other important negative influences on success.

IMPACT & SUSTAINABILITY. Its description and rubrics will be nuanced to focus attention on the need to (i) connect impact and sustainability, whether during the planning or execution phases, or afterwards - and hence include 'Impact sustainability'; (ii) examine with vigour any (potentially) neutralising effects that can emerge from foreseen or unforeseen negative consequences or impact ripples (the ICSU approach described here and here can again be very helpful); and (iii) Ecological sustainability, to reinforce the importance of environmental considerations for sustainable development.

DEVELOPMENT EFFECTIVENESS. Its description and rubrics will be nuanced to ensure attention to summative judgments that (i) pull together the assessments based on individual criteria, and (ii) consider, as relevant, intervention as well as development models and trajectories at national and/or regional levels. We should be challenged to engage - at least to some extent - with the sum of what we have assessed, and what it means for a country or region at different stages of its development within its chosen development path (or lack thereof). This will challenge us to clarify and make explicit the values and beliefs about 'development' that underpin our evaluative judgments.

Finally …

Each description and use of the set of criteria has to ensure that some attention is shifted back to the merit of the design and execution approaches, and to preconditions that are essential and sufficient for development success. An obsession with 'Impact' in isolation of everything else can lead to very wrong conclusions about the extent to which certain efforts have supported development that is likely to sustain.

Thinking about these matters, and about this ambitious framing of potential criteria for our use, should also challenge us to consider the limits of the value we are in a position to add as evaluators to countries' and regions' development efforts. And whether we should do more to ensure that we add more value than we have done to date.