The DAC criteria, Part 4. Flexible criteria

Our development evaluation criteria should not be determined solely by stakeholder-determined interests, or by pragmatism resulting from constrained resources. Development of nation-states is a complicated matter. If we are serious about ensuring that evaluation plays an effective role in it, there are questions and criteria that we should treat as essential while others can be more flexibly applied.

But how do we determine which evaluation criteria to use under which circumstances? Instead of making some ad hoc choices or mechanically applying a set of pre-determined criteria, we need a credible framework that can direct our understanding in this regard.

Characteristics of a framework for evaluation criteria

Let us consider some of the possible determinants of a framework for one or more sets of revised criteria. There are many possibilities, but based on an analysis of existing challenges related to the DAC criteria (post 2 in this series), at least six imperatives will determine the character of such a set:

1. Flexibility per context: The set of criteria will enable comparison between evaluations, yet also ensure the necessary flexibility to accommodate different mandates, norms and stakeholder interests. A certain part of the criteria will have to be flexible enough to be tailored to circumstance.
2. A non-negotiable core set resulting from the nature of development. The set of criteria will ensure that we address critical elements of "sustainable development" - otherwise we will not be evaluating for This demands engagement with the conceptualisation of "development", and with development as complex adaptive system.
3. Wide applicability: The set will be suitable for a variety of evaluands, beyond projects and programmes, and across organisational, sector and geographic boundaries.
4. Nuance: They will have definitions and descriptions that are detailed and hence nuanced enough to improve application and help drive good performance standards.
5. Supporting conceptualisation, execution and results: They will shift some attention away from an obsession with "impact" towards smarter design and implementation efforts. Evaluation priorities and methodologies will better support intervention and systems design and implementation.
6. Enabling meaningful synthesis: They will facilitate meaningful synthesis judgments across criteria to get a reasonably holistic picture of the extent to which (sustainable) development or humanitarian action has been improved.

Flexible sets of criteria, tailored to context

As I noted in post 3 in this series, there are two categories of criteria that can be considered as flexible. Which ones are used depends on the context and circumstances.

This set is determined by important global, regional and/or societal values and norms during a specific period, reflected within an organisational mandate. It will highlight the frameworks or models - and hence the ideologies - about development that a society or organisation and its partners hold dear. It will resonate with global and regional conventions such as the 2030 Agenda for Sustainable Development with its SDGs, and the Agenda 2063 of the African Union.

This set of criteria is only to some extent flexible. There is an imperative to reflect the norms of the day as determined by the mandates and strategies of each organisation or partnership involved in an evaluation.

The following are well-known as well as lesser known examples; the list can be much more extensive

Among the less common criteria:

Inclusivity, Reach and Coverage all relate to the important norm of "no-one left behind" in the 2030 Agenda for Sustainable Development. Though usually treated as the same concept, they can also have slightly different nuances, where for example Reach reflects the boundaries of a targeted group within a country or region (e.g. a major population in need of water and sanitation), while Coverage reflects the extent to which different groupings within the targeted population have been adequately reached (e.g. households in distant villages; the most poverty-stricken households).

Legitimacy is a value-laden concept that relates to the credibility of interventions and processes of, for example, planning or engagement, as seen through the eyes of different stakeholder groupings.

An aspect that is usually neglected by well-meaning outsiders is that different parts of the world are likely to prioritize the criteria in such a set differently, based on their societal philosophies and values - if they are allowed the space to do so. In certain parts of Asia, Africa and among indigenous peoples, the interests of the society as a collective and/or as ecosystem, usually based on intricate webs of relationships, are deemed so important that it equals and might even trump individual rights. Harmony in Society is for example a very important ideal in East Asia, rooted in Confucian philosophy. It seeks to balance and reconcile differences and conflicts through creativity and mutual transformation rather than conflict. Harmony with Nature, with strong roots in Taoism and other indigenous philosophies, is in line with sustainable development as reflected in the SDGs. It is a concept set to become more important across the world. Our existing sets of criteria seldom give prominence to such differences.

These criteria are determined by stakeholder interests and concerns at a particular moment. There is no other imperative to include any of these in a set of criteria, and they are therefore even more flexible than the set based on mandates and norms. Examples drawn from current practice include

However, there is in my view also a core set of non-negotiable criteria that needs to be considered. This very important and likely controversial issue will be the focus of my next post.