Challenging narratives: Capacity development

How good are we at defining and judging 'success' - the essence of what evaluators do? Let us consider a pertinent example: Every year the aid industry spends a quarter of its funding, around US$15 billion, on 'capacity development' (CD). So-called 'state-building' is especially popular. Inevitably, evaluations in the Global South very frequently focus on this aspect.

A 2017 report by the Secure Livelihoods Research Consortium highlights a series of studies conducted over six years on diverse international efforts to build state capacity for service delivery in eight fragile, conflict-affected countries. Service delivery is a core function of a modern state, and the study was aimed at understanding whether the dominant models used for capacity development were fit for this important purpose.

Flaws in CD strategies, and the implications for evaluation

Their findings have the following implications for evaluators and evaluation commissioners, with reference to state capacity development, but the issues are applicable in other contexts too:

Evaluations need to focus on how change really happens, beyond training and 'gaining knowledge'. "Too often, grand efforts to rebuild states and societies boil down to training or providing equipment and resources." Programming tends to favour short-term training, and its tangible and quantifiable approaches. Yet getting more knowledge is only part of a much bigger picture. Many of the training initiatives have internal design weaknesses such as flawed learning methodologies, too short timeframes and insufficient localisation. The authorising environment in which new skills and knowledge have to be applied has to be taken into account. Intangibles such as values, vision, leadership, management style, and organisational culture are at least as important as 'countable' CD activities.

Evaluations need to consider the underlying power, politics and systems that prevent success. "Power and politics are central to how services are delivered, but capacity development often concentrates on technical aspects only." State capacity to delivery services is shaped by factors related to power and politics, such as elite behaviour or the invisible forces of informal institutions and social norms. The incentives and influence of local power brokers are ignored, or the asymmetries in relationships within the household, for example.

Evaluations need to focus beyond the obvious in understanding how new capacities are applied in practice, recognising existing limitations. In particular in the illustrative cases in the report, relational aspects of development, for example how citizens view, trust and interact with different parts of government across different locations are neglected, despite playing a strong role in people's perception and experience of services. People's decision-making incentives and the breadth of providers people rely on, often extend beyond the state. Under-investments are made at district or provincial level.

Evaluations need to think more holistically about the capacity of systems. "By focusing on individuals and organisations, capacity development is typically approached as a modular exercise, assuming micro-capacities naturally aggregate up to build better systems." With a focus on individuals or organisations, the systems themselves - say a network of organisations with formal regulations and mechanisms, and their power relations, informal institutions, social interactions and other contextual influences - receive inadequate attention. Tweaking one part of a system is not necessarily enough. It is important to consider what else needs to be in place for the new capacities to make a real difference.

Limitations persist because the aid system itself is flawed. Staff are hired for their technical skills rather than contextual or conceptual knowledge. Short timeframes have to support long-term change processes. There is a focus on the tangible and quantifiable. Accountability for aid is to donor publics, not to beneficiaries. There is a push for results and a diminishing appetite for risk. Social change is projectised. "The aid industry turns fundamentally political processes of social and institutional change into projects - which are in turn cloaked in value-neutral technocracy."

What does this mean for our understanding and assessment of 'success'?

The problem is that I hardly ever see evaluations of capacity development with a robust focus on these issues. Yet many CD evaluations make quite positive judgments of progress and results without every studying the issues discussed in this report.

This is an example of a critical matter that deserves our attention: 'Success' is often defined in a way that is too limited, even just plain wrong. So our assessment of it is flawed.

Success is most often seen as achieving objectives or getting within a specified timeframe to some pre-conceived results (outcomes or impacts) without considering whether they have been appropriately conceptualized in the first place.

Or it is seen as any kind of positive impact - even if the 'impact' is only a very incremental, even minute change, or one that is easily reversed (cue: what if in month or year's time there is no sign of the change, or of other positive changes flowing from the initial change?).

Or when an intervention satisfies some of the (DAC) criteria, without any summative assessment of contributions to development for example (where the understanding of "development" is clearly defined).

And as this report points out, without considering important aspects known to prevent success in a specific domain: misunderstanding how change really happens; forgetting about power asymmetries; or ignoring the larger systems within which new capacities need to be applied. People need to have the opportunity to exert their agency.

And capacity development is by no means the only domain with these problems.

In essence - many of us do not do enough systems thinking when we commission or do evaluations. And as the report points out, there may be reasons why people choose to ignore a better definition of 'success'.

Then there is the fact that we do not have the incentives - or is it knowledge and methods? - to question the overarching models that direct ideas about why, how, when, for whom, and what capacities are supposed to be developed. For example, conventional models argue that strong institutions are an imperative for development. China's development over three decades has just convincingly proven the opposite. Yet there is still a major focus on 'institution building' in the aid environment. But that is another story ….