Is there a standard basis against which any aid project can be judged?

It is a truism that any individual project takes place within a unique context in time and space, and so if the question is about a definitive or standard set of ‘indicators’ that can be applied across multiple projects, then my experience suggests that this is problematic.  However, if the question is more about a conceptual framework through which the performance of individual projects can be assessed, then I think the answer is ‘yes’.

The DAC has defined five criteria for evaluating aid projects: relevance, efficiency, effectiveness, impact, sustainability.  Definitions for these terms are available at http://www.oecd.org/dataoecd/29/21/2754804.pdf.  These criteria are probably the most widely adopted in aid evaluation, and while I frequently apply them, I have personally been influenced by Soft Systems Methodology (SSM).  The SSM literature argues that the performance of any ‘system’ (e.g. an aid project system) can be judged against the three criteria: efficiency, efficacy and effectiveness.  These terms are all derived from the Latin efficere, to accomplish, but within the context of project performance have subtly different interpretations.  Both efficiency and effectiveness are widely used in general management literature, and also aid project management literature, however, their use has frequently been imprecise.  Efficacy, while not as widely used, offers a third dimension to the concept of project performance.

·       Efficiency (‘doing the thing right’; ‘is there minimum use of resources?’) concerns cost and process management, and is a core emphasis of the managerialist paradigm in general and project management literature in particular.  Much work has been done in developing methods to increase the control exerted by project implementation teams, and hence improve project efficiency.  A project is efficient if it delivers the planned outputs within the budgeted inputs and on or ahead of schedule.  Within certain constraints, the efficiency of a project is a function of the management capacity of the implementation team.  SSM is concerned with efficiency since a system is likely to fail to achieve the desired ends without an economy of resource usage.

·       Efficacy (‘doing a successful thing’; ‘does the means work?’) concerns the merit of the theory of change of a given project.  Every aid project is based on an implicit ‘theory of change’ that assumes that the deliverables (outputs) of the project will foster changes in knowledge, attitude or practice (outcomes) among people/communities with whom the project interacts (direct beneficiaries).   That is, the extent to which the ‘means’ produce the anticipated ‘ends’.  Efficacy tends to be the focus of most evaluation literature and is a function of the project design—the extent to which the causality of the project is grounded in well-established knowledge and utilises appropriate mechanisms of social transformation .  A project design may be deemed efficacious when the outputs of the project result in the anticipated outcomes.  SSM is concerned with efficacy since a system (project) could fail if the means selected to bring about the planned transformation does not work.

·       Effectiveness (‘doing the right thing’; ‘is this the right thing to be doing?’) concerns the philosophical and developmental worthiness or appropriateness of an initiative (Crawford and Bryce 2003).  Ultimately, effectiveness is determined by the ecological, social and economic sustainability of interventions and hence is a function of the policies and strategies adopted by the project implementing agency.  Whereas efficacy is concerned with the performance of a single project in fostering social transformation, effectiveness is concerned with the performance of the whole system (i.e. the whole aid program or approach) to which any given project makes a contribution.  That is, given a project deemed to be efficacious, effectiveness is concerned with whether or not the project outcomes contribute to the anticipated impact.  Checkland’s ‘monitoring and control subsystem’ is concerned with effectiveness since the system under study may be deemed to have failed when it is perceived to have not contributed to high-level, longer-term goals (e.g. the agency’s mission).  In other words, effectiveness is a measure of the extent to which a particular system contributes to the wider context that gives that system purpose.[1]

I keep coming back to these criteria because I find them to be simple/succinct and yet comprehensive:

·       They are simple because they cut through some of the ambiguity and overlap between the five DAC criteria (NB note that even the DAC definition of ‘effectiveness’ recognises two possible interpretations).  I’ve found that non-English speakers sometimes struggle to differentiate the subtle differences between ‘relevance’, ‘effectiveness’, ‘impact’ and ‘sustainabiliity’). 

·       They are comprehensive because thee overall success of an aid project requires satisfactory performance against each of the three ‘Es’.