Zen and the art of monitoring & evaluation

The 'theory of change' approach

pcrawford@aid-it.com.au — Mon, 13 Oct 2008 14:00:00 GMT

For a long time I've been using the phrase 'theory of change' to express the idea that a project is essentially a social experiment, and that M&E is about testing the hypotheses implicit in the social experiment. Recently I was challenged to succinctly ellaborate what I thought embodied the 'theory of change' approach. The following points provide an overivew of the concept, and the practical and philosophical elements, as I see them:

THEORY OF CHANGE

Broad Concept

International aid projects exist to create social change. Any project implicitly aligns with a ‘theory’ about how desirable social change might be achieved—a ‘theory of change’.
To bring about social change, human actors interact within a social system through time.
A role of project design is to articulate the ‘theory’—the temporal sequence of relationships, and how these are expected to influence desirable change.
A role of M&E is to test the ‘theory’—to judge the extent and merit of the changes fostered by the project. These judgements are required to satisfy demands for accountability (‘to prove’) and learning (‘to improve’).

Philosophical Aspects

The ‘theory of change approach’ recognises that, by definition, social change takes place through the interaction of human actors. No humans, no change. This is an ‘actor centric’ or ‘interpretist’ perspective.
Any ‘project’ is conceived as a ‘social experiment’ within which the ‘theory of change’ is tested. This tends to foster greater curiosity in the efficacy of the design (and less defensiveness among designers and implementers), and hence, greater reflection and learning about what works and what doesn’t work.

Practical Aspects

The design logic explicitly recognises the temporal sequence of relationships that effect the desired change: implementing team --> direct beneficiaries (‘boundary partners’) --> ultimate beneficiaries.
Outputs, Outcomes and Impact are defined in terms of changes effected by, or experienced by, each of the classes of human actor (respectively: implementing team, direct beneficiaries, ultimate beneficiaries).
M&E explores the role and experience of each class of human actor in the change process; and explicitly captures evidence of when change has been less than anticipated (i.e. where risks have been borne out).

The '3Es': a useful conceptual framework through which to judge any project performance

pcrawford@aid-it.com.au — Sun, 07 Sep 2008 13:00:00 GMT

Is there a standard basis against which any aid project can be judged?

It is a truism that any individual project takes place within a unique context in time and space, and so if the question is about a definitive or standard set of ‘indicators’ that can be applied across multiple projects, then my experience suggests that this is problematic. However, if the question is more about a conceptual framework through which the performance of individual projects can be assessed, then I think the answer is ‘yes’.

The DAC has defined five criteria for evaluating aid projects: relevance, efficiency, effectiveness, impact, sustainability. Definitions for these terms are available at http://www.oecd.org/dataoecd/29/21/2754804.pdf. These criteria are probably the most widely adopted in aid evaluation, and while I frequently apply them, I have personally been influenced by Soft Systems Methodology (SSM). The SSM literature argues that the performance of any ‘system’ (e.g. an aid project system) can be judged against the three criteria: efficiency, efficacy and effectiveness. These terms are all derived from the Latin efficere, to accomplish, but within the context of project performance have subtly different interpretations. Both efficiency and effectiveness are widely used in general management literature, and also aid project management literature, however, their use has frequently been imprecise. Efficacy, while not as widely used, offers a third dimension to the concept of project performance.

· Efficiency (‘doing the thing right’; ‘is there minimum use of resources?’) concerns cost and process management, and is a core emphasis of the managerialist paradigm in general and project management literature in particular. Much work has been done in developing methods to increase the control exerted by project implementation teams, and hence improve project efficiency. A project is efficient if it delivers the planned outputs within the budgeted inputs and on or ahead of schedule. Within certain constraints, the efficiency of a project is a function of the management capacity of the implementation team. SSM is concerned with efficiency since a system is likely to fail to achieve the desired ends without an economy of resource usage.

· Efficacy (‘doing a successful thing’; ‘does the means work?’) concerns the merit of the theory of change of a given project. Every aid project is based on an implicit ‘theory of change’ that assumes that the deliverables (outputs) of the project will foster changes in knowledge, attitude or practice (outcomes) among people/communities with whom the project interacts (direct beneficiaries). That is, the extent to which the ‘means’ produce the anticipated ‘ends’. Efficacy tends to be the focus of most evaluation literature and is a function of the project design—the extent to which the causality of the project is grounded in well-established knowledge and utilises appropriate mechanisms of social transformation . A project design may be deemed efficacious when the outputs of the project result in the anticipated outcomes. SSM is concerned with efficacy since a system (project) could fail if the means selected to bring about the planned transformation does not work.

· Effectiveness (‘doing the right thing’; ‘is this the right thing to be doing?’) concerns the philosophical and developmental worthiness or appropriateness of an initiative (Crawford and Bryce 2003). Ultimately, effectiveness is determined by the ecological, social and economic sustainability of interventions and hence is a function of the policies and strategies adopted by the project implementing agency. Whereas efficacy is concerned with the performance of a single project in fostering social transformation, effectiveness is concerned with the performance of the whole system (i.e. the whole aid program or approach) to which any given project makes a contribution. That is, given a project deemed to be efficacious, effectiveness is concerned with whether or not the project outcomes contribute to the anticipated impact. Checkland’s ‘monitoring and control subsystem’ is concerned with effectiveness since the system under study may be deemed to have failed when it is perceived to have not contributed to high-level, longer-term goals (e.g. the agency’s mission). In other words, effectiveness is a measure of the extent to which a particular system contributes to the wider context that gives that system purpose.[1]

I keep coming back to these criteria because I find them to be simple/succinct and yet comprehensive:

· They are simple because they cut through some of the ambiguity and overlap between the five DAC criteria (NB note that even the DAC definition of ‘effectiveness’ recognises two possible interpretations). I’ve found that non-English speakers sometimes struggle to differentiate the subtle differences between ‘relevance’, ‘effectiveness’, ‘impact’ and ‘sustainabiliity’).

· They are comprehensive because thee overall success of an aid project requires satisfactory performance against each of the three ‘Es’.

www.paddledogs.org

pcrawford@aid-it.com.au — Sun, 07 Sep 2008 13:00:00 GMT

Some friends and I will paddle our sea kayaks across Bass Strait (from Victoria to Tasmania, Australia) in March 2009. Why? We keep asking ourselves that question too…and it’s something to do with proving to ourselves that life after 40 is not so bad…and raising awareness and funds for bipolar disorder.

Check out our website at www.paddledogs.org/

The Black Dog Institute is doing great work to learn more about bipolar disorder, and we’d like to raise $10,000 to support research that leads to early detection in kids and adolescents.

As part of our training program we’ll be running across the Sydney Harbour Bridge in the 9km Blackmores Sydney Running Festival event…in a canoe! Yes, we’re probably mad, but we’ll run in a modified canoe ‘Fred Flintstone style’ for the whole 9km.

Please make a donation to the Black Dog Institute by clicking on this link: http://www.gofundraise.com.au/PaulCrawford or alternatively, contact me directly to discuss corporate or tax deductible donations (NB corporate donations of over $1000 can be featured on our home page).

Thanks in advance for supporting this worthy cause, and our crazy plan!

Cheers,

Paul

Bursting the M&E bubble

pcrawford@aid-it.com.au — Tue, 13 May 2008 14:00:00 GMT

I recently had a conversation with a senior bilateral aid donor official that confirmed a growing feeling I've had that the 'M&E bubble' may burst sometime soon.

In recent years the field of 'M&E' has escalated from a peripheral area to a highly sought-after discipline. Demand for greater accountability and evidence of aid effectiveness has been a driving force for more, and better M&E.

But I've begun getting a sense that the pendulum may be reaching the end of its arc. The recent conversation with the bilateral donor official mentioned above indicated that there was a growing frustration with the inadequacy of M&E information to support its own fundamental tenet...to inform judgements about aid project performance. The risk then is that the whole concept of M&E is rejected outright, rather than discriminaing between good and bad quality M&E (aka 'throwing the baby out with the bathwater'!).

There are at least two reasons why the M&E bubble may burst...

1) Unrealistic expectations about what M&E can achieve...particularly an expectation that unmeasurable things should be measured, and frustration when they are not
2) Poor quality work by M&E professionals...including 'data dumping' (overloading stakeholders with a mountain of data, rather than exerting the effort to synthesise the key issues in a more accessible form)

In order for M&E not to be rejected outright, it is the responsibility of M&E professionals to do good work with integrity, and to constantly educate stakeholders about the limits of M&E as a 'science'.

Don't average categorical or ordinal data!

pcrawford@aid-it.com.au — Wed, 23 Apr 2008 13:00:00 GMT

A surprisingly common mistake that is made with the analysis and interpretation of M&E data is to try to treat categorical or ordinal data as though it is quantitative data (i.e. interval or ratio data). This is simply wrong. It is bad numeracy!

First some background...

Categorical data (also called 'nominal data') describes an 'exclusive category'. For example, consider a survey question:

What pets do you own: a) dog; b) cat; c) goldfish; d) none

These are categorical responses. You either do or do not own a cat.

Ordinal data is the same as categorical data, except that the order of the responses matters. Consider a survey question:

How do you feel today: a) lousy; b) ok; c) great; d) never better

These are ordinal responses because most people would agree that they represent a progression of feelings. BUT, the difference between the states is debatable. My 'gap' between 'lousy' and 'ok' may be different to your gap between 'lousy' and 'ok'. And furthermore, how I interpret the gap between 'lousy' and 'ok' today may be different to how I interpret it tomorrow! But I will always agree that 'lousy' is worse than 'ok'.

Now to the common error...

For the sake of convenience, people often 'code' the responses in surveys with numbers, such as: 1) lousy; 2) ok; 3) great; 4) never better. Then the tempation arises, to start manipulating the data quantitatively. For example, someone might try to average all of the responses to the question "how do you feel today" and conclude that because the number '3.4' can be derived, that the 'average' feeling is "great".

The way to check that your analysis is sensical, is to substitute the numbers for letters and see if you can still perform the same analysis, and derive the same conclusion. For instance:

A + B / 2 = ??

'Cat' + 'Dog'/2 = ??

'lousy' + 'ok' = ??

Useful analysis of categorical and ordinal data includes:

Mode: what is the most frequent response (e.g. more people answered 'dog' than any other response)
Frequency: the percentage of respondents that chose 'goldfish'; or the distribution of frequencies for all responses
Crosstabulation: e.g. 'of those people that said they own a "dog", what proportion said that they felt "never better"?'

Risk Management: Reactive, Proactive and Predictive

pcrawford@aid-it.com.au — Sun, 23 Mar 2008 13:00:00 GMT

Risk management involves 'identifying risks' (i.e. detecting an event of significance), and then 'mitigating the risk' (i.e. responding appropriately).

Risk management has increasingly become recognised as an integral part of good practice M&E. But in the international aid industry we still have a lot to learn about risk management. The realisation of risks can be seen as a 'failure', but it represents a clear opportunity to learn. One of my favourite quotes is from Gharajedaghi, J. (1999):

"Learning results from being surprised: detecting a mismatch between what was expected to happen and what actually did happen. If one understands why the mismatch occurred (diagnosis) and is able to do things in a way that avoids a mismatch in the future (prescription), one has learned."

One of the problems I frequently observe with risk management in the aid industry is that it is too blunt an instrument to be of much value. Rarely do we apply a 'typology' of risks to make sense of them. So a consequence is that we end up with an ocean of un-interpretable risks...very little value!

One way to help with this is to ground the risks in the human reality of social change. What I mean by this is to assign a 'human face' to risk...i.e. ask 'risk to whom?'

The theory of change implicit in most international aid projects can be conceived as a temporal sequence of relationships between a series of human actors: i) the implementing team --> ii) the direct beneficiaries/partner organisations/boundary partners --> iii) the ulitimate beneficiares. (NB this temporal sequence of human relationships also corresponds to the Outputs, Outcomes and Impact in most project designs).

Risks can then be conceived as inhibitors of the overall 'theory of change' by appreciating who is most directly affected. Following from above, there are then three classes of risk that correspond to the three classes of human actor implicit in the theory of change. Risks may cause the overall theory of change to fail by directly affecting the i) implementing team, ii) direct beneficiaries/partners or iii) the ulitmate beneficiaries. For example:

Managment Risk: events that directly affect the ability of the implementing team to deliver on the terms of the project management plan (e.g. unforeseen flooding means that a particular training event in a target village cannot be conducted by the implementing team).
Intervention Risk: events that affect the direct beneficiaries/partners' ability to realise intermediate changes despite successful work by the implementing team (e.g. government extension workers trained in extension methodologies by the implementing team are unable to implement their new skills because they are not allocted sufficient resources for travel and field work by their department).
Development Risk: events that affect the ulimtate beneficiares of projects despite successful project management and the realisation of intermediate changes (e.g. smallholder famers do not adopt improved agronomic practices promoted by extension staff because of deep cultural beliefs).

The value of applying a rational 'risk typology' such as the one briefly outlined above is that risk information can be of greater practical value. One construct that can help us consider the role of risk identification is consider how it can help us to be:

Reactive
Proactive
Predictive

These three forms of risk mitigation correspond to the three classes of risk outlined above. Identifying 'management risks' can help us be reactive to issues that have emergend that will affect the delivery of project Outputs. Identifying 'intervention risks' can help us to be proactive about emerging risks that may affect the success of the project design in fostering Outcomes. Identifying possible 'development risks' can help us to predict likely factors that may affect the longterm Impact and sustainability of the whole project.

So there's the challenge with risk management: be reactive, proactive and predictive!

Logframe logic

pcrawford@aid-it.com.au — Sun, 02 Mar 2008 13:00:00 GMT

Recently Rick Davies eloquently wrote in his blog ('Rick on the Road': http://mandenews.blogspot.com/) about different ways that people articulate program logic in logframes.

This is an issue that I've also battled, and have written about elsewhere (http://www.aid-it.comhttp://www.aid-it.com.au/Portals/0/Documents/070105_Aristotle%20and%20Plato%20at%20it%20again.pdf).

Many users of logframes seem to lose sight of the fact that the 'vertical logic' is supposed to describe the temporal sequence of change. That is, how a particular project is anticipated to contribute to social change through time. Instead, some folk use the vertical logic to systematically disaggregate the problem...in what Rick succinctly decribes as a "hirerearchy of inclusion". An analogy of this approach to program logic is:

* the Goal = a wall
* the Purpose = the bricks that make up the wall
* the Outputs = the sand and cement that make up the bricks
* the Acivities = the molecules that makes up the grains of sand and cement
etc...

This approach to logic derives from Aristotle (and has been called 'functionalism'), and holds a certain appeal for organising the 'the problem' into bite size pieces. But this approach loses sight of the mechanics of social change...which afterall is what the logframe is supposed to describe. If you draw a line along a wall that represents the linear passage of time, and then cut out the 'cells' of a logframe prepared using this form of logic, in many cases you would find it difficult to distribute the 'cells' meaningfully along the timeline. They would tend to cluster around a paticular point in time rather than elaborating a process of social change through time. Also the the interplay betweeen key classes of human actor would likely be abstract...i.e. it would not necessarily be explicit which classes of human actor are invovled at each stage of change.

An alternative approach to logic derives from Plato and has been called 'interpretivism', since it aims to 'interpret' logic through the eyes of human actors. This approach to logic makes explicit the role of key classes of human actor within the overall social change process. Each class of human actor either influences or is influenced, or both...

* Goal = significant and lasting changes anticipated in the lives of the ultimate beneficiaries (e.g. households with improved livelihoods)
* Outcomes = changes in knowledge/attitude/practice influenced among intermediary stakeholders (e.g. government extension workers with better capacity)
* Outputs = deliverables of the project implementing team aimed to iniate the change process (e.g. training of governement extension workers)

Can IT really help M&E?

pcrawford@aid-it.com.au — Sun, 20 Jan 2008 14:00:00 GMT

Yes, yes...I know I haven't been a good boy in the blogging department! And I've promised Santa that I'll be better this year. No excuse really except for the familiar cries of too busy, too much international travel, a new baby...

Anyway, enough with excuses.

An issue that has come up several times recently is the role of information systems in support of M&E; specifically, the relative merit of technology-based systems.

As one of the founders of Aid-IT Solutions, I obviously have a view that IT systems can help to improve M&E outcomes. But it is clear that IT is no 'magic bullet'.The world learned a very harsh lesson about being unrealistic in this regard during the Dot.Com mania of the new millennium!

A technology-supported M&E system does not change the fundamental problem facing M&E practitioners...that measuring amorphous social change and atributing this to particular interventions is difficult. Computers can't help with this. It is ludicrous to believe that a M&E system can dissolve this problem.

Nevertheless, well placed IT can definitely assist humans with some practical aspects of M&E work. The simple fact is that irrespective of whether M&E is conducted through paper-based, oral, visual or technology-supported systems, there are many mundane and practical challenges. Well planned IT systems can assist with tackling these challenges.

Technology systems are most successful when they acknowledge and are grounded in human realities...when they help individuals to save time/effort and/or attract esteem.

Some of the practical things that a good M&E information system can help with include: a) streamlining the capture of raw data; b) assisting with the analysis, storage and retrieval of information; c) efficiently disseminating performance information to globally distributed stakeholders.

But beyond these practical benefits, the ultimate challenge for all M&E arrangements is to ensure that the information is actually utilised. With escalating stakeholder demands for evidence of aid effectiveness has come a dramatic increase in the requirement for information synthesis.

Synthesis is not easy for humans or technology alone. It requires an intelligent interaction. Synthesis remains the strategic challenge for M&E practitioners. The multi-award winning scientist, Professor Edward Wilson captures this challenge succinctly:

"Thanks to science and technology, access to factual knowledge of all kinds is rising exponentially while dropping in unit cost. It is destined to become global and democratic. Soon it will be available everywhere on television and computer screens. What then? The answer is clear: synthesis. We are drowning in information, while starving for wisdom. The world henceforth will be run by synthesisers, people able to put together the right information at the right time, think critically about it, and make important choices wisely."

Wilson, E. (1998) Consilience: the unity of knowledge, Abacus, London, p 300

Yes, but what do you actually do?

pcrawford@aid-it.com.au — Sun, 01 Jul 2007 13:00:00 GMT

Outputs are universally defined as the tangible/quantifiable deliverables for which an implementation team can be held responsible. There are two things that seem to commonly confound the way we define 'outputs' for M&E purposes.

We confuse 'outputs' (conceived as the deliverables of the implementation team), with 'outputs' conveived as the anticipated deliverables of the project (i.e. the outcomes)...more on this another time!
We tend to get a little verbose in the way we describe what we are actually delivering. Part of this stems from professional pride and the need to be offering something fresh and innovative. But it also partly stems from the 'spin' that is sometimes associated with a project proposal submitted to a donor. We feel the need to sound sophisticated, and to align with popular (or politically correct) terminology.

In either case, what we end up with is vague (and hence for M&E purposes...) unattainable outputs. The acid test is to imagine an implementation team member coming home at the end of a day in the field and being asked by his/her partner, "Hi honey, what did you do today?". If the outputs (as they are phrased) render a vague or unbelievable response, then chances are, for M&E purposes, they are not very useful.

For example, if someone working in an advocacy project declares to their partner, "today I've been lobbying". This will almost certainly lead to a follow-up question..."yes, but what did you actually do?".

Work by the International Development & Research Centre (IDRC) offers a useful framework to help guide our thinking in terms of what we actually do in aid projects. They argue that there are three broad types of strategy that development agencies employ:

* Causal: strategies aimed at directly changing a situation
* Persuasive: strategies aimed at influencing a situation or thinking about a situation
* Supportive: strategies aimed at influencing the environment with which a focal problem is situated

When we apply this framework, we find that there are literally a limited set of things that any development agency can actually do to foster desirabel changes in the world:

Causal

Resources: equipment, literature, tools...
Infrastructure: major assets, buildings, structures...

Persuasive

Training: lectures, demonstrations, exchange visits, workshops...
Mentoring: informal training such as on-the-job coaching...

Supportive

Special events: meetings, conferences, workshops, public forums...
Information campaigns: advertising, media releases, meetings with key change agents...

This may seem a little boring compared to nice fluffy statements about what we are doing. But for M&E purposes it grounds our work and locates the anticipated changes of our projects in a pragmatic framework.

The reason this is important is that when we come to evaluate the impact of a project, we can appreciate any significant and lasting changes observed among the ultimate beneficiaries in the context of what has actually been done to foster those changes (i.e. the effort invested by the implementation team). This then allows more meaningful debate about what has worked or not worked.

'Outputs' more than the boring quantitative stuff

pcrawford@aid-it.com.au — Thu, 21 Jun 2007 14:00:00 GMT

In recent years 'Outputs' have become the pariah of M&E. There's good reason for this of course...because in the bad old days, people tended to get stuck on only reporting the widgets that projects produced, and in so doing, lost sight of the bigger picture of the human changes that were expected to result from these widgets...the outcomes and impact.

But, I think that we've somehow thrown the baby out with the bathwater. There is a sense the M&E of outputs is only concerned with the boring 'quantitative' data about projects, and so tends to be downplayed by 'enlightened' M&E practitioners. But in fact there can be a lot of qualitative value in ensuring the 'outputs' remain one element within our M&E arrangements. This can be a source of valuable information for learning about the quality of our work...and what works or doesn't work in terms of fostering the desired outcomes.

Let me use an example from my personal life to explain...

My partner and I recently had a baby. As the tradition dictates, I made an announcement when the baby was born (by email). Can you imagine the uproar if I had simply reported the quantitative element? "A baby has arrived". I'm sure I would have been flooded with demands for more qualitative information. What gender? What birth weight? How is the mother? Name? Who does he/she look like?

And so it is with project outputs. There is an 'Output Type'...the simple description of what has been delivered ('a baby'). Then there are the 'Output Qualities'...all of the intersting information that ellaborates the quantitative fact.

A real example from a recent Oxfam project...

Output Type: 'Disaster Commitee Training Session Facilitated"

Output Qualities: Date, location, number of participants (male/female), facilitator, topics presented, particpant feedback...etc.

By capturing both the output type and the output qualities, we can build up a rich understanding of what is actuallying being done in the project.

Incidentally...Laria Maree Crawford was born on 9th May 11:17AM

Weight: 3.9kg (8.6lb)

Length: 52 cm

Birth method: Caesarian

Status: Mother and baby doing well