For the past month I've been compiling the impact analaysis data for PACAP (wwww.pacap.org.ph). It has proven to be an interesting and challenging assigment to assimilate data from a diverse protfolio of projects, and to make 'sense' of the impact of the program. It has occured to me that the concept of 'impact' can be interpretted in terms of three dimesions: * Magnitude: the scale of the changes effected. *Timeframe: the amount of time that an intervention may take to realise the intended changes. * Breadth: the extent to which benefits are experienced by a narrow or broad base of beneficiaries.

A key element in the success or failure of surveys is the way that questions are worded. De Vaus (2002) provides a handy checklist that can help mitigate poorly pitched questions... The following checklist to guide the wording of questions may be useful : • Is the language simple? • Can the question be shortened? • Is the question double-barrelled? • Is the question leading? • Is the question negative? • Is the respondent likely to have the necessary knowledge? • Will the words have the same meaning to everyone? • Is there a prestige bias? • Is the question ambiguous? • Is the question too precise? • Is the frame of reference for the question sufficiently clear? • Does the question artificially create opinions? • Is personal or impersonal wording preferable? • Is the question wording unnecessarily detailed or objectionable? • Does the question have dangling alternatives? • Does the question contain gratuitous qualifiers? • Is ...

For a long time I've been using the phrase 'theory of change' to express the idea that a project is essentially a social experiment, and that M&E is about testing the hypotheses implicit in the social experiment.  Recently I was challenged to succinctly ellaborate what I thought embodied the 'theory of change' approach.  The following points provide an overivew of the concept, and the practical and philosophical elements, as I see them: THEORY OF CHANGE Broad Concept International aid projects exist to create social change.  Any project implicitly aligns with a ‘theory’ about how desirable social change might be achieved—a ‘theory of change’.  To bring about social change, human actors interact within a social system through time. A role of project design is to articulate the ‘theory’—the temporal sequence of relationships, and how these are expected to influence desirable change. A role of M&E is to test ...

Is there a standard basis against which any aid project can be judged?   It is a truism that any individual project takes place within a unique context in time and space, and so if the question is about a definitive or standard set of ‘indicators’ that can be applied across multiple projects, then my experience suggests that this is problematic.  However, if the question is more about a conceptual framework through which the performance of individual projects can be assessed, then I think the answer is ‘yes’.   The DAC has defined five criteria for evaluating aid projects: relevance, efficiency, effectiveness, impact, sustainability.  Definitions for these terms are available at http://www.oecd.org/dataoecd/29/21/2754804.pdf.  These criteria are probably the most widely adopted in aid evaluation, and while I frequently apply them, I have personally been influenced by Soft Systems Methodology (SSM).  The SSM literature argues that the performance of any ...

I recently had a conversation with a senior bilateral aid donor official that confirmed a growing feeling I've had that the 'M&E bubble' may burst sometime soon.  In recent years the field of 'M&E' has escalated from a peripheral area to a highly sought-after discipline.  Demand for greater accountability and evidence of aid effectiveness has been a driving force for more, and better M&E. But I've begun getting a sense that the pendulum may be reaching the end of its arc.  The recent conversation with the bilateral donor official mentioned above indicated that there was a growing frustration with the inadequacy of M&E information to support its own fundamental tenet...to inform judgements about aid project performance.  The risk then is that the whole concept of M&E is rejected outright, rather than discriminaing between good and bad quality M&E (aka 'throwing the baby out with the bathwater'!). There are at least two reasons why the M&E bubble may burst... ...

A surprisingly common mistake that is made with the analysis and interpretation of M&E data is to try to treat categorical or ordinal data as though it is quantitative data (i.e. interval or ratio data).  This is simply wrong.  It is bad numeracy! First some background... Categorical data (also called 'nominal data') describes an 'exclusive category'.  For example, consider a survey question: What pets do you own: a) dog; b) cat; c) goldfish; d) none These are categorical responses.  You either do or do not own a cat.  Ordinal data is the same as categorical data, except that the order of the responses matters.  Consider a survey question: How do you feel today: a) lousy; b) ok; c) great; d) never better These are ordinal responses because most people would agree that they represent a progression of feelings.  BUT, the difference between the states is debatable.  My 'gap' between 'lousy' and 'ok' may be different to your gap between 'lousy' and 'ok ...

Risk management involves 'identifying risks' (i.e. detecting an event of significance), and then 'mitigating the risk' (i.e. responding appropriately). Risk management has increasingly become recognised as an integral part of good practice M&E.  But in the international aid industry we still have a lot to learn about risk management.  The realisation of risks can be seen as a 'failure', but it represents a clear opportunity to learn.  One of my favourite quotes is from Gharajedaghi, J. (1999): "Learning results from being surprised: detecting a mismatch between what was expected to happen and what actually did happen.  If one understands why the mismatch occurred (diagnosis) and is able to do things in a way that avoids a mismatch in the future (prescription), one has learned." One of the problems I frequently observe with risk management in the aid industry is that it is too blunt an instrument to be of much value.  Rarely do we apply a 'typology' of risks to make sense of the ...

Recently Rick Davies eloquently wrote in his blog ('Rick on the Road': http://mandenews.blogspot.com/) about different ways that people articulate program logic in logframes. This is an issue that I've also battled, and have written about elsewhere (http://www.aid-it.com/Portals/0/Documents/070105_Aristotle%20and%20Plato%20at%20it%20again.pdf).  Many users of logframes seem to lose sight of the fact that the 'vertical logic' is supposed to describe the temporal sequence of change.  That is, how a particular project is anticipated to contribute to social change through time.  Instead, some folk use the vertical logic to systematically disaggregate the problem...in what Rick succinctly decribes as a "hirerearchy of inclusion".  An analogy of this approach to program logic is: * the Goal = a wall * the Purpose = the bricks that make up the wall * the Outputs = the sand and cement that make up the bricks * the Acivities = the molecules that makes up the grains of sand and cement etc... ...

Yes, yes...I know I haven't been a good boy in the blogging department!  And I've promised Santa that I'll be better this year.  No excuse really except for the familiar cries of too busy, too much international travel, a new baby... Anyway, enough with excuses. An issue that has come up several times recently is the role of information systems in support of M&E; specifically, the relative merit of technology-based systems. As one of the founders of Aid-IT Solutions, I obviously have a view that IT systems can help to improve M&E outcomes.  But it is clear that IT is no 'magic bullet'.The world learned a very harsh lesson about being unrealistic in this regard during the Dot.Com mania of the new millennium! A technology-supported M&E system does not change the fundamental problem facing M&E practitioners...that measuring amorphous social change and atributing this to particular interventions is difficult.  Computers can't help with this.  It is ludicrous to believe ...