I occasionally come across debate about how to distinguish 'monitoring' (M) from 'evaluation' (E).  I have to admit that I have also engaged in this debate in the past, but in the end I've found most defintions unhelpful for practical purposes.  As a result I tend to side-step the issue by avoiding a distinction and simply using the combined label 'M&E' rather than distinguishing 'M' from 'E'.  Whichever way someone chooses to define 'M' v 'E', for practical purposes, both processes end up involving the identifcation --> capture --> analysis --> dissemination --> utilisation of information for accountability and for learning.

Most people attempt to differentiate 'M' from 'E' in one of four ways:

• The nature/focus of the data collected?
• How often it happens?
• Who collects the data?
  Who uses the information?

The raison d'etre of M&E is to enable accountability ('to prove') and learning ('to improve').  This can only be done if M&E data is used...in other words, it is supplied to interested stakeholders in a accurate, relevant and timely form.

But who are these "interested stakeholders"?

It has occured to me that M&E information is required by stakeholders in four 'directions':

• Downwards: M&E information commonly follows a uni-directional flow ‘upwards’ from beneficiaries to donors.  But it is essential that information be fed back ‘down’ to contractors, partners and counterparts.  Where feedback does not occur, it inevitably limits opportunities for debate and learning.  Further, it ignores the reality that all stakeholders depend on timely, accurate and relevant information to perform. 

I've been doing work for two different clients who would like to implement a standardised or generic evaluation method or instrument across portfolios of projects implemented by unrelated NGOs.  Both clients are eager to have a consistent way to conduct comparative analysis of individual projects, and sets of projects.  They want to know if individual projects have achieved what was anticipated; and if the portfolio of projects as a whole has fostered desireable outcomes.

One client is predominantly focussed on gathering (to the extent possible) quantitative performance data; the other is predominantly focussed on qualitative perceptions gathered from a range of stakeholders.  But both assume that it is possible to obtain ‘scalable’ performance information.  In other words, information that will be meaningful at any scale: individuals, communities, projects and programs.  The other complication is that this ...

According to the State of Texas, Office of the State Auditor, (A Guide to Measure the Performance of Information Systems, April 1991) the following items are critical to successful implementation of a performance management system (similar issues apply to implementing a M&E system):

• Management support and involvement from all levels in the organization to counteract resistance to change associated with introduction of new policies.
• Appropriate measures that include defining the goals and objectives, identifying the measures to be used, evaluating the costs and benefits, and then implementing the cost-effective measures
• Start small and measure the processes, not the people.  Fewer measures mean less initial cost; measures can be added.  Measure how things are done, and the result, not the people.
• Provide feedback by using the results and reporting where improvements are ...

M&E is not an end in itself.  It should serve a means...a means to learning and being accountable. 

So, unless M&E processes/tools render information that is actually used by someone to further these ends, they are an unethical waste of resources.

So what does 'utilisable' M&E information actually mean?

I suspect that there are at least fourcontributing factors:

• Relevant: does the information fundamentally meet a need?  Is it perceived as important/valuable by someone (an information 'client')?
• Accurate: can the information be trusted? 
• Timely: is the information delivered (to the information 'client') within a timeframe that permits action to be taken? 
• Accessible: is the information provid ...

While the concept of organisational learning has been popularised within the international aid industry in recent years, the persistent challenge is to ground the concept.  In practical terms, what is learning?  What actual mechanisms can be employed to promote learning in a structured way?  How can learning be moved from a tacit process within individuals to a shared process among team members? 

One practical understanding of learning has been proposed by Gharajedaghi:

“Learning results from being surprised: detecting a mismatch between what was expected to happen and what actually did happen.  If one understands why the mismatch occurred (diagnosis) and is able to do things in a way that avoids a mismatch in the future (prescription), one has learned.”

The above quotation contains three practical mechanisms that an aid organisation can employ to operationalise the concept o ...

It is sometimes difficult to decide what data and/or methods should be used for M&E.  There seems to be so many perspectives and issues to consider.  I've found the following 4 points helpful to think through the range of issues, and as the basis seeking consensus among stakeholders:

• Reliability: Do the questions have the same meaning to the same respondents at different times?
• Validity: Do the answers tell us what we want to know? Do the questions have the same meaning to different respondents at the same time?
• Scalability: Can comparative analysis of the findings be carried out at all the required levels of dis-aggregation?
• Affordability: Can the method be implemented for reasonable cost?

Every now and then I observe robust discussion about the merit of quantiative vs qualitative methods for M&E.  Some people seem to feel passionately about one or the other.  The pragmatic reality that both approaches have merit in different contexts, and can complement eachother, is recognised by Patton (1997, p 266), who states that the debate about evaluation paradigms “has run out of intellectual steam".

My own experience suggests that the distinction between 'quantitative' and 'qualitative' is, in practice, a bit academic.  I will illustrate by way of example:

a)       A focus group question (i.e. a ‘qualitative method’) “how many households in this village?” would yield a ‘quantitative’ answer
b)       A survey question (i.e ...

Over the past few weeks I've been writing up an evaluation report.  Part of the evaluation involved a review of the M&E arranagements employed by four prominent international NGOs.  It was encouraging to find that all four agencies were committed (at least intellectually) to learning and improving aid effectiveness.  And it was encouraging that all had taken positive steps towards implementing M&E processes.  However, it was also disappointing to find that the efforts to implement an 'M&E system' seemed to post hoc.  They seemed to lack an overall 'information architecture' to give the M&E processes some coherence and purpose.

I may have developed an inadequate appreciation for the situation, but on reflection it seems that the focus of M&E remains very much on 'data' rathern than on 'knowledge'. 

As noted by Henri Poincaré, the great French ...

I've been periodically engaged by AusAID since 2005, working on the development of a quality framework for M&E.  In other words, a set of criteria to articulate what AusAID considers to be the basis for good quality M&E arrangements.  This framework and the associated good practice guide are now at final draft status within AusAID.

One of the 6 broad criteria for good quality M&E arrangements (and arguably the most challenging) requires that M&E practitioners ensure that M&E arrangements are both comprehensive and efficient.

On the surface, this seems perfectly reasonable.  But as with many things in life, a dig slightly below the surface reveals a surprising amount of complexity.