LEARNING

PAPER

Chiefs from different ethnic groups in Wau, South Sudan, work differently – together – to promote peace. Photo: M Perkins

DOING THINGS DIFFERENTLY

Rethinking monitoring and evaluation to understand change

Over the past four years, Saferworld has put

Key messages
in place a way of monitoring, evaluating and
Our approach is centred on collecting and
learning from our work focused on behaviour analysing evidence together about what others do
and relationship change. This paper outlines differently – and determining how far that is
the process we have gone through to adapt, because of our work. The approach is different
embed, and embrace an approach inspired because it is flexible and straightforward enough to
by Outcome Mapping1 and Outcome be used in complex, rapidly-changing contexts. It
empowers front-line staff, communities and
Harvesting (OH)2. We hope it shows other partners to ‘monitor’ what matters to them.
interested parties how we have transformed
Key benefits:
the way we understand, monitor and collect
evidence of change in our work.  It is simple, but promotes complex discussion
and analysis.
“This is a beautiful, exemplary case of an organisation  It allows conflict- and gender-sensitivity to be
taking Outcome Harvesting, and customising it to built into programmes; it promotes analysis of
specific needs and context. The danger is that the and adaptation to context.
essence of OH can easily get ‘lost in translation’, but
Saferworld has retained the essential principles while  Bringing front-line staff and partners into wider
making the approach their own.” conversations with others substantially
increases cross-organisational learning.
– Ricardo Wilson-Grau Learning points:
 Like most change processes, using the
1
See Outcome Mapping http://www.outcomemapping.ca/ approach requires leadership, guidance, and
2
See Outcome Harvesting practice – one-off trainings don’t work.
http://www.betterevaluation.org/plan/approach/outcome_harvesting
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 2 of 12

Dedication
In 2012 Saferworld appointed its first MEL coordinator based in a
country team. Ramesh Nidhi Bista (pictured, right) had spent many,
many years in Nepal working on M&E, and had tried pretty much
everything to support partners in dealing with the language of
goals, objectives, outputs, outcomes, indicators, targets, and
milestones that donors and international NGOs are so fond of.
Not much seemed to work to help them to understand, measure
and report on changes. However, he had hit on this definition –
“an outcome is something that others do differently” – and found
that it made sense. The minute he said it to me, I thought – that’s just what I’ve been looking for. Ramesh,
dear colleague and friend to the Nepal programme, died suddenly in November 2013. This paper is
dedicated to him.
Madeline Church

behaviour- and relationship-change, by and between

Introduction certain ‘actor categories’4. It had to be straightforward
enough for communities, partners, and front-line staff
“An outcome is something that others do differently (some with English as a second, or even third
(individuals, groups, institutions, parts of institutions)” language) to use to learn about their work, build on
and improve it, and be able to communicate about it
– Ramesh Nidhi Bista
effectively to others. We know from experience that
In 2011, Saferworld had one Planning and Evaluation the worst type of instrument for tracking the effect of
Adviser. She was the sole staff member in a rather our work in complex and changing environments is a
grandly titled Organisational Development Unit complicated set of things to measure that are hard to
(ODU), which was tasked with driving a big interpret. What we were looking for was something
institutional push towards better Monitoring, that focused on demonstrable change, and that was
Evaluation and Learning (MEL) approaches. Our new simple enough for all staff to work with.
Strategic Plan provided the impetus and leadership
that the organisation needed to systematise its “MEL was an area I had difficulty to follow and have a
accountability and learning, so we could know picture in my head of all the statistical data, log frame,
whether we were meeting our ambitious plans. And a framework and so on. The outcome harvesting
new Programme Partnership Agreement (PPA) with approach makes MEL easier for people like me to
the Department for International Development (DFID) understand and plan activities better. I am glad that
(over three years) provided strategic funds to support we have this process beside the data gathering for
these new commitments. the log frame.”
– Khirad Kargasov, Area Programme
Saferworld works in complex, conflict-ridden
Manager, South Sudan team
environments, with fast-changing dynamics,
alongside communities that have historically been
abandoned, marginalised, or ignored by their local
authorities and security providers, and feel a profound
Rethinking ‘M&E’
sense of insecurity. The organisation prides itself on In determining how to move forward to meet our MEL
linking this community-level work (primarily commitments, we looked at our current practice and
Community Security3) with an ‘evidence-gathering’ tried to understand what the organisation really meant
agenda, to show that things can change for these when it talked about ‘M&E’ (as it did then). In
communities if they are empowered to lead particular we looked at whether this was understood
constructively on what’s important for them. It also exclusively as an ‘enforcement’ or ‘accountability’
helps to demonstrate positive change to authorities exercise or whether we were learning from what we
and policy-makers. We were, therefore, seeking a did in a structured way that went beyond reporting to
MEL approach that would reflect this organisational donors and commissioning external reviews or
agenda and concentrate on meaningful evidence, by evaluations.
listening to and valuing communities’ own definitions
We noticed that in the main, staff understood ‘M&E’ to
of what constitutes success.
mean ‘reporting to donors against indicators in the
log-frame’. We found that most staff and partners
We wanted the approach to be appropriately focused
didn’t really understand what was meant by an
on what Saferworld actually does – influence
indicator, often because different donors used the

3 4
See our Community Security Handbook Saferworld works to influence four actor categories: individuals and
http://www.saferworld.org.uk/resources/view-resource/806-community- communities; civil society; authorities; external actors. See our Change
security-handbook Model, http//saferworld.org.uk/images/saferworld-change-model.png
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 3 of 12

term differently, or because indicators had been ‘outcomes’ are, for instance, all translated as
chosen or written by others. Many early hours of the ‘results’ in Arabic and Russian, showing just how
ODU were spent clarifying the difference between an difficult it is to differentiate between them. It was
indicator, milestone, target, and ‘objectively verifiable more useful for us to distinguish between what we
indicators of achievement’. and our partners do (activities) and what others
do differently as a result of these activities
Some of the elements we found least helpful about
(outcome).
the ‘status quo’ were:
 the lack of capacity in MEL at the right level in
 the focus on activity reporting. This meant that
Saferworld – i.e near to the partners, to the
many staff were largely reporting on what they
communities, and to the national offices.
and their partners had spent their time doing. We
had plenty of data on the ‘what?’, but not enough In sum, we needed an appropriate MEL approach for
on the ‘so what?’, i.e what difference did it make? our change agenda, which allowed for flexibility, a
focus on behaviour and relationship change, and the
“[Outcome monitoring] has helped us focus more on
ability to adapt to quickly changing environments with
the changes we are trying to achieve over activities.”
many political dynamics at work. We needed
– Shelagh Daley, UK Advocacy Coordinator something that would build in conflict- and gender-
sensitivity and support adaptive management.
 the static and linear nature of log-frames,
which doesn’t fit with Saferworld’s work on And in line with the projected growth of Saferworld –
complex social change in shifting and evolving working at a greater level of scale and intensity – we
conflict contexts. While log-frames were intended had a vision of a devolved organisation, with more
to be ‘tools for thinking and strategising’ they had country offices, country managers, and increasing
become rigid project implementation templates, numbers of national staff. That needed national MEL
which failed to take into account the systemic advisers in teams working closer to the action.
nature of many conflicts. Almost all our funders
had log-frames attached to agreements (the What is Outcome Harvesting?
Dutch Ministry of Foreign Affairs being a notable
exception), which meant that ‘delivering on the We came to Outcome Harvesting through the
log-frame’ and ‘checking off the indicators’ had Outcome Mapping community.5 Outcome Mapping
become the central purpose of monitoring. was designed to rethink the way in which
behaviour-change work is designed, planned and
 the largely unhelpful nature of quantitative described. The focus of Outcome Mapping is on
measures, given that so much of our work hoped-for changes in behaviour any programme is
focuses on behaviour and relationship change, in working to achieve through its influence, using
a particular context. These quantitative indicators outcome challenge statements to describe what
tended to be default, as counting is considered to these are.
be somehow more revealing and/or easier,
despite the wealth of alternative qualitative Outcome Harvesting is an evaluation approach
approaches that would make more appropriate inspired by Outcome Mapping and Utilization-
bedfellows. On the whole, counting often can’t Focused Evaluation. Unlike other evaluation
answer the important questions that Saferworld, methods it doesn’t start with predetermined
its partners, and the communities we work with, outcomes, and measure progress towards them,
want to ask. Lack of reliable data is common in but rather collects evidence of what has been
many conflict contexts, so long-term changes or achieved in the programme or project area, and
wider effects are often hard to understand without works backwards to determine whether and how
huge investment in expensive research. Teams the project or intervention contributed to the
were either collecting lots of data that were of change.
questionable use, often because that data was At its most elegant, it is really simple. It asks a few
easy to collect, or not collecting any data at all. core questions for data-gathering about the
Few were asking themselves what constituted change in behaviour:
reliable evidence of change.
 Who did what, when and where?
 confusing language used by the sector, and
by funders. This included the mixed use of ‘goal’,  How significant is this change?
‘purpose’, ‘objective’, ‘outcome’, ‘indicators’,  What contribution was made by the
‘targets’, ‘results’, ‘outputs’, and ‘evidence’, programme to this change?
combined with a new fascination with ‘theory of
change’. This was a particular challenge as
Saferworld employs largely local staff in
programmes, and/or works primarily in local
partnerships, with English as a second or third
5
language in all instances. ‘Results’, ‘outputs’ and for a full description of the approach see
http://www.betterevaluation.org/plan/approach/outcome_harvesting
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 4 of 12

In particular it works well when you want to focus

on outcomes, rather than activities, and it is
suitable for evaluating in complex programming
contexts ‘where relations of cause and effect are
not fully understood. Conventional monitoring and
evaluation aimed at determining results compares
planned outcomes with what is actually achieved.
In complex environments, however, objectives and
the paths to achieve them are largely
unpredictable and predefined objectives and
theories of change must be modified over time to
respond to changes in the context’ (emphasis
added)6.
Outcome harvesting is flexible enough to be used in
complex, rapidly-changing contexts. Political violence
in Bangladesh in the run up to the 2014 elections
Adapting and using Outcome changed action group priorities in some project sites.
Photo: Saferworld/Altaf Hossain
Harvesting
Given Saferworld’s primary focus on behaviour and
relationship change in complex contexts – by and “The approach helps us to tell a story of how our
between social actors, working through on-the-ground actions have prompted changes in behaviour and
programming, policy and advocacy – Outcome improvements in relationships.”
Harvesting offers a way for us to get to the heart of
– Luke Errington-Barnes, Funding Coordinator
what matters. It particularly appeals as a
straightforward and easy-to-understand way of
shifting attention away from activities (what did we
“This approach makes it possible to more clearly
do?), onto the practical, observable shifts in actions
explain our goals and to justify interventions and our
by the actors our programmes are seeking to
responses on specific local community security
influence (what did they do differently?).
needs. For beneficiaries (Local Crime Prevention
Centres – LCPC) it is easier to understand the
relationship between analysis they conduct and to
“It helps me determine whether an activity
select effective measures that will have an impact on
implemented is relevant to the context or not.”
the situation. We introduced outcome statements in
– Peter Machar, Project Coordinator, the LCPCs plans, and this helps them to understand
Wau, South Sudan the direction and nature of response that could result
effectively in a change in the situation.”
– Stefan Stoyanov, Project Manager, Central Asia
Although Outcome Harvesting is largely practiced as
an evaluation approach, we wanted to see if it would At the same time we wanted to focus our attention on
help us in our regular monitoring. We wanted teams evidence, and to find a way to present our claims in a
to be able to discuss ‘outcomes’ with their partners way that could be tested and verified by other
and communities regularly, document them, review external reviewers or evaluators. Most external
them, understand them, and build on them. Our evaluations Saferworld had commissioned in the past
Community Security theory of change is founded on relied on paid consultants reading strategies, plans,
using joint action-planning to create trust, through project proposals and donor reports, and then being
which change occurs to relationships, and expected to collect primary evidence. We wanted to
behaviours, which in turn creates more trust, be able to provide a review or evaluation team with a
confidence, agency, ownership, and transparency. At package of verifiable claims and evidence that they
the same time, common action creates tangible could then put to the test by interviewing key
changes. This work is highly reliant on skilled, informants and focus groups, rather than having to do
knowledgeable staff and partners working with key all the work uncovering results and evidence
stakeholders to create the conditions for this to themselves.
flourish. To do so, they need to be able to identify and
So, as part of our new framework for MEL, we
codify the changes they are seeing in a manner
constructed a mixed approach around a central core
straightforward enough to stimulate learning and
focus on what others do differently as a result of our
adaptation.
work. It is – and is meant to be – pretty
straightforward.

6
ibid
 You’re trying to record something Who, or which institution Follow with recording
that another person did differently. or group, is this outcome exactly what they did. Use
about? active verbs to describe this.
Start with naming the person, Dr Manga, The Minister ‘He instructed’, ‘she provided’.
institution or group. Use as much of Interior for Zang, from Limit yourself to describing
detail as possible.  the Union of People party. the action.

What did he do differently?

The Minister of Interior for Zang Where and when did this happen? He instructed the Chief of
instructed the Chief of Police for Maple Be as specific as possible about Police for Maple County to
County to introduce a monthly meeting where and when. The date, or time introduce a monthly meeting
with communities across Maple County. period is really important, so don’t with communities across
Date: July 2014 miss this out. Your outcome should Maple County.
Place: Mobo Capital City. now look like this:

Highlight why this This change is significant because this So, to decide its
change is worth Minister has consistently refused to accept significance, ask
noting, important or that the communities have anything your team some
meaningful to contribute (it is a change in his questions:
significant.
consistent practice).
• Is this the first time?
You are seeking to
The situation in Maple County is very bad • Does it link into the
record the connection
when it comes to police-community relations. conflict analysis you
between the context, and
The police are often drunk and abusive. The have?
the change. For instance, if this
clan relations mean that the communities
is the first time the Minister has taken • Is it big? Small?
don’t want to complain, because the
action it is probably significant. If he • Does it seem to be
Minister of Interior comes from there, and
has given this instruction many times a systemic change?
his reputation could be at risk if he lets the
before, and nothing has changed, then
communities say what is happening. • Does it seem to be
it is not. If the Minister is powerful, and
difficult to influence, then it might be a policy change?
The communities have been saying for many
significant. If the Minister is not the one • Are there fruits of
months now that the local Chief of Police
with the real power in the situation, then sustainability in
doesn’t listen to their concerns, but the
it probably isn’t. there?
Minister has refused to talk to the partner.

What did Saferworld, Keep an ‘evidence box’, physical or

our partners and/or the “Saferworld made all electronic, in which you can store
the calls to encourage emails, notes, phonecalls,
communities do to
the Minister to meet with etc, which demonstrate
contribute to this change?
the partner. The partner the changes that you
trained the communities are referring to. These
Here you can start to record
in advocacy. The can be added to the
the activities or resources
communities wrote the outcome forms as pieces
that the team, the partners
paper that the Minister of evidence to substantiate
and/or the communities put into
read to convince him”. claims, by embedding them, or
making the change possible.
uploading them electronically.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 6 of 12

How does it get used in practice?

What this means in practice is that all teams are now
expected to do the following as a matter of routine:
 keep records of who has done what differently as a
result of our work (noting where they did it, and
when). These are kept in notebooks, in evaluations
of community action plans, in partner reports, in
Word and on-line forms.
 hold review conversations with their partners at
least twice a year, to discuss these and other
outcomes.
A ‘smartie map’ maps the significance and
 note down why they think these changes are contribution of changes seen in one project in South
significant, linked to the conflict and context Sudan.
analyses that form part of their strategies and
which they update. evaluation data for community security action
 bring all this information together at least twice a plans.
year in a face-to-face workshop setting with their What we have found is that by attending to the detail
programme/policy/advocacy colleagues. In this of outcomes per social actor, at different levels of a
they discuss the meaning of the outcomes, and social system, we can create more comprehensive
they record the significance of the outcomes in theories of change for our work. We piloted this with
relation to context. our Central Asia evaluation in 2012.
 rank the outcomes, as a team, according to levels “The team and I went back over all our activity reports
of significance and the contribution made by (partner reports, planning and review meeting notes,
Saferworld and partners to that outcome (low, staff activity reports) and filled in ‘outcome forms’,
medium or high). This is not a personal opinion, but which stated and analysed any changes in behaviour
a team agreement, and is a tool to stimulate and relationships that we had noticed, for each
discussion. Each team creates a ‘smartie map’ community that we worked with and for each
which displays this information visually (see picture community security cycle that we had facilitated with
top right). them. We also attached all the evidence that we had
 upload any outcomes, positive or negative, into collected. This was hugely enlightening for the
Saferworld’s intranet system, Karacel, where they programme (and also motivating!) in terms of us all
can be seen by everyone in the organisation. All seeing how each community had learnt to work on
outcomes are displayed on relevant team pages, more sensitive issues and how difficult relationships
on the home page, and in the ‘results’ section of had changed over time. This was a really useful
the system. They can also be exported as reports. process to go through and not only resulted in a very
useful external evaluation that asked the right
 include their ‘outcome harvest’ documentation in questions and gave us helpful recommendations, but
their twice-yearly internal reports (again done also helped us to articulate our programme aims far
through our intranet system). more clearly to partners and everyone else we were
 use this material for their donor reports, and case trying to influence.”
studies, and for their part in our PPA reporting. – Maija Paasiaro, then Regional Conflict /Security
Staff have access to guidance materials, agendas Adviser
and session plans for triannual or biannual reviews, Since then we have created a comprehensive MEL
on-line and off-line outcome forms, and every other framework for this and our South Sudan programmes,
team’s outcomes via Karacel. based on mapping desired outcomes per actor,
When a mid-term review or evaluation is being theories of change, and a detailed plan for data
planned by a team, and will involve an external collection. We have provided outcomes harvested
person, we encourage the team to undertake a through our monitoring to external consultants
documentation exercise to ensure that we have a conducting reviews of our Central Asia, Bangladesh,
dataset to provide for them to analyse and test. On South Sudan, Yemen, and Sudan programmes, for
top of normal documentation this can include: them to test and verify.
 completing an updated outcome harvesting “The... big thing for me was the M&E framework. It so
exercise with teams, partners and/or communities. clearly articulates all the intangible things we need to
measure”
 completing extended ‘outcome forms’ which have
more detail in them, and are used to gather – Katie Morris, Europe and Central Asia Team
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 7 of 12

Ayak Mou Kuot, Deputy Chairperson of a Learning from pioneers in South

community security working group in Warrap state, Sudan
South Sudan, talks to group members about safety
concerns and changes they are seeing in their Our South Sudan programme has grown over the
community. Photo: Marcus Perkins/Saferworld last four years from work in three locations in two
states, to ten locations in eight states. The team
has grown from a staff group of 6 in 2011, to over
However, as Saferworld does all its programming 40 staff, with eight project coordinators based in
work with and through partners, any approach we use the field.
has to be something we can ask our partners to sign
up to as well. This means another layer of people, The first three locations have been running for two
organisations and understanding to develop when years longer than the most recently added,
trying to systematise and institutionalise an approach providing an interesting comparison between the
to MEL that links up across communities, partners two. This was not intentionally set up as a ‘phased
and staff. approach’ but has turned out that way, as our
funder has seen the fruits of the work.
“It is a participatory tool that enhances quick
identification of changes with less effort – no All staff have now come together three times to
scratching the head when generating outcomes any ‘harvest’ outcomes. They spend up to two days
more. It generates adequate information on complex looking at their conflict contexts, listing and refining
behaviour change and relationship among many their statements about what others have done
actors and serves as an analytical tool – by differently, discussing the significance of these
disaggregating outcomes per actors, and outcomes changes, highlighting the areas that have potential
per level of influence (local, sub national, national, but are not yet outcomes, and those that have
regional and international). As a process one cannot hints, or foundations of sustainability.
exhaust harvesting outcomes because staff become “What was really noticeable this time was how the
sensitive to any emerging, growing and mature in second ‘batch’ had really learned something about
different actors during the project implementation, it how to make bigger changes from the pioneers.
triggers cross-learning among the team and across While they still had to put in the early leg-work,
projects and it can demonstrate whether Theory of developing trust and relationships with local
Change (ToC) is working at early stages of the project communities, they all had a sense of what might
implementation. be possible once the foundations were laid,
– Thomas Kimathi Nyagah, MEL Officer, Kenya Team because they could see how things had changed
over the previous two years in the first states.
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 8 of 12

They had understood how the way of working don’t understand that point that M&E suffers. I think
significantly affects the outcomes, and had that comes from constant engagement and
managed to build that in from the start.” discussions, and the use of tools that show the
value.”
– Madeline Church, Head of
Organisational Development – Katie Morris, Europe and Central Asia Team
“‘Due to the follow up on actions/responses made  It enables staff to become more expert at spotting
by individuals or institutions like police for change, and finding ways to increase potential.
example, representatives/leaders of those Our South Sudan team, for example, have started
institutions often think twice before pledging keeping notes about things that are not yet quite
responses/actions for a community outcomes, but could be, a kind of ‘Outcome
security/safety’s concerns as our outcome Watch’, which is a way of keeping an eye on things
monitoring and harvesting mechanism requires us that have potential. The Nepal team discuss
to make a follow on those pledges and changes changes they are seeing at their weekly staff
registered by programmes.” meeting, to ensure the whole team understands
what the programme is seeking to achieve.
– Peter Machar, Project Coordinator,
South Sudan “As a manager and mentor I believe that outcome
harvesting has motivated and empowered my team
In such a conflict-affected environment what seem
members to more successfully implement their
like positive changes can easily generate
programmes and more effectively engage with
additional unforeseen negative effects. The
partners and stakeholders at all levels of our
outcome monitoring approach allows Saferworld to
intervention.”
surface these early, and discuss how to add in any
further strategies to mitigate them, or adapt the – Tamara Duffey-Janser, Head of Programme,
work – building in an important conflict sensitivity Great Lakes and Sudans
lens.
 The approach is appropriate for our model of
“What does it add to our cross-organisational ‘community security’, which prioritises community
learning? – “We have a tendency to get sucked empowerment, and the need for citizens to take
into the places where we work, so when we come more control of their own agenda through active
together and use this as a process to talk about involvement.
change and about how we achieve those changes,
“The outcome harvesting approach forces us to focus
..to do that in a collaborative way then inspires
on the connections between what we are doing and
other people to think about their work differently,
how things have changed for communities. This
and think about different approaches they might
means that we focus on what truly matters, which is
integrate into their work… it’s just been a really
how our work meets the needs of communities.”
useful collaborative and facilitative tool that really
engages people in those really important learning – Deepti Sastry, Impact and Accountability Adviser
discussions.”
“I like it because it is very simple, it focuses on the
– Ariana Martini, Grant Manager community, and the impact of the project on the
community, while the approach that I have used
before it only focused on the numbers of trainings
conducted, the number of participants attending, but
The benefits our approach is really very good, cause it focuses on
A number of benefits from this outcome monitoring the changes that is taking place in the lives of the
and harvesting approach have become clear over the individual and the communities where we are
last few years. working, that is really the aim of the project, to work
with the local authorities, to reduce the incidences of
 It reinforces that regular MEL practice is a job that insecurity happening in the community.”
everyone needs to be involved in. It shows that it
can contribute to improved programming, and that – Phoebe Egbalia Manza, Project Coordinator,
all staff have a role to play in collecting and South Sudan Team
analysing data and evidence that shows changes.  The straightforward approach, with in-built time for
It motivates staff, some of whom have actually reflection and discussion, is critical for conflict- and
been heard to say that they really like M&E, and gender-sensitivity. We use our conflict analyses to
finally see how it relates to their programme determine the significance of the outcomes. This
development. allows for learning and adaptation by frontline staff.
“It made me realise the absolute importance of M&E, “I think one of the things that I really like about it
and how it is really a key programmatic tool, rather compared to other systems is that it’s very easy to
than something that is added on for bureaucracy or use, it’s not technical like other systems.”
for donors more broadly. And it’s only when people
– Bonita Ayuko, Project Coordinator, Kenya Team
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 9 of 12

Identifying policy success

 In highly volatile environments, it enables the For our Advocacy and Policy work, the approach
teams to take corrective action and understand
has been much easier to incorporate. These
where perverse incentives or negative effects
teams regularly review their outcomes together,
might be occurring for some, from what look like
and keep an ‘evidence box’ that helps to
positive results for others. This reinforces the
substantiate their claims.
importance of being context-and gender- specific.
As an initial foray into systematising reporting on
 It challenges all staff and partners to think about
our advocacy and policy work we developed a
evidence and evaluation, and not just leave it to
matrix to track six types of evidence linking to five
‘evaluators’ to work out whether the programme is
stages of success. Our outcome harvesting
making a difference or not. The format of thinking
approach then allows us to see how we are
through the outcomes ensures that teams state
managing to move through these stages, from
clearly the claims we are making, and the evidence
improved credibility, to relevance, access to
we have, which can then be verified or tested by
decision-makers, increased support for our
outsiders.
messages, to policy change.
“Although the impact of such high-level advocacy
“I think it has definitely helped people to focus more can be difficult to measure, outcome monitoring
on actual changes rather than outputs. That helps to allows us to reflect on whether the advocacy
make us more ambitious as we’re more aware that approach we have adopted and the activities we
people just saying nice things about our work doesn’t have carried out have been relevant and effective.”
constitute an outcome. At the same time it can be a
– Kloe Tricot O'Farrell, EU Advocacy Officer
bit demoralising as it makes you realise it takes ages
sometimes for any outcomes to materialise,
especially when you’re aiming for policy change.”
– Hannah Wright, Gender, peace and security
Our learning
Adviser Over the past few years we have learned a lot about
how to implement the approach sustainably.

“It constantly reminds me to be on the lookout for  Clear, simple language does wonders to kick-start
changes in behaviour, content, patterns, etc. as meaningful conversations about change.
opposed to assuming that the amount of work I do “The definition of outcome here is very simple and
equates to success. It helps because having had to everyone is able to think about the outcomes of their
continually go through this process, I'm usually one of projects.”
the people in the room who has a positive answer at
hand when the question comes up ‘how do we know – Posha Raj Adhikari, MEL Coordinator,
what we're doing works?’ It's amazing how much that Nepal Team
sets us apart”  It takes a lot of repetition, and practice, rather than
– David Alpher, Washington Associate just ‘training’, to shift attention solely from activities,
so that thinking about change and evidence in this
 It provides useful documentation for our internal way becomes natural. Establishing a routine and
and external reporting, for identifying good material systems is really important – you can’t do this once
for case studies, cross organisational learning, and and expect it to ‘work’.
for external reviewers and evaluators. It is
invaluable material for our cross-organisational  Staff training is also needed. It’s useful if this
grant reporting, such as for strategic funding, which includes seeing the approach in action in other
requires an overview of the kinds of changes we’re teams and participating in the reviews those teams
seeing in very different contexts. hold.
 You need people in place who understand the
approach, and work with and support teams, at the
“Outcome harvesting not only helps teams to identify ground level, to make sure that the focus is right.
the changes they are seeing, but also provides the This is hard to do from a distant HQ or centre.
starting point for wider stories of success and learning Having country-level MEL coordinators or advisers
that we can share externally. If anything the challenge is essential – where we have them, the approach is
now is the sheer amount of material available – a very much more systematically applied.
different situation to four years ago!”
 Working through outcomes with partners, in
– Simon Moore, Head of Communications conversation with them, rather than expecting them
to complete reports in writing, produces much
better material. Partners often have a clear
understanding of what others have done
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 10 of 12

Saferworld and Bangladesh partner BRAC at a

joint advocacy workshop: understanding the
outcome harvesting process has taken time, but
BRAC is now using it within their strategic
planning. Photo: Shoeb Ahmed.

Partnership working in Bangladesh

Saferworld works with 16 community groups in five
districts in Bangladesh, with a very large partner,
BRAC. BRAC has its own systems, and our
challenge in this work was to influence the way in
which they conceptualised their M&E in relation to
this project. This involved engaging with and
training the BRAC project staff, their M&E
department, and the leaders of the Community
Empowerment Programme. Their monitoring
system focused almost exclusively on quantitative
data collection at the level of activities and outputs,
but didn’t really have a process for capturing
outcomes, discussing them, and deciding how to
build on them.
We worked to adapt reporting mechanisms, train
the project staff, conduct annual learning events,
and draw in the senior management to
understand, see the value and advocate for the
differently, but they don’t realise that that is approach. We worked intensely to develop
important. They have been encouraged in the past extended outcome monitoring for all the
to report mainly as an accountability exercise led communities, and build a dataset for each
by a need to ensure transparency about the money community action group that focused on the core
they’ve spent. This has led to an obvious link changes the communities themselves wished to
between activity and expenditure, with less of a see. We made all this available to our external
focus on outcomes. reviewer.

 Bringing front-line staff and partners into wider “It took at least one year cycle to understand the
conversations with others in the organisation whole thing, which was a very rigorous process.
substantially increases what we can learn from our We were in the field first, we sat down beside the
work. Saferworld’s focus on working at several partners to support them, and then they did it on
levels, and influencing multiple actors, means we their own. At the end of this year, through
need a way to bring evidence together from across practising, we identified that this is a really, really,
our programming, policy / advocacy and powerful tool, to show what can be documented,
communications work. Where we can, we what can be visible, and how people can at least
encourage our policy and advocacy staff to join understand how the change process is on-going.
programming outcome harvesting sessions, and The partner BRAC also marketed it, sharing it
vice versa. It requires all staff to think ‘MEL’, and to internally, and externally to the donor. And now
make time for analysing and processing their work they’re thinking about this in their strategic
– something that can be a challenge. planning, so they’ll replicate this. It’s not an easy
process, it needs time, and it needs both parties.
“I find it innovative. Many different MEL approaches But it’s an empowering tool.”
that I’ve used in the past tend to stay at the output
level, not really digging deeper into whether there are – Bibhash Chakraborty, Bangladesh
changes happening because of their work... the Programme Manager
approach is essentially a qualitative tool to measure
changes, and it also helps us as project coordinators We have also identified a number of internal and
in the field, to keep an eye on what we need to do in external factors that made the change in approach
order to motivate our stakeholders in realising the possible.
changes that we’d like them to embrace. I find this  The new strategic plan highlighted just how much
more enriching, as many other models tend to stay on our work is about influencing different sets of
outputs.” actors. This was a powerful catalyst to find a
– Galdino Joseph Sakondo, Project Coordinator, complementary approach in our MEL.
South Sudan Team
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 11 of 12

 PPA funding, and a commitment to

institutionalising a ‘framework’, meant that the
organisation set up an Organisational
Development Unit, and employed dedicated MEL
staff. This unit led on the development of Karacel
(the intranet), and ensured that it had in-built
potential to manage our MEL work and make data
more widely available across the organisation. The
ODU has also led on coordinating and facilitating
our Community Security Learning and Practice
Group.
 Cross-organisational grants allowed for
experimentation, and enabled learning groups to
run for long enough (four years). Behaving differently: the chief of the district police
 There was important buy-in and commitment from addresses the community's local security concerns
senior management to collect and use evidence to and the need for better police-public partnership in
support learning. Core funds were allocated to an event organised by a community security
support this learning among frontline staff. The working group in Kalika, Bardiya, Nepal. Photo:
Executive Director truly felt that this approach Saferworld/Anil Poudel
could have a transformative effect.
 A new communications team supported the focus Stories of success in Nepal
on plain English and the use of uncomplicated In Nepal, we have used the outcome harvesting
language. focus to help us develop stories of success that
communities themselves wish to communicate.
 Saferworld staff were open to trying new things.
The very simple focus on who, what, where, when,
significance and contribution allows for groups of
Looking to the future people to sit together and describe, in quite rich
detail, the nature of the change and the
The wider success of this approach into the future importance they attach to it.
also relies on donors seeing us as partners, and
This ‘formula’ is something that USAID, our Nepal
using the evidence we provide them with to see the
programme donor, is proposing now to learn from
quality of our work; with evaluators able to test it.
to improve the case studies that they ask all their
Funders also need to enable peacebuilding and partners to contribute (and is an ‘outcome’ for us!).
conflict prevention organisations to take a flexible,
“Oftentimes you can get boxed into the project and
adaptive approach to programming, based on
not have the whole team discuss what we’re really
demonstrable changes linked to objectives and
doing in Nepal, how we’re looking forward to
theories of change, rather than a predefined and
change, or bringing about change in the
linear activity, output, outcome framework. While the
communities, whereas this outcome harvesting
target-driven approach may work well in command
process really forces us to think about that, it’s
and control input-output interventions, where
more than just a log-frame with indicators, it’s
strategies are known, and processes are known to
more about coming together and setting time
lead to certain outcomes, this is generally not a
aside, to really think about, ok, what’s the impact?”
helpful structure for peace-building and conflict
prevention interventions working in conflict-affected – Ojaswi Shah, Project Manager, Nepal Team
states.
Saferworld needs to continue building on how we use
this data and information actively to communicate and
work with communities, our partners, and our donors
to improve our programming, and to influence policy
agendas both nationally and internationally.
We need to involve our community partners more in
monitoring the changes they want to see, and
encourage them to use their influence with decision-
makers. For further information, discussion, input and
feedback, please contact Madeline Church at
We need to continue to do things a little differently. Saferworld, [email protected]
Saferworld learning paper: Doing things differently – rethinking monitoring and evaluation to understand change Page 12 of 12

How does this approach feed into and complement other MEL-related elements
that Saferworld has put in place over this same period?
During 2011-15, Saferworld has put in place a set of new components in our planning, content and strategy
development, learning, and evaluation that complement the focus on behaviour and relationship change.
 Theory of change: we have had a number of goes at knitting theory of change approaches into our
focus on key actors, and changes in behaviour and relationships. Through many iterations, we have
found that teams find it easiest to examine their inherent assumptions about change by starting out
articulating the changes that they wish to see in each actor category in the form of outcome (challenge)
statements (adapted from Outcome Mapping). We then work outwards to investigate why what we do
works to create such changes. The outcome harvesting approach helps us to see whether this ‘theory’
is born out in practice, and hopefully helps us to articulate our theory better.
 Gender-sensitive conflict analysis, plus actor mapping: our conflict and insecurity analysis is used
not only for our own strategy development, but also to do assessments at community level for each
community we work with. Mapping the influence different actors have in each environment is crucial for
working for more sustainable, systemic change.
 Policy/Advocacy Matrix: this identifies five stages of success (improved credibility, improved
relevance, improved access to decision-makers, increased support for our messages, policy change),
using six types of evidence.
 Our Community Security Learning and Practice Group has had a significant learning effect across
the organisation by bringing together our project officers, coordinators, policy, advocacy, and
communications staff. We learn together about how best to work with communities on conflict, insecurity
and empowerment. We have consolidated this learning into our Community Security Handbook.
 Cross-organisational participation in evaluation teams: we enable, where we can, those working in
one region to join any evaluation team in another. We develop internal Terms of Reference for these
staff, based on a focused aspect that we want to learn about, and share, from one location to another.

About Saferworld
Saferworld is an independent international organisation working to prevent violent conflict and build safer
lives. We work with local people affected by conflict to improve their safety and sense of security, and
conduct wider research and analysis. We use this evidence and learning to improve local, national and
international policies and practices that can help build lasting peace. Our priority is people – we believe that
everyone should be able to lead peaceful, fulfilling lives, free from insecurity and violent conflict.
We are a not-for-profit organisation with programmes in nearly 20 countries and territories across Africa, the
Middle East, Asia and Europe.
Saferworld – 28 Charles Square, London N1 6HT, UK
Registered Charity no 1043843
Company limited by guarantee no 3015948
Tel: +44 (0)20 7324 4646 | Fax: +44 (0)20 7324 4647
Email: [email protected]
Web: www.saferworld.org.uk