Course Notes 1

Download as docx, pdf, or txt
Download as docx, pdf, or txt
You are on page 1of 4

INTRODUCTION TO IMPACT EVALUATION (CLASS NOTES-1)

why impact evaluation is important and how it fits within the context of
evidence-based policy making

What Do We Mean by “Impact Evaluation”?

Impact evaluation investigates the changes brought about by an intervention.

Impact evaluation can be undertaken on interventions at any scale:

A small, local disease prevention project;

An entire civil society strengthening program of a Non-Government


Organizations;

A sequence of natural resource management projects undertaken in a


geographic area;

A collection of concurrent activities by different organizations aimed at


improving a community’s capacity.

The expected results of an intervention are an important part of an impact


evaluation, but it is important to also investigate unexpected results.

Impacts are defined as: The positive and negative, intended and unintended,
direct and indirect, primary and secondary effects produced by an intervention
(OECD Development Assistance Committee definition)

Impacts are usually understood to occur later than intermediate outcomes. In


practice, final outcome includes both outcomes and impacts.

An impact evaluation includes any evaluation that systematically and


empirically investigates the impacts produced by an intervention.

There are narrower definitions of impact evaluation: In these definitions

Only include evaluations containing a counterfactual of some kind (an


estimate of what would have happened if the intervention had not
occurred) or

A particular sort of counterfactual (for example, comparisons with a


group who did not receive the intervention)

1
Other definitions of Impact evaluations:

Impact evaluations measure the change in a development outcome that


is attributable to a defined intervention;

Impact evaluations are based on models of cause and effect and require
a credible and rigorously defined counterfactual to control for factors
other than the intervention that might account for the observed change.

These different definitions are important when deciding what methods or


research designs will be considered credible by the intended users of the
evaluation or by partners or funders.

Why Should We Do Impact Evaluation?

Some common reasons for doing impact evaluation include:

To decide whether to fund an intervention

“ex-ante evaluation” is conducted before an intervention is


implemented, to estimate its likely impacts and inform funding decisions.

To decide whether or not to continue or expand an intervention

To learn how to replicate or scale up a pilot project

To learn how to successfully adapt a successful intervention to suit another


context

To reassure funders, including donors and taxpayers (upward accountability),


that money is being wisely invested

including that the organization is learning what does and doesn’t work,
and is using this information to improve future implementation and
investment decisions.

To inform intended beneficiaries and communities (downward accountability)


about whether or not, and in what ways, a program is benefiting the
community.

2
What Is Impact Evaluation?

Impact evaluation figures among a broad range of complementary methods


that support evidence-based policy.

Several approaches can be used to evaluate programs:

Monitoring tracks key indicators of progress over the course of a program as a


basis on which to evaluate outcomes of the intervention.

Evaluation examines how effectively programs were implemented and whether


there are gaps between planned and realized outcomes.

Impact evaluation studies examine whether the changes in well-being are


indeed due to the program intervention and not to other factors.

Monitoring

Monitoring is a continuous process that tracks what is happening within a


program and uses the data collected to inform program implementation and
day-to-day management and decisions.

Using mostly administrative data, monitoring tracks program performance


against expected results, makes comparisons across programs, and analyzes
trends over time.

Usually, monitoring tracks inputs, activities, and outputs, though occasionally it


can include outcomes, such as progress toward national development goals.

Evaluation

Evaluations are periodic, objective assessments of a planned, ongoing, or


completed project, program, or policy.

Evaluations are used to answer specific questions related to design,


implementation, and results.

In contrast to continuous monitoring, they are carried out at discrete points in


time and often seek an outside perspective from technical experts.

Their design, method, and cost vary substantially depending on the type of
question the evaluation is trying to answer.

3
Broadly speaking, evaluations can address three types of questions (Imas and
Rist 2009):

 Descriptive questions. The evaluation seeks to determine what is taking


place and describes processes, conditions, organizational relationships,
and stakeholder views.
 Normative questions. The evaluation compares what is taking place to
what should be taking place; it assesses activities and whether or not
targets are accomplished. Normative questions can apply to inputs,
activities, and outputs.
 Cause-and-effect questions. The evaluation examines outcomes and tries
to assess what difference the intervention makes in outcomes.

Impact evaluation

Impact evaluations are a particular type of evaluation that seeks to answer


cause-and-effect questions. The basic impact evaluation question can be
formulated as:

What is the impact (or causal effect) of a program on an outcome of interest?

The basic evaluation question can be applied to many contexts.

For instance:

 What is the causal effect of scholarships on school attendance and


academic achievement?
 What is the impact on access to health care of contracting out primary
care to private providers?
 If dirt floors are replaced with cement floors, what will be the impact on
children’s health?
 Do improved roads increase access to labor markets and raise
households’ income, and if so, by how much?
 Does class size influence student achievement, and if it does, by how
much?

You might also like