1

Towards a new EU framework for reporting on environmental policies and measures (Reporting on
environmental measures - REM)

PAPER I:

DEFINING CRITERIA FOR EVALUATING THE EFFECTIVENESS OF EU
ENVIRONMENTAL MEASURES

1. Introduction

This paper arises from a discussion at the REM [Reporting on Evaluation of
Measures] Steering Group meeting on 10 November 1999 concerning the criteria that
should be used for judging the effectiveness of EU environmental measures. A
specific question that was posed was whether social and economic impacts should
also be considered, in addition to environmental.

Discussions about evaluation often cause confusion, for two important reasons:

different types of evaluations ask a wide variety of different questions, and use
widely differing methodologies;
the terms that are employed eg effectiveness, effects, efficiency, output,
impact etc are often used inconsistently.

This paper seeks to distinguish between the different sorts of evaluative questions, and
to clarify the terms that are used, as the basis for the considering the criteria that
should be used to judge whether a measure is effective.

As part of its mission to encourage an evaluation culture throughout the
Commission, the Evaluation Team in DGXIX has attempted to standardise evaluation
terms and methodologies in two useful publications (1,2). In order to maintain
consistency between the current REM project and evaluation work being undertaken
by the Commission, the discussion that follows uses the terms and approaches set out
in these publications.

2. The Evaluation Framework

The sorts of questions that are posed in most evaluations fall into three basic
categories:

1. Descriptive questions intended simply to observe and measure changes ( ie what
happened after the measure was put in place?)

2. Causal questions that seek to analyse what happened in terms of cause and effect
(ie to what extent are these observed changes attributable to the measure, and why did
that particular measure have those particular effects in those particular
circumstances?). Assessing the actual and projected effects of environmental measures
is necessary for developing models or scenarios in relation to future trends in the state
of the environment.

2
3. Normative are these results satisfactory? Normative questions require a
judgement to be made against some prior explicit objective or benchmark. Questions
within this category may relate to:

Effectiveness: To what extent has the measure achieved its intended objectives, in
relation either to outcomes (ie changes in the behaviour of socio-economic actors,
and/or impacts (on the state of the bio-physical environment)?
Relevance: To what extent do these objectives adequately address the needs of
the issue or problem?
Efficiency: Have these objectives been achieved at lowest cost?
Utility: Have the overall effects of the measure both intended and unintended,
good and bad contributed to a net increase in social welfare? (This is the kind of
question posed in cost-benefit analysis (CBA)).

How these normative questions relate to the different aspects of the policy,
programme or project is illustrated in Figure I.

Normative questions are of most interest to those EU institutions which have formal
responsibility for the design, management and review of EU policies. Causal and
descriptive questions are of interest to those agencies and organisations reporting on
current and future trends in the state of the environment, pressures, and driving forces.

3. Scope of an evaluation

3.1 Evaluating effectiveness in relation to objectives

Using the above definition of the term, evaluations of effectiveness must be based
upon comparing the effects of a measure (ie outcomes and/or impacts) to its explicitly
stated objectives. These objectives may be expressed in general or specific terms, but
the most useful evaluations of effectiveness require objectives to be expressed as
clearly as possible in the legislation, including quantitative objectives.

In some policy areas, EU measures have multiple objectives, and their effectiveness
therefore needs to be assessed in relation to all of them. An example is the EUs
recent Rural Development Regulation 1257/1999 , the objectives of which include:

- the improvement of agricultural incomes;
- the maintenance of a viable social fabric in rural areas;
- the improvement of working and living conditions;
- protection of the environment;
- equal opportunities for men and women.

Evaluation guidelines issued to Member States by the Commissions Directorate-
General for Agriculture take account of such wide-ranging objectives, and include as
many as 50 common evaluation questions that mid-term and ex post evaluations of
rural development programmes need to address (3).

By contrast, the objectives of almost all EU environmental measures are exclusively
environmental. Although Article 2 of the Amsterdam Treaty establishes as one of the
3
overarching tasks of the Community to promote the sustainable development of
economic activities, this has not yet been translated into specific and/or operational
4


Needs
Objectives Inputs Outputs
Outcomes
Impacts
Relevance
Utility
Efficiency
Effectiveness

Needs, Problems, Issues
(Societal, Economic and
Environmental)

[RELEVANCE, UTILITY]

Policy / Programme / Project

[EFFECTIVENESS,
EFFICIENCY]

Evaluation Questions
Figure I : Policy Evaluation
Effects
Relevance:
To what extent are the
objectives justified in
relation to needs?
Effectiveness:
To what extent have the
expected objectives been
achieved?
Utility:
Are the expected or
unexpected effects
contributing to a net
increase in social
welfare?
Efficiency:
Have the objectives been
achieved at lowest cost?
5
socio-economic objectives that environmental measures should achieve. It is true that
- for example - local air quality action plans developed by Member States in the
framework of the Air Quality Framework Directive 96/62 will undoubtedly have
effects on local economic development and the social incidence of urban air pollution.
However, the Directive sets no explicit socio-economic objectives against which its
effectiveness (as defined above) may be judged. It may be that one consequence of
the development of an EU sustainable development strategy will be that such explicit
objectives will in future become increasingly common.

However, in current circumstances, the criteria for evaluating the effectiveness of EU
environmental measures must necessarily remain limited to environmental outcomes
and impacts. Moreover, any extension of these criteria to social or economic effects
raises questions of cost and usefulness, which are further discussed below.

3.2 Evaluating utility

While evaluations of effectiveness can be undertaken only in relation to the explicit
objectives of a policy, evaluations of utility seek to identify all the effects of the
measure, intended and unintended, in relation to a wide range of issues social,
economic, environmental, cultural etc - with a view to arriving at some judgement
about its contribution to overall social welfare.

Evaluations of EU environmental measures might therefore include broader
evaluations of utility. Indeed, under the EUs GHG Monitoring Mechanism, Member
States are curently required to assess not only the effects of policies and measures on
emissions of GHGs, but also the wider economic impact of such measures.

However, the major drawback with evaluations of utility is that the breadth of
questions they seek to address means that their costs are high while their practical
usefulness is uncertain. Hence, the selection of questions to be addressed by any
evaluation needs to be guided by three practical considerations:

Use: Will the information and/or judgements generated by the evaluation be used
by decision-makers. Do they fulfil a real need or legal requirement?

Evaluability: Can the questions posed be answered within the constraints of
available data and the willingness of Member States to collect it ( in the light of
increasing reporting fatigue)?

Cost: Can the information be collected without disproportionate expenditure by
Member States and EU institutions of staff time and resources?

These considerations suggest that evaluations of overall utility should be required only
in exceptional, and well-justified, circumstances.

4. Conclusions

6
In the absence of any social, economic or other objectives explicitly articulated in the
legislation, evaluations of the effectiveness of EU environmental measures will need
to be restricted to the consideration of environmental objectives alone.

Evaluating the broader question of the overall utility of EU environmental measures
by assessing all effects across a wide range of criteria would be an interesting (and
challenging) exercise, but would stretch the skills and resources of most Member
States to the limit. In these circumstances, evaluations of utility should be undertaken
only when they can be shown to fulfil a clear need; where they are practicable; and
where they can be undertaken without disproportionate cost. The development of a
more streamlined reporting system - one of the objectives of the REM project would
suggest that evaluations should be made as manageable as possible.




References

1.European Commission (DGXIX/02) Evaluating EU expenditure Prgrammes - A
Guide: Ex post and intermediate evaluation January 1997

2. European Commission The MEANS collection: Evaluating socio-economic
programmes 6 volumes, 1999.

3. European Commission Directorate-General for Agriculture Evaluation of Rural
development Programmes 2000-2006 supported from the European Agricultural
Guidance and Guarantee Fund, 1999


I:\IEEP\PROJECTS\REPORTIN.EEA\FINAL\STGPAPR1.DOC