Impact Assessment and Indicators

Dr. Effie Amanatidou, ERA-LEARN / University of Manchester

2020 Water JPI Workshop on International Cooperation, 26 May 2020

• We ultimately monitor and evaluate to draw useful lessons on how to
The first step….Setting up a M&E system
improve Page 2
• By whom should the required data be collected
• Thus we not only need to estimate what but also to understand why
(e.g. central management team, project team, a
and how
centralised P2P systems)?
• Numbers and shares monitor things but do not evaluate – evaluation is
• Is the required capacity available?
about critical judgement based on synthesis of indicators, synthesis of
• Who will verify the data for accuracy and
findings, accumulated intelligence, benchmarking, counterfactual
consistency with requirements
exercises, etc.
• Engaging the relevant stakeholders is key in this process
who
• What data needs to be collected (inputs,
outputs, outcomes, impacts)? What
• When should the relevant data be additional data needs to be collected
collected (during the monitoring (ad hoc)?
phase, reporting phase, ex-post, • What are suitable methods for

•
how often, etc.)?
When should ad hoc data be
when why what collecting, storing and analysing data?
What are the necessary data protection
collected? (ex ante, ex-post, other?) protocols to be applied?

• How much will the setting up of the M&E system

• How / where will required data be gathered and stored?
cost and what resources will be needed to run it? how • Can the process be aligned with the reporting schedule
• Are these resources (human, financial, time) in
for the evaluation/impact assessment?
place or can they be ensured?
• How will the data be verified for accuracy and
• Can this M&E system be aligned with other
consistency with requirements?
existing systems (e.g. national level?)
Overall Water JPI Logic Frame and Contribution of Internationalisation Page 3

(Global)
Challenges Objectives Inputs Activities Outputs Outcomes
impacts

Internationalisation
Internationalisation Internationalisation
outputs, outcomes &
strategy inputs and activities
impacts

Contribution, i.e. integration of the internationalisation dimension, instead of

addition or supplement, in the overall Water JPI Agenda
Linking challenges to objectives – multiple reference levels Page 4

EU/International
policy context

Cross-national
policy context

JPI objectives

National
policy
context

Why was the JPI established? Which challenge, problem, or situation does it aim to address?

What are the short-term/operational, the medium-term/intermediate and longer-term/global objectives of the specific JPI?

to edit the footer only use "Insert --> Header & Footer"
Water JPI Objectives by 2020 Page 5

Objectives need to be SMART (Specific, Measurable, Accepted, Realistic and Time-dependent)

Involving water end-users for effective RDI Reaching effective, sustainable coordination Harmonising National water RDI activities in
results uptake of European water RDI Partner Countries
• JPI governance bodies, but also at the ▬ A permanent dialogue between Member • a catalogue of jointly programmed
Partner country level. While Europe is the States, Associated Countries and EC. activities whose global budget amounts to
natural domain of this initiative, global at least 20 % of the total water RDI
▬ benchmark water RDI programmes within
impact will also be targeted budget of partner Programmes
Internationalisation dimension and internationalisation-specific
Europe, facilitate access to research and objectives
development results and scientific • joint calls for proposals, mobility schemes
Attaining critical mass of research infrastructure, and promote innovation and infrastructure actions under the
programmes Harmonising National water RDI agendas in variable geometry principle

• involve at least two-thirds of the public Partner Countries Supporting European leadership in science
National water RDI investment in Europe. ▬ RDI agendas of Partner Countries and the and technology
JPI Strategic Agenda will show effective • maintain the current European world
harmonization leadership in water related publications
• doubling multinational European
authorship of scientific publications
Source: Water JPI Vision Document
Water JPI : linking challenges to objectives and activities or the Objectives Hierarchy Page 6

Involving end-users Europe the most Achieving Sustainable

Overall Challenge
Intermediate challenges
Water JPI Objectives
Activities-specific Objectives

Joint calls
Critical mass competitive water Water Systems for a
Exploratory sector Sustainable Economy
workshops Coordinate European in Europe and Abroad
water RDI Capacity of landscape
Knowledge hubs and water ecosystems,
Networking Harmonise RDI biodiversity and
workshops agendas services
Harmonise Clean drinking water
Monitoring of
supported projects RDI activities and proper sanitation
Internationalisation dimension Technology
and internationalisation-specific
European leadership objectives
Water JPI SRIA deployment in the
in water STI
Strategy activities water sector
Mobility &
Infrastructures
Horizontal workshops

The clearer and more verifiable the objectives of a programme, the more useful its evaluation;

ERA-LEARN Guide for P2P Impact Assessment: Step 2 (Linking Challenges to Objectives) and Step 3 (Inputs, activities)
What we can evaluate…evaluation questions
Network health and connectivity Page 7

Added value

(Global)
Challenges Objectives Inputs Activities Outputs Outcomes
impacts

effectiveness

relevance efficiency
What does that mean? Also in relation to internationalisation Page 8

Effectiveness Added value

▬ To what extent do the effects (outputs, outcomes, and impacts) ▬ What is the additional value resulting from the P2P,
induced by the P2P correspond with its objectives? compared to what could be achieved by Member States
▬ To what extent have the internationalisation results (outputs, alone at national and/or regional levels?
outcomes and impacts) contributed to the overall P2P objectives? ▬ To what extent do the problems/challenges addressed by
Relevance the intervention require action at EU level?

▬ To what extent are the P2P objectives relevant with respect to the ▬ What would be the most likely consequences of stopping
needs, problems and issues identified? or withdrawing the existing policy intervention?

▬ To what extent are the internationalisation objectives relevant ▬ What is the added value of the internationalisation
with the neds, problems and issues addressed? activities?

Efficiency
▬ How economically have the resources used been converted into
effects?
▬ How economically have the resources used been converted into
internationalisation effects?
What does that mean? Also in relation to internationalisation Page 9

‘Network health’: ability to engage its members, sustain their ‘Network connectivity’: the extent to which the members’ ties
engagement, and adapt as needed. to each other are resulting in efficient and effective
“pathways” for shared learning and action.
▬ What are the network’s governance rules and are they
effective? ▬ Has the P2P assembled members with the capacities
needed to meet network goals (experience, skills,
▬ Do decision-making processes encourage members to
contribute and collaborate? connections, resources)?
▬ Who is connected to whom?
▬ How are the network’s internal systems and structures adapting
over time? ▬ Who is not connected but should be?
▬ Do all members share a common purpose for the network? Are ▬ Is membership adjusted to meet changing network
all members working together to achieve shared goals, needs?
including goals that emerge over time?
▬ What are the number, quality, and configuration of
▬ Are members achieving more together than they could alone? network ties?
▬ Has a sense of trust developed amongst the network ▬ How dependent is the network on a small number of
participants? individuals?
▬ Has the P2P secured the necessary resources (capacities, ▬ Is the network structure adjusted to meet changing
money, and infrastructure) to become self-sustained? network needs and priorities?
Horizon Europe impact pathways’ indicators Page 10
ANNEX IV of Regulation establishing Horizon Europe Page 11

to edit the footer only use "Insert --> Header & Footer"
ANNEX IV of Regulation establishing Horizon Europe Page 12

to edit the footer only use "Insert --> Header & Footer"
ANNEX IV of Regulation establishing Horizon Europe Page 13

to edit the footer only use "Insert --> Header & Footer"
Indicators: features and examples Page 14

All indicators should be ‘RACER’, i.e.:

• Relevant to the objectives and should measure the right thing;
• Accepted (e.g. by staff, stakeholders who hold responsibility)
• Credible for non-experts, unambiguous and easy to interpret.
• Relatively easy to monitor (e.g. data collection should be possible
at low cost).
The RIPE toolkit help
• Robust against manipulation (e.g. If the target is to reduce
administrative burdens to businesses, the burdens might not • https://www.era-learn.eu/support-for-
be reduced, but just shifted from businesses to public
p2ps/monitoring-and-assessment/r-i-partnership-
administration).
evaluation-toolkit-ripe/carrying-out-the-evaluation-
of-partnerships-in-r-i/collecting-data-and-
information/defining-indicators
 Page 15

• Rationale
• Assumptions
• Data information needs and resources
• Who is providing the needed information
• Methodology and frequency for indicator
measurement
• Assessment of indicator quality and comparability with
existing monitoring systems of EC and Member
States
• Estimated cost of data collection (incl. access to
external databases)
• Level of reporting burden for beneficiaries

• Summary of key data needs for monitoring progress

on scientific impact

to edit the footer only use "Insert --> Header & Footer"
 Page 16

• You still need to check the appropriateness of what is proposed and this is generic enough to allow adjustment to
different cases. The appropriateness of indicators is case and context dependent.
• Results span different levels (project level & network level, national & trans-national level).
• Indicators are subject to a number of limitations but don’t fall into the trap that ‘Impact is only what we can
measure’. The combination of quantitative and qualitative information is invaluable.
• You still need to select the appropriate methods for collection and elaboration of data and indicators.

The R&I Partnership Evaluation (RIPE) Toolkit

https://www.era-learn.eu/support-for-p2ps/monitoring-and-assessment/r-i-
partnership-evaluation-toolkit-ripe
presents a complete monitoring and evaluation methodology with concrete steps, examples, templates and good practice tips based on
the work of ERA-LEARN over the years in supporting the P2Ps in their monitoring and evaluation activities.

to edit the footer only use "Insert --> Header & Footer"
Thank you!
[email protected]

[email protected]
Monitoring: Project level information collection Page 18

• Call / activity budget (National contributions pre-call/actual) Outputs

• Applications received / approved • Publications (articles, conference proceedings, books, book
chapters, reports, grey literature, datasets, etc.)
• Types of participants
• Conference/workshop attendances
• Etc.
• Project meetings
• Degree theses
Extracts from proposals
• Products, process etc. (licensed/patented or otherwise)
• Level of staff receiving support
• Student/staff exchanges
• Sources/amounts of co-funding for the project
• Contributions to standards, public awareness, policy
• Level of prior contact with other project participants
• Further development of research networks, etc.
• Centrality of research project to core activities of organisations
• Information about academic/industrial relevance;
• Information about intended dissemination and follow-up research;
• Information about availability of resources.
• Etc.
Background to the ERA-LEARN Guide for P2P Impact Assessment: Section 6
Monitoring: P2P level information collection Page 19

• Information about the networks (types of networks, funding sources, budgets, national contributions, funding modes, countries
represented, network objectives, activities, thematic priorities and S&T fields addressed),
• Information about network partners (number, types, contact data, role of organisation in network, funding source of organisation for
the participation in the network, etc.)
• joint activities (number, types of activities, types of research and research fields addressed, sources of funds, national budget, EU
budget, other budget, funding mode)
• joint calls (number, types of research and research fields addressed, sources of funds, national budget, EU budget, other budget,
funding mode)

Interim evaluation/monitoring
• Planned activities against those materialised (number, type, outcomes)
• The network budget (absorbed against overall)
• Stages in networks development achieved
• Etc.

Background to the ERA-LEARN Guide for P2P Impact Assessment: Section 6

Data collection and analysis methods: what you need to know before deciding Page 20

For each indicator, how can they best be measured/captured?

• Secondary data: national / European / international statistics (R&I indicators); thematic data (publications, patents, employment, etc.)
• Primary data: collection through surveys of value judgements but also facts (publications, collaborations, patents, etc.)
• Importance of monitoring systems established at the start of the activity
What is the added value of applying a quantifiable or a qualitative measurement or a combined approach in measurement?
• Snap-shot in time vs. longitudinal trends
• what vs. why and how
• A number vs. a narrative of a chain of impacts
The issue of attribution – establishing cause-effect relationships
• Cannot be too ambitious – certain correlations can indeed be made – narratives of impact chains are equally important

The importance of monitoring and keeping track of possible impact pathways can never be overestimated

ERA-LEARN Guide for P2P Impact Assessment: Step 8: Data collection and analysis methods
Data collection and analysis methods: examples Page 21

Collection methods Analysis methods

• Existing surveys / databases • Case studies
• Participant surveys • Network analysis
• Non-participants surveys • Econometric analysis
• Focus groups / meetings / workshops • Descriptive statistics
• Technometrics/Bibliometrics analysis • Input / output analysis
• Document search • Document / Content analysis
• Monitoring data • Control group approach
• Counter-factual analysis
• Cost/benefit analysis

ERA-LEARN Guide for P2P Impact Assessment: Step 8: Data collection and analysis methods
Indicators – what you need to pay attention to… Page 22

• Before proposing new data requirements, you should carefully assess to what extent the existing data reflect the objectives set and
whether the missing key data can be collected via existing monitoring structures.

• It is essential to understand that indicators are subject to a number of limitations. They cannot measure all aspects of the reality
while indicators that are defined ex-ante can only capture intended impacts. Societal impacts appear especially difficult to measure
but don’t fall into the trap that ‘Impact is only what we can measure’!

• It may be the case that the most accurate indicators are extremely resource intensive to collect; thus a balance will have to be
struck between indicator suitability and ease of collection.

• Qualitative indicators can be highly illustrative of the outputs and impacts of activities but are more difficult to aggregate and to
subject to quantitative analyses.

• The appropriateness of indicators is case and context dependent.

ERA-LEARN Guide for P2P Impact Assessment: Step 7: Defining output, outcome and impact indicators
From activities to outputs, outcomes and impacts Page 23

We need to consider that networks have a “chain of impact” that includes

• the network’s impact on its members (network level)
• the members’ impacts on their local environments, (national level)
• the members’ combined impact on their broader environment (trans-national)
Impacts (global,
Evaluations designed to examine impact must understand the long-term)
Outcomes • Economic impacts
relationship between these three levels and (intermediate
impacts) • Societal impacts
be clear about where their focus lies. • Innovation
• S&T impacts impacts
• Organisational • Policy/conceptual
The first step: sharing the right Outputs • Capacity building impacts
common understanding of the • SR(I)As • Structural impacts • Organisational
terminology! • Training • Economic impacts • Etc.
• Databases • Symbolic
• Etc. • Etc.

ERA-LEARN Guide for P2P Impact Assessment: Step 4 & 5: What are the output, intermediate and global impacts?
 Examples of indicators from P2Ps (network level)…etc. Page 24

Activity Sub-activity Output Indicators / nature Outcome Indicators / nature Impact indicators / nature Source of Timing
information of eval.
Mapping Mapping • No of attendants (quant.) • Identification of common • Critical mass of research in Monitoring/ Interim/
national/trans- workshops/ • Quality of report/ deliverable areas of interest certain areas (both quant. questionnaire ex-post
national activities meetings (qual.) (qualitative) qual.)
• Programme clustering (qual.)

Foresight and Foresight exercise • No of attendants (quant.) • Identification of common • Inform national and Monitoring/ Interim/
common vision Vision building ws • Quality of report/deliverable areas of interest European policies questionnaire ex-post
(qualitative) (qualitative) (qualitative)

Strategic Research Interaction with • No of attendants (quant.) • Identification of themes • Specific strategies for Monitoring/ Interim/
Agenda / AB, stakeholders • Quality of discussions (qual.) for calls (qual.) certain areas (qual.) questionnaire ex-post
Implementation Plan Specific surveys • Quality / level of approval of • changes in research • Influence national
SRA (qual.) priorities of agencies strategies/policies/
(qual.) programmes (qual.)
• alignment of research • Changes in national
strategies (qual.) budgets (quant.)
Joint calls Building a portal • User-friendliness of portal • Promotion of research • Common rules, Monitoring/ Interim/
Call management (quant. qual.) area at national levels procedures, timing, and questionnaire ex-post
Evaluation of prop. • No of proposals submitted/ (quant.) evaluation panels
approved (quant.) • Change of national rules, (qualitative)
• Time to contract (quant.) timings (qual.) • Changes in legislation to
• Multinational evaluation allow payments to foreign
schemes (qual.) researchers
ERA-LEARN Background(qual.)Document to the Guide: section 7
 Examples of indicators from P2Ps (project level)…etc. Page 25

Project activity Output Indicators / nature Outcome Indicators / nature Impact indicators / nature Source of Timing of
information evaluation
Research Publications (quant.) Changes to research programmes of New research trajectories / new areas of Monitoring/ Interim
collaboration New staff, students, employees linked to organisations (qual.) research (quant./qual.) questionnaire Ex post
project/theme (quant.); New methods, Increased collaborations (quant.) Solutions to challenges (qual)
services, products (quant/qual); Co- Higher-research ranking (quant.) international profile (quant./qual)
authorships (quant.); New joint Increased reputation (qual.) Increased long-standing collabs (quant./qual)
proposals/projects (quant.) Access to extra R&I funding (quant.)
Research Industry/HE co-publications (quant.) New methods/products/services Solutions to challenges (qual.) monitoring/ Interim
collaboration Prototypes of new (quant.qual); Spin-offs (quant./ qual) Increased industry competitiveness questionnaire Ex post
Academia – methods/products/services (quant.qual) Market share figures (quant./qual) (quant/qual)
industry Patents, licenses, leasing, etc. (quant) Commercial returns – turnover – Improved business models (qual.)
employment (quant.)
Reduced operating costs (quant.)
Results diss. Raising awareness in society Change consumers behaviour More informed / concerned citizens Monitoring/ Interim
society (quant./qual) (quant./qual) (quant./qual) questionnaire Ex post
Results diss. Inputs to standards (qual.) White papers, draft regulations Solutions to challenges (qual./quant) questionnaire Ex post
policy (quant./qual) Improved policy-making (qual.)
Changes in policies / regulations Improved service quality (qual)
(quant/qual.) Reduced environmental impacts (quant.qual)
Capacity Training schemes/activities Improved capacities at organisational Improved national capacity / performance in Monitoring/ Interim
building (quant./qual.); Masters/PhD students level (quant./qual.) specific area (quant./qual) questionnaire Ex post
knowl. (quant.); Conferences, workshops, Changes to human resources New practices for research organisation
transfer seminars (quant./qual.) Organisational changes (quant./qual) (qual) ERA-LEARN Background Document to the Guide: section 7
Examples of outcomes and impacts per different type of beneficiary Page 26

Type of Beneficiary
Outcomes Intermediate Impacts Global Impacts
additional research income, new research trajectories, new
Research new technology, new data/method, commercial income, increased solutions for socio-environmental
organisation formal publications, patents research capacity, spin-off challenges, economic spill-overs to
businesses, enhanced reputation industry
new product/service, new technical increased turnover/profit, new jobs, economic spill-overs to other
Industrial
process, new organisational process, protection of existing jobs, increased businesses, new solutions for socio-
organisation
patent, improved capacities market share, geographic expansion economic challenges
Public service new methods/services, new improved service quality, reduced cost improved health, safety, security
organisation organisational process of service delivery and/or quality of life for citizens
improved governance, reduced
improved scientific evidence, new improved economic, social and/or
Public administration administration costs, evidence-based
organisational process environmental impacts
policy making
improved scientific evidence,
improved standards/regulations,
Societal organisation improved services, improved increased influence
improved quality of life
capacities
improved scientific evidence,
Environmental improved standards/regulations,
improved services, improved Increased influence
organisation reduced environmental impacts
capacities
ERA-LEARN Background Document to the Guide: section 7
 Examples of types and timing of impacts Page 27

Definitions and a typology of impacts

in the background Document of
ERA-LEARN Guide (section 3.d
‘Outputs, outcomes and impacts of
P2Ps’)

ERA-LEARN Guide for P2P Impact Assessment: pp. 14-15

What are the inputs and activities that will achieve the P2P objectives
***
Page 28

Inputs

• Financial, human resources, skills, infrastructures, ‘costs’ of beneficiaries and end-users, but also

• network structures and processes, governance and decision-making procedures, rights, obligations, rules

Activities

• Implementing transnational calls; additional joint calls

• Dissemination / Up-Take of research results

• Foresight and common vision building / Strategic Research Agenda / Implementation Plan

• Mapping national/trans-national activities

• Knowledge sharing amongst researchers, Mobility and training

• Research infrastructures; Widening participation; Internationalisation

• Monitoring and evaluation/assessment activities

ERA-LEARN Guide for P2P Impact Assessment: Step 3

But a Logic Frame is NOT the Intervention Logic Page 29

Intervention Logic – main underpinning assumptions

• Brain drain - repatriation The main reasons for researchers leaving is lack of
Challenge/Objective
professional opportunities in their home country
The offered post-docs and exchange visits cover their
• National R&D needs in terms of opportunities offered and career
Inputs budget/strategy prospects
A post-doc position or exchange visit can act as showcase
• Post-docs of benefits if they return home
Activities
• Exchange visits
The working conditions in the home country are more
appreciated than those abroad
• No & types of Post-docs
Outputs
• Co-authored publications The personal / family opportunities in the home country are
more appreciated than those abroad
• Return of ?% of expat The political / economic situation in the country can ensure
Outcome researchers a well-paid tenure position for people to return
The conditions in the institutions in the home country can
• Increased international ensure a well-paid, well-framed position for people to return
Impact collaboration
People will retain their links abroad when they return home
…
Some more tips Page 30

Continuously running ERA-LEARN central survey

for project impact assessment based on harmonised
questionnaire developed by ERA-LEARN in consultation
with P2Ps. ERA-LEARN Guide for P2P Impact Assessment
(Guide and Background document) downloadable at
https://www.era-learn.eu/support-for-p2ps/monitoring-
Contact Hayley Welsh Optimat UK and-assessment.
[email protected]

All ERA-LEARN Policy briefs on impacts on P2Ps

and their projects https://www.era-learn.eu/support-for-
p2ps/monitoring-and-assessment