Indicators for promoting and

monitoring Responsible
Research and Innovation
Report from the Expert Group on Policy Indicators
for Responsible Research
and Innovation

EUR 26866 EN

EUROPEAN COMMISSION
Directorate-General for Research and Innovation
Directorate B Innovation Union and European Research Area
Unit B.7 Science with and for Society
E-mail: [email protected]
[email protected]
European Commission
B-1049 Brussels

EUROPEAN COMMISSION

Indicators for promoting and

monitoring Responsible
Research and Innovation
Report from the Expert Group on Policy
Indicators for Responsible Research and
Innovation

Chair: Roger Strand

Rapporteur: Jack Spaapen
Members: Martin W Bauer, Ela Hogan, Gema Revuelta, Sigrid Stagl
Contributors: Lino Paula, ngela Guimares Pereira

June 2015

2015

Directorate-General for Research and Innovation

Science with and for Society

EUR 26866 EN

EUROPE DIRECT is a service to help you find answers

to your questions about the European Union
Freephone number (*):
00 800 6 7 8 9 10 11
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you)

LEGAL NOTICE
This document has been prepared for the European Commission however it reflects the views only of the
authors, and the Commission cannot be held responsible for any use which may be made of the information
contained therein. More information on the European Union is available on the Internet (http://europa.eu).
Luxembourg: Publications Office of the European Union, 2015.
ISBN 978-92-79-43169-2
ISSN 1831-9424
doi 10.2777/9742
European Union, 2015.
Reproduction is authorised provided the source is acknowledged.

Contents
1.

RESPONSIBLE RESEARCH AND INNOVATION ACTIVITIES AND THEIR MONITORING

VIA INDICATORS: PRINCIPLES, GAPS AND CHALLENGES ........................................... 9
1.1.

Objective and purpose of the expert group and this report ................................ 9

1.2.

Responsible Research and Innovation: concepts and practices ........................... 9

1.3.

The policy context: promoting and monitoring RRI ......................................... 11

1.4.

Intended and unintended consequences of monitoring.................................... 13

1.5.

Monitoring RRI activities and their impacts with indicators for outcomes,
processes and perceptions ......................................................................... 15

2. INDICATOR CRITERIA: THE SIX KEYS AND BEYOND ................................................... 18

2.1.

Governance ............................................................................................. 18

2.2.

Public engagement .................................................................................... 21

2.3.

Gender equality ........................................................................................ 26

2.4.

Science education ..................................................................................... 28

2.5.

Open access/open science .......................................................................... 31

2.6.

Ethics...................................................................................................... 33

2.7.

Sustainability ........................................................................................... 36

2.8.

Social justice/inclusion............................................................................... 38

3. PRIORITISED LIST OF INDICATORS TO BE DEPLOYED ................................................ 41

Table

Table 1 :

Indicator framework for Responsible Research and

Innovation.

Table 2.1:

Proposed indicators for governance.

Table 2.2:

Proposed indicators for public engagement.

Table 2.3:

Proposed indicators for gender equality.

Table 2.4:

Proposed indicators for science education.

Table 2.5:

Proposed indicators for open access/open science.

Table 2.6:

Proposed indicators for ethics.

Table 3.1:

Prioritised indicators for Responsible Research and

Innovation.

Executive summary

Early in 2014 the European Commission appointed an expert group to identify and propose
indicators and other effective means to monitor and assess the impacts of Responsible Research
and Innovation (RRI) initiatives, and evaluate their performance in relation to general and specific
RRI objectives. This report presents the results of the work of the expert group. It contains three
parts: first a conceptual introduction of RRI; second a detailed review of possible indicators in eight
key areas for RRI policy; and third a number of concrete proposals for indicator design and
implementation.
Six of the key areas are defined by the European Commission: public engagement (PE), gender
equality, science education, open access, ethics and governance. As RRI is a policy concept
experiencing a dynamic development, as witnessed by the recent Rome Declaration on Responsible
Research and Innovation in Europe (1), the group accordingly considered two more areas of
relevance to RRI, sustainability and social justice/inclusion, two of the three overarching goals that
are the backbone of the Europe 2020 strategy. A dynamic framework for RRI indicators is
accordingly recommended in order to capture the full range of ways to implement RRI as a crosscutting issue.
In part 1 the expert group looked at the various ways RRI has been defined in the scientific
literature and in EU policy reports. It found that there is as yet no clear consensus about what RRI
exactly entails, nor about how to measure its impact. The expert group took the following definition
of RRI as a guiding principle: a transparent, interactive process by which societal actors and
innovators become mutually responsive to each other with a view on the (ethical) acceptability,
sustainability and societal desirability of the innovation process and its marketable products (Von
Schomberg 2011). This means that the group sees RRI from a network perspective, consisting of
stakeholders jointly working on a set of principles guided by the RRI keys. When monitoring RRI
policy, the prime focus therefore should be on the development of RRI agendas in these networks.
In other words, the focus of monitoring and indicators should be on the governance of RRI in these
networks, and in particular on what kind of RRI policies are developed (best practices). Secondly,
one should discuss the kind of indicators best fitting research and innovation (R & I)practices and
goals within these networks, and then decide upon a limited set of indicators (because things have
to be manageable). Such indicators are thus highly contextual, meaning that there will be no one
list for all. This of course does not mean that all lists of indicators will be completely different. And,
to be sure, when we talk about indicators, we think not only of quantitative but also of qualitative
data. The latter might be even preferable, given the early stage of development of RRI policy.
The expert group decided upon a structure for indicator development and implementation. RRI is in
its view a matter of the interface and interplay between R & I and the context in which it takes
place, and hence the group has considered indicators both of action (in terms of processes and
outcomes) within the R & I sector and of its perception by other actors and society in general.
Finding indicators for impact should also embrace the interactive character of most innovations,
that is, they dont follow a linear pattern from basic research to application and use by society.
Indicators should explicitly or implicitly refer to the iterative character of innovation. An additional
value of involving stakeholders in indicator development will be the fact that if the stakeholders
become the owner of the monitoring they will be more ready to accept this as a valuable
instrument to improve their performance.
In short, the group sees the development of indicators as a bottom-up process, guided by the
collaboration between relevant stakeholders. In this process, indicators are considered for eight
key areas in three categories: R & I processes; their outcomes; and how such processes and
outcomes are perceived (perception). The ultimate set should be limited for the sake of
manageability.
In part 2, the group looked into the eight key areas and considered the options for indicators in
each of the three categories. It divided the eight issues into three sections: first, (good)
governance as an overarching principle for formal and informal R & I networks; second, PE, gender

( 1)

http://ec.europa.eu/research/swafs/pdf/rome_declaration_RRI_final_21_November.pdf
5

equality, science education, open access and ethics as the five main keys for governance; and
third, sustainability and social justice/inclusion as more general policy goals. Each criterion is
critically reviewed with regard to its current state of development, followed by suggestions for
potential process, outcome and perception indicators.
On governance: From a network perspective, RRI is governed through the active participation of all
relevant stakeholders in developing a monitoring policy and indicators. Frameworks in which
stakeholders can collaborate to that effect are developing at all hierarchical levels of the science
and innovation system, for example the joint programming initiatives, or various publicprivate
partnerships. Indicators should be oriented towards the development of these kinds of networks
and the exchanges that take place between different stakeholders. These stakeholders should
jointly decide what indicators best represent the kind of R & I that takes place in their particular
network. Indicator example: identification of formal and informal networks of R & I that promote
RRI, at both the national and the EU level.
On public engagement: The group has distinguished three dimensions to develop PE indicators: (i)
policies, regulation and frameworks; (ii) event/initiative making and attention creation; (iii)
competence building. For each of these three dimensions we distinguish performance indicators of
process and outcome and perceptions indicators (or indications), and a range of actors to which
these activities and commitments are attributable. Indicator examples: science events; citizen
science initiatives.
On gender equality: In terms of monitoring RRI policies on gender equality, the group concludes
that the main focus should be on processes of institutional change to see whether these general
ambitions are translated into concrete forms of action. Regarding indicator categories of
performance and perception, this means that the group suggests looking at: (i) changes in
institutional processes and structures that govern and influence the career progression of women in
research institutions; (ii) cultural change in institutions that reduces gender bias and promotes
gender equality; (iii) addressing the unconscious gender bias that favours one sex over another,
e.g. perception of womens achievements in science, technology, engineering and mathematics
(STEM); and (iv) changes in workplace arrangements to support female researchers, as it is usually
women who are discriminated against by such arrangements. Indicator example: percentage of
Member State funding programmes explicitly including gender requirements.
On science education: the group concludes that two goals can be identified at the level of the EU:
(i) the enhancement of education so that future researchers and other societal actors are going to
be equipped to become good RRI actors; and (ii) boosting interest in science among children and
young people with the purpose of either recruiting them to a research career or allowing them to
contribute to a science-literate society, that is, to become scientific citizens. Indicators currently in
use (research proposals including at least one participant from formal and informal science
education organisations; research projects that involve STEM teachers or students; PE events
aimed at young people; science education projects registered in the Scientix collaboration) are
predominantly proxies for the goal of boosting interest. With regard to (ii), the group feels that this
goal belongs to PE rather than to science education. Indicator example: at the level of education
institutions and/or research disciplines, the occurrence of RRI education/training (yes/no;
availability of courses; if they are mandatory or not; percentage of funds).
On open science/open access: The mandate of the expert group is to identify gaps and challenges
in the monitoring of RRI and propose indicators that help bridge the gaps and meet the challenges.
In the case of open access in the strict sense, the expert group acknowledges that the
recommendation from the European Commission of 17 July 2012 on access to and preservation of
scientific information (European Commission, 2012) to a large degree covers the gaps and
challenges. However, the group proposes to extend the monitoring of RRI to encompass the
ongoing phenomenon of open science. Following Winfield, the group distinguishes three levels of
open science: (i) Level 0: maintenance (including frequent updates) of project websites; Level 1
equals Level 0 plus: project blogs; post-project movie clips and blog posts with explanation and
commentary; Level 2 equals Level 1 plus: routine upload of experimental datasets to project
websites, with explanatory notes and commentary, daily laboratory notebooks, regular project
dialogue, etc. Indicator example: percentage of research projects with a virtual environment that is
updated and actively used with a threshold frequency (to be defined).
On ethics: In the broad RRI context ethics can be divided into three subfields: (i) research integrity
and good research practice, which is concerned with issues such as scientific misconduct and

questionable research practices (e.g. plagiarism, fabrication, fraud, authorship and intellectual
property and citation/acknowledgement practices, scientific neutrality, conflicts of interest in peer
review and scientific advice and others); (ii) research ethics for the protection of the objects of
research, which is a well-developed dimension with institutions and practices for such protection
the ultimate goal of policy in this field is that human beings, animals and other objects of research
are duly protected; (iii) societal relevance and ethical acceptability of R & I outcomes. This
dimension as an RRI key is the one that is closest to the general policy of RRI as a cross-cutting
principle and the one for which the European Union has the most distinct role to play. This field is
the one warranting the highest interest in the monitoring of ethics as an RRI key. Indicator
example: presence of mechanisms for multi-stakeholder and/or transdisciplinary processes of
appraisal of societal relevance and ethical acceptability (presence/frequency; qualitative
descriptions; best practices).
The concept of RRI being fairly recent in EU policy, the expert group expects that it will undergo a
dynamic development in the years to come. In light of the Rome Declaration on RRI and the
Europe 2020 ambition to achieve smart, sustainable and inclusive growth, the expert group
anticipates the future relevance of sustainability and social justice/inclusion as potential key areas
to promote and monitor in the context of RRI.
On sustainability: RRI indicators for sustainability should address the following question: to what
extent does a research field, a research programme or an RRI initiative contribute to sustainable
growth? While there is a growing body of relevant research-based knowledge, there is also a need
for further research to develop novel indicators for this field. One option would be to develop an
indicator framework that monitors R & I activities in terms of their effects on the socioecological
metabolism of the EU and the earth. This would entail monitoring stocks, flows and funds
(renewable and non-renewable resources; their rates of consumption and regeneration; the impact
of labour and technology on these rates), as well as ecosystem services and their effect on human
well-being. Process indicators can be defined in terms of milestones on specified pathways that
have an effect on specified stock-flow interactions, while R & I actors perceptions may be indicated
in terms of their anticipation and imagination of pathways, milestones and the ultimate effect on
specified stock-flow interactions. As of today there is no obvious place for such indicators in current
policy practice; however, the expert group has found it reasonable also to take into account
possible future developments in the concept of RRI.
On social justice/inclusion: While social justice directly in the context of research activities can be
considered from two perspectives: (i) the relationship between the researchers and the research
subjects; and (ii) the participation of social groups in benefits arising from research, the group
concluded that the issue that should be monitored is the impact of research and its effect on social
justice/inclusion. Issues that could be monitored here include, amongst others, the access and
affordability of products and services developed as a result of R & I activities for different social
groups. Measuring the impact should focus on two issues: (i) whether researchers consider at all
the impact of their research on social justice; and (ii) whether they have taken any steps either to
extend the impact of their research to a larger population or to minimise potential unintended
negative consequences in relation to social justice.
In part 3, the group concludes that it cannot offer a general prioritised list of indicators for actors in
the European Research Area. National and regional actors, universities and research institutes, civil
society organisations, funding agencies and others should devise their own process of deliberation
in order to choose and tailor the indicators proposed in Chapter 2, and add their own indicators
according to their own needs, goals and concerns. For this purpose the group distinguishes
between three levels.
First, there is the level of individual RRI criteria, such as PE, gender equality or sustainability.
Although the eight criteria overlap to some degree, each of them is subject to its own policy
development, policy action and monitoring. At this level, we recommend that attempts be made to
make full use of the suite of indicators proposed by the report. Secondly, there is the level of
concern for the successful implementation and development of RRI as a cross-cutting principle of
Horizon 2020. For those working at this level, the full set of 100 indicators is unlikely to be
practicable or even interesting. Rather, one should choose a smaller set of indicators that ideally
should:

include indicators for all eight criteria for RRI;

have a balance between outcome, process and perception indicators;

focus on performance (outcome and process) indicators for states of affair that are targeted (or
should have been targeted) by major RRI actions and initiatives;

be meaningful and informative to various R & I actors and conducive to good processes that
promote and develop RRI as a policy principle that is, rather than the emphasis being on
hard facts, chosen because they are easy to quantify, to be fed into an illusory command-andcontrol mode of governance, it should be on information that is helpful in collaborative modes of
governance, developing trust, best practices and mutual institutional change.

Finally, the Directorate-General for Research and Innovation should make its discretional choice in
the identification of this smaller set, as it is the directorate-general and not the expert group that
owns its policy priorities. Based upon our knowledge of the current policy context and the mandate
for the expert group, the group makes a proposal for such a smaller set. This is given in the table
at the end of this report.

1. Responsible Research and Innovation activities and their

monitoring via indicators: principles, gaps and challenges
1.1.

Objective and purpose of the expert group and this report

The European Commission, EU Member States and associated countries have launched various
initiatives and activities under the name of Responsible Research and Innovation (RRI), a term that
found its way from academic literature into R & I policy in 2011. Above all, RRI has acquired
prominence by its status as a cross-cutting issue of the EU framework programme for R & I,
Horizon 2020, as well as its central place among the objectives of the Science with and for society
programme within Horizon 2020.
The concept of responsibility is easy to endorse and difficult to define. Prima facie, to act
responsibly entails two aspects: being responsible and being seen to be responsible. The act
requires care for both actions and perceptions, as aspects of the modality of the act. Irrespective of
the particular choice of definition of RRI the complexities of which we shall return to below
this newly shaped principle of policy comes with its own objectives and purposes, which at its most
general may be stated as follows: the purpose of RRI is to achieve a better alignment of R & I
programmes and agendas with societal needs and concerns. As with any other policy, it is part of
good governance to monitor and assess the impacts that may result from the RRI initiatives and
activities. The specificity of the purpose of RRI, however, warrants attention and care in the choice
of methods for the monitoring and assessment of RRI impacts.
Consequently, the European Commission appointed in 2014 an expert group to identify and
propose indicators and other effective means to monitor and assess the impacts of RRI initiatives
and evaluate their performance in relation to general and specific RRI objectives (terms of
reference). This report presents the results of the work of the expert group. In its final chapters,
the report presents a number of concrete proposals for indicator design and implementation.
In order to arrive at concrete and attainable indicators, it is necessary to have a precise
understanding of the outcome variables (impacts) that the indicators are supposed to indicate.
While this may appear trivial in the case of chemical indicators or at least in some cases
performance indicators in a production line, it is anything but trivial in the case of RRI impacts. As
a policy principle, RRI is young and unconsolidated in the sense that there is neither an
authoritative definition nor a consensus on how to understand it. The same can be said about RRI
impacts, which are wrought with their own uncertainties and indeterminacies. It is difficult,
however, to specify a precise, valid and robust indicator for something that is imprecise and
changing. In order to achieve its objective, the expert group accordingly has had to clarify the
concept of RRI to make it fit for measuring and monitoring. Furthermore, we have had to clarify
the nature of the impacts to be indicated and monitored, as well as their type. The results of this
work are presented in the first two chapters of the report.

1.2.

Responsible Research and Innovation: concepts and practices

The main task of the expert group is to identify and propose indicators that can measure impacts of
RRI initiatives, RRI actions and RRI activities. Before we can specify the type of impacts and the
opportunities and methods to measure them, we need to know what an RRI
initiative/action/activity is and how it can be recognised.
RRI does not correspond to one fixed definition. First and foremost, it is now a term in the legal
text of Horizon 2020 from the European Parliament and the Council, introduced as follows in
Preamble 22 (our emphasis).
(22) With the aim of deepening the relationship between science and society and reinforcing public
confidence in science, Horizon 2020 should foster the informed engagement of citizens and civil
society in R & I matters by promoting science education, by making scientific knowledge more
accessible, by developing Responsible Research and Innovation agendas that meet citizens and
civil societys concerns and expectations and by facilitating their participation in Horizon 2020

activities. The engagement of citizens and civil society should be coupled with public outreach
activities to generate and sustain public support for Horizon 2020. (European Parliament and
Council 2013)
In 2012, the European Commission explained RRI as follows.
Responsible Research and Innovation means that societal actors work together during the whole
research and innovation process in order to better align both the process and its outcomes, with
the values, needs and expectations of European society. RRI is an ambitious challenge for the
creation of a Research and Innovation policy driven by the needs of society and engaging all
societal actors via inclusive participatory approaches. (European Commission 2012b)
The Commission then went on to specify that the RRI framework consists of six keys, as shown
below.
1.
2.
3.
4.
5.
6.

Public engagement.
Gender equality.
Science education.
Open access.
Ethics.
Governance.

This expert group has been asked to propose indicators for these six keys. At the same time, EU
policies and scholarship on RRI extend beyond these keys. This can already be seen in the first set
of work programmes of Horizon 2020, in which RRI as a cross-cutting issue is being interpreted
and implemented in a variety of ways. Here is an example from a Horizon 2020 work programme
on food security.
Proposals should address Responsible Research and Innovation aspects by taking account of
specific nutritional requirements, dietary behaviours and preferences, sensory aspects, the gender
dimension, ethical, socioeconomic and cultural aspects.
The effort to take specific nutritional requirements, dietary behaviour and preferences of citizens
into account in ones research is not unrelated to the six keys. Aspects of gender equality, public
engagement (PE) and ethics can clearly be invoked in this case. However, the effort cannot be
subsumed to any particular set of the six keys, at least not as they are usually understood and
practised. Still, it can easily be argued that the call indeed is an effort to develop an R & I agenda
that meets citizens and civil societys concerns and that tries to align it with the values, needs and
expectations of EU society. In this sense, the call makes a reasonable implementation based on an
interpretation of RRI as a principle and a concept in itself and not as a list of six keys.
We draw two preliminary conclusions here. First, we set out the six keys for RRI, but at the same
time we should be open to the possibility that indicators for RRI can also be found outside these
keys. As RRI is a dynamic concept, other ways might occur to implement RRI as a cross-cutting
issue and support the dynamic development of RRI policies and practices. Second, it makes sense
to have the monitoring of the development of RRI agendas as a primary, overarching
indicator. This would fit with the current phase of development in which much of RRI policy is, and
would enable us to compare the different practices and the intentions behind them more easily.
Meet civil societys concerns and expectations and align R & I with the values, needs and
expectations of EU society are, however, formulations that are too general to allow for monitoring
and the design of meaningful indicators. One EU society is something that one might aspire to, but
that does not yet exist. Nevertheless, one can have a discussion on what key EU values are or
should be, but then one would have to acknowledge the different perspectives and stages of RRI
development that exist in the EU. Moreover, the indicators should be able to provide meaningful
information also when applied to initiatives and practices that are not necessarily labelled RRI.
Obviously, the issues of responsibility are also important in the vast majority of EU R & I that is
neither funded by Horizon 2020 nor subject to the RRI initiatives executed from within Horizon
2020. In addition to the explicit policies of the EU, the expert group highlights two sources of
knowledge as particularly useful in its work to obtain a better precision of RRI, as shown below.
(a) We have built upon the work of the Expert Group on the State of Art in Europe on Responsible
Research and Innovation (European Commission, 2013). While that expert group did not
10

propose a succinct, operational definition of RRI, it did explain the concept of RRI as referring
to ways of proceeding in research and innovation that allow those who initiate and are involved
in the processes of research and innovation at an early stage (A) to obtain relevant knowledge
on the consequences of the outcomes of their actions and on the range of options open to them
and (B) to effectively evaluate both outcomes and options in terms of moral values (including,
but not limited to wellbeing, justice, equality, privacy, autonomy, safety, security,
sustainability, accountability, democracy and efficiency) and (C) to use these considerations
(under A and B) as functional requirements for design and development of new research,
products and services (European Commission, 2013; p. 55). Furthermore, RRI requires the
research and innovation process to be designed in a way that allows for the consideration of
ethical aspects and societal needs. This implies an issue orientation of research and innovation
and calls for stakeholder involvement in these processes. Accordingly, the group saw RRI as
signified by being anticipatory, inclusive, reflexive and responsive. It should be added
that this explanation of RRI on one hand characterised by a focus on ethical aspects and
societal needs and on the other a set of norms or even virtues for practice is highly
consistent with the research literature on the subject.
(b) We have considered the following proposed definition of RRI: a transparent, interactive
process by which societal actors and innovators become mutually responsive to each other with
a view on the (ethical) acceptability, sustainability and societal desirability of the innovation
process and its marketable products (Von Schomberg, 2011). Ethical acceptability and societal
desirability are also large and general concepts. Von Schomberg has proposed that they may
be operationalised in an EU context by holding R & I agendas and practices up against the
Charter of Fundamental Rights of the European Union. It may be argued that this is what has
been done by the Europe 2020 strategy in its goal of smart, sustainable and inclusive growth.
In conclusion, the responsibility in RRI is a matter of outcomes as well as characteristics of the
processes that lead to the outcomes, and so we are considering indicators both for outcomes and
for processes. We focus on the interface between R & I and the society in which it takes place, and
hence we have considered indicators both of actors and action within the R & I sector, but also the
perception by other actors and society in general. Clearly, the dynamics of interface and interplay
between actors differ from country to country and from sector to sector. Finally, it is a matter of a
number of substantive values that to some degree are captured by the six keys, but not in an
exhaustive manner. In particular, the aspects of sustainability and social justice/inclusion deserve
special attention, not only because the EU has committed itself to these aspects on the most
general level (in the Charter of Fundamental Rights) but also because they are central to the
Europe 2020 strategy of smart, inclusive and sustainable growth to which Horizon 2020 (and,
consequently, RRI policy) is a means. They also speak to the political guidelines for the new
Juncker Commission, which present an agenda for jobs and growth that has a clear eye for fairness
and democratic change. Accordingly, the expert group has considered future indicators addressing
the following additional aspects.
1. Sustainability.
2. Social justice/inclusion.

1.3.

The policy context: promoting and monitoring RRI

The purpose of indicators is to support and develop good governance of the sector of society that is
being monitored. Good governance of the European Union is a matter of five substantive principles:
openness, participation, accountability, effectiveness and coherence (European Commission, 2001).
Effectiveness also demands that knowledge about how the R & I sector works and can be governed
be taken into account.
Ever since Vannevar Bush (1945) declared science to be the last and unlimited frontier, modern
states have treated the R & I sector as a primary force in the development of societies. Bush
recommended governance in which it was a governmental task to supply the R & I sector with
ample funding but to leave most of the concrete decisions to academic freedom. For Bush and his
contemporaries, the value of research was unquestionable. In the final quarter of the last century
this principle started to waver, and since the beginning of the 21st century national governments
and the Commission have been demanding evidence for the value that research has for society. In
the knowledge society research is not only seen as a source of new knowledge, indispensable for

11

innovation, but also as a way to improve policymaking, and more specifically education and
learning, and socioeconomic welfare and quality of life in general.
Scientific research helps us to understand the dynamics behind the societal challenges that we are
currently facing, such as health issues, climate change, clean energy, social exclusion, etc., and it
is expected to offer us solutions that can be implemented at the local, national, EU and global
levels. The purpose of RRI policy is to help achieve these solutions in interaction with relevant
stakeholders in society in ways that adhere to democratic norms and values (ethical, sustainable,
transparent, accountable, etc.). The governance of science and innovation then becomes of central
importance in this process. The question is, how does governance work in such dynamic and
heterogeneous networks?
Kuhlmann and Rip (2014) point out that the grand societal challenges that are formulated by the
Commission create another grand challenge for R & I policy and practice because these challenges
are of a different nature than science, technology and innovation (STI) concerns in the past.
Instead of clear targets (man on the moon, AIDS vaccine), current challenges are much more open
ended and often concern the socioeconomic system as a whole, and might even require system
transformation (healthy ageing, inclusive society, etc.). Grand challenges, in their view, pertain to
heterogeneous elements and forces, which have to be mobilised, guided and integrated, and
include social innovation. Many different actors need to be involved, and the perspectives on what
is the problem and what constitutes its resolution differ across various societal groups.
What is the implication of Kuhlmann and Rips analysis? First of all it may be taken as a warning
against too strong beliefs in top-down governance of R & I in which top-level authorities set
objectives for the sector, implement them through top-down policies and expect them to be
achievable and measurable in a linear fashion. Research and innovation is a complex system and
governance in complexity is a wiser strategy than an attempt at governance of complexity.
This quite general point is confirmed empirically by science policy studies. Impact assessment
methods for scientific research have been widely studied and practised (see for instance Stevens et
al., 2013). Impact studies can be divided into two broad areas, those that focus on the effects of
scientific research in the scientific community and those that look at wider effects in society, that is
industry, public organisations and the public at large. The former type of impact is usually
measured through some kind of bibliometric method and therefore limited to fields that
communicate their results through international journals. There is growing opposition worldwide to
the use of such methods, in particular methods based on the impact factor of journals (San
Francisco Declaration). It has been argued that such measurements have serious flaws (for
example an article profits from the impact factor of the journal it is published in, whether it is
read/cited or not), and also that the use of these quantitative measures distract attention from the
quality of research. Nevertheless, such measures are attractive for policymakers because they turn
huge and complex datasets into simple numbers. However, policymakers are also aware of the
changing context, and evaluation systems tend to become more open and comprehensive, for
example with criteria emphasising societal impact next to scientific publications, as the new Dutch
Standard Evaluation Protocol 2015-2012 shows (2). The other type of impact study regards the
societal effects of scientific research. Here one can find a plethora of approaches and methods, and
a slowly growing consensus on what is important and what is not (see for example Merkx, 2007;
Spaapen and Van Drooge, 2011). Contextual variation between research fields is part of the reason
why consensus is difficult to reach: the societal impact of research in, for example, the medical
fields takes place in a context of pharmaceutical industry, patient organisations, professionals
(doctors, nurses, etc.), insurance companies, legislators, etc., while architects, for instance, will
have to deal with city governments, urban developers, citizens, small businesses, building
companies, etc. But there are other constraints too. Among these, the two most mentioned are
temporality (it may take up to 15 years before a research idea materialises in some kind of
application) and attribution (impact is the result of a network of interactions between a variety of
stakeholders; contribution is arguably a better concept here). Two provisional conclusions may be
drawn here. First, it is imperative to raise awareness and understanding of the R & I network
context, and in particular of the kind of interactions between the most important stakeholders in
that network. Second, it would be more realistic and valuable to focus on short- and medium-term
effects/goals; on intermediate impacts. Taking these two conclusions together, the emphasis of

( 2)

http://www.vsnu.nl/sep
12

impact evaluation is shifting from (end) product to process, and from verdicts/judgments to
learning and improving.
In such a view, the concept of impact needs to be adapted. Research and innovation takes place in
a societal context in a process of interaction between multiple stakeholders, and the outcome of
this process is social innovation, i.e. a mixture of technological, behavioural and institutional
changes. The linear concept of impact evaluation needs to be replaced by concepts that represent
the interaction in the network in which R & I takes place. The success of the various RRI topics and
aspects then becomes a joint responsibility, and governance takes place in a decentralised context.
In its 2007 report on the evaluation of the framework programmes, the European Court of Auditors
concluded that the practices the European Commission built into the programme to collect data on
dissemination and use of results were inadequate and applied at the wrong time (Special Report
No 9/2007). Since then, the situation has improved considerably, and funding bodies at national
and regional levels and research-performing organisations are trying to learn from research done
on this topic. In the seventh framework programme (FP7) Siampi project, which aimed at finding
indicators for societal impact measurement, a network approach was developed that focused on
productive interactions between research and its societal environment (Spaapen and Van Drooge,
2011). Starting from a common mission to address a particular problem in society, the method
identified various sorts of interaction between relevant stakeholders in three categories: (1)
personal contacts, (2) media exchanges and (3) financial and material support. In other words, in
this approach the process of R & I in its societal context was captured. Furthermore, the method
entailed a search for instances of societal impact (good practices) and the involvement of
stakeholders in developing R & I agendas. In that, Siampi combined a quantitative approach with a
qualitative one, and this combination also raised the commitment of the involved actors. Arguably,
some of the key elements of this approach can be valuable for the development of RRI indicators.
These regard the focus on process indicators, collaboration between stakeholders in developing a
joint mission (for example on gender equality or PE), the identification of instances of societal
impact or intermediate indicators (goals that need to be reached in 2 or 3 years).
To summarise, we emphasise the value of indicators that document interaction on a short and
intermediate time scale (and they do not necessarily have to be of the process type). Even more
importantly, we recommend that the use of the indicators be taken to embody the principles of
good governance, notably openness and participation through a network approach rather than a
linear, top-down chain of command. In this way, indicators may help promote RRI by facilitating
monitoring at a variety of levels, but also more directly by offering criteria in the continuous
development of practice.

1.4.

Intended and unintended consequences of monitoring

Continuing the line of reasoning from the previous section, part of our recommendation is to offer a
warning about the potential risks of the use of indicators. There is quite a rich discussion in
various fields of enquiry examining the unintended consequences of using indicators of all kinds (3).
Implementing an indicator system is one thing; observing whether it is achieving its goal is another
(for an early observation to this effect see Merton, 1937).
The use of indicators to evaluate R & I that gained popularity in the second half of the 20th century
can be seen as the consequence of the fact that science has become big business, following the
knowledge societys demands for a highly educated labour force and excellent research to be able
to compete on a global scale. Universities have been transformed from elite institutions to mass
education and research enterprises in which it is no longer deemed possible to rely on mutual trust
to assess performance. Rising levels of input and output can only be reviewed with the use of
aggregated data and indicators based upon them. The consequential systems of performance

( 3)

The San Francisco Declaration on Research Assessment (DORA), initiated by the American Society for Cell
Biology (ASCB) together with a group of editors and publishers of scholarly journals, recognises the need
to improve the ways in which the outputs of scientific research are evaluated. The group met in December
2012 during the ASCB Annual Meeting in San Francisco and subsequently circulated a draft declaration
among various stakeholders. It is a worldwide initiative covering all scholarly disciplines.
13

indicators are often perceived as an expression of mistrust from those who will use these indicators
towards those who are performing; the observer mistrusts the actor to regulate his or her affairs
autonomously. Where there was self-regulation, there shall be oversight. Thus, systems replace
trust with a set of indicators of oversight. Where there is trust there is no need for such indicators.
Indicators can be destabilising and take away legitimacy from current practices. Constructing an
indicator system is an audit process in order to provide assurance to an outside and overseeing
observer that things are done responsibly and are moving in the right direction. However, to
strengthen the legitimacy and use of indicators, it is necessary that the R & I community somehow
assumes a sense of ownership; then the two dimensions involved in this process performance
and perception will arguably get a meaningful interpretation and implementation.
Performance indicators are supposed to shape behaviour and practices in some desirable direction
in our case into a system of R & I that acts responsibly. In an ideal world, a system of
indicators does contribute to this steering, but there a number of ways in which this goal might be
missed.
Colonisation. To be responsible requires autonomy to act in the responsible mode. However, the
required compliance with a set of criteria reduces this autonomy, and it might in the end no longer
be clear whether the actor has retained autonomy or only compliance. In other words, the
imposition of a system of performance indicators is always the imposition of external control. This
attempt to gain external control over a set of activities will create resistance and attempts to
counter-control.
Colonisation is not only an imposition from outside, but might be welcomed by some insiders who
will use the indicators to try to change the organisation in line with their own design. Indicator
systems are often welcomed as setting the right incentives, putting significance on things that were
hitherto neglected and helping create new mentalities and a culture change, which are all
considered good things in the right direction. However, as the saying goes, the road to hell is
paved with good intentions.
Decoupling: perception only, no performance. This refers to cases when a system is entirely
compliant to external criteria and thus changes its behaviour, but only in relation to the indicators,
not in the direction the indicators actually want to achieve. Behaviour is changed, not in line with
responsibility, but in a perfunctory ritual of compliance. In extreme cases, an organisation can set
up a team of people to produce data for the indicators, but these data have little or nothing to do
with what happens on the ground. The changing behaviour is not correlated to the overall goal.
Distraction from the real thing: working towards the measure and not towards the goal
of activity. An effect of measuring human activities relates to the fact that performance measures
are always approximations of a goal concept; it is an operationalisation that is convenient and costeffective, rather than perfect. This leaves any measure with a gap between data and concept/goal.
But the measure easily becomes the goal, and the original purpose gets lost and falls out of sight.
The measure takes the place of what it purports to measure; the means becomes the goal.
Unintended consequences of indicator systems. Measuring an activity on something that is
measurable can have perverse effects by setting incentives for part activities that were not
intended to be enhanced. For example, in the university sector, the quest for measuring research
excellence as a fair way of distributing money to the right people has led to the incentivising of
write more and teach less to such an extent that teaching quality has collapsed, which requires
another set of indicators on teaching quality, which then separates teaching excellence from
research excellence and leaves little time to be good citizens of the university, which then requires
an increase in administrative staff, who operate to different performance criteria such as providing
performance indicators for academic staff, and so on.
Erosion of intrinsic motivation with incentive schemes. A psychological effect of measuring
activities by an external criterion is the erosion of intrinsic motivation. If someone does something
naturally and in an unquestioned manner out of habit, pleasure or conviction, that is called intrinsic
motivation. If a system now begins to assess and reward these actions, the person may begin
focusing on the assessment, and the pleasure in the activity as such is eroded. In substituting
intrinsic motivation with reward it will become much more costly to sustain the same level of
achievement. This is well known in the logic of work design and work incentive systems.
Economists talk of the crowding out of intrinsic motivation (Sandel, 2012, p. 61 et seq.).

14

The costs of collecting indicator data. Collecting indicator data will have costs either for the
agency which collects the data, for example the Commission, or for the institution that is to provide
these data, for example via a questionnaire to fill in. Filling in a questionnaire will take somebodys
time and effort. Costs arise from the types of data but also the level of precision that is asked for.
More precise data will take more time to collect and its collection is likely to be more costly. Any
audit process at some point will cost more than the additional benefits that can be expected from
using the data. Auditing follows the rule of diminishing returns: to have some data is better than
no data; to have more data is not necessarily cost-effective (see Power, p. 77).

1.5.

Monitoring RRI activities and their impacts with indicators for outcomes,
processes and perceptions

The general objective of the expert group is to help the Commission identify existing indicators
and to propose new indicators that can measure impacts of RRI activities in qualitative and
quantitative terms (terms of reference). RRI activities and initiatives in the context of Horizon
2020 include:

the introduction of RRI as a cross-cutting principle in Horizon 2020 in its entirety;

specific actions and initiatives from the Commission to implement RRI as a cross-cutting
principle in the various work programmes of Horizon 2020;

activities funded by the various work programmes of Horizon 2020 in which RRI aspects have
been part of the objective or expected impact of the call;

specific actions and initiatives by DG Research and Innovation on the six keys of RRI;

the implementation of the Science with and for society programme of Horizon 2020;

the activities funded by the Science with and for society programme of Horizon 2020.

Many other activities and initiatives can be recognised as RRI activities, however, in particular in
the context of R & I policy and practices outside of Horizon 2020, on an EU level and in the various
Member States and associated countries. Examples would include the plethora of PE initiatives in
various countries; institutional practices for ethics review; gender equality legislation, regulation
and practice; the funding and/or requirement of including ELSA (ethical, legal and social aspects)
research in research on emerging science and technology; etc. (4).
One assumption underlying our objective is the possibility of causal relationships between RRI
activities and initiatives and their impacts or effects on R & I in terms of its degree of becoming
(more) responsible. The objective could in principle be addressed from two angles. One could focus
on the RRI activities conducted by the Commission in the context of Horizon 2020, and try to map
and evaluate the range of their causal effects. Such a task would be quite difficult. It would not
only be a matter of monitoring the state of RRI in terms of outcomes, processes and perceptions;
it would also have to demonstrate causality, which in itself is a research task of its own. The
second angle regards a focus on indicators for impacts referring to outcomes, processes and
perceptions of RRI. The causality between a given RRI activity and a given RRI impact would then
be an empirical question to be investigated in part by use of the indicators, but also by a careful
study of other activities, initiatives and developments that might influence the state of affairs. The
goal of such an operation would then shift from accountability to learning from developing practices
and perceptions, which arguably would help to build a European Union RRI awareness and culture.
We believe that the latter angle is the better one given the stage of development of RRI, also
because it would enlarge the chance of genuine ownership by the R & I community.
In the following paragraph we operationalise the conceptual implications above into a framework
for RRI indicators. It follows from the clarifications that responsibility is linguistically and
substantively an attribute of R & I. Horizon 2020 requires this attribute of R & I in addition to

( 4)

Other examples are national RRI initiatives by funding bodies such as the Engineering and Physical
Sciences Research Council in the UK, the Rgion le de France in France (AIR, Repere) and the Dutch
Research council NWO (Netherlands Organisation for Scientific Research).
15

the objectives of promoting good/excellent/relevant research. This means that rather than simply
and automatically being valuable in themselves, to perform R & I in the EU context they have to
answer to some additional specific criteria of responsibility. Perform should be taken in a broad
sense, both in terms of the performance of individuals and organisations and in terms of the
performance of the network in which the innovation takes place. Researchers and research
institutions, as well as innovative enterprises, are clearly relevant actors in R & I networks, but so
are potentially many others (financiers, regulators, policymakers, overseeing institutions, etc.).
Producing RRI thus becomes the result of the collaborative effort of actors in a network.
While this can make the assessment of RRI a demanding exercise, we believe that it can be done in
an organised way, as shown in the matrix form below. The columns specify how R & I is perceived
and performed (in terms of processes and outcomes) by which key actors. The rows specify the
criteria by which the perception and performance is judged to be indicating responsibility or not.
Defining the rows: criteria of RRI. Our report so far has presented eight criteria, namely the six
keys of RRI and the two additional aspects of sustainability and social justice. More criteria could
have been suggested and we do not claim that the list necessarily should be limited to the ones
presented in this report. We anticipate that the list of criteria will develop as the concept and
practices of RRI develop.
The eight criteria we use show some overlap (which will become clear in the next chapter), but
they also differ in a number of ways. Some are more connected to the implications of R & I while
others refer more to the intrinsic features of R & I practice. Some are quite specific (e.g. open
access) while others are more general and overarching (e.g. ethics and governance). Indeed, we
were in doubt as to whether to include governance as a separate criterion (in the sense of a row in
the matrix) or to monitor governance in the context of RRI mainly in terms of process (i.e. a
column in the matrix). When we have kept it as a separate criterion, this is also because a certain
redundancy in the indicator framework may provide flexibility.
We believe that RRI indicators can be relevant to a number of policy levels and contexts.
Accordingly, we present a generic framework that may be used meaningfully beyond the particular
task of supporting the Commissions need for evidence-based policymaking in the context of
Horizon 2020, across geographical scales, political levels and sectors of research, science and
innovation. The RRI indicator framework is designed to be generally relevant throughout the
European Research Area. For this reason, a flexible framework is desirable. This means that we
see the framework as a toolbox more than a tick box. Users should use this framework to
pick and choose those indicators that fit their activities and those of their R & I network the best.
What counts is that they show their RRI performance in a way that makes sense in their context.
That is why they should do this together, in a process that will arguably also raise the commitment
among stakeholders for RRI. The choices they make in this process are not inconsequential,
because the set of indicators they choose form a specific framework of accountability.
Defining the columns: performance, perceptions and key actors. To be responsible in
general and in the specific terms of RRI includes three dimensions: performance, perception and
key actors. RRI performance depends both on the processes that promote RRI activities and on the
effects that these processes have: outcome. Acting responsibly defines who we are we are
acting in a certain manner (performance). But responsibility also includes the key element of
perception to be seen to act responsibly. The weight given to process, outcome and perception
indicators for a given issue depends upon the nature of that issue. Again, what we offer here is a
generic and flexible framework that in itself can be applied responsibly as well as irresponsibly.

16

Performance indicators
Criteria

Process
indicators

Outcome
indicators

Perception
indicators

Key actors

Public
engagement

Gender equality

Science education
Open access
Ethics
Governance
Sustainability

Social
justice/inclusion

Table 1: Indicator framework for Responsible Research and Innovation.

The way this framework works depends to a large extent on the key actors. We hope that by
developing this together, therefore bottom up instead of top down, not only will the commitment
grow but it will also benefit the implementation and efficacy of the framework. RRI and its
evaluation are also about ownership. Therefore, it is up to the key actors to put the topic of RRI on
the agenda and to develop a responsible framework for impact measurement that fits the purpose
of their network in the best possible way. This includes the identification of RRI activities relevant
to all stakeholders and the selection of indicators or other effective means that measure impact.
Stakeholders should find ways to discuss the best set of indicators for their purpose and make sure
that robust data are going to be available to monitor the impact of RRI activities. This is easier said
than done, however we believe that RRI is only possible if stakeholders collectively agree about its
necessity, and consequently feel responsible for RRI being an integral part of their activities. To
make things easier in developing this tool box, it would be good to start with the first two columns,
the process and outcome indicators. This is relatively easy, because quite some indicators are
already available there, and we will suggest some new ones. Regarding the other two columns, it is
of course necessary to identify the key actors and their roles in the RRI process, but they do not
necessarily need to be captured with indicators (although network techniques might be helpful to
identify stakeholders who are not usual suspects). Finally, the perception indicators are, in our
view, important, but more in some cases than in others, in particular when controversial subjects
are researched, for example fracking or EHEC bacteria. Decisions about this should also be made in
joint meetings of involved actors. In what follows we have summarised suggested indicators for
most topics in a reduced table, either with the three indicator columns or in some cases only two of
them. The detailing of key actors is, as explained, an exercise that needs to be done for the
specific policy context and preferably by an inclusive process.

17

2. Indicator criteria: the six keys and beyond

As explained in Chapter 1, the expert group has decided to review the criteria for RRI indicators
and propose options for monitoring indicators for eight criteria of RRI, which include the following.
1.
2.
3.
4.
5.
6.
7.
8.

Governance.
Public engagement.
Gender equality.
Science education.
Open access/open science.
Ethics.
Sustainability.
Social justice/inclusion.

As will be evident from the analysis below, the set of criteria is diverse and heterogeneous. At the
same time, there is also overlap between criteria, and some (in particular ethics, sustainability and
social justice/inclusion) may be thought of as being more overarching and encompassing than
certain others (PE, science education and open access). Governance is in part a criterion of its own,
in part an aspect of all criteria. That is why we begin our proposition with the governance criterion,
because in good governance lies the key to success of all the other aspects of RRI, while bad
governance may create obstacles for even the best analysis of criteria and how they may be
indicated, monitored and used.
Some key indicators of RRI proposed in this report will be experimental in nature. This is a
consequence of acknowledging the need for moving beyond command and control towards a more
dynamic governance of science in society, as it was put in FP7. It is also underlined by our
preference for a network approach and the lack of currently accepted indicator set. This is true in
particular for the more overarching criteria, such as governance, ethics and sustainability, as will
be seen below.
In what follows we address each criterion in turn by first reviewing the current state of
development of the key and discussing some of the main elements and issues involved, and then
proposing relevant process, outcome and perception indicators. In the final section of this chapter
we summarise our analysis and propose a framework for indicators.

2.1.

Governance

Current state
In the first chapter we noted that the R & I process is characterised by collaborative efforts of a
variety of stakeholders who each have a particular interest in this process. Overall goals are usually
formulated in general terms and therefore arguably meet consensus among most stakeholders (for
example green transport, healthy ageing), policymakers may encounter difficulties in the control
and organisation of this process, including intellectual, financial and other material contributions.
The question of how to govern such R & I networks from the perspective of funding bodies and/or
government (local, national and supranational) is rapidly transforming from policy perspectives
based on central control and accountability to a perspective where coordination and stimulation are
key concepts.
In the expert report on the global governance of science (European Commission, 2009),
governance was described as entailing multiple processes of control and management and
involving directing or setting goals, selecting means, regulating their operation and verifying
results. However, 3 years later, in the EU report on ethical and regulatory challenges (European
Commission, 2012), the focus of governance shifted to reaching a consensus in a network of
relevant stakeholders. In relation to governance in the context of RRI, this development is reflected
in the well-known definition of RRI by von Schomberg (2011).
Responsible Research and Innovation is a transparent, interactive process by which societal actors
and innovators become mutually responsive to each other with a view to the (ethical) acceptability,
18

sustainability and societal desirability of the innovation process and its marketable products (in
order to allow a proper embedding of scientific and technological advances in our society).
The question then, of course, is how such an interactive process can be governed, especially
because it is based on the assumption of trustworthy relationships among all societal actors. The
solution in our view has to be sought in the active participation of all relevant stakeholders in
developing an RRI policy. Frameworks in which stakeholders can collaborate to that effect are
developing at all hierarchical levels of the science and innovation system. The two aforementioned
EU reports regard relations at high aggregation levels (between nations), but also national and
local/urban level governments and other organisations see governance more and more from a
network perspective (5). However, while these frameworks offer opportunities, the awareness
about the importance of RRI is still underdeveloped in many cases. Consequently, the first priority
seems to be in raising the level of awareness in these frameworks, both in the sense of peoples
responsiveness to the RRI criteria and of the RRI rules and regulations within these frameworks.
Additionally, we want to make sure here that we understand innovation both in terms of the
scientific and technological advances that von Schomberg is referring to in the above definition
and also in terms of social innovation, that is new forms of organising social relations between
people and organisations. We see social innovation in a broad sense, referring not only to work but
also to politics, culture and the broad social sphere, including for example social media, crowd
funding and Google art. The interplay between social and technological innovation is more and
more becoming a central issue in the RRI criteria. For example, the replacement of people by
robots in the healthcare system raises not only the immediate issue of employment but also wider
questions of ethics, science education, sustainability, social justice and governance.
When we look at the question of indicators to describe and measure the governance of these
processes, linear models focusing on impact are not useful in describing the dynamics of the
interactions taking place between stakeholders. They also miss out on the temporality of
innovation, meaning the time it takes to develop new ideas, products, services, organisations, etc.
into practice. Innovation is a long-term iterative process; new ideas, products and services are
developed by a dynamic group of stakeholders that arguably might be very different at the start
than at the end. Ergo there is a need for indicators that fit into interactive approaches that do
justice to the contributions of the relevant stakeholders in the network. Government (and
governance) policies arguably need to pay more attention to supporting the activities in these
networks in order to optimise the outcomes, and consequently they will also have to reflect on their
own role and responsibilities.
In developing such network-oriented perspectives, we can rely on the one hand on a host of
theoretical perspectives that have been developed over the last 10-15 years on network
governance, following the declining force of the new public management theories. On the other
hand, work in the social studies of science can be helpful to develop perspectives on what has been
referred to as the new production of knowledge that takes place in these networks. Knowledge
production is multiactor, transdisciplinary and socially robust (Gibbons et al., 1994; Nowotny et al.,
2001). Participants in these networks come from academia, industry, public organisations and
society at large. These innovation networks are formal and non-formal publicprivate partnerships.
While these partnerships have been in existence for decades, they seem to become the main form
in current R & I policies. Governments in most EU countries and also at the EU level have
established policy schemes that stimulate the collaboration of academic research and industry and
other partners witness the top-sector policy in the Netherlands and the grand societal challenges
of Horizon 2020.
From a practical point of view, it would be helpful to look at examples from the emerging practices
and identify how these new kinds of heterogeneous networks are developed and coordinated
(governed). Furthermore, there are various projects that aim at developing interactive indicators
(e.g. Siampi and the Dutch ERiC project (6)). Also, social network and stakeholder analysis might

( 5)

( 6)

Examples from the Netherlands regard the top-sector policy, nine economic sectors with importance for
the Dutch society, which are jointly governed by science, industry and society, and economic boards that
some cities have established in which partners from different institutional background (universities,
professional schools, employers, policymakers) discuss innovation in the city and wider region.
For Siampi see http://www.siampi.eu; for ERiC see
http://www.rathenau.nl/en/themes/theme/project/eric-evaluating-research-in-context.html
19

be helpful to develop ideas about parameters that work in heterogeneous networks. In developing
RRI requirements, we would also have to consider that some indicators are likely to be more
meaningful on a higher aggregation level than others. For instance, governance indicators or PE
indicators are perhaps likely to be less meaningful on the lowest aggregation levels, while perhaps
some indicators monitoring the development of ethics and gender equality policies are more likely
to be meaningful on these levels (if only to raise awareness in the work space).

Towards indicators
Given the conceptual approach we are taking, the development of indicators should be network
oriented, and the outcome should be based on an iterative process in which various stakeholders
jointly decide what indicators represent the best kind of R & I that takes place in their particular
network. There are various methodological ways to do this, some of them based on self-organising
principles, others more guided, for example focus groups (like in the Siampi project) or Delphi-like
techniques, some of which are now available in digital forms to realise the participation of many
stakeholders in distributed settings.
If we look at some indicators currently in use and review them with regard to their usefulness, we
come to the following conclusions for each indicator. For most and perhaps all of them the basic
principle is that qualitative indicators give more insight than quantitative ones. Below we highlight
the strengths and weaknesses of individual indicators currently in use and present options for
improvement.
Percentage (number) of R & I networks that implement actions to promote RRI.
Trying to identify innovation networks clearly fits into the network approach. However, these
networks are inherently unstable and fluid, and therefore hard to define in any precise way. One
option would be to try and distinguish between formal and informal networks, because formal
networks are likely to have research and other goals laid down on paper. And they are likely to
have governing boards that to a certain extent can be held responsible for the development of RRI
policies. However, even formal networks are often more dynamic than more traditional research
projects. Our suggestion here is to use narratives and/or case studies to describe network activities
for RRI according to a fixed set of proxy indicators, for example the existence of budgetary items
assigned to RRI or policy implementation plans to develop RRI policy.
Percentage (number) of JTIs, JPIs and PPPs that apply RRI principles
The number of joint technology initiatives (JTIs) and joint programming initiatives (JPIs) is
available, and because it is a relatively small number, the use of percentages makes no sense.
Qualitative information about the implementation of RRI principles could perhaps be combined with
quantitative information about certain elements of RRI like gender equality goals. It should not be
too difficult wonder if it could be made mandatory to have these formal for these networks to have
an RRI policy, and have them report on RRI policy. For publicprivate partnerships (PPPs) this
might not be possible.
Millions of euros spent by networks of research funding organisations and industry for
co-financing specific projects with relevant RRI part V aspects
It will probably be very difficult to find reliable numbers for this indicator, especially from industry.
Besides, when the investment is in kind (people, sharing of facilities, etc.), numbers will perhaps
be even harder to get. On the other hand, it might be important to get some sense of the actual
input of participants, and in some cases proxy indicators are available or can be developed.
RRI contribution to Europe 2020, STEM literacy objective
It is unclear how this can be measured in a robust way, and what this might mean in relation to
specific RRI activities.

Undeniably, to monitor the impact of RRI activities in terms of governance, the main focus should
be on identifying networks of stakeholders and the ways they collectively assume the responsibility
of raising awareness regarding RRI and devising policies for the promotion of the key RRI elements
(PE, science education, gender equality, etc.).
In the following table we present some suggestions for how this could be done for the categories
we have distinguished above.

20

Criterion

Governance

Governance

Performance indicators
Process indicators

Outcome
indicators

Identification of
formal and informal
networks of R & I that
promote RRI, at both
the national and the
EU level

For each of these

networks:

Activities of funders to
promote RRI

number of RRI
debates

number of RRI
protocols

number of RRI
policies

number of RRI
agreements

Number of funding
mechanisms to
support RRI
activities
Number of euros
invested in RRI
projects

Perception
indicators

Key actors

Involvement of the
wider public in RRI
debates, measured
for example through
social media

National and
supranational
governments,
major
stakeholders in
science and
society

Involvement of the
wider public in RRI
policy, the
development of
policy, protocols

Number of
references in
applications to RRI

Funding
organisations,
stakeholders

Number of
collaborative RRI
projects

Table 2.1: Proposed indicators for governance.

2.2.

Public engagement

Current state
The desire to strengthen the relationship between science and society has brought developments of
PE over the past decades. From a narrow perspective centred on the need to educate society in
order to gain its approval for science and technology developments, to a perspective focused on
the quality and benefits of the effective participation of society, one can now find a range of
strategies, actions and activities regarding PE. This history is often told in terms of moving from a
focus on literacy to a focus on public attitudes, both embedding the idea of worrying deficits on the
part of the public, and then to a focus on a public dialogue that is more concerned with a deficit of
scientists and innovators and their institutions with regard to their dialogue with society. We
consider these shifts of foci less a history of progress, i.e. the new agenda displacing the older one,
than one of broadening the remit of science communication and the relationship between science
and society with different buzzwords (see Bauer, Allum and Miller, 2007). Annex 1 provides a
more detailed narrative of the development of the concept of PE.
We might define PE as a societal commitment to provide encouragement, opportunities and
competences in order to empower citizens to participate in debates around R & I, with potential
feedback and feed-forward for the scientific process. Deeper forms of engagement in science and
technology, where citizens are peers in the knowledge production, assessment and governance
processes, also deserve attention. This is described through non-equivalent expressions of different
degrees of agency such as citizen science, science in transition, do-it-yourself, fablabs, hacker

21

spaces, maker spaces, etc. many supported by the digital culture. PE is also a key element in
R & I policies in the EU (7).
However, there is little consensus on how to measure and monitor PE. There remains ambiguity in
the use of the concept and thus choice of measures. Preliminary research is inconclusive: opinions
and demands of the public are often ignored at the level of policymaking; bi-directional
communication is not a reality in many cases. There are differences between countries though,
both in definition and in implementation of PE (see Gonalves and Castro, 2009). In some countries
PE presents signs of fatigue (see for example Cooke and Kothari 2001; Horst, 2010), in others
engagement with science and technology never developed. The participatory turn of science and
technology appears contested by many scholars (Jasanoff, 2003; Wynne, 2007; Chilvers, 2008;
Saurugger, 2010).
Measures of public interest have been used to indicate PE as a measure of motivation. The
monitoring of media coverage has been used to indicate public interest regarding attention to
science. These indicators monitor in part the sciencesociety relationship, but do not yet indicate
the quality of dialogical forms of public participation. Indicators of public perceptions are probably
the best documented, but are poorly analysed (Bauer and Falade, 2014). They point to changes in
interest, knowledge and attitudes towards science and technology over the past decades (e.g.
Eurobarometer surveys on interest in R & I issues, trust in science, degrees of
optimism/pessimism, etc.).

Towards indicators
We consider the process of interaction between science and society and include the following three
dimensions to develop PE indicators:

policies, regulation and frameworks,

event/initiative making and attention creation,

competence building.

For each of these three dimensions, which partly overlap, we distinguish performance indicators of
process and outcome and perception indicators. A range of actors to which these activities and
commitments are attributable (see Table 2.2) is also proposed. Below we briefly explain the
rationale of these proposed indicators and highlight where existing or preliminary data are available
and where instead basic work on the construction of indicators is still required.

Dimension 1: Policies, regulation and frameworks

As process indicators for socially engaged R & I, we suggest looking at formal commitments as
evidenced by the mission statements of state ministries, regional or city authorities or universities
(e.g. the three pillars of research, education and outreach), or at the level of major research
projects. Comparative indicators have been described. See, for instance, the indicators proposed
by the projects MASIS (Mejlgaard, 2012) and MoRRI/Res-AGorA (8), or by Neresini and Bucchi
(2010). The focus is first on the types of commitments, and second on their number and whether
this makes sense. Content-oriented narratives might sometimes make more sense.
Outcome indicators are, for example, the level of PE funding as a percentage of R & I spend.
Examples of this might be found in the 5 % ELSE allocations of the Human Genome Project (1990-

( 7)

( 8)

Looking at the semiotics of these developments at EU level, we see through the preposition changes of the
sciencesociety research framework programmes that a deepening of such interfaces has been sought:
first science and society, then science in society, then science with society, and in Horizon 2020 science in
and with and for society (Owen et al., 2012). On the contorted history of attempts to measure the larger
contexts of science, see Godin (2005).
http://www.morri.res-agora.eu
22

2000), or similar allocations in FP7 and Horizon 2020 programmes. Miller et al. (2002) highlighted
as a best practice the case of Portugal, a country that in 1995 devoted 5 % of its research and
technological development spending to PE-related activities (Ciencia VIVA, including informal
science education, science public awareness and science centres). The most important outcome
indicators, however, must evidence the involvement of citizens, which can influence research and
technology developments (e.g. changes to the research agenda as a consequence of PE). Models of
this might be found in literature on how public debates and controversies changed nuclear power
policies on a global scale (Ruedig, 1990; Wynne, 2007).
Perception indicators can be found in various surveys, measuring public expectations of being
involved in public consultations. Such expected involvement cannot be taken for granted, and
might be a historical variable. Absence or declining public expectations of being involved might be
an indicator of the acceptance of technocracy (Castell et al., 2014, p. 98; Eurobarometer 2005,
2010 and 2013).

Dimension 2: Science events/initiatives and attention creation

Science events/initiatives and public attention raising have been strong development areas over
the past 20 years. Lists of these formats could be assembled comparatively easily, together with
additional quantitative and qualitative markers to construct process indicators. Key performers in
this are scientists themselves (Bentley and Kyvik, 2011), but increasingly these events are
outsourced. Formats include, without being limited to, science events (occasional or with an annual
cycle such as Science Weeks; see EUSCEA, 2005), public debates and any kind of participatory
format like consensus conferences (Einsiedel, 2014) and political referenda (Buchmann, 1995).
There has been an increase in the number of science museums, interactive science centres and
mobile exhibition spaces used as formal and informal settings for PE. Finally, two different
initiatives could also be included as particular formats of PE: crowd funded science and citizen
science. In the context of declining public funding, researchers and others are exploring new and
innovative ways of fundraising through mobilising the wisdom of the many in crowd funding
schemes. Through internet presence and offline campaigns citizens are invited to take a financial
stake in research ideas beyond the tradition of charitable giving and TV charity campaigns (e.g.
telethons). Astronomy, biology and other sciences are also increasingly experimenting with citizen
science, a deep form of engagement in which citizens participate in knowledge production by being
involved in data collection or/and actual research. Media coverage of science is subject to national
and international studies (Schaefer, 2012).
The outcome of these event and initiative performances would generally be assessed through
indirect figures such as indicators of public mobilisation, including mass media coverage in the
press or on radio or television of general or specific topics (9). Some indicators of short-term,
immediate attention are new social media references (on Facebook, Twitter or similar), which can
be monitored through keywords searches for intensity, thematic co-occurrence statistics and
sentiment analyses. The outcome of museum and other exhibitory activities is indirectly assessed
by monthly or annual visitor numbers. Also, more sophisticated measures of the science museum
activities and their outcomes and impacts have been or are being used (European Network of
Science Centres and Museums, 2008). Civil society activism can be assessed by membership
figures and mass media attention on particular groups. Many of the previous indicators are already
in use and give some insight about PE, but normally in an indirect and partial manner. Their use in
public policy will require the balanced selection of indicators of complementary aspects.
Perception indicators of event/initiative making can be found in individuals reports about taking
part in such events. Various surveys use items on a ladder of participation, including noticing an
issue in the mass media, talking about it with friends and family, sympathising with civil society
organisations dealing with the issue, being a member of a civil society organisation and
participating in events (see Mejlgaard and Stares, 2010; Revuelta, 2014). A portfolio of science
events/initiatives in a particular context a community, a city, a region will contribute to a
feeling of being engulfed by an atmosphere of scientific culture, such as a buzz of conversation
and an awareness of old and new developments.

( 9)

See the MACAS project: http://cordis.europa.eu/result/rcn/149614_en.html

23

Dimension 3: Competence building

Our third dimension of PE concerns competence-building processes. A key competence-building
process can be seen in the training of communicators and science mediators. Since the early
1990s, special science communication programmes have proliferated across the world at university
level. These programmes are variously documented, and in some cases evaluated over the long
term (PCST, 2014; ENSCOT, 2003; Miller et al., 2002; Mellor, 2013). They train students to
become specialist mediators between science and society, traditionally as science journalists, and
more recently as science communicators and science event/initiative makers. These programmes
are also involved in providing training modules for scientists and engineers as part of their basic
education. The penetration of science communication training into normal university science
curricula is an indicator of this competence-building process. There are also some initial
experiences of RRI training in higher education institutions (including PE training). It is not difficult
to imagine a future indicator measuring the penetration and development of such training.
The established competence, used as an outcome indicator, can be assessed in the level and type
of staffing of the communication function of large research projects, research institutes and
universities. R & I institutions are increasingly professionalising their communication function
beyond the traditional role of the press officer, including staff competent in science event/initiative
making. Here we consider the changing affairs of science journalism (see Bauer et al., 2012). A key
indicator of this competence is the degree to which this is performed in-house or outsourced. Here
we need to list and monitor the intercultural and multidisciplinary collaborations and partnerships
between natural and social sciences, or between sciences and designers and artists, as emerging
movements and strong trends (see for example the recently created University of Amersfoort in the
Netherlands (10). The expanding demand for science event making is increasingly covered by small
spin-offs or the diversification efforts of existing marketing and public affairs consultancies building
up science groups. As with most indicators, its interpretation will depend on the context and the
question to be answered. A high level of investment in external consultants may indicate a
commitment to high-quality communication work. However, it may more often than not also be
seen as not being conducive to RRI, in the sense that it creates a division of labour and an
increased distance from the public, and thereby reduces the sense of ownership and responsibility
among researchers and innovators.
Perception indicators are classical indicators of the public understanding of science. Such indicators
include knowledge of science in terms of textbook facts, methodological processes and awareness
of and beliefs about institutional functioning, thus citizens might be scientifically savvy to varying
degrees. Traditional literacy indicators, widely used across the EU, China, India and the United
States, along with the Organisation for Economic Cooperation and Developments Programme for
International Student Assessment (PISA) (Sjoeberg, 2012), are usefully reframed as competence
indicators of PE (Mejlgaard and Stares, 2010). Knowledge is not a driver of positive attitudes but a
cognitive component of public perceptions. Eurobarometer surveys thus speak of a local science
culture that is already comparable across the EU.
Engagement with science is constrained by attitudes, another classical indicator of the public
understanding of science. Under attitudes to science we consider, on the one hand, utilitarian
expectations invested in science (e.g. science will improve health, human welfare, life comfort,
jobs; science is part of the solution rather than the problem). On the other hand, attitudes to
science include more fundamental dimensions explored under the notion of instrumental-pragmatic
(DaVINCI) versus transcendental-aesthetical (Pascal) relations with the world (Blumenberg, 2010).
The latter is less well developed and under construction (Bauer, Shukla and Allum, 2012); see also
the MACAS project (11).

(10) http://universiteitamersfoort.nl
(11) http://cordis.europa.eu/result/rcn/149614_en.html
24

Performance indicators
Criteria

Process
indicators

Outcome
PE funding
percentage from
R&I

Policies,
regulation
and
frameworks

Formal
commitment

Science events
and cycles

Event and
initiative
making
Attention
creation

Referenda and
Danish-model
activities. Organised
debates
Museums/science
centres Informal
settings
Citizen science
initiatives
Crowdfunded
science and
technology
development

Public influence on
research agendas
Share of PE in
R & I projects
based on
consultation,
deliberation or
collaboration

Media coverage
Social media/web
2.0 attention
Museum visits
and impacts (on
visitors,
stakeholders, local
communities)
Civil society
organisation
activities and
impacts

PR staffing

Competence
building

Training of
communicators

Social scientists
collaboration

Training of
scientists/engineers

Inhouse/outsourced
consultancies

Mediators
Grass roots

The state of
science
journalism

Perception
indicators

Key actors

Public
expectations of
involvement
Researchers
openness to
pursue PE
Interest of
publics

Engagement
activities
(ladder)
Interest in
science
Issue
discrimination
Image of an
atmosphere of
scientific
culture

States
Regions
Cities
Universities
University
departments
Research centres
Research
projects
Sections of the
public
Civil society
organisations

Knowledge,
beliefs
Trust,
confidence
Attitudes
(utilitarian
expectations,
fundamental
orientations)

Table 2.2: Proposed indicators for public engagement.

In bold are indicators already developed at least to some degree and for which we can refer
to literature (scientific and/or grey literature); in black are indicators that still need to be
developed. See Bauer and Falade (2014) and Bauer, Shukla and Allum (2012).

25

2.3.

Gender equality

Current state
The European Commission has published the She figures (12) report every 3 years since 2003. The
reports, addressed particularly at policymakers, researchers and their employers, monitor human
resources statistics and indicators in the research and technological development sector and
gender equality in science. The main questions addressed are as follows.

What is the proportion of female to male researchers in the EU, and how is this proportion
evolving over time?

In which scientific fields are women better represented?

Do the career paths of female and male researchers follow similar patterns?

Are statistics on women in science comparable across the EU?

How many women occupy senior positions in scientific research in the EU?

What are the proportions of men and women in a typical academic career, students and
academic staff?

What is the proportion of female heads of institutions in the Higher Education Sector?

What is the evolution in research funding success rate differences between women and men?

While the indicators used provide a good overview of the participation of women and men in
different sectors and at different levels, they do not seem to provide insight into the cultural issues
associated with gender inequality. Similarly, they do not go into depth with regard to factors
influencing womens promotion and progression in research careers, nor do they offer much insight
into institutional arrangements and mechanisms for promoting gender balance. And since our main
priority is the development of RRI policies in institutions and programmes, we choose to focus our
indicators differently.
Gender equality in the context of RRI policy has two dimensions: promoting the equal participation
of men and women in research activities (the human capital dimension); and the inclusion and
integration of gender perspectives in R & I content. The need to monitor the development of
gender equality policy is underpinned by evidence that research performance is limited by direct
and indirect sex discrimination, that gender equality at all levels contributes to achieving excellence
and efficiency (European Commission, 2012c) and that policy at different levels of the R & I system
is slow to develop (Wynne, 1991). The main problems in advancing the gender equality agenda
include: a lack of clarity in decision-making (which affects structures and processes within the
research system and often reinforces status quo, for example old boys networks); informal
institutional practices and organisational culture (which often hides unconscious bias against
women); unconscious gender bias in the assessment of excellence and the process of peer-review,
especially in STEM areas; and the structuring of the workplace and the gender pay gap in academia
(including research), which favours men and creates difficulties for women.
Gender bias may also have implications for the content of science itself. The integration of sex and
gender analysis can increase the quality and relevance of research and its applicability, especially
where gender differences play a major role, such as in the medical sciences.
The overarching goal of the EU policy on gender equality in the context of RRI is gender
mainstreaming in R & I, which includes both the equal participation of men and women in R & I and
reviewing research content from a gender perspective. In the FP7 project EGERA (13), the
policy/implementation options for these aspects are researched.
The EU strategy for gender mainstreaming (European Commission, 2012c) includes: gender
requirements for all funding programmes at EU and Member State levels; creating well-funded,
dedicated programmes to promote structural change in research institutions (also including training
programmes for leaders, decision-makers, evaluators and experts); identifying, publicising and

(12) http://ec.europa.eu/research/science-society/document_library/pdf_06/she-figures-2012_en.pdf
(13) http://www.egera.eu
26

promoting gender best practices; ensuring that researcher mobility measures incorporate the
gender dimension; and introducing legislation and policies for adopting gender equality plans.

Towards indicators
In terms of monitoring RRI policies on gender equality, we conclude that the focus should be on
processes of institutional change to see whether these general ambitions are translated into
concrete forms of action. Regarding our indicator categories of performance and perception, this
means that we suggest looking at:

changes in institutional processes and structures that govern and influence the career
progression of women in research institutions;

cultural change in institutions that reduces gender bias and promotes gender equality;

addressing the unconscious gender bias that favours one sex over another, e.g. perception of
women achievements in STEM;

changes in workplace arrangements to support female researchers, as it is usually women who

are discriminated against by such arrangements.

The typical solution has been to monitor (or require) the representation of women in various
positions and levels of decision-making, both as an indicator of a desirable outcome and, probably,
as a proxy for processes of institutional change. While not perfect and involving a number of risks
of the side effects of monitoring (e.g. by leading to disproportionate administrative burdens on
female researchers), current indicators have the advantage of being well established and easily
estimated by data. We will accordingly support the continued use of such indicators.
The second dimension of gender equality as a criterion of RRI is, as stated above:

the width and breadth of penetration of gender perspectives in research content.

This dimension can be indicated by the number of research projects that include gender analysis or
gender dimensions. Clearly (as with many other indicators) its use will require a qualitative
judgement about what qualifies as a gender dimension. Such procedures of judgement are in
themselves opportunities for reflection and responsibility, and we do not recommend subjective
attempts at standardisation to try eliminate them.
We propose the following indicators for gender equality as a criterion of RRI.

27

Performance indicators
Criteria
Process indicators

Outcome indicators

Perception
indicators

Percentage of Member State

funding programmes explicitly
including gender requirements
Percentage of research
institutions (including
universities) that (a) have gender
equality plans and (b) provide
documentation of their
implementation
Percentage of research
institutions that document
specific actions that minimise
/reduce barriers in work
environment that disadvantage
one sex (e.g. flexibility of
working hours)

Gender
equality

Percentage of research
institutions that document
specific actions aiming to change
aspects of their organisational
culture that reinforce gender bias
Percentage of research
institutions that provide
training/support for researchers
in regard to the inclusion of
gender dimensions in the content
of research
Percentage of schools (primary
and secondary) that have
programmes promoting gender
equality issues in regard to
career choices

Percentage of women on
advisory committees
Percentage of women in
expert groups
Percentage of women on
proposal evaluation panels
Percentage of women in
projects throughout the
whole life cycle (in full-time
equivalent)
Percentage of women that
are principal investigators on
a project
Percentage of women that
are first authors on research
papers
Percentage of research
projects including gender
analysis/gender dimensions
in the content of research
Percentage of women taking
part in research mobility
programmes

Perception of gender
roles in science
amongst young people
and their parents, e.g.
percentage of young
people who believe
that science careers
are equally suitable for
both women and men;
percentage of parents
who believe their
children (daughters)
will have equal
opportunities to pursue
a career in STEM
Perception of people
working in the area of
R & I in regard to
gender equality, e.g.
percentage of women
in R & I, who believe
they have equal
opportunities to pursue
their careers in R & I in
comparison to men

Table 2.3: Proposed indicators for gender equality.

The table summarises the considered range of process, outcome and perception indicators for
gender equality. Key actors are implicitly specified by most indicators (research institutions;
researchers and innovators; members of the public (young people and parents)).

2.4.

Science education

Current state
In ongoing and planned monitoring of RRI and the Science with and for society programme,
science education as an RRI key has been quite narrowly construed as research activities that
aim at promoting an interest in science education, in particular among young people, and the
involvement of practices and institutions that organise such activities (science education remits).
Science education remits have been defined as including science museums, science centres,
departments of science communication, schools, academies, science magazines, science blogs and
living labs. Accordingly, current monitoring in the Commission is aimed at indicators that are

28

defined by quantification of the involvement of science education remits or targeted actors

(students, teachers, citizens) in research activities. This includes:

the percentage of research proposals/projects that include at least one participant with a
science education remit;

the percentage of research projects with at least one educational resource deliverable;

the percentage of research projects that involve STEM teachers or students;

the number of public dissemination events (open days, participation in festivals, prizes and
competitions);

the number of projects registered in the Scientix collaboration.

A first observation is therefore that science education as an RRI key is something rather different
and much more limited than science education as a policy object as such. There are good reasons
for such a narrow framing of this key, and we will return to these reasons in the next paragraph. A
second observation is that science education as an RRI key is related to the key of PE and that the
distinction between them is porous if at all well defined. We shall also return to this issue below.
The European Commission (2012b) explained the RRI key of science education as follows (our
emphasis):
Europe must not only increase its number of researchers, it also needs to enhance the current
education process to better equip future researchers and other societal actors with the
necessary knowledge and tools to fully participate and take responsibility in the research
and innovation process. There is an urgent need to boost the interest of children and youth in
maths, science and technology, so they can become the researchers of tomorrow, and
contribute to a science-literate society. Creative thinking calls for science education as a means to
make change happen.
If we analyse the quote, two goals can be identified, as shown below.
1. Enhance education so that (a) future researchers and (b) other societal actors are equipped
to become good RRI actors.
2. Boost the interest in science among children and young people, with the purpose of either
recruiting them to a research career or allowing them to contribute to a science-literate
society, that is, become scientific citizens.
The two goals are legitimate within their appropriate contexts, but there is no immediate
consistency. Indicators currently in use (research proposals including at least one participant from
formal and informal science education organisations; research projects that involve STEM teachers
or students; PE events aimed at young people; science education projects registered in the Scientix
collaboration) are predominantly proxies for the goal of boosting interest.

Towards indicators
A clear gap can be identified: indicators for the enhancement of education of (1a) future (and
present) researchers and (1b) other societal actors in order to enable them to enact the principle of
RRI actors. As for point (1b), we believe that this sub-goal is better addressed under the RRI key
of PE, and we will leave it here without proposing indicators. Point (1a), however, is not obviously
addressed in any of the other keys, though it could overlap with indicators for capacity building in
the context of ethics. If we apply the processoutcomeperception distinction to sub-goal (1a),
potential indicators are as follows.

Process

The inclusion of an initiative or requirement for RRI-related training in a research

strategy/call/work programme, etc. (yes/no, percentage).

Capacity building for RRI-related training (existence, percentage of funds allocated).

29

Outcome

At the EU and national levels, RRI descriptors in the qualification frameworks for lower and
higher education (quantitative: yes/no; qualitative: degree of relevance).

At the level of education institutions and/or research disciplines, RRI education/training (yes/no,
availability of courses, if they are mandatory or not, percentage of funds).

At the level of R & I projects, whether they encourage or require young researchers to take RRI
education/training and to apply it in the project (e.g. in an integrated ELSA model).

Perception
Perception indicators may be pursued along the lines of the two following questions: are R & I
actors and stakeholders:

knowledgeable of EU values and the needs and concerns of EU citizens?

sensitive to EU values and the needs and concerns of EU citizens?

Although such information is clearly policy relevant, we do not foresee its regular use in indicators.
Rather, one can see the need for research into such matters. We will accordingly not include such
indicators in the table below.
As for the second sub-goal of boosting interest, we identified two dimensions above: boosting
interest in young people and educating scientific citizens. The second dimension we covered under
PE. The discussion of how to monitor the first dimension would be better handled from the overall
policy perspective of education and not DG Research and Innovation. The perception and
outcome indicators overlap because the outcome is to raise the interest of young people. As for
the process indicators one would have to monitor the performance of these limited activities.
Three of the currently designed indicators (mentioned in Chapter 1) can be used for this purpose:

the percentage of research projects with at least one educational resource deliverable;

the percentage of research projects that involve STEM teachers or students;

the number of projects registered in the Scientix collaboration.

30

A summary of our analysis is provided below in matrix form.

Performance indicators
Criteria
Process indicators

Outcome indicators
EU and national levels: presence of
RRI descriptors in the qualification
frameworks for lower and higher
education

The inclusion of an initiative or

requirement for RRI-related training
in a research strategy/call/work
programme, etc. (yes/no,
percentage)

Science education

Capacity building for RRI-related

training (existence, percentage of
funds allocated)

Education institutions/research
disciplines: presence of RRI
education/training
R & I project level: do they
encourage or require RRI
education/training (e.g. in an
integrated ELSA model)?
Percentage of research projects with
at least one educational resource
deliverable
Percentage of research projects
involving STEM teachers or students
Number of projects registered in the
Scientix collaboration

Table 2.4: Proposed indicators for science education.

2.5.

Open access/open science

Current state
Open access/open science, while an RRI key pillar, is also a policy goal on its own for publicly
funded research, at both national and EU levels. In the EU context, monitoring of open access is
specifically provided for by the recommendation from the European Commission (2012d) on access
to and preservation of scientific information, in which various sets of indicators are specified
focusing mainly on outcomes, i.e. scientific publications and data.
To our knowledge, the appropriate work on developing indicators as provided for by the
Commission is being undertaken within DG Research and Innovation, also building upon existing
indicators for open access in use for the European Research Area progress reports (e.g. the
number of scientific publications as an outcome indicator; the share of funders that require open
access for publications as a process indicator).
The mandate of the expert group is to identify gaps and challenges in the monitoring of RRI and
propose indicators that help bridge the gaps and meet the challenges. In the case of open access in
the strict sense, the expert group acknowledges that the 2012 Commission recommendation on
access to and preservation of scientific information to a large degree covers the gaps and
challenges. However, we would like to propose to extend the monitoring of RRI to encompass the

31

ongoing phenomenon of open science, also reflecting the fact that the European Commission itself
increasingly refers to this phenomenon (14). Open science can be defined as follows:
Open science is a practice in which the scientific process is shared completely and in real time. It
offers the potential to support information flow, collaboration and dialogue among professional and
non-professional participants. (Grand et al., 2014)
Another, similar, definition is an emerging approach to the conduct of science, technology and
engineering projects, in which information about the whole of an ongoing investigation is made
available on and through the internet (Grand et al., 2010). Winfield (2014) has distinguished
between three levels of open science, as shown below.

Level 0 open science: maintenance (including frequent updates) of project websites; deposition
of papers (i.e. accepted draft) in publicly accessible repositories; inclusion of datasets with
publications; publication in open access journals.

Level 1 open science equals level 0 plus the following: project blogs, and respond to comments
or feedback; post project movie clips to a project YouTube or other video channel, with links to
project website and blog posts with explanation and commentary.

Level 2 open science equals level 1 plus the following: routinely upload experimental datasets to
project websites, with explanatory notes (i.e. the values in each field) and commentary; daily
laboratory notebooks are written online and publicly accessible in real time; regular project
dialogue, i.e. discussion between researchers, partners and collaborators through a project wiki,
is publicly accessible; employ rich virtual environments for processes of social learning and
innovation.

Level 0 open science is essentially what is provided for in current EU open access policies. In the
context of RRI, open access is not an end in itself but a means to achieve the goal of better
alignment of R & I with societal values, needs and concerns. This goal requires that the openness
actually be used and useful. The expert group would therefore like to propose the further
development of this RRI topic and propose indicators for level 1 and level 2 open science, where
appropriate.

Towards indicators
Possible process indicators for open science levels 1 and 2 could be measures of the existence of
open science policies, mechanisms for establishing, enforcing and monitoring open science,
mechanisms for learning from open science experience and the inclusion of open science measures
in research policies and calls for proposals. They can be numeric and can be applied at different
levels of aggregation. Key actors would include policy institutions, universities and other
institutional R & I actors.
Possible outcome indicators for open science encompass the vitality of repositories, blogs and
other virtual environments for exchange, social learning and social innovation. Indicators might
include (as a percentage): research projects with a virtual environment that is updated and
actively used with a certain frequency (defined as a threshold level); data repositories that include
explanation and commentary to facilitate use; research projects with daily laboratory notebooks
online; research projects that report real added value by an open science mechanism (either for
themselves or other actors); journals that maintain extended peer review practice. Key actors
would include R & I actors at various levels of aggregation, for instance research institutions,
funding programmes, research areas, etc.
Possible perception indicators for open science levels 1 and 2 could be defined around public
awareness of the existence of research projects with an active virtual environment; public
awareness of the intentions of such virtual environments (perfunctory/symbolic, promoting actual
social learning and innovation, etc.); or the extent to which members of the public has visited or
made use of such environments and found them useful. Of these indicators, the latter number of

(14) See, for example, http://ec.europa.eu/programmes/horizon2020/en/h2020-section/open-science-openaccess

32

visits and appraisal of utility would be the easiest to implement, for instance by direct feedback
collection on websites. We will accordingly only include this latter indicator in our summary table
below. The proposed indicators are listed in the matrix below.
Performance indicators
Criteria

Process indicators
Documentation of open
science policies

Open
access/open
science

Documentation of
institutional mechanisms
for promoting open
science
Documentation of
mechanisms for learning
from open science
experience
Inclusion of open science
measures in research
policies and calls for
proposals

Outcome indicators
Percentage of research projects with
a virtual environment that is
updated and actively used with a
threshold frequency (to be defined)
Percentage of data repositories that
include explanation and commentary
to facilitate use
Percentage of research projects with
daily laboratory notebooks online
Percentage of research projects that
report real added value by an open
science mechanism (for themselves
and/or other actors)

Perception
indicators

The extent to
which members
of the public
has visited such
environments
and found them
useful

Table 2.5: Proposed indicators for open access/open science.

2.6. Ethics
Current state
The European Commission (2012b) introduces ethics as an RRI key in the following way:
European society is based on shared values. In order to adequately respond to societal
challenges, research and innovation must respect fundamental rights and the highest ethical
standards. Beyond the mandatory legal aspects, this aims to ensure increased societal
relevance and acceptability of research and innovation outcomes. Ethics should not be
perceived as a constraint to research and innovation, but rather as a way of ensuring high
quality results.
Ethics in the context of research may be seen as a complex field in which internal norms and
values relating to conduct, practice, culture and organisation operate together with the norms,
values, practices and structures that society imposes on research through a variety of mechanisms.
In the broad RRI context ethics can be divided into the following three subfields.

Research integrity and good research practice, which is concerned with issues such as scientific
misconduct and questionable research practices (e.g. plagiarism, fabrication, fraud, authorship
and intellectual property, and citation/acknowledgement practices, scientific neutrality, conflicts
of interest in peer review and scientific advice, etc.). There are three main dimensions that can
be monitored here: the gap between codified rules and the actual norms and values of scientific
communities as expressed in practice; new organisational measures to improve accountability
with respect to research integrity (and overlaps to some extent with open access/open science);
and neutrality and conflict of interest and bias as an ethical as well as a quality problem.

Research ethics for the protection of the objects of research is a well-developed dimension with
institutions and practices for such protection. The ultimate goal of policy in this field is that
human beings, animals and other objects of research are duly protected. The existence and
33

proper functioning of institutional procedures are clearly relevant measures for this goal. Their
amount or the intensity of the work, however, is not very informative of proper functioning.

Societal relevance and ethical acceptability of R & I outcomes. This dimension as an RRI key is
the one that is closest to the general policy of RRI as a cross-cutting principle and the one for
which the European Union has its most distinct role to play. This field is the one warranting the
highest interest in the monitoring of ethics as an RRI key. It is also a field that has experienced
an expansion over the latter decades in terms of specific, concrete issues considered, for
example in ethics reviews. It seems likely, in particular in light of developments in ethics
research and scholarship, that this expansion will continue. Specifically, many scholars would
argue that the novel topics presented in this report (Sections 2.7 and 2.8 on sustainable
development and social justice and inclusion, respectively) both belong to ethics proper. The
readers of this report may accordingly consider the recommendations in this section also as
relevant for ethics policies, practices and indicators.

Current indicators for ethics focus on measuring the proportion of funded or proposed projects that
satisfy some criterion of ethical awareness or activity, or also being ethically problematic. Amongst
them are the following.

Percentage of research proposals that flag at least one issue in the ethics issues table; that
address such issues in text; that require ethics screening; that require ethics assessment; and
that require changes in grant application or second ethics assessment. Also the number of
ethics issues per proposal has been mentioned as a possible indicator.

Percentage of research projects that undergo ethical check/audit; that are not funded because
of ethical concerns.

Percentage of research projects that include a work package for ethics; number of project
publications discussing or otherwise attending to ethical aspects of the research project.

With the exception of the very last indicator in this list, the expert group is not convinced that any
of these indicators are suitable for the monitoring and promotion of RRI. Rather, they measure the
state of more or less arbitrary administrative practices and grant-writing strategies. Accordingly,
we do not include any of them in our recommendations.
Ethics is one of the RRI criteria in greatest need of new concepts and designs for indicators. It is
outside the scope of this expert group to test and implement new indicators; nevertheless we
provide general concepts and recommendations for implementation under the three subfields
identified above.

Towards indicators
As the main issue of research integrity is to monitor the level of awareness and ability to
adequately handle the tensions and discrepancies between official norms and actual practices, as
well as the tensions between different norms and values, the key indicators proposed are process
and perception indicators rather than outcome indicators. The following dimension is particularly
relevant.

Documentation of institutional attention to normative tensions related to research integrity

policies and actions. (This may manifest itself in a number of ways, e.g. activities such as open
meetings or seminar series on ethics and research integrity and the inclusion of such
discussions in research strategy papers, research proposals.) This is a qualitative indicator that
will have to be based on strategic sampling of information process indicator.

Research ethics for the protection of objects of research is not the main current challenge
for ethics as a criterion of RRI in the context of the EU. Rather, our analysis above implies that the
main challenge is to prevent mandatory institutional ethics procedures from degenerating into
perfunctory exercises. Simple quantitative indicators measuring only the level of activity for such
procedures increase rather than decrease that risk. We accordingly discourage the widespread use
of simple quantitative indicators of the number of ethical issues declared, the percentage of
projects that undergo ethical review, etc. and propose the adoption of qualitative indicators that
necessarily will involve the exercise of judgement on behalf of the data provider or analyst.
Relevant indicators include:
34

The percentage of research proposals for which the ethics review/Internal Review Board (IRB)
clearance process requires substantive changes in grant application or second ethics assessment
outcome indicator.

The formal and actual scope of the ethics review/IRB clearance (e.g. whether a committee or a
reviewer is free to identify any type of ethical issue or concern or has to limit the analysis to a
predefined set, and the degree to which this freedom is enacted) process indicator.

Societal relevance and ethical acceptability of R & I outcomes is a main issue in the
promotion of RRI policies and indicators as it should be designed in accordance with the
understanding that this issue is a challenge of governance in complexity that calls for a network
approach. Meaningful indicators are likely to be qualitative. Their main utility is to provide a
substrate and a template for meaningful deliberation and interaction between actors within the
networks. Relevant networks may be found at the level of (large) research proposals and projects,
but even more so at the level of funding programmes, R & I fields, research institutions and other
actors at a higher level of aggregation.
To make the point more concrete, instead of futile attempts at collecting data from below for a topdown command-and-control system, we recommend that indicator designs focus on bringing actors
together to discuss the state of affairs as a part of good governance. For RRI in general, and in
particular for the more overarching criteria such as ethics, indicators accordingly will and should be
experimental in nature. In what follows we provide a list that combines simple, quantitative
suggestions with qualitative and more experimental ones, as shown below.

Documented change in R & I priorities (research or research funding) attributable to multistakeholder and/or transdisciplinary processes of appraisal of societal relevance and ethical
acceptability (presence/frequency; qualitative descriptions; best practices) process indicator.

Presence of mechanisms for multi-stakeholder and/or transdisciplinary processes of appraisal of

societal relevance and ethical acceptability (presence/frequency; qualitative descriptions; best
practices) process indicator.

In research projects, the existence of an ELSI/ELSA (15) project component and/or

transdisciplinary component that addresses societal relevance and ethical acceptability
(presence/frequency; qualitative descriptions; best practices).

Public awareness and evaluation of mechanisms for multi-stakeholder and/or transdisciplinary

processes of appraisal of societal relevance and ethical acceptability this may be developed
into a perception indicator, however we will not include it in our main recommendations for
routine use.

(15) ELSI/ELSA is a common acronym for ethical, legal and social/societal implications/issues/aspects of a
given field of research or innovation.
35

Performance indicators

Criteria
Process indicators

Mechanisms for multistakeholder/transdisciplinary processes

of appraisal of ethical acceptability (best
practices)

Ethics

Documented ELSI/ELSA project

component for ethical acceptability (best
practices)
Documentation regarding normative
tensions related to research integrity
policies and actions

Outcome indicators

Documented change in R & I

priorities attributable to appraisal
of ethical acceptability
Percentage of research proposals
for which ethics review/IRB
clearance process requires
substantive changes in grant
application or second ethics
assessment

Formal and actual scope of ethics

review/IRB clearance

Table 2.6: Proposed indicators for ethics.

The table lists in a keyword fashion the main indicators proposed by the expert group. Please
consult the text for details.

2.7.

Sustainability

Motivation for an integrated perspective

In the introductory chapter (Section 1.2) we briefly presented our argument for indicators for the
aspects of sustainability and social justice/inclusion. In what follows we will detail the argument
and provide a few initial reflections on how such indicators could be devised as RRI as a policy
concept evolves in the European Research Area.
The rationale of the Europe 2020 strategy is to address and overcome the shortcomings of the
current growth model in order to achieve smart, sustainable and inclusive growth. To this end
the strategy includes headline targets in five areas: employment, research and development,
climate/energy, social inclusion and poverty reduction. While sectorial strategies have been
formulated for each of these areas, they are clearly not independent. Measures in one sector
addressing one headline target will frequently, if not always, have direct or indirect effects relevant
to other headline targets.
The innovation union and Horizon 2020 are EUs main initiatives in the area of research and
development. They will be monitored, governed and evaluated on the basis of their headline target,
formulated as the level of investment in R & I as a proportion of EUs GDP, and also on the basis of
performance indicators defined in terms of their general and specific objectives.
The performance indicators provided for Horizon 2020 are accordingly not unrelated to the overall
Europe 2020 objective of smart, inclusive and sustainable growth. The target that 60 % of Horizon
2020 expenditure should be related to sustainable development bears clear witness to this
interrelationship. Performance indicators that measure the means by which these ends are
addressed provide information also of relevance for Europe 2020.
It is quite common, however, for ungoverned science, research and innovation as such not to
necessarily produce the societal effects that are desired. This was implicitly recognised when
Horizon 2020 set out specific priorities, such as societal challenges, and specific objectives for
these priorities, such as secure, clean and efficient energy and others. Still, we believe it is a fair
criticism to say that one main focus of present and planned Horizon 2020 monitoring is on
performance indicators that measure the extent to which planned programme activities have been
36

carried out. The second main focus is on the contribution of these activities to the overarching goal
of macroeconomic success measured as employment and growth, though not necessarily inclusive
or sustainable.
Also, if one considers Eurostats indicators for research and development (16) they are mostly
concerned with the headline target for R & I. They do present one indicator more directly targeted
towards sustainability, namely the number of patent applications of technologies or applications for
mitigation or adaptation against climate change. It remains an empirical hypothesis, however, that
actual climate change is affected by this number. There is a knowledge gap between the headline
targets for inclusive and sustainable growth, measured at a societal level of aggregation, and the
performance indicators for R & I.
RRI as a policy principle is concerned with and addresses this knowledge gap. The Science with
and for society programme, as did its predecessors, sets out to provide research-based knowledge
and best practices for more dynamic governance that will align R & I better with societal needs and
goals. RRI as a cross-cutting principle throughout Horizon 2020 is intended to contribute to such
governance by the actual development of RRI agendas. This important function of RRI should be
reflected in RRI indicators and monitoring practices. While many, perhaps all, of the six original RRI
keys are to some extent related to aspects of inclusion and sustainability, indicators for these keys
cannot answer the following questions: to what extent does a research field, a research
programme or an RRI initiative contribute to inclusive and sustainable growth, and how can this be
assessed and monitored? Such questions are undoubtedly highly relevant and important and can
be asked about all activities and initiatives that are derived from the Europe 2020 strategy, and yet
they are difficult to answer. Horizon 2020 being what it is an EU contribution to the knowledge
society it is a good place to pursue such difficult questions that involve knowledge challenges.
Furthermore, there is a substantive body of research-based knowledge that can be applied in the
development of novel indicators in this field. In this report, we mainly point out the directions
where the knowledge and the indicators may be found; there is also a need for further research
and development in order to reduce the recognised knowledge gap in this field.

An initial step towards indicators for sustainability

Above we asked the questions: to what extent does a research field, a research programme or an
RRI initiative contribute to inclusive and sustainable growth, and how can this be assessed and
monitored?
The topic is extremely important essential for the future wealth and health of the planet and its
people. However, to answer it, original research is needed a task beyond the remit of our expert
group. A useful first approximation to the topic is provided by Kettner, Kppl and Stagl (2014).
Below, we will only draw upon the conclusion of their work, in terms of the type of indicator
framework that is needed. A comprehensive implementation of such a framework would amount to
a number of requirements:

monitoring of stocks (renewable and non-renewable resources);

monitoring of flows (consumption and regeneration of stocks);

mapping and monitoring of stock-flow interactions;

mapping of fund elements (labour and technology) and how they influence the stock-flow
interactions;

monitoring of ecosystem services and their effect on human well-being.

This would be implemented not only with respect to R & I activities in isolation but to the entire
socioecological metabolism of the EU and, even better, planet earth.
Such a comprehensive scheme goes beyond what can be expected in the short term for the routine
monitoring of Horizon 2020 and other R & I policies. Rather, what might be considered is (a) a
long-term process by which the monitoring and governance of RRI includes and gathers experience

(16) http://ec.europa.eu/eurostat/help/first-visit/content
37

with a gradually more comprehensive version of the framework and (b) beginning in the short term
with a minimal set of indicators.
The construction of outcome indicators for socioecological metabolism is a research field of its own.
Perception and process indicators for sustainability as a dimension of RRI, however, need not be
that difficult to construct. Perception indicators can easily be defined by inquiring into different
actors sociotechnical imaginaries (Jasanoff and Kim, 2009) with respect to the R & I activity in
question. One could simply ask: what is the anticipated effect of this research development on
stock-flow interactions? For instance, both research funding programmes and research proposals
make frequent claims on how the anticipated research will contribute to sustainability by resource
and energy decoupling. Such claims are empty unless they suggest possible pathways towards
these effects. Process indicators can be defined to monitor the efforts and developments being
made towards the expected outcomes. When taken together, perception and process indicators
may provide a basis for RRI governance in the sense of improved responsiveness and
accountability among R & I actors. Ex ante sustainability assessments and sociotechnical
imaginaries (promises) may be held accountable by process results.
To sum up, process indicators can be defined in terms of milestones on specified pathways that
have an effect on specified stock-flow interactions, while R & I actors perceptions may be indicated
in terms of their anticipation and imagination of pathways, milestones and the ultimate effect on
specified stock-flow interactions. As of today there is no obvious place for such indicators in current
policy practice. For this reason we do not include the concrete indicator proposals in our overall
table in Chapter 3. However, as noted above, the expert group has found it reasonable also to take
into account what we consider to be likely future developments of the concept of RRI by including
reflections on design principles for such indicators.

2.8.

Social justice/inclusion

Current state
Social justice can be defined as an ideal condition in which all individual citizens have equal rights,
equality of opportunity, and equal access to social resources (Maschi and Youdin, 2012). National
social justice policies focus on investing in achieving inclusion rather than compensating for
exclusion. The effectiveness of such policies is measured by monitoring progress in six dimensions:
poverty prevention, access to education, labour market inclusion, social cohesion and nondiscrimination, health and intergenerational justice (OECD, 2011).
The role of science and technology in promoting social justice is very important. Social justice,
although not explicit, is a transversal theme running through most, if not all, societal challenges of
the Horizon 2020 framework. However, to date no attempts to measure how social justice is
actually addressed through R & I activities have been observed. The connection between science
and technology and social justice is recognised through acknowledging the role of science and
technology education (Dy, 1994) and technological developments, especially in the area of
information and communications technology (ICT), in promoting social justice (Vrasidas, Zembylas
and Glass, 2009), as well as the consideration of ethical issues and values in the design,
development and implementation of new technologies (17).
Social justice directly in the context of research activities can be considered from two perspectives:
(a) the relationship between the researchers and the research subjects; and (b) the participation of
social groups in benefits arising from research. The first perspective is concerned with researchers
unfairly taking advantage of research subjects and imposing unfair burdens on them for their own

(17)

See for instance the value Ageing project: incorporating European fundamental values into ICT for
ageing: a vital political, ethical, technological, and industrial challenge (http://www.value-ageing.eu).

38

benefit or the benefit of others. The second involves the potential unfair exclusion of particular
groups from either participation in research and/or access to benefits arising from research
(European Commission, 2010).
These two perspectives are key to developing indicators to address social justice issues in the
context of R & I. The first perspective the relationship between the researchers and the research
subjects lies firmly within the field of research ethics and should be incorporated in the
indicators for the ethics key. The second perspective of equal participation of social groups in
benefits arising from research goes beyond what is usually included in the ethics key as currently
practiced. In what follows we suggest initial steps towards a set of specific indicators that would
enable monitoring of the progress of R & I activities and their contribution in achieving social
justice/inclusion. One possibility would be to include such aspects in institutional ethics practices
such as ethics reviews.

An initial step towards indicators for social justice and inclusion

The issue that should be monitored is the impact of research and its effects on social
justice/inclusion. Monitoring could/should answer questions such as the following (non-exhaustive
list).

Is the new technology/product accessible/affordable to wide variety of different social groups?

Is the research problem addressing an access problem of a disadvantaged social group, such as
disabled people, illiterate people, migrants, elderly people, etc.?

Does the research have the potential to impact negatively on some social groups?

Measuring the impact should focus on two issues: (a) whether researchers consider at all the
impact of their research on social justice; and (b) whether they have taken any steps to either
extend the impact of their research to a larger population or to minimise potential unintended
negative consequences in relation to social justice. Following this argument, a next step towards
indicators could be to pursue the following directions.
Process indicators

The number/percentage of funding calls that explicitly require impact statements to consider
social justice/inclusion issues (if percentages are used they should be the percentage of calls in
homogenous scientific areas to allow meaningful comparisons).

The percentage of research institutions that have procedures that encourage/oblige researchers
to consider the impact of their research on social justice/inclusion, in regard to both the
participation of excluded groups in the research and the potential research impact on such
groups (e.g. training, ethics reviews).

The percentage of research institutions that have mechanisms that assist researchers in the
recruitment of research participants from socially excluded groups (e.g. databases of potential
participants, strong links with representative bodies).

The number of stakeholders who actively review/show interest in research results that have an
impact on social justice. Qualitative indicators should also be used to examine the use of such
results for policymaking processes.

Outcome indicators

The percentage of research proposals considering the impact of the research on different
aspects of social justice (six dimensions used for social justice indicators could be used here:
poverty prevention, access to education, labour market inclusion, social cohesion and nondiscrimination, health and intergenerational justice).

The percentage of research projects that modified their methodology/implementation of

research to improve their impact on social justice (e.g. including research participants from a
wider social groupings to address broader perspectives/needs).

The percentage of projects that may have unintended negative effects on social justice (e.g. .
projects that have benefits for only small portion of the general population or projects that may

39

create additional barriers). Qualitative measurement may be employed to identify potential

unintended negative effects on social justice to inform future funding calls/policymakers.

Performance indicators
The percentage of researchers/research institutions who believe that it is important to
consider/address issues related to social justice/inclusion in their research in regard to (a) research
methodology and implementation and (b) research results. Such indicators should measure the
level of importance together with commitment.
The percentage of public that believes that research activities (a) actively promote/contribute to
achieving social justice/inclusion and (b) have a negative effect on social justice. Qualitative
indicators should be employed to identify best and worst practices.
The indicators listed above require substantial resources to be monitored and can be meaningfully
monitored only within fields where the link between research and social justice is found to be
evident or at least relevant (several scientific fields may be excluded here). For this reason we do
not include the concrete indicator proposals into our overall table in Chapter 3 as is the case
with the need for an indicator framework for sustainability as an aspect of RRI, we are providing
resources for likely future developments of RRI concept and policy, and our analysis should be seen
as conditional upon this development.

40

3. Prioritised list of indicators to be deployed

In Chapter 2 we presented a large number of RRI indicators and dimensions for indicators to cover
the various RRI criteria. Some of the proposed indicators are quantitative, others qualitative; some
are already in use or readily implemented, others require further methodological development or
even further research. Our ambition has been to present the European Commission as well as other
actors within the European Research Area with a toolbox from which they may choose and tailor
sets of indicators for the monitoring, promotion and development of RRI. It is obvious that one
cannot create a prioritised list of indicators without explicitly or implicitly prioritising the
objectives to be achieved within a particular policy context.
For this reason, we cannot offer a general prioritised list of indicators for actors in the European
Research Area. National and regional actors, universities and research institutes, civil society
organisations, funding agencies and others should devise their own process of deliberation in order
to choose and tailor the indicators proposed in Chapter 2, and add their own indicators according to
their own needs, goals and concerns.
The policy context of the European Commission, and in particular DG Research and Innovation,
offers directions and constraints that have enabled the group to give more substantive advice on
which indicators to prioritise. For this purpose we may distinguish between three levels.
First, there is the level of individual RRI criteria, such as PE, gender equality or sustainability.
Although the criteria overlap to some degree, each of them is subject to its own policy
development, policy action and monitoring. At this level, we recommend that attempts be made to
make full use of the suite of indicators proposed by the report. Some of these attempts are going
to be exploratory; data may be missing or hard to obtain, and methodologies may be immature.
Still, we have only proposed those indicators that we have considered necessary for a
comprehensive view of the matters of fact to be monitored and acted upon, among the many other
alternatives considered and left out. The general regulative principle for those working at this level
should be that of saturation and complementarity: unless there are good arguments to the
contrary, try to obtain a balance and complementarity between process, outcome and perception
indicators, and to cover the main substantive issues at stake.
Secondly, there is the level of concern for the successful implementation and development of RRI
as a cross-cutting principle of Horizon 2020. This level is important, not the least for the mandate
of this expert group. For those working at this level, the full set of 100 indicators is unlikely to be
practicable or even interesting. Rather, one should choose a smaller set of indicators that ideally
should:

include indicators for all RRI criteria;

have a balance between outcome, process and perception indicators;

focus on performance (outcome and process) indicators for states of affair that are targeted (or
should have been targeted) by major RRI actions and initiatives;

be meaningful and informative to various R & I actors and conducive to good processes that
promote and develop RRI as a policy principle that is, rather than the emphasis being on
hard facts, chosen because they are easy to quantify, to be fed into an illusory command-andcontrol mode of governance, it should be on information that is helpful in collaborative modes of
governance, developing trust, best practices and mutual institutional change.

Ultimately, DG Research and Innovation should make its discretional choice in the identification of
this smaller set, as it this the DG and not the expert group that owns its policy priorities. Based
upon our knowledge of the current policy context and the mandate for the expert group, we have
felt entitled, however, to make our proposal for such a smaller set. This is given in the table below.
In this table we have not included our tentative indicators for sustainability and social
justice/inclusion. The reason is that their use would be conditional upon a policy development that,
however likely and logical, remains outside the remit of this expert group. We repeat, however,
that they to a large degree follow logically from existing EU policies from which RRI and Horizon
2020 are derived, not the least from the Charter of Fundamental Rights and from Europe 2020. The
direct relationship between RRI and the fundamental rights and values of the EU was also
emphasised by the Rome Declaration on RRI.

41

Finally, there is the third policy level of Horizon 2020 in its entirety, of which RRI is but one
element, an element that itself is in need of legitimacy and justification, among many other
elements, and is to some degree in tension with some of them. We see four complementary
approaches to the prioritisation of indicators at this level.
1.
2.

3.

4.

Table 3.1 below represents a suite of indicators for RRI as a cross-cutting principle for the whole of
Horizon 2020, and it provides information on the state of RRI as judged by the normative principle
of RRI itself (internal criteria).
The Science with and for society programme should (and will) be evaluated and monitored also by
standard criteria for any subprogramme of a European Union framework programme for R & I, to
be broadly characterised as attempts at measuring the effectiveness and efficiency of the funded
activities with respect to achieving their expected impact. This approach will be pursued also
without the efforts of this expert group, and will provide important information.
For some actors, RRI may appear to be in tension with other goals and desires for R & I policy. One
argument sometimes encountered is that concerns about ethics, gender equality, open access, etc.
are distractions that impede the effectiveness and efficiency of R & I and hence the maximum
potential realisation of economic growth. A similar argument is that RRI has no direct bearing on
the headline target for research and development of Europe 2020. However, such arguments do
not take into account the Charter of Fundamental Rights, nor that the overall objective of Europe
2020 is smart, inclusive and sustainable growth, nor the growing body of research-based
knowledge on the relationships between the R & I sector, society and the ecosystem (see Section
2.8). We believe that the comprehensive indicator framework for sustainability presented in
Section 2.8 is what needs to be developed and deployed in order to develop a knowledge base that
is sufficient for an unbiased and integrated perspective on the legitimacy and justification of RRI
also in pragmatic terms.
Finally, public perception indicators are particularly important for considerations on legitimacy and
justification, also of RRI. A number of such indicators are presented in the various sections of
Chapter 2.

42

Performance indicators
Criteria

Perception indicators
Process indicators
Number and degree of development of formal
procedures for citizens involvement (consensus
conferences, referendum, etc.)

Public
engagement

Number of citizen science projects,

discriminating from those supported by
institutions and those that are created at
grassroots level, by field

Outcome indicators
Number (absolute and percentage with respect to the
total) and the percentage in terms of funding of
projects and initiatives (a) led by citizens or civil
society organisations and (b) including research done
by citizens or civil society organisations (citizen
science)
Number of advisory committees including citizens
and/or civil society organisations
Percentage of citizens and civil society organisations
with special responsibilities within advisory boards,
committees and consultant bodies (chair, rapporteur,
etc.)

Gender
equality

Percentage of research institutions that

document specific actions that aim to change
aspects of their organisational culture that
reinforces gender bias

Degree of public interest in science

and technology issues: percentage of
the total population declaring
themselves interested; percentage of
citizens indirectly showing interest in
science and technology (percentage
visiting science centres, percentage
participating in demonstrations about
scientific issues, etc.)
Expectations of responsible science:
percentage of population that sees
science as part of the solution rather
than the problem; percentage of
population with high expectation

Number of citizens engaged in citizen science projects

Percentage of women that are principal investigators
on a project
Percentage of women that are first authors on research
papers
Percentage of research projects including gender
analysis/gender dimensions in the content of research

Science
education

The inclusion of an initiative or requirement for

RRI-related training in a research
strategy/call/work programme, etc. (yes/no,
percentage)

Percentage of research projects with at least one

educational resource deliverable
Inclusion of open science measures in research
policies and calls for proposals

Open access

At the level of R & I projects, whether they encourage

or require young researchers to take RRI-related
education/training and to apply it in the project (e.g. in
an integrated ELSA model)

Percentage of research projects that report real added

value by an open science mechanism (for themselves
and/or other actors)

43

The extent to which members of the

public have visited vital virtual
project environments and found
them useful

Ethics

Governance

Documented ELSI/ELSA project component

and/or transdisciplinary component that
addresses societal relevance and ethical
acceptability (presence/frequency; qualitative
descriptions; best practices)

Documented change in R & I priorities (research or

research funding) attributable to multi-stakeholder
and/or transdisciplinary processes of appraisal of
societal relevance and ethical acceptability.
(presence/frequency; qualitative descriptions; best
practices)

Identification of formal and informal networks of

R & I that promote RRI, at both the national and
the EU level

For each of these networks:

number
number
number
number

of
of
of
of

RRI
RRI
RRI
RRI

Involvement of the wider public in

RRI debates measured, for example,
through social media

debates
protocols
policies
agreements

Involvement of the wider public in

RRI policy, the development of
policy, protocols

Table 3.1: Prioritised indicators for Responsible Research and Innovation.

44

[Catalogue

References
Bauer, M. W., Howard, S., Romo Ramos, Y. J., Massarani, L. and Amorin, L. (2012), Science
journalism across the world working conditions, practices, professional ethos and future
expectations, SciDev.Net and LSE, London. See: http://www.scidev.net/en/content/our-

learning-series/#X346620B030F0470C9BD5FB18DC433272
Bauer, M. W., Allum, N. and Miller, S. (2007), What can we learn from 25 years of PUS research?
Liberating and widening the agenda, Public Understanding of Science, Vol. 16, No 1, pp. 79-95.
Bauer, M. W. and Falade, B. (2014), Public understanding of science: survey research around the
world, in Bucchi, M. and Trench, B. (eds), Handbook of public communication of science and
technology, second revised edition, Routledge, New York, pp. 140-159.
Bauer, M. W., Shukla, R. and Allum, N. (2012), The culture of science How the public relates to
science across the globe, Routledge, New York.
Bentley, P. and Kyvik, S. (2011), Academic staff and public communication: a survey of popular
science publishing across 13 countries, Public Understanding of Science, Vol. 20, No 1, pp. 48-63.
Blumenberg, H. (2010), Theorie der Lebenswelt Lebenswelt und Technisierung under Aspekten
der Phaenomenologie, Suhrkamp, Berlin, pp. 157-180 (original 1959).
Buchmann, M. (1995), The impact of resistance to biotechnology in Switzerland: a sociological
view of the recent referendum, in: Bauer, M. W. (ed.), Resistance to new technology, CUP,
Cambridge, pp. 207-225.
Bush, V. (1945), Science, the endless frontier A report to the President, U.S. Government
Printing Office, Washington DC, OCLC 1594001.
Castell, S., Charlton, A., Clemence, M., Pettigrew, N., Pope, S., Quigley, A., Navin Shah, J. and
Silman, T. (2014), Public attitudes to science 2014 Main report, IPSOS Mori, Social Research
Institute, London, ESRC and BIS.
Chilvers, J. (2008), Deliberating competence: theoretical and practitioner perspectives on effective
participatory appraisal practice, Science, Technology & Human Values, Vol. 33, No 2, pp. 155-185.

Cooke, B. and Kothari, U., (eds) (2001), Participation: the new tyranny?, Zed Books, London.
Doran, G. T., (1981), Theres a S.M.A.R.T. way to write managements goals and objectives,
Management Review, Vol. 70, No 11(AMA FORUM), pp. 35-36.
Dy, M. B., Jr. (ed.) (1994), Values in Philippine culture and education, Philippine philosophical
studies, 1., Council for Research in Values and Philosophy, Washington.
Einsiedel, E. (2014), Publics and their participation in science and technology: changing roles,
blurring boundaries, in Bucchi, M. and Trench, B. (eds), Handbook of public communication of
science and technology, Routledge, London, pp. 125-149.
Enscot (2003), The European Network of Science Communication Teachers, Public Understanding
of Science, Vol. 12, pp. 167-181.
Eurobarometer 2005 63.1; Eurobarometer 2010 78.1; Eurobarometer 2013 79.2 See

http://ec.europa.eu/public_opinion/index_en.htm
European Commission (2001), European governance A White Paper, Official Journal of the
European Communities C 287, 12.10.2001, p. 1.
European Commission (2009), Ozolia, ., Mitchan, C. and Stilgoe, J., Global governance of
science Report of the Expert Group on Global Governance of Science to the Science, Economy
and Society Directorate, Directorate-General for Research, European Commission. Available at:
http://ec.europa.eu/research/science-society/document_library/pdf_06/global-governance020609_en.pdf
European Commission (2010), European textbook on ethics in research, Publications Office of the
European Union, Luxembourg.
45

European Commission (2012), Ethical and regulatory challenges to science and research policy at
the global level, DG Research and Innovation.
European Commission (2012b): Responsible Research and Innovation: Europes ability to respond
to societal challenges, DG Research and Innovation.
European Commission (2012c), Structural change in research institutions: enhancing excellence,
gender equality and efficiency in research and innovation, European Commission, Brussels.
European Commission (2012d), Commission recommendation of 17.72012 on access to and
preservation of scientific information, C(2012) 4890.
European Commission (2013), van den Hoven, J., Jacob, K., Options for strengthening Responsible
Research and Innovation Report of the Expert Group on the State of Art in Europe on
Responsible Research and Innovation, DG Research and Innovation.
European Court of Auditors, Special Report No 9/2007 concerning Evaluating the EU research and
technological development (RTD) framework programmes could the Commissions approach be
improved?
European Network of Science Centres and Museums (2008), The impact of science and discovery
centres A review of worldwide studies, Ecsite, United Kingdom.
European Parliament and Council (2013), Regulation (EU) No 1291/2013 of the European
Parliament and of the Council of 11 December 2013 establishing Horizon 2020 the framework
programme for research and innovation (2014-2020) and repealing Decision No 1982/2006/EC,
Official Journal of the European Union L 347, 20.12.2013, p. 104.
EUSCEA (2005), Science communication events in Europe EUSCEA white book, Gteborg.
Evans, G. and Durant, J. (1995), The relationship between knowledge and attitudes in the public
understanding of science in Britain, Public Understanding of Science, Vol. 4, pp. 5774.
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P. and Trow, M. (1994), The new
production of knowledge The dynamics of research in contemporary societies, Sage, London.
Godin, B. (2005), Measurement and statistics on science and technology 1920 to the present,
Routledge, London/New York, coll. Routledge studies in the history of science, technology and
medicine, Vol. 22, p. 360.
Gonalves, M. E. and Castro, P. (2009), Local is beautiful? Governing science-society relations in
Europe, Portuguese Journal of Social Science, Vol. 8, pp. 191-207.
Grand, A., Wilkinson, C., Bultitude, K. and Winfield, A. F. (2010), On open science and public
engagement with engineering.
Grand, A., Wilkinson, C., Bultitude, K. and Winfield, A. F. (2014), Mapping the hinterland: Data
issues in open science, Public Understanding of Science.
Horst, M. (2010), Collective closure? Public debate as the solution to controversies about science
and technology, Acta Sociologica, Vol. 53, No 3, pp. 195-211.
Irwin, A. and Wynne, B. (1996), Misunderstanding science? The public reconstruction of science
and technology, Cambridge University Press, Cambridge.
Jasanoff, S. (2003), Technologies of humility: citizen participation in governing science, Minerva,
Vol. 41, pp. 223-244.
Jasanoff, S. and Kim, S. H. (2009), Sociotechnical imaginaries and nuclear power in the United
States and South Korea, Minerva, Vol. 47, pp. 119-146.
Kettner, C., Kpple, A. and Stagl, S. (2014), Towards an operational measurement of socioecological performance Working paper No 52, WWWfor Europe. Available at:
http://www.foreurope.eu

46

Kuhlmann, S. and Rip, A. (2014), The challenge of addressing grand challenges A think piece on
how innovation can be driven towards the grand challenges as defined under the prospective
European Union framework programme Horizon 2020, University of Twente, the Netherlands.
Maschi, T. and Youdin, R. (2012), Social worker as researcher Integrating research with
advocacy, Peachpit Press.
Mejlgaard N., Bloch, C., Degn, L., Ravn, T. and Nielsen, M. W. (2012), Monitoring policy and
research activities on science in society in Europe (MASIS) Final synthesis report, DG Research
and Innovation. Available at: http://ec.europa.eu/research/sciencesociety/document_library/pdf_06/monitoring-policy-research-activities-on-sis_en.pdf
Mejlgaard, N. and Stares, S. (2010), Participation and competence as joint components in a crossnational analysis of scientific citizenship, Public Understanding of Science, Vol. 19, pp. 545-561.
Mellor, F. (2013), Twenty years of teaching science communication: a case study of Imperial
Colleges masters programme, Public Understanding of Science, Vol. 22, pp. 916-926.
Merkx, F., van der Weijden, I., Oostveen, A.-M., van den Besselaar, P. and Spaapen, J. (2007),
Evaluation of research in context: a quick scan of an emerging field, Rathenau Institute, The
Hague.
Merton, R. K. (1937), The unanticipated consequences of purposive social action, American
Sociological Review, Vol. 1, No 6, December, pp. 894-904.
Miller, S., Caro, P., Koulaidis, V., de Semir, V., Staveloz, W. and Vargas, R. (2002), Report from
the expert group Benchmarking the promotion of RTD culture and public understanding of
science, European Commission, Brussels. Available at:
ftp://ftp.cordis.lu/pub/era/docs/bench_pus_0702.pdf
Neresini, F. and Bucchi, M. (2010), Which indicators for the new public engagement activities? An
exploratory study of European research institutions, Public Understanding of Science, Vol. 20,
pp. 64-79.
Nowotny, H., Scott, P. and Gibbons, M. (2001), Re-thinking science, Polity Press, London.
OECD (2011), Social justice in the OECD How do the member states compare? Sustainable
governance indicators 2011, Bertelsmann Stiftung, Gtersloh.
Owen, R., PM. Macnaghten and J. Stilgoe (2012) Responsible Research and Innovation: From
Science in Society to Science for Society, with Society. Science and Public Policy 39(6): 751-760.
Pardo R. and Calvo, F. (2004), The cognitive dimension of public perceptions of science:
methodological issues, Public Understanding of Science, Vol. 13, pp. 203-227.
PCST (2014), Proceedings of the 13th International Public Communication of Science and
Technology Conference (PCST2014); 5-8 May 2014, Salvador, Brazil.
Revuelta, G. (2014), Impacts of science communication on publics, cities and actors, Journal of
Science Communication, Vol. 13, No 1.
Ruedig, W. (1990), Anti-nuclear movements: a world survey of opposition to nuclear energy,
Longman, Harlow.
Sandel, M. (2012), What money cant buy The moral limits of markets, Allen Lane, London.
Saurugger, S. (2010) The social construction of the participatory turn: the emergence of a norm in
the European Union, European Journal of Political Research, Vol. 49, pp. 471-495.
Schaefer, M. (2012), Taking stock: a meta-analysis of studies on the medias coverage of science,
Public Understanding of Science, Vol. 21, pp. 650-663.
Siune, A., Markus, E., Calloni, M., Felt, U., Gorski, A., Grunwald, A., Rip, A., de Semir, V. and
Wyatt, S. (2009), Challenging futures of science in society Emerging trends and cutting-edge
issues, MASIS Expert Group, European Commission.

47

Sjoeberg, S. (2012), PISA: Politics, fundamental problems and intriguing results, La Revue,
Recherches en Education, No 14, September.
Spaapen, J. and van Drooge, L. (2011), Introducing productive interactions in social impact
assessment, Research Evaluation, Vol. 20, No 3, September, pp. 211-218
Stevens, H., Dean, A. and Wykes, M. (2013), Describe project Definitions, evidence and
structures to capture research impacts and benefits Final project report, University of Exeter.
Sturgis, P. and Allum, N. (2004). Science in Society: re-evaluating the deficit model of public
attitudes, Public Understanding of Science, Vol. 13, No 1, pp. 55-74.
von Schomberg, R., (2011), Prospects for Technology Assessment in a framework of responsible
research and innovation, in Technikfolgen abschtzen lehren: Bildungspotenziale transdisziplinrer
Methode, Springer VS, Wiesbaden.
Vrasidas, C., Zembylas, M. and Glass, V. G. (2009), ICT for education, development and social
justice, IAP.
Winfield, A. F. (2014), Open science a three level approach. Presentation at the conference
Science, Innovation and Society Responsible Research and Innovation, Rome, 20 November.
Wynne, B. (1991), Knowledge in context, Science, Technology, & Human Values, Vol. 16, pp. 111121.
Wynne, B. (2007), Public participation in science and technology: performing and obscuring a
political-conceptual category mistake, East Asian Science, Technology and Society: An
International Journal, Vol. 1, pp. 99-110.

48

ANNEX 1. PUBLIC ENGAGEMENT: APPROACHES AND ACTIVITIES

The very first movements of what is now called public engagement had their focus quite exclusively
on the need for societys acquisition of scientific knowledge. This view, also called the science
literacy approach, is still present in many activities within the field (even in some that call
themselves participatory activities). Actions taking this perspective are mainly based on top-down
communication to the public, as well as on involving scientists and engineers in communication
training programmes. Despite this approach being widely used, many authors criticised it in the
1990s, arguing that once we move outside a simple cognitive deficit model of the public
understanding of science, we become increasingly aware of the range and variety of possible
interactions between peoples existing understanding of particular situations and those that
emanate from science (Wynne, 1991). This model, also called in a depictive way the deficit model
of science communication, starts from a too-simplistic view that states that the more knowledge
and information society has, the greater the appreciation is and the lower the rejection of the
applications arising from science and technology; statistics, however, do not sustain this linear
relationship. For instance, those who know the most about a technology are the ones that can also
see its risks, and thus may not be the most optimistic about it. There are certainly relationships
between knowledge and attitudes, but they are much more sophisticated and complex than the
linear model assumes (Evans and Durant, 1995; Pardo, 2004). In contrast to the rather simplistic
deficit model that has traditionally characterised discussions of this relationship, this analysis
highlights the complex and interacting nature of the knowledgeattitude interface, concluded
other authors after deep analysis of this subject (Sturgis and Allum, 2004).

In the EU, the UK BSE scandal of the mid 1980s to mid 1990s is often cited as pivotal in the
change of direction in the relationship between science and policymaking. A key moment was the
publication of the 2000 House of Lords report on Science and society, followed a year later by the
European Commissions Science and society action plan, as well as the EU fifth framework
research programmes Raising awareness of science and technology activity of the late 1990s. In
fact, in the late 1990s and early 2000s, a discourse dominated by a more dialogic and participative
view became more and more present in EU documents and initiatives and in scientific literature
(see, for example, European Commission, 2001). Emphasis was put on dialogue, engagement and
participation, and the terms used to define it went from public dialogue to public participation, or
the more generic public engagement or public engagement upstream. Public engagement
corresponds to the need to establish a dialogue between scientists and the public, and the need for
questions, opinions, expectations and values of citizens and different actors to reach decisionmaking bodies, in this case about the development of science and technology. It also recognises
the desirability of citizens effective participation in the process of science, with participation being
a democratic right and duty. This need seems particularly important in the development of
emerging technologies, so that the process is set in a framework of alignment with social needs
and values, as stated more recently in the definition of RRI set out by the European Commission
(European Commission, 2012). In contrast to the deficit model, a contextual model of
interpretation of the relationship between science and society is presented at this stage, picking
approaches anticipated by Wynne (1991), Layton (1993) or Irwin and Wynne (1996).

Public engagement, therefore, includes activities not only pursuing a dialogue (among researchers,
citizens and other stakeholders), but also searching for a democratic participation of citizenship in
decision-making process.

49

50

How to obtain EU publications

Free publications:
one copy:
via EU Bookshop (http://bookshop.europa.eu);
more than one copy or posters/maps:
from the European Unions representations (http://ec.europa.eu/represent_en.htm);
from the delegations in non-EU countries (http://eeas.europa.eu/delegations/index_en.htm);
by contacting the Europe Direct service (http://europa.eu/europedirect/index_en.htm) or
calling 00 800 6 7 8 9 10 11 (freephone number from anywhere in the EU) (*).
(*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you).

Priced publications:
via EU Bookshop (http://bookshop.europa.eu).

51

KI-NA-26-866-EN-N

The expert group concludes that, for the sake of manageability, actors
should use a limited tailored set of indicators according to their own needs,
goals and concerns. It makes a tentative proposal for a smaller set of
indicators that could be used in a European R&I policy context, presented
as a table at the end of the report.
The expert group sees the development of indicators as a bottom-up
process, guided by the collaboration between relevant stakeholders. It
recommends that a set of indicators should be chosen that include
indicators for all the dimensions of RRI and that focus on performance
(outcome and process). Indicators should furthermore be meaningful and
conducive to good processes that develop RRI as a policy principle. That
is, rather than hard facts, chosen because they are easy to quantify,
emphasis should be placed on information that is helpful in collaborative
modes of governance, developing trust, best practices and mutual
institutional change.

Studies and reports

doi 10.2777/9742

[Catalogue

This report of the Expert Group on Policy Indicators for Responsible

Research and Innovation, considers options for RRI indicators. It divides
such indicators into three sections: i) (good) governance as an
overarching principle for R&I networks; ii), public engagement, gender
equality, science education, open access and ethics, as the five main keys
for governance; and iii), sustainability and social justice/inclusion as a
more general policy goal.