Perspective

https://doi.org/10.1038/s41893-019-0448-2

Principles for knowledge co-production in

sustainability research
Albert V. Norström   1*, Christopher Cvitanovic2,3, Marie F. Löf   4, Simon West   1,5,6, Carina Wyborn   7,8,
Patricia Balvanera   9, Angela T. Bednarek10, Elena M. Bennett11, Reinette Biggs1,12, Ariane de Bremond13,14,
Bruce M. Campbell15, Josep G. Canadell   16, Stephen R. Carpenter   17, Carl Folke   1,18,
Elizabeth A. Fulton   3,19, Owen Gaffney1,20, Stefan Gelcich   21, Jean-Baptiste Jouffray   1,22,
Melissa Leach23, Martin Le Tissier   24, Berta Martín-López   25, Elena Louder26, Marie-France Loutre   27,
Alison M. Meadow   28, Harini Nagendra   29, Davnah Payne   30, Garry D. Peterson   1,
Belinda Reyers   1,31, Robert Scholes32, Chinwe Ifejika Speranza   33, Marja Spierenburg34,35,
Mark Stafford-Smith36, Maria Tengö   1, Sandra van der Hel   37, Ingrid van Putten3,19 and
Henrik Österblom   1

Research practice, funding agencies and global science organizations suggest that research aimed at addressing sustainability
challenges is most effective when ‘co-produced’ by academics and non-academics. Co-production promises to address the
complex nature of contemporary sustainability challenges better than more traditional scientific approaches. But definitions
of knowledge co-production are diverse and often contradictory. We propose a set of four general principles that underlie high-
quality knowledge co-production for sustainability research. Using these principles, we offer practical guidance on how to
engage in meaningful co-productive practices, and how to evaluate their quality and success.

H
uman domination of the biosphere has led to substantial with different needs and interests and are beset by social, political
gains in human welfare and economic development, but and administrative uncertainty5.
simultaneously threatens the planetary conditions that Researchers and practitioners alike are turning to knowledge
underpin societal wellbeing and prosperity1–3. Emerging challenges, co-production as a promising approach to make progress in this
including water scarcity, food security issues and biodiversity loss, complex space. Conceptually, knowledge co-production is part of
are intractable, interconnected and influenced by a range of cross- a loosely linked and evolving cluster of participatory and transdis-
scale drivers and complex feedback mechanisms4. These challenges, ciplinary research approaches that have emerged in recent decades.
and attempts to address them, involve multiple groups of people These approaches reject the notion that scientists alone identify the

1
Stockholm Resilience Centre, Stockholm University, Stockholm, Sweden. 2Australian National Centre for the Public Awareness of Science, Australian
National University, Canberra, Australian Capital Territory, Australia. 3Centre for Marine Socioecology, University of Tasmania, Hobart, Tasmania, Australia.
4
Baltic Sea Centre, Stockholm University, Stockholm, Sweden. 5Fenner School of Environment and Society, Australian National University, Canberra,
Australian Capital Territory, Australia. 6Northern Institute, Charles Darwin University, Darwin, Northern Territory, Australia. 7Luc Hoffmann Institute,
IUCN Conservation Centre, Gland, Switzerland. 8W.E. Frankie College of Forestry and Conservation, University of Montana, Missoula, MT, USA. 9Instituto
de Investigaciones en Ecosistemas y Sustentabilidad, Universidad Nacional Autónoma de México, Morelia, Michoacán, Mexico. 10The Pew Charitable
Trusts, Washington, DC, USA. 11Department of Natural Resource Science and McGill School of Environment, McGill University, Montreal, Quebec, Canada.
12
Centre for Complex Systems in Transition, Stellenbosch University, Stellenbosch, South Africa. 13Centre for Development and Environment, Global Land
Programme, University of Bern, Bern, Switzerland. 14Department of Geographical Sciences, University of Maryland, College Park, MD, USA. 15CGIAR
Research Program on Climate Change, Agriculture and Food Security, International Center for Tropical Agriculture, University of Copenhagen, Copenhagen,
Denmark. 16Global Carbon Project, CSIRO Oceans and Atmosphere, Canberra, Australian Capital Territory, Australia. 17Center for Limnology, University of
Wisconsin-Madison, Madison, WI, USA. 18The Beijer Institute, Royal Swedish Academy of Sciences, Stockholm, Sweden. 19CSIRO Oceans and Atmosphere,
Hobart, Tasmania, Australia. 20Potsdam Institute for Climate Impact Research, Potsdam, Germany. 21Departamento de Ecología, Center of Applied Ecology
and Sustainability, Center for the Study of Multiple-Drivers on Marine Socio-Ecological Systems, Pontificia Universidad Católica de Chile, Santiago,
Chile. 22Global Economic Dynamics and the Biosphere Academy Programme, Royal Swedish Academy of Sciences, Stockholm, Sweden. 23Institute of
Development Studies, University of Sussex, Brighton, UK. 24Future Earth Coasts, MaREI Centre for Marine and Renewable Energy, Environmental Research
Institute, University College Cork, Cork, Ireland. 25Faculty of Sustainability, Leuphana University of Lüneburg, Lüneburg, Germany. 26School of Geography
and Development, University of Arizona, Tucson, AZ, USA. 27PAGES International Project Office, Bern, Switzerland. 28Institute of the Environment,
University of Arizona, Tucson, AZ, USA. 29School of Development, Azim Premji University, Bangalore, India. 30Global Mountain Biodiversity Assessment,
University of Bern, Bern, Switzerland. 31Department of Conservation Ecology, Stellenbosch University, Stellenbosch, South Africa. 32Global Change Institute,
University of the Witwatersrand, Johannesburg, South Africa. 33Institute of Geography, University of Bern, Bern, Switzerland. 34Radboud University
Nijmegen, Nijmegen, the Netherlands. 35Stellenbosch University, Stellenbosch, South Africa. 36CSIRO Land and Water, Canberra, Australian Capital
Territory, Australia. 37Copernicus Institute of Sustainable Development, Utrecht University, Utrecht, the Netherlands. *e-mail: [email protected]

Nature Sustainability | www.nature.com/natsustain

Perspective NatURe SUstainaBility

Box 1 | The history of knowledge co-production

Knowledge co-production is part of a loosely linked and were not simply provided by government agencies to society in a
evolving cluster of approaches, including participatory research, one-way flow, but rather were a distributed, collaborative product
mode 2 science, interactive research, civic science, post-normal of society as a whole. Ostrom introduced the idea of citizens as
science, transdisciplinary and joint knowledge production, action ‘co-producers of public services’ to demonstrate that the provision
research, translational ecology and engaged scholarship that of basic public services like policing relied on both police (to
have become increasingly prominent during the past 40 years84–87. investigate crimes and arrest suspects) and citizens (to report
Such new forms of knowledge production began to emerge as crimes and share information)92. The language of co-production
a response to the complexity and social relevance of emerging spread quickly within the field of public administration, and
challenges, including environmental concerns, economic develop­ currently encompasses both state-initiated (top-down) and more
ment and social upheaval. They are context-driven, problem- radical social-movement (bottom-up) initiated processes93.
focused and require the engagement of multiple disciplines. As the notion of knowledge co-production has gained
In parallel, an expansion of the involvement of non-academic popularity in sustainability research, two broad approaches have
actors in knowledge generation and research activities has taken emerged. The first, labelled ‘normative’, is more pragmatic and
place. Participatory research, for example, has its roots in the work regards co-production as a deliberate collaboration between
of Kurt Lewin88, and developed further across multiple fields in different people to achieve common goals94,95. The second, referred
the 1970s, such as research in industrial organization89, adaptive to as ‘descriptive’, examines how science and society constantly
environmental management and assessment90, as well as through shape each other in expected but also unexpected ways. This
studies of oppressed communities in developing regions91. Post- approach regards all knowledge as being continually shaped and
normal science introduced the idea of participation through co-produced by the current social order96. Despite some tensions
an extended peer community to deal with societal problems among the approaches they often merge with each other in
characterized by high uncertainties and high decision stakes85. various ways11,97. For instance, many approaches to knowledge
Further development of this new state of science, that stressed co-production begin with a descriptive account (for example, the
mission-oriented problem-solving was articulated under the name world is complex, science and society are constantly shaping each
of ‘mode 2’ knowledge production84. other), and use this as the basis for a normative account of what
The concept of co-production encompasses all of these ideas. should occur (for example, science–society interactions should be
One of the first uses of the term ‘co-production’ was by Elinor and developed in terms of participation/inclusion and acceptance of
Vincent Ostrom in the 1970s as a way to describe how public services the validity of multiple perspectives).

issue, research the problem, and then deliver knowledge to society, Four principles of knowledge co-production
in favour of more interactive arrangements between academic and Based on the literature and experiences and perspectives of leading
non-academic actors (Box 1). Over the past decade, knowledge researchers and practitioners engaged in knowledge co-production
co-production has shifted from niche areas of scientific practice around the world (for methodological details, see Supplementary
towards the mainstream6. Within international science and policy Information), we define knowledge co-production in the context of
fora there is a growing expectation that shifting towards co-pro- sustainability research as:
duction will enable science to have greater impact on sustainable ‘Iterative and collaborative processes involving diverse types of
development outcomes. This has led to substantial commitments to expertise, knowledge and actors to produce context-specific knowl-
knowledge co-production. Examples include the strategic plans for edge and pathways towards a sustainable future.’
sustainability research in countries such as Switzerland, Australia, We describe co-production processes as ‘iterative’ because we
the United Kingdom and Germany; the decadal strategic plan of the find that there is no single approach for success, and ‘collaborative’
US Global Change Research Program7 and the focus of international because the act of engagement across domains and disciplines can
research networks such as the Programme on Ecosystem Change be as important for the pursuit of sustainability as the production
and Society8, the Global Land Programme9 and Future Earth10. of knowledge. Compared to disciplinary research processes, knowl-
However, these commitments outpace the development of guiding edge co-production extends from a collaborative stage of problem
definitions of what knowledge co-production is and frameworks framing and trust building, through knowledge gene­ration, to a
to assess its quality or success. Indeed, while the term has become phase of exploring the practical impacts of the process. Our defini-
commonplace in sustainability research, the ways in which it is con- tion emphasizes that co-production processes produce more than
ceptualized and implemented are highly variable11. This contributes just knowledge; they develop capacity, build networks, foster social
to the creative use of the concept, but also limits the ability to assess capital, and implement actions that contribute to sustainability.
and learn from the outcomes and thus improve practice. The high context-specificity associated with knowledge co-produc­
In this Perspective, we draw upon our collective experiences work- tion precludes a more prescriptive definition. However, we propose
ing within diverse sustainability co-production processes—as well as four general principles that contribute to high quality co-production
recent theory and empirical practice from fields such as participa- for sustainability. Specifically, we suggest that processes should be:
tory research and transdisciplinary research—to propose a definition (1) context-based; (2) pluralistic; (3) goal-oriented; and (4) interac-
of knowledge co-production for sustainability research. We identify tive (Fig. 1). We explore these principles in more detail in this sec-
four principles that underpin high-quality co-production that can tion and describe three case studies of knowledge co-production to
guide researchers, practitioners, programme managers and funders explicitly highlight some of the practical nuances in applying the prin-
seeking to engage in co-produced sustainability research. We con- ciples (Boxes 2–4). We recognize that there is some overlap between
tinue by presenting a set of considerations for monitoring and evalu- the principles. Even though some principles underpin the application
ating how these principles are put into practice. Finally, we identify of others, they are deliberately not presented in a sequential manner.
key advances that will improve the abilities of researchers, practitio-
ners and funders to engage in meaningful co-productive practices (1) Context-based. Co-production processes should be consi­dered
and address the sustainability challenges of the Anthropocene. and situated within the particular social, economic and eco­logical

Nature Sustainability | www.nature.com/natsustain

NatURe SUstainaBility Perspective
Knowledge co-production for sustainability research

Context- Goal-
based Pluralistic oriented Interactive

Situate the process in a Explicitly recognize the Articulate clearly defined, Allow for ongoing
particular context, place, multiple ways of shared and meaningful learning among actors,
or issue knowing and doing goals that are related to the active engagement and
challenge at hand frequent interactions

Fig. 1 | Principles for knowledge co-production in sustainability research. High-quality knowledge co-production for sustainability should be context-
based, pluralistic, goal-oriented and interactive.

Box 2 | Future-proofing conservation in Colombia

This project involved researchers, civil society, local practitioners landscapes. Some elements of interaction were made difficult by
and conservation advisers to develop new ways of understanding the dispersed locations of the project team (Colombia, United
and managing Colombian protected areas in the face of ongoing Kingdom, Switzerland and Australia). Regular conference calls
ecological change. The process was catalysed by the Luc Hoff- helped to support dialogue and biannual face-to-face meetings in
mann Institute, a boundary organization that partners closely with Colombia built shared understandings, addressed tensions, and
World Wide Fund for Nature (WWF) offices around the world. supported progress towards project goals. On reflection, these
The project was situated within the context of the goals required a degree of interaction and iteration that would be
REDPARQUES Declaration, a commitment made by 18 Latin more suited to a project team that was co-located (or at least on the
American countries at the 2015 United Nations Climate Change same continent or time zone). This learning suggests that teams
Conference in Paris to integrate protected areas into climate should carefully consider project goals in relation to resources and
mitigation and adaptation. This provided a political window capacities required to develop shared perspectives and activities.
of opportunity, together with national processes such as the The project attempted to utilize participatory evaluation
development of Colombia’s Nationally Determined Contribution scorecards to facilitate dialogue and learning. While this tool was
(NDC) and a process to revise the Colombian protected area often successful, it was sometimes difficult to dedicate sufficient
planning and management framework (PAMF). The project built time to complete the scorecard. This points to a tension between
on a longstanding relationship between WWF Colombia and the the need for reflection, and continuing progress within short
Colombian protected area agency around climate change. The time horizons of a project. An external evaluation (that drew on
close collaboration with WWF enabled an international research project documentation and interviews) was conducted prior to
team who did not speak Spanish, nor had prior history of working the project’s completion, and missed some of the broader impacts
in Colombia, to participate in this project. because they had yet to take effect.
Project goals were co-developed over a one-year period, While this project faced challenges, it is widely heralded by
building on the existing work of individuals and organizations partners as a success. The methodology has been completed98,
within the team. This created a co-dependency within the project and elements of it are now being used, adapted and further deve­
goals, whereby policy and research objectives could not be realized loped by project partners. Four factors were critical to the project’s
in isolation. The project would have been designed very differently success: (1) alignment between project goals and the individual
were it simply a policy project or a research project and at times motivations and organizational incentive structures of participants;
there were challenges for both researchers and practitioners to (2) the political window of opportunity created by Colombia’s
appreciate the complexities of each other’s worlds. NDC and the REDPARQUES Declaration and the revision of
Then followed a two-year co-production process (2015–2017) the PAMF; (3) iterative and flexible methodology; and (4) the
where the team developed and piloted a methodology in two commitment and respect for diverse perspectives within the team.

contexts in which they are embedded, and the associated How will policy, regulatory, institutional and cultural factors shape
confines and opportunities of the surrounding circumstances. the process and the realization of desired outcomes? For example,
A co-production process can be place-based, but ‘context’ is not the co-production of drought information systems by Native
synonymous with ‘local’; it could be national, regional, global or American communities and researchers in the US Southwest was
even scale-agnostic, but restricted to a defined set of issues. Context- shaped by financial (for example, no investments for more weather
based co-production also means taking into account the different stations) and technological factors (for example, slow Internet con-
needs, interests and beliefs of the different social groups who are nections and limited data-handling infrastructure), which resulted
invested in or affected by the challenge at hand12,13. in context-specific solutions of combining local observations with
Situating a co-production process involves asking questions a structured monitoring framework14. Identifying policy windows
about how a particular challenge has emerged, and how changing or entry points within a given context can provide a tangible start-
circumstances are likely to influence the work (Boxes 2–4). Such ing place to consider how a knowledge co-production process
questions include: who will be impacted or affected by the process can contribute to the pre-existing goals and objectives of partners
and its outcomes? Who has the power to enable or constrain action? (Box 2). It is also critical that co-production processes are described

Nature Sustainability | www.nature.com/natsustain

Perspective NatURe SUstainaBility

Box 3 | Alternative livelihoods in Papua New Guinea

The borderland region of the South Fly, Papua New Guinea (PNG) pluralism to be reflected in the actors involved in co-production. For
is one of the poorest regions in the world. This project wanted to instance, in a traditional society like PNG some aspects of decision
understand how sustainable fisheries and community well-being making are gender specific, and females may not traditionally
in the region could be assured while at the same time reducing participate. They are nevertheless key to ensure the workability of
illegal and unsustainable activities, such as illegal trade in fisheries a co-produced solution. Access to societal groups which are more
supply chains99. difficult to engage, was facilitated by female researchers connecting
Community leaders and non-governmental organizations with females in the community who subsequently transferred
(NGOs) were instrumental in connecting a multi-cultural group the perceived trustworthiness of project team members to local
of researchers (from Australia, Indonesia and PNG), to managers community members. On the flipside, co-production could have
(mainly from the PNG fisheries agency), local translators and been derailed if trust was broken by inadequate knowledge of local
interviewers, actors in the fisheries sector and supply chain (fishers, customs and taboos. Sensitivity around getting the right mix of
middlemen/women, buyers and exporters), and community people in the room was facilitated by having separate aspects of
members. Project ownership was partly achieved through early co-production in gender safe spaces. Taking a gender-sensitive
involvement of local grassroots NGOs and community leaders in approach revealed some important gender-related consequences
collaboratively designing the research and developing agendas for in marine product trade. In the co-production process, it was also
meetings and workshops. important to be sensitive to different learning methods. Simple
Situating this project was not straightforward, given the remote and printed infographics provided important leverage points for
economically disadvantaged nature of PNG, where education levels discussion and co-production of results.
and literacy rates are very low, and local cultural traditions are strong. Ultimately, solutions to local fisheries and supply chain
The project benefitted from prior relationships; the continuity in the issues were co-produced with local PNG fishermen and women,
researchers returning to the area multiple times per year over at least middlemen/women and traders. Twelve solutions were identified
five years (for different projects and purposes) meant they became drawing on different knowledge systems, perspectives and
familiar faces and led to more trust and consequent engagement. understanding of the key issues. This approach has led to increased
Because the project focused on the sensitive issue of illegal trade, this local understanding of complex supply chain links and present
trust also reduced the fear of being prosecuted or being exposed to trade dynamics. The belief that the project could bring tangible
adverse project consequences. Aside from familiarity, support for the changes to the communities (in the long term) meant continued
project was created through productive and respectful connections and enthusiastic participation. Moreover, the lengthy and
to key individuals (community ‘gatekeepers’) which was essential for continuous involvement of project proponents with local people
gaining representative community interactions. in the communities, their connections to several institutions and
A diverse team with a good mix of ages, seniority, gender and local NGOs meant they were well informed of extant local issues
cultural backgrounds was key to encouraging a similar level of and potential points of conflict.

Box 4 | Managing ecosystem services in Canada

The Montérégie Connection project focused on improving land- The project continued with four years of field research, and
use management for the provision of multiple ecosystem services research into historical land-use change and provision of ecosystem
in the Vallée-du-Richelieu Municipalité Régionale de Comté services in the past. This was followed by community-based scenario
(MRC), a 75 km2 regional governance body involving 13 towns development and modelling of future ecosystem service provision
southeast of Montréal. The project was developed by a pluralistic under each set of scenarios. Researchers interacted frequently with
collaboration of researchers from multiple departments at McGill landowners on whose property they were undertaking fieldwork,
University, alongside members of the boundary organization met with other formal partners (co-signatories on the grant that
Centre de la Nature de Mont Saint Hilaire (CN) and a variety of funded the work) annually or semi-annually, and convened a
local actors, including farmers, mayors and other government larger group of local actors at least annually. At those meetings,
representatives, land-use planners and NGOs100. feedback on the project progress was collected, helping to adjust
The process was situated in a context in which political action, the project as it progressed to make sure it was meeting contextual
including Montréal’s commitment to the Aichi target of preserving needs and goals. The Montérégie Connection project received
17% greenspace, was pulling towards recognition of the value of the Alice Johannsen award for contribution to the protection of
greenspace. Local land-use planners recognized that the science local nature, and many participants spoke about a change in their
needed to make good decisions about which 17% to preserve was attitude towards landscape sustainability and ecosystem services
not as coherent as they would like. This provided an entry point as a result of having participated in the project.
and a policy window that eased the process of agreeing on an One of the critical factors in the success of the Montérégie
overarching goal, despite the pre-existing divergence of goals and Connection project was the strong relationship with the boundary
objectives of partners. organization CN. For 15 years prior to the start of the project, the
However, coming to an agreed-upon set of goals required CN had worked to raise public awareness of the importance and
the whole first year of the project to be dedicated to working vulnerability of the local natural heritage, focusing on the Mont
interactively with the community to determine project goals Saint Hilaire Biosphere Reserve and surrounding communities.
that were both scientifically compelling and useful for them. The CN had also been working with researchers at McGill and
Most of those interactions were spearheaded by the CN, understood both the scientific process and needs of the scientists
who helped researchers understand how land-use planning working on the project, in addition to the needs of the community.
decisions were made in the region, and to identify key actors to This allowed the CN to bring both together around a common set
engage in the process. Actors were asked about their concerns for of goals. The CN was also able to foresee many potential issues
the future of the region during small workshops or one-on-one such as power differentials or political roadblocks which could
meetings. have stalled the project.

Nature Sustainability | www.nature.com/natsustain

NatURe SUstainaBility Perspective
in contextually relevant language and based on a shared under- creation of new relationships and networks of collaboration37. Once
standing of key concepts and terminology8. overarching goals are established, participants should work together
to identify meaningful milestones (that is, stepping-stone goals) to
(2) Pluralistic. Co-production of knowledge must explicitly achieve and monitor progress. This facilitates learning, increas-
recog­nize the multiple ways of knowing and doing. Feminist and ing the likelihood of achieving the desired broad-scale outcomes38
standpoint theorists, among others, have persuasively argued that and reducing the potential for hidden agendas to undermine the
all knowledge is inevitably situated and partial15, highlighting progress and the legitimacy of co-production processes39. As in
the practical and ethical importance of ensuring a range of perspec- transdisciplinary research, sequenced milestones should be identi-
tives on a given issue. Achieving pluralistic co-production entails fied and revised in iterative processes through which the different
bringing together academics (from different disciplines) and people resources, aims and values at stake are negotiated and discussed40.
from other sectors (from for example, government, business, civil It is important to recognize that there are often multiple possible
society, local and indigenous communities) to generate knowledge pathways to reach an agreed goal41.
and catalyse change. It is important to ensure that those involved When setting goals and milestones, it is important to carefully
represent a range of skills (for example, analysis, translation, syn- avoid the trap of focusing only on what is measurable42. The com-
thesis, facilitation and evaluation) and types of knowledge and plexity of the contexts in which co-production processes typically
expertise (for example, experiential, local, traditional, academic take place makes it difficult to draw direct causal relations between
and official). This diversity generates an enriched understanding of actions and outcomes. Moreover, important outcomes, such as
the ecological, political and technical aspects of a sustainability increased agency of previously marginalized actors, are difficult to
challenge16. Moreover, research suggests that under the right measure, not the least because such outcomes might only become
conditions, knowledge outcomes are enhanced by including visible over time and are therefore not captured in the timescale of
various other dimensions of diversity, such as gender, ethnicity, age a project12. Finally, recognizing that visions and goals often evolve
and nationality17. during a project, and that goals are sometimes contested, it is essen-
Pluralistic processes do pose risks and challenges not present tial to collectively revisit the goals regularly in an adaptive approach
in more traditional research practices, and can increase transac- and allow for iteration and reflexivity. A ‘theory of change’ that gen-
tion costs18–20. For example, the convenor of a co-production pro- erates a hypothesis and describes assumptions about how a set of
cess faces the task of assembling an appropriately broad coalition activities will contribute to a desired change can be used to develop
of relevant actors, while keeping the process manageable within a shared understanding of goals, objectives, metrics for success and
practical and strategic limits21,22 (Box 3). Techniques such as stake- the design of co-production processes43.
holder mapping23 and social network analysis24 can help address
this challenge. A step-wise approach to participation can reduce (4) Interactive. High-quality co-production requires frequent
potential points of conflict or allow for some facilitating steps to be interactions among participants to occur throughout the process,
undertaken before a broader group is engaged (Box 3). Preferably, extending from collaboratively framing and designing the research
the coalition will include the relevant expertise, experience and agenda, to conducting the research, and jointly using and dissemi-
interests that are needed to tackle the sustainability challenge in a nating the knowledge generated. It is critical to avoid token partici-
way that provides solutions and contributes to the related scientific pation (for example, passive engagement at the onset or completion
body of knowledge12. Another challenge relates to power dynam- of a project, with one-way communication flows) and instead cre-
ics in participatory processes. A failure to sufficiently engage with ate active engagement through frequent interactions and repeated
power imbalances lessens the quality of the engagement and pro- conversations44. A growing body of literature sheds light on how
cess outcomes, and can derail and undermine the entire exercise25. the amount, timing and type of interactions influence the quality of
Asymmetrical power relations can prevent some actors from engag- knowledge co-production processes45,46.
ing in knowledge co-production26 and will reproduce knowledge High levels of participant interaction throughout a co-produc-
hierarchies, in which certain knowledge and expertise are seen as tion process nurture ongoing learning among participants39 and can
being more legitimate than others27,28. Tools like the ‘Power Cube’ build trust through dialogue. Interaction also increases the likeli-
(www.powercube.net) can help participants to map the different hood that the resulting knowledge is perceived by participants and
ways power manifests itself29. other end-users as credible (scientifically robust arguments and
While helpful, these tools will not provide easy solutions and in outputs), salient (relevant to user needs), and legitimate (the extent
the real world choices often have to be made quickly on the basis to which the information is perceived as fair and respectful of all
of practical judgement. Systematically reflecting on and reviewing actors). This makes the knowledge more likely to be incorporated
the process is vital for unearthing the visions, understandings and into decision-making processes8, and increases the chances that
values of the actors involved27,28,30, identifying positions of power and the knowledge produced meets specific needs and expectations45
sources of inequity, and developing ameliorative strategies31. There (Box 2). Finally, sharing experiences, ideas and values through
is also growing evidence that pluralistic processes can be improved frequent interactions can facilitate collective action47, which is
by individuals, teams or organizations that possess broad knowledge critical to address sustainability challenges that are mired in uncer-
across multiple domains (for example, science, policy and practice) tainty, change and surprise42.
and skills that foster learning, trust and conflict resolution. These
roles have been variously referred to as boundary spanners/orga- Assessing knowledge co-production
nizations32,33 (Box 4), bridging organizations34, knowledge brokers35 We have suggested that high-quality knowledge co-production is
and epistemediators36. context-based, pluralistic, goal-oriented and interactive. How these
principles are put into practice will differ depending on the spe-
(3) Goal-oriented. Knowledge co-production for sustainability is cific challenge at hand, the actors involved, the social and politi-
problem-focused and benefits from clearly defined and meaningful cal contexts, and the scale of the project. The principles are not a
goals shared among participants (Box 3). It is important to develop definitive list, and we hope that they serve as a stimulus for further
a collective understanding among all participants of the challenge(s) discussion and their continued refinement. Improving the practice
at hand, as well as an agreed measure of success (that is, the over- of knowledge co-production also requires better monitoring and
arching goal). Success can take many forms, and includes changes in evaluation of co-production practices, and in particular practices
policies and practices, changes in attitudes and perceptions, or the that can capture complexity and manage for emergent outcomes42,48.

Nature Sustainability | www.nature.com/natsustain

Perspective NatURe SUstainaBility

Co-production is founded on the assumption that the relationships formative evaluation (that is, evaluation that is performed while
between science, policy and practice are complex, multi-pathway co-production is in progress) conducted by an extended peer
and nonlinear; thus, evaluation frameworks need to align with these group comprising experts from both science and practice. Such
assumptions. Monitoring and evaluation will also need to account iterative learning also enables the subsequent steps and phases to
for different conceptions of ‘success’ among participants and be reshaped if necessary12.
projects. For example, knowledge co-production may be pursued
as a way to enhance the legitimacy of research outcomes, to Assessing the context-based principle. Monitoring and evaluation
ensure the implementation of scientific knowledge in society, or in of context-based quality will focus on the degree to which a co-pro-
recognition of the limits of scientific expertise and the value of duction process is effectively situated within a particular place, set
complementary perspectives10. In search of such approaches, we of relationships or a particular issue. For example, did the request
turn to recent work on the evaluation of research impact38,49 and for co-production originate from an actor already encountering
transdisciplinary practice50–54. the problem addressed, such as a community organization or gov-
Impact is conceptualized and defined in many different ways. ernment department? Are the goals of the co-production process
For some, ultimate impact is changes in ecosystem health55, soci- linked to the existing priorities and activities of partners directly
etal change54 or changing peoples’ lives56. Others break impact down working in the particular context? Is the process utilizing, building
into components or dimensions such as research quality, research upon and strengthening existing skills and relationships between
relevance, stakeholder knowledge and stakeholder practices57. participants already working in the context? Are the skills and out-
Wiek et  al.58 divide impacts into direct and indirect effects, to puts (for example, co-produced solutions) developed during the
incorporate intangible impacts such as building networks and process still being used and implemented by the community of par-
capacities, alongside the development of more concrete products ticipants after the initial project is finished18?
and outputs. Pitt et  al.59 identify a discrepancy between processes It is vital that this evaluation is itself conducted according to con-
geared towards producing high-quality research outputs, versus cepts and language relevant for the place, issue and participating
those focused on creating changes in policy and practice, or those actors. Participatory evaluation frameworks and methods are use-
aiming at enabling or informing decision making processes. Within ful for ensuring that the terms of assessment are negotiated by the
this literature, it is clear that all approaches assessing research actors involved65. Such approaches help to capture the true value of a
impact are underpinned by particular philosophical assumptions60, co-production process for those working within the particular con-
and that there is no ‘best procedure’51. text or issue, reveal unexpected impacts of the work and prompt the
Within transdisciplinary research, the articulation of guiding articulation of new context-specific projects and knowledge needs.
principles12,13 has led to a more nuanced approach to evaluation51,61.
For example, several studies have used the degree of stakeholder Assessing the pluralistic principle. Metrics of pluralistic quality
participation, information flow and levels of collaboration across will capture the different elements of diversity within a co-produc-
the entire transdisciplinary research process to establish analytical tion process. This may include simple measures of inclusiveness that
frameworks62. Similarly, evaluations of different transdisciplinary capture the involvement of actors across multiple axes (for example,
research efforts have allowed for the identification of key factors disciplines, sectors, countries/regions, gender and age) and proce­
supporting successful practice; for example, the establishment dural justice (for example, number of contributions by different
of communities of research and practice where participants can types of actors). It may also include considering the degree to
build mutual trust, interact with different knowledge systems, and which the process enables participants to build trust and develop
jointly develop a shared understanding of the problem at hand63. shared perspectives and understandings63, and potentially more
Others present principles of quality drawing from the cred- complex metrics that assess diversity in mental models and
ible, salient, legitimate criteria53–55 and then present indicators for knowledge systems66.
evaluation within those. Wickson and Carew52, for example, pres- While such quantitative indicators are important, they can-
ent the following principles for evaluating socially responsible not capture the full breadth and depth of a pluralistic process67.
innovation: socially relevant and solution oriented; sustainability Evaluating this principle will therefore also require qualitative indi-
and future scanning; diverse and deliberative; reflexive and respon- cators and approaches that capture whether the process is allow-
sive; rigorous and robust; creative and elegant; honest and account- ing the knowledge and perceptions of different participants to be
able. The authors then present a rubric with ranges of quality from mobilized and articulated into forms that can be shared with others16.
‘exemplary’ to ‘routine’. Still other scholars arrange indicators into Evaluation and assessment methods may include unconventional
categories of some variation of ‘impact dimension,’ including con- forms of collecting evidence, such as narrative indicators, written
text, process, outcome and impact48; or research problem, research reflections and blogs. For example, video diaries have been shown
process and research results64. to be effective when running throughout a co-production process
Across these approaches to evaluation, most include metrics as a form of live evaluation as opposed to participant interviews
that focus on the process, outputs, outcomes and impacts of the conducted at the end of a project68. Similarly, short, periodic sur-
co-production or transdisciplinary research process. Complexity- veys to evaluate the group dynamics of participatory research efforts
oriented evaluation frameworks emphasize the importance of can ensure the project is on track to meet participants’ needs and
learning and change over time, and focus on evaluating the qual- learning objectives69.
ity of processes, relationships and networks48,59. We build on these
approaches to identify critical aspects of evaluation strategies Assessing the goal-oriented principle. Evaluation of goal-oriented
for context-based, pluralistic, goal-oriented and interactive knowl- quality will focus on the degree to which a co-production process
edge co-production. As with our definition and principles, our enables the articulation, revision and achievement of desired goals,
guidance for monitoring and evaluation is necessarily broad outcomes and impacts. Goals will include both tangible and intan-
given the context-specificity of all co-production processes, and gible outcomes. The former can be assessed by metrics that capture
is intended to be illustrative rather than an exhaustive list. While if knowledge generated by the process is directly informing
monitoring and evaluation of co-production faces many practi- management or policy decisions48. However, not all co-produced
cal challenges, it should not require a disproportionate share of research will necessarily lead to policy impacts. Impacts can be dif-
resources or overburden participants. The main purpose of moni- ficult to measure with any certainty, because of the complex link-
toring should be to improve the ongoing process. This requires ages between knowledge and action70,71 and the long timescales

Nature Sustainability | www.nature.com/natsustain

NatURe SUstainaBility Perspective
over which impacts arise, which make attribution challenging72. The intertwined social and ecological dynamics of the
However, focusing on the goals, for example, building relationships, Anthropocene mean that local and place-contexts are influenced by
will help identify appropriate evaluation approaches49. multiple drivers at larger scales, and have complex connections to
A high-quality co-production process may often lead to less other places. In a globalized world, these drivers (for example, trade,
conventional outcomes such as building new understandings, rela- international commodity prices, technological improvements,
tionships and social networks. While these outcomes may be at large-scale land acquisitions, fishing and agriculture) are themselves
the level of individual participants and their understanding, rather shaped by a complex array of forces. State power and supranational
than at the scale of an entire social–ecological system, they can co-operations (for example, the European Union) coexist alongside
pave the way for larger-scale and longer-term changes. Evaluation powerful multinational corporations. At the same time, research
approaches that measure contribution to change throughout a institutions and NGOs increasingly operate at a global scale. How­
process can therefore be more appropriate than those focused on ever, knowledge co-production processes to date have predomi-
direct attribution of cause and effect at the end of a process73. It is nantly involved pluralistic coalitions at a local to regional level
also critical to develop temporally dynamic approaches that gener- between academics and actors from other sectors, such as local or
ate rapid feedback and real-time data to enable a project to con- national government representatives, business representatives, local
tinually evolve to meet its goals, such as developmental evaluation and regional NGOs, and natural resource managers (for example,
approaches74, and that track impact from short to long term (for Boxes 2–4). For sustainability researchers committed to instigating
example, monthly to decadal)75,76. Relevant short-term indicators change, this requires considering such cross-scale dynamics, how to
can include enhanced capacity to address the sustainability chal- engage in co-production, and with whom.
lenge and increased attention of non-academic actors or media Approaches to knowledge co-production for a sustainable
attention. Medium-term indicators relate to the degree to which the Anthropocene may entail new alliances and more direct engage-
co-production process or its outputs are used by partners or other ment with global corporate actors to leverage their unique ability to
non-academic actors to inform actions, strategies, policies or prac- influence change. For example, Österblom et al.77 described an ongo-
tice. Finally, longer-term indicators concern the attainment of social ing co-production process where researchers are actively engaging
or environmental goals. However, given that longer-term impacts with keystone actors that shape marine ecosystems, to collabora-
will typically be realized well beyond the life of a project, there is a tively develop solutions to ocean sustainability challenges. This pro-
need to be pragmatic about what can be measured during the pro- cess has led to the establishment of a unique global ocean initiative,
cess itself. Consideration must also be given to ongoing monitor- where science and business collaborate toward the United Nations
ing following the completion of a co-production process. Tracking Sustainable Development Goals (www.keystonedialogues.earth).
progress against indicators that align with the project theory of While emerging forms of co-production may produce remarkable
change enables monitoring against goals at different temporal scales gains, they will bring new risks and potential pitfalls. Engaging with
within the project timeframe. industry can influence perceived scientific credibility and also chal-
lenge previously established power dynamics.
Assessing the interactive principle. Metrics of interactive qual- We envision a new and significant role of knowledge co-pro-
ity will capture the nature, frequency and quality of interactions duction in the Anthropocene to navigate transformations towards
between participants in a co-production process. Simple quantita- sustainability78. Transformations that set us on sustainable tra-
tive tools (such as attendance lists and meeting minutes) can be used jectories need to happen rapidly, and will require massive social
to measure frequency and timing of encounters. Richer qualitative changes (shifts in underlying values, assumptions, cultures and
approaches can be used to assess the quality of interactions, such as worldviews) coupled to technological progress and biosphere
interviews and surveys to assess if actors perceive they had equitable stewardship. Research on socio–technical transitions79, social–eco-
opportunities to participate in project activities48. logical transformations80 and leverage points81 suggests that many
Assessing this principle should also focus on capturing learning, sustainability interventions applied to date have been easy to make,
how the perceptions of actors change throughout the process and but with limited potential for transformational change. If knowl-
the degree to which a shared perspective on the problem emerges. edge co-production processes are to contribute to the call for deep
The process of interactively engaging in co-production should sustainability transformations78, they need to recalibrate goals
change how participants view a problem as they learn through from dealing with proximal problems and quick fixes, towards
inquiry, analysis and the sharing of and triangulation amongst more explicitly targeting the underpinning, ultimate drivers of
diverse perspectives, and potentially trigger reflexive processes current trajectories.
about the kinds of knowledge and action needed to address sustain- Deeper leverage points for transformational change include
ability challenges. In cases where differences in perspective remain, reconnecting people to nature, restructuring institutions, and
there should be evidence of respect for the diversity of perspectives. rethinking how knowledge is created and used in pursuit of sus-
Other indicators could focus on assessing whether the co-produc- tainability81. Inevitably this means that knowledge co-production in
tion process has resulted in a mutual understanding in the face of sustainability research will require more attention to mechanisms
differences between actors (for example, through communication of social change including values, politics and power. Researchers
that uses a language and terms that can be understood by all actors) engaged in co-production of knowledge will become partners in
and whether criteria of credibility (of different knowledge systems) value-laden and contested societal change processes. These nor-
have been dealt with in a respectful way16. mative and political dimensions should not be brushed under the
carpet, but rather addressed transparently and head on. For
Knowledge co-production in the Anthropocene example, co-produced research on the emerging global networks
The Anthropocene is the first time in the Earth’s history that human addressing interlinked environmental, human and animal health—
activities are influencing the global functioning of the Earth system1. so-called ‘One Health’ and ‘Planetary Health’ agendas—has explic-
Actions occurring over the next decades could significantly influ- itly explored the hierarchies and power relations between medical,
ence the long-term trajectory of the Earth system, and potentially veterinary, ecological and social scientists and policymakers, as
lead to conditions that would challenge the existence of human well as the gaps between global concepts and locally situated knowl-
civilization4. We conclude by identifying some particular opportu- edge and practices82. Doing so requires an open, deliberative and
nities and challenges for sustainability knowledge co-production reflexive attitude of sustainability researchers addressing values
within this context. and politics in knowledge co-production83.

Nature Sustainability | www.nature.com/natsustain

Perspective NatURe SUstainaBility

Finally, we urge for a rapid development of new institutions 23. Reed, M. S. et al. Who’s in and why? A typology of stakeholder analysis
and incentive structures across the science–business–policy– methods for natural resource management. J. Environ. Manag. 90,
1933–1949 (2009).
practice spectrum. While co-production has gained currency in 24. Bodin, Ö., Crona, B. & Ernstson, H. Social networks in natural resource
recent decades, many researchers still face incentive structures that management: what is there to learn from a structural perspective? Ecol. Soc.
primarily reward disciplinary science that does not engage with 11, r2 (2006).
society45. At the same time, many practitioners work within organi- 25. Brandt, F., Josefsson, J. & Spierenburg, M. Power and politics in stakeholder
engagement: farm dweller (in)visibility and conversions to game farming in
zations that do not incentivize critical reflection, ongoing learning
South Africa. Ecol. Soc. 23, 32 (2018).
and revision of actions. Moreover, the development of a new social 26. Mobjörk, M. Consulting versus participatory transdisciplinarity: a refined
contract based around co-production requires changes among those classification of transdisciplinary research. Futures 42, 866–873 (2010).
who have the potential power to reshape institutions. For instance, 27. Wittmayer, J. M. & Schäpke, N. Action, research and participation: roles of
co-production demands that the scientific community incorporates researchers in sustainability transitions. Sustain. Sci. 9, 483–496 (2014).
28. Reed, M. S., Stringer, L. C., Fazey, I., Evely, A. C. & Kruijsen, J. H. J. Five
notions of reflexive practice and multiple knowledges into their principles for the practice of knowledge exchange in environmental
working practices. While such shifts may be challenging, they are management. J. Environ. Manag. 146, 337–345 (2014).
necessary if co-produced knowledge is to grow fast enough to meet 29. Gaventa, J. Finding the spaces for change: a power analysis. IDS Bull. 37,
the sustainability challenges presented by a rapidly changing world. 23–33 (2006).
30. Popa, F., Guillermin, M. & Dedeurwaerdere, T. A pragmatist approach to
Received: 12 December 2018; Accepted: 12 November 2019; transdisciplinarity in sustainability research: from complex systems theory
to reflexive science. Futures 65, 45–56 (2015).
Published: xx xx xxxx 31. Polk, M. Transdisciplinary co-production: designing and testing a
transdisciplinary research framework for societal problem solving. Futures
References 65, 110–122 (2015).
1. Rockström, J. et al. A safe operating space for humanity. Nature 461, 32. Bednarek, A. T. et al. Boundary spanning at the science–policy interface:
472–475 (2009). the practitioners’ perspectives. Sustain. Sci. 13, 1175–1183 (2018).
2. Raudsepp-Hearne, C. et al. Untangling the environmentalist’s paradox: why 33. Cvitanovic, C., Löf, M. F., Norström, A. V. & Reed, M. S. Building
is human well-being increasing as ecosystem services degrade? BioScience university-based boundary organisations that facilitate impacts on
60, 576–589 (2010). environmental policy and practice. PLOS ONE 13, e0203752 (2018).
3. Nash, K. L. et al. Planetary boundaries for a blue planet. Nat. Ecol. Evol. 1, 34. Hahn, T., Olsson, P., Folke, C. & Johansson, K. Trust-building, knowledge
1625–1634 (2017). generation and organizational innovations: the role of a bridging
4. Steffen, W. et al. Trajectories of the Earth system in the Anthropocene. organization for adaptive comanagement of a wetland landscape around
Proc. Natl Acad. Sci. USA 115, 8252–8259 (2018). Kristianstad, Sweden. Hum. Ecol. 34, 573–592 (2006).
5. Cash, D. W. et al. Scale and cross-scale dynamics: governance and 35. Miller, T. R. et al. The future of sustainability science: a solutions-oriented
information in a multilevel world. Ecol. Soc. 11, 8 (2006). research agenda. Sustain. Sci. 9, 239–246 (2014).
6. The best research is produced when researchers and communities work 36. Wiek, A. Challenges of transdisciplinary research as interactive knowledge
together. Nature 562, 7 (2018). generation – experiences from transdisciplinary case study research.
7. Weaver, C. P. et al. From global change science to action with social GAIA - Ecol. Perspect. Sci. Soc. 16, 52–57 (2007).
sciences. Nat. Clim. Change 4, 656–659 (2014). 37. Leach, M. et al. Local disease–ecosystem–livelihood dynamics: reflections
8. Balvanera, P. et al. Key features for more successful place-based from comparative case studies in. Africa. Philos. Trans. Royal Soc. B 372,
sustainability research on social-ecological systems: a Programme on 20160163 (2017).
Ecosystem Change and Society (PECS) perspective. Ecol. Soc. 22, 38. Earl, S., Carden, F. & Smutylo, T. Outcome Mapping: Building Learning and
14 (2017). Reflection into Development Programs (IDRC, 2001).
9. Verburg, P. H. et al. Land system science and sustainable development of 39. Moser, S. C. Can science on transformation transform science? Lessons
the earth system: a global land project perspective. Anthropocene 12, from co-design. Curr. Opin. Environ. Sustain. 20, 106–115 (2016).
29–41 (2015). 40. Wiesmann, U. et al. in Handbook of Transdisciplinary Research (eds Hadorn,
10. van der Hel, S. New science for global sustainability? The G. H. et al.) 433–441 (Springer, 2008).
institutionalisation of knowledge co-production in Future Earth. Environ. 41. Leach, M., Stirling, A. C. & Scoones, I. C. Dynamic Sustainabilities
Sci. Policy 61, 165–175 (2016). (Routledge, 2010).
11. Bremer, S. & Meisch, S. Co-production in climate change research: reviewing 42. Carpenter, S. R., Folke, C., Scheffer, M. & Westley, F. Resilience: accounting
different perspectives. Wiley Interdiscip. Rev. Clim. Change 8, e482 (2017). for the noncomputable. Ecol. Soc. 14, 13 (2009).
12. Lang, D. J. et al. Transdisciplinary research in sustainability science: 43. Archibald, T., Sharrock, G., Buckley, J. & Cook, N. Assumptions, conjectures,
practice, principles, and challenges. Sustain. Sci. 7, 25–43 (2012). and other miracles: the application of evaluative thinking to theory of change
13. Pohl, C. & Hadorn, G. H. Frameworks for transdisciplinary research: models in community development. Eval. Program Plan. 59, 119–127 (2016).
framework #1. GAIA - Ecol. Perspect. Sci. Soc. 26, 232–232 (2017). 44. Reid, R. S. et al. Evolution of models to support community and policy
14. Ferguson, D. B., Masayesva, A., Meadow, A. M. & Crimmins, M. A. Rain action with science: balancing pastoral livelihoods and wildlife conservation
gauges to range conditions: collaborative development of a drought in savannas of East Africa. Proc. Natl Acad. Sci. USA 113, 4579–4584 (2016).
information system to support local decision-making. Weather Clim. Soc. 8, 45. Dilling, L. & Lemos, M. C. Creating usable science: opportunities and
345–359 (2016). constraints for climate knowledge use and their implications for science
15. Haraway, D. Situated knowledges: the science question in feminism and the policy. Glob. Environ. Change 21, 680–689 (2011).
privilege of partial perspective. Fem. Stud. 14, 575–589 (1988). 46. Sarkki, S. et al. Adding ‘iterativity’ to the credibility, relevance, legitimacy: a
16. Tengö, M. et al. Weaving knowledge systems in IPBES, CBD and novel scheme to highlight dynamic aspects of science-policy interfaces.
beyond—lessons learned for sustainability. Curr. Opin. Environ. Sustain. Environ. Sci. Policy 54, 505–512 (2015).
26–27, 17–25 (2017). 47. Steyaert, P. & Jiggins, J. Governance of complex environmental situations
17. Nielsen, W. et al. Gender diversity leads to better science. Proc. Natl Acad. through social learning: a synthesis of SLIM’s lessons for research, policy
Sci. USA 114, 1740–1742 (2017). and practice. Environ. Sci. Policy 10, 575–586 (2007).
18. Greenwood, D. & Levin, M. Introduction to Action Research. Introduction to 48. Wall, T. U., Meadow, A. M. & Horganic, A. Developing evaluation
Action Reasearch (SAGE Publications, 2007). indicators to improve the process of coproducing usable climate science.
19. Pohl, C. et al. Researchers’ roles in knowledge co-production: experience Weather Clim. Soc. 9, 95–107 (2017).
from sustainability research in Kenya, Switzerland, Bolivia and Nepal. 49. Meagher, L. R. & Martin, U. Slightly dirty maths: the richly textured
Sci. Public Policy 37, 267–281 (2010). mechanisms of impact. Res. Eval. 26, 15–27 (2017).
20. Harvey, B., Cochrane, L. & Van Epp, M. Charting knowledge co‐production 50. Walter, A. I., Helgenberger, S., Wiek, A. & Scholz, R. W. Measuring societal
pathways in climate and development. Environ. Policy Gov. 29, effects of transdisciplinary research projects: design and application of an
107–117 (2019). evaluation method. Eval. Program Plan. 30, 325–338 (2007).
21. Hurlbert, M. & Gupta, J. The split ladder of participation: a diagnostic, 51. Klein, J. T. Evaluation of interdisciplinary and transdisciplinary research.
strategic, and evaluation tool to assess when participation is necessary. Am. J. Prev. Med. 35, S116–S123 (2008).
Environ. Sci. Policy 50, 100–113 (2015). 52. Wickson, F. & Carew, A. L. Quality criteria and indicators for responsible
22. Nel, J. L. et al. Knowledge co-production and boundary work to promote research and innovation: learning from transdisciplinarity. J. Responsible
implementation of conservation plans. Conserv. Biol. 30, 176–188 (2016). Innov. 1, 254–273 (2014).

Nature Sustainability | www.nature.com/natsustain

NatURe SUstainaBility Perspective
53. Belcher, B. M., Rasmussen, K. E., Kemshaw, M. R. & Zornes, D. A. Defining 83. van der Hel, S. Science for change: a survey on the normative and political
and assessing research quality in a transdisciplinary context. Res. Eval. 25, dimensions of global sustainability research. Glob. Environ. Change 52,
1–17 (2016). 248–258 (2018).
54. Hansson, S. & Polk, M. Assessing the impact of transdisciplinary research: 84. Gibbons, M. et al. The New Production of Knowledge: The Dynamics of
the usefulness of relevance, credibility, and legitimacy for understanding the Science and Research in Contemporary Societies (Sage, 1994).
link between process and impact. Res. Eval. 27, 132–144 (2018). 85. Funtowicz, S. O. & Ravetz, J. R. in Perspectives on Ecological Integrity
55. Posner, S. M., McKenzie, E. & Ricketts, T. H. Policy impacts of ecosystem (eds Westra, L. & Lemons, J.) 146–161 (Springer, 1995).
services knowledge. Proc. Natl Acad. Sci. USA 113, 1760–1765 (2016). 86. Lubchenco, J. Entering the century of the environment: a new social
56. Lebel, J. & McLean, R. A better measure of research from the global south. contract for science. Science 279, 491–497 (1998).
Nature 559, 23–26 (2018). 87. Leemans, R. The lessons learned from shifting from global-change research
57. Phillipson, J., Lowe, P., Proctor, A. & Ruto, E. Stakeholder engagement and programmes to transdisciplinary sustainability science. Curr. Opin. Environ.
knowledge exchange in environmental research. J. Environ. Manag. 95, Sustain. 19, 103–110 (2016).
56–65 (2012). 88. Lewin, K. Action research and minority problems. J. Soc. Issues 2,
58. Wiek, A., Talwar, S., O’Shea, M. & Robinson, J. Toward a methodological 34–46 (1946).
scheme for capturing societal effects of participatory sustainability research. 89. Checkland, P. Soft systems methodology: a thirty year retrospective.
Res. Eval. 23, 117–132 (2014). Syst. Res. Behav. Sci. 17, 11–58 (2000).
59. Pitt, R. et al. Wrestling with the complexity of evaluation for organizations 90. Holling, C. Adaptive Environmental Assessment and Management
at the boundary of science, policy, and practice. Conserv. Biol. 32, (John Wiley & Sons, 1978).
998–1006 (2018). 91. Brown, L. & Tandon, R. Ideology and political economy in inquiry: action
60. Greenhalgh, T., Raftery, J., Hanney, S. & Glover, M. Research impact: a research and participatory research. J. Appl. Behav. Sci. 19, 277–294 (1983).
narrative review. BMC Med. 14, 78 (2016). 92. Ostrom, E. Crossing the great divide: Coproduction, synergy, and
61. Wolf, B., Lindenthal, T., Szerencsits, M., Holbrook, J. B. & Heβ, J. Evaluating development. World Dev. 24, 1073–1087 (1996).
research beyond scientific impact: how to include criteria for productive 93. Watson, V. Co-production and collaboration in planning – the difference.
interactions and impact on practice and society. Gaia 22, 104–114 (2013). Plan. Theory Pract. 15, 62–76 (2014).
62. Barreteau, O., Bots, P. W. G. & Daniell, K. A. A framework for clarifying 94. Clark, W. C. & Dickson, N. M. Sustainability science: the emerging research
‘participation’ in participatory research to prevent its rejection for the program. Proc. Natl Acad. Sci. USA 100, 8059–8061 (2003).
wrong reasons. Ecol. Soc. 15, 1 (2010). 95. Lemos, M. C. & Morehouse, B. J. The co-production of science and policy
63. Tobias, S., Ströbele, M. F. & Buser, T. How transdisciplinary projects in integrated climate assessments. Glob. Environ. Change 15, 57–68 (2005).
influence participants’ ways of thinking: a case study on future landscape 96. Jasanoff, S. in States of Knowledge: The Co-Production of Science and Social
development. Sustain. Sci. 14, 405–419 (2019). Order (ed. Jasanoff, S.) 1–12 (Routledge, 2004).
64. Jahn, T. & Keil, F. An actor-specific guideline for quality assurance in 97. Miller, C. A. & Wyborn, C. Co-production in global sustainability:
transdisciplinary research. Futures 65, 195–208 (2015). histories and theories. Environ. Sci. Policy https://doi.org/10.1016/j.
65. Brisolara, S. The history of participatory evaluation and current debates in envsci.2018.01.016 (2018).
the field. New Dir. Eval. 1998, 25–41 (1998). 98. van Kerkhoff, L. et al. Towards future-oriented conservation: managing
66. Gray, S., Chan, A., Clark, D. & Jordan, R. Modeling the integration of protected areas in an era of climate change. Ambio 48, 699–713 (2019).
stakeholder knowledge in social–ecological decision-making: benefits and 99. Busilacchi, S., Butler, J., Van Putten, I., Maru, Y. & Posu, J. Asymmetrical
limitations to knowledge diversity. Ecol. Modell. 229, 88–96 (2012). development across transboundary regions: the case of the Torres Strait Treaty
67. Yamineva, Y. Lessons from the Intergovernmental Panel on Climate Change Region (Australia and Papua New Guinea). Sustainability 10, 4200 (2018).
on inclusiveness across geographies and stakeholders. Environ. Sci. Policy 100. Mitchell, M. G. E. et al. The Montérégie Connection: linking landscapes,
77, 244–251 (2017). biodiversity, and ecosystem services to improve decision making. Ecol. Soc.
68. Woolrych, R. & Sixsmith, J. Placing well-being and participation within 20, 15 (2015).
processes of urban regeneration. Int. J. Public Sect. Manag. 26, 216–231 (2013).
69. Schulz, A. J., Israel, B. A. & Lantz, P. Instrument for evaluating dimensions Acknowledgements
of group dynamics within community-based participatory research A.V.N. received support by the Swedish Research Council Formas (grant number 2017-
partnerships. Eval. Program Plan. 26, 249–262 (2003). 01326) and the GRAID programme at SRC. M.F.L. received support by the foundation
70. Bednarek, A. T., Shouse, B., Hudson, C. G. & Goldburg, R. Science-policy BalticSea2020 and the Stockholm University Baltic Sea Centre. S.W. received support
intermediaries from a practitioner’s perspective: the Lenfest Ocean Program by the Swedish Research Council Formas (mobility starting grant 2017-01631). H.Ö.
experience. Sci. Public Policy 43, 291–300 (2016). received support from the Walton Family Foundation (grants 2017-693 and 2018-1371),
71. Wyborn, C. et al. Understanding the impacts of research synthesis. The David and Lucile Packard Foundation (grants 2017-66205 and 2019-68336), and
Environ. Sci. Policy 86, 72–84 (2018). the Gordon and Betty Moore Foundation (grants GBMF5668.01 and GBMF5668.02).
72. Chowdhury, G., Koya, K. & Philipson, P. Measuring the impact of research: R.B. received support from the South African Research Chairs Initiative (SARChI) of
lessons from the UK’s research excellence framework 2014. PLOS ONE 11, the Department of Science and Technology and National Research Foundation of South
e0156978 (2016). Africa (grant 98766); the GRAID programme at SRC; and the Swedish Research Council
73. Mayne, J. Contribution analysis: coming of age? Evaluation 18, (grant 621-2014-5137). M.-F.L. received support by the Swiss Academy of Sciences
270–280 (2012). (SCNAT) and the Chinese Academy of Sciences (CAS). B.M.C. received support from the
74. Patton, M. Q. Developmental Evaluation: Applying Complexity Concepts to CGIAR Trust Fund and through bilateral funding agreements. This research contributes
Enhance Innovation and Use (Guilford Press, 2011). to the Program on Ecosystem Change and Society (www.pecs-science.org).
75. Alcamo, J. Evaluating the impacts of global environmental assessments.
Environ. Sci. Policy 77, 268–272 (2017).
76. Olsen, S. B., Page, G. G. & Ochoa, E. The Analysis of Governance Responses Author contributions
to Ecosystem Change: A Handbook for Assembling a Baseline LOICZ Reports The design, development and writing of the manuscript were co-led by authors A.V.N.,
and Studies No. 34 (LOICZ, 2009). C.C., M.F.L., C.W. and S.W. The work was coordinated by A.V.N. All authors wrote and
77. Österblom, H., Jouffray, J.-B., Folke, C. & Rockström, J. Emergence of a commented on the manuscript.
global science–business initiative for ocean stewardship. Proc. Natl Acad.
Sci. USA 114, 9038–9043 (2017). Competing interests
78. Westley, F. et al. Tipping toward sustainability: emerging pathways of The authors declare no competing interests.
transformation. Ambio 40, 762–780 (2011).
79. Geels, F. W. Technological transitions as evolutionary reconfiguration
processes: a multi-level perspective and a case-study. Res. Policy 31, Additional information
1257–1274 (2002). Supplementary information is available for this paper at https://doi.org/10.1038/
80. Moore, M. et al. Studying the complexity of change: toward an analytical s41893-019-0448-2.
framework for understanding deliberate social-ecological transformations. Correspondence should be addressed to A.V.N.
Ecol. Soc. 19, 54 (2014). Reprints and permissions information is available at www.nature.com/reprints.
81. Abson, D. J. et al. Leverage points for sustainability transformation. Ambio
46, 30–39 (2017). Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in
82. Galaz, V., Leach, M. & Scoones, I. in One Health: Science, Politics and published maps and institutional affiliations.
Zoonotic Disease in Africa (ed. Bardosh, K.) 21–37 (Routledge, 2016). © Springer Nature Limited 2020

Nature Sustainability | www.nature.com/natsustain