See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/330753955

How open science norms improve scientific practices

Conference Paper · January 2019

CITATIONS READS

3 431

2 authors:

Ljiljana Lazarevic Iris Žeželj

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Effects of benevolent/dangerous worldview on belief in conspiracy theories View project

PhD thesis: Mechanisms through which academic gender stereotypes influence educational outcomes in elementary education View project

All content following this page was uploaded by Iris Žeželj on 31 January 2019.

The user has requested enhancement of the downloaded file.

Abstract: Empirical findings indicate that various [6]. In economics, 39% of the replicated studies did not
scientific fields are in serious replication crisis. One have significant effects in the same direction as original
source of low reproducibility can be attributed to the so- studies [7]. Having public health in mind, the most
called Questionable Research Practices. The wider disturbing estimate comes from the biomedical research:
incentive structures prioritizing novel and positive the percentage of non-reproducible findings ranges from
findings over negative ones also influence the rate of 75% to 90% [8].
false-positive effects in published papers. We are It seemed that the climate in empirical science did not
discussing the open science norms and good scientific foster cumulative, reproducible knowledge. There were
practices targeted to improve methods, reporting, no mechanisms in place that would guarantee the
dissemination and incentives for researchers. Among necessary self-correctiveness. In fact, it was just the
proposed norms are pre-registration of study protocols, oposite.
results-blind peer reviews, collaboration and team-
science, and improvement of statistical and
methodological training. Moreover, important II. Questionable Research Practices
stakeholders are recognizing the importance of open
science, and incentives for researchers to openly share Scientific practices that contributed to such sad state
work are developing. Finally, we are discussing some of of affairs were somewhat euphemistically labelled
the remaining challenges in the quest for open, Questionable Research Practices (QRP) [2][9]. Studies
reproducible, and credible science. show that low reproducibility rate of scientific findings
resulted from a manifold of practices: too small sample
Keywords: open science; reproducibility; sizes, searching for statistically significant results (p-
questionable scientific practices; good scientific hacking), post hoc hypothesizing (HARK-ing:
practices. Hypothesizing After the Results are Known), diverging
from originally planned research designs and analytical
strategies, selectively reporting studies with positive
I. Introduction
results and not studies with negative results, etc. [2][3]
Ideally, science should be an unbiased, collaborative [10][11].
quest for truth, for the benefit of the whole society. To It would not be fair to attribute the responsibility for
accomplish that ideal, it should be cumulative and self- such practices solely to scientists: it was the system that
corrective. It means that over time, credible scientific rewarded flashy positive findings and marginalized
findings should be independently replicated and negative ones. That resulted in self-censoring by
preserved, while so-called false-positive findings should scientists, further biased selection by journal editors,
be refuted [1]. It is important to note that the self- project funders and employers, which inevitably lead to a
correction principle does not imply necessarily that all grossly inflated rate of false-positive effects in published
published science is credible and replicable – it implies papers [11][12].
that it only such findings should stand the test of time. If the system is to blame, then the system needs to
The estimate of the proportion of the correct first- change, and there seemed to be a substantial room for
proposed results varies between 0% and 100%. However, improvement in research and publishing practices. So far,
more importantly, the proportion of the correct results a number of innovations were introduced to this end (for
after meta-analytically analyzing all available evidence an overview, see [11]).
still varies between 0% and 100% [2].
Novel analyses from different empirical scientific
fields, revealed a serious replication crisis [3][4]. In III. Open science norms and good scientific
psychology, a large-scale collaboration successfully practices
replicated only around 50% of the selected published
studies [4][5]. A recent high-powered effort replicating The credibility of scientific findings depends on the
experimental studies in the social sciences published in transparency of all elements of the scientific process [13].
the most prestigious outlets - Science and Nature - In this paper, we will briefly present the most prominent
between 2010 and 2015 reported that the average effect practices aimed to improve methods, reporting,
size of the replications was 50% of the original effect size dissemination and incentives for research.

DOI: 10.5281/zenodo.1411157 13
A. Methods diverse group of stakeholders and launched a worldwide
campaign for open access (OA) to all peer-reviewed
As science is a human endeavour, scientists are prone
research. A year later, the Berlin Declaration on open
to biases in formulating hypotheses, conducting data
access, signed by nearly 300 research institutions,
collection and data preparation - they typically, and most
funding bodies, libraries, museums, and governments,
often not intentionally, favour confirming their starting
was based on the BOAI and called for the research results
hypotheses [11][14]. As a countermeasure, a pre-
to be publicly available.
registration of the study design, hypothesis, analytic
In recent years, important stakeholders started
strategy, and measured outcomes to the journal for peer
demanding from researchers to share materials,
review before beginning the study was proposed [11][15].
databases and analytical scripts. A survey evaluating the
Some scientific disciplines, like clinical medicine or
effects of BOAI after 15 years, indicated the transition
stomatology, introduced a mandatory pre-registration of
from establishing open access as a concept (in 2002) to
study protocols to address publication bias and analytical
making open the default [30]. However, this survey
flexibility [11][16][17]. In social and behavioural
showed two important remaining challenges: a)
sciences, pre-registration is becoming more frequent: the
researchers lack meaningful incentives and rewards to
journals are adopting and promoting the Registered
openly share their work, and b) lack of sufficient funding
Reports publishing format encouraging pre-registration
to pay for open access-related costs.
and results-blind peer review [11].
There are recent attempts to further formalize the
In addition, in an attempt to avoid publication biases,
process of sharing in research planning and reporting,
journals are now introducing a policy of results blind
such as Transparency and Openness Promotion (TOP)
evaluation of manuscripts submitted to journals [18], for
guidelines. These are comprehensive sets of standards for
the illustration of the process, see [19][20]. Results blind
journals, funding bodies, institutions, professional
evaluation implies that the review process is two-staged:
societies, reviewers, and authors [31]. Until now, more
in the first step, the editor distributes the manuscript
than 5000 journals endorsed TOP guidelines.
containing the Introduction and the Methods section for
In addition to researchers sharing their outputs, open
external review; if the decision of the first stage review is
science practices assume that, once published, the
positive, it proceeds to the second step, in which the full
content of scientific articles should be free and
manuscript is once again evaluated.
accessible. Hence, there is an increasing pressure on
To address the problem of low statistical power (too
publishers to provide open access to publications. For
small samples to allow reliable conclusions) which
instance, the National Institute of Health (NIH), National
increases the likelihood of obtaining both false-positive
Science Foundation (NSF), German Psychological
and false-negative results [21], researchers suggested to
Association demand that data collected as part of publicly
join forces. Collaboration across many sites enabled
funded research are made open to all interested parties.
high-powered study designs, stringent analytic strategy,
European Commission, as part of innovation funding
and more economic use of resources. A very good
Horizon 2020 program, made open access to publications
example of efficient collaborations in social and
mandatory and launched a pilot project to open up
behavioural sciences are The Many Labs projects [5][22]
publicly funded research data available from 2013
[23] or other large-scale collaborations (e.g., [24]);
onwards.
Psychological Science Accelerator: [25]. In other
disciplines, there are older and more prominent
C. Incentives
examples: The LIGO scientific collaboration [26]; The
GENOME project [27]; The CERN project [28]. Symbolic rewards. For example, the Center for Open
Finally, most “reformers” agree that more rigorous Science proposed that journals appoint badges to articles
statistical and methodological training and research endorsing open practices, i.e., open data, open materials,
methods is fundamental for high-quality research. and pre-registration. Empirical evaluation of this practice
Although most of the research programs do contain these demonstrated that introduction of open practices badges
courses, they should be fundamentally reinvented; some had a positive impact of scholars and lead to an increase
authors argue that “departments need to begin teaching of data sharing [32].
statistical thinking, not rituals” ([29], p. 198). As a good Employment policies. There is a growing trend of
example of a crowdsourced project with an emphasis on universities, institutes and funding bodies requiring
the replication and pedagogy we can list an initiative, researchers to demonstrate their devotion to open science.
launched in 2012, Collaborative Replications and
Education Project (CREP; https://osf.io/wfc6u/) aimed to
strengthen undergraduate students’ knowledge and IV. Conclusion
expertise in statistics and research methods.
What we presented is not an exhaustive list of
B. Reporting and Dissemination proposed measures, rather the measures that were shown
to have an effect in raising scientific standards. Open
First global initiatives for more sharing and Science movement had numerous positive outcomes.
transparency appeared almost 20 years ago. In 2002, the Funding bodies started adopting transparency
Budapest Open Access Initiative (BOAI) gathered a requirements and supporting financially replication

14
 studies. Institutions started providing the scholars with [12] R. Joober, N. Schmitz, L. Annable, and P. Boksa, "Publication
bias: what are the challenges and can they be overcome?" Journal
the infrastructure for data sharing, and what is more of Psychiatry and Neuroscience, vol. 37, pp. 149-152, 2012
important, started changing hiring standards and putting [13] J. P. A. Ioannidis, "Why science is not necessarily self-correcting,"
more weight to the open science values. Journals are Perspectives on Psychological Science, vol. 7, pp. 645–654, 2012
adopting badges to acknowledge open practices, [14] R. MacCoun, and S. Perlmutter, "Blind analysis: hide results to
Registered Reports and TOP guidelines. seek the truth," Nature, vol. 526, pp. 187–189, 2015
However, there are still some unresolved issues that [15] J. E. Gonzales, and C. A. Cunningham, "The promise of pre-
we would like to tackle. First is to establish a system of registration in psychological research," Psychological Science
Agenda, 2015. [Online]. Available
incentives that would encourage good scientific practices http://www.apa.org/science/about/psa/2015/08/pre-
and be endorsed by the scholars as sufficiently rewarding. registration.aspx [Accessed Aug. 20, 2018]
We do not argue that novelty should not be rewarded, on [16] J. Lenzer, J. R. Hoffman, C. D. Furberg, J. P. Ioannidis, and G. P.
the contrary. We argue that it should not be the only thing R. W. Grp, "Ensuring the integrity of clinical practice guidelines: a
tool for protecting patients," British Medical Journal, vol. 347, pp.
that is rewarded. Novelty versus reproducibility is a false F5535, 2013
dichotomy and goods science requires both. Second [17] F. Schwendicke, F., and N. Opdam, "Clinical studies in restorative
challenge draws from the fact that resource allocation is dentistry: design, conduct, analysis," Dental Materials, vol. 24,
not only required of scientists, it is required of publishers, pp. 29–39, 2017
too. Even though most of them declaratively support the [18] J. J. Locascio, "Results blind science publishing," Basic and
new norms, the current business model is far more Applied Social Psychology, vol. 39, pp. 239-246, 2017
lucrative than the open-access model and this still needs [19] I. L. Žeželj, and B. R. Jokić, "Replication of experiments
evaluating impact of psychological distance on moral judgment,"
to be addressed. Social Psychology, vol. 45, pp. 223-231, 2014a
In spite of the challenges, it seems that the current [20] I. L. Žeželj, and B. R. Jokić, "A rejoinder to comments by Eyal,
climate is pushing all stakeholders towards more Liberman, & Trope and Gong & Medin," Social Psychology, vol.
reproducible, credible and accessible science. We can 45, no. 4, pp. 332-334, 2014b
only hope the winds will not turn. [21] K. S. Button, J. P. Ioannidis, C. Mokrysz, B. A. Nosek, J. Flint, E.
S. Robinson, and M. R. Munafò, "Power failure: why small
sample size undermines the reliability of neuroscience," Nature
References Reviews Neuroscience, vol. 14, no. 5, pp. 365, 2013
[22] C. R. Ebersole, et al. "Many Labs 3: evaluating participant pool
[1] K. Popper, Conjectures and refutations: The growth of scientific quality across the academic semester via replication," Journal of
knowledge. London: Routledge and Kegan Paul, 1963 Experimental Social Psychology, 67, pp. 68–82, 2016
[2] J. P. A. Ioannidis, "Why most published research findings are [23] R. A. Klein, K. A. Ratliff, M. Vianello, R. B. Adams Jr, Š. Bahník,
false," PLoS Medicine, vol. 2, pp. E124, 2005 M. J. Bernstein,... and Z. Cemalcilar, "Investigating variation in
[3] M. R., Munafò, B. A., Nosek, D. V., Bishop, K. S., Button, C. D., replicability," Social psychology, vol. 45, pp. 142–152, 2014
Chambers, N. P., du Sert, ... and J. P., Ioannidis, "A manifesto for [24] J. F. Landy, M. Jia, I. L. Ding, D. Viganola, W. Tierney, ..., and E.
reproducible science," Nature Human Behaviour, vol. 1, pp. 0021, L. Uhlmann, "Crowdsourcing hypothesis tests: Making
2017 transparent how design choices shape research results."
[4] Open Science Collaboration, "Estimating the reproducibility of Manuscript submitted for publication, 2018
psychological science," Science, vol. 349, pp. Aac4716, 2015 [25] H. Moshontz, L. Campbell, C. R. Ebersole, H,. IJzerman, H. L.
[5] R. A. Klein, et al., "Many Labs 2: investigating variation in Urry, P. S. Forscher,..., and C. R. Chartier, "The Psychological
replicability across sample and setting," Advances in Methods and Science Accelerator: Advancing Psychology through a Distributed
Practices Psychological Science, In press Collaborative Network," Advances in Methods and Practices in
[6] C. F. Camerer, A. Dreber, F. Holzmeister , T. Ho, J. Huber, M. Psychological Science, In press
Johannesson , M. Kirchler, G., Nave, B. A. Nosek , T. Pfeiffer , A. [26] B.T. Abbott, R. Abbott, R. Adhikari, P. Ajith, B. Allen, G. Allen, ...
Altmejd, N. Buttrick, T. Chan, Y. Chen, E. Forsell, A. Gampa, E. & M. Araya, "LIGO: the laser interferometer gravitational-wave
Heikensten, L. Hummer, T. Imai , S. Isaksson, D. Manfredi, J. observatory," Reports on Progress in Physics, vol 72, pp. 076901,
Rose, E. Wagenmakers, and H. Wu, "Evaluating the replicability 2009
of social science experiments in Nature and Science between 2010 [27] T. S.Heng, M. W. Painter, K.Elpek, V. Lukacs-Kornek, N.
and 2015". Nature Human Behavior (Letters), doi: Mauermann, S.J. Turley, ... & S. Davis, "The Immunological
10.1038.s41562-018-0399-z, 2018 Genome Project: networks of gene expression in immune cells,"
[7] C. F. Camerer, A. Dreber, E. Forsell, T. H. Ho, J. Huber, M. Nature immunology, vol 9, pp. 1091-1094, 2008
Johannesson,... and E. Heikensten, "Evaluating replicability of [28] https://home.cern/ [Accessed Aug. 20, 2018]
laboratory experiments in economics," Science, vol. 351, pp.
1433-1436, 2016 [29] G. Gigerenzer, "Statistical Rituals: The Replication Delusion and
How We Got There", Advances in Methods and Practices in
[8] C. G. Begley, and J. P. Ioannidis, "Reproducibility in science: Psychological Science, vol 1, pp. 198-218, 201
improving the standard for basic and preclinical research,"
Circulation research, vol.116, pp. 116-126, 2015 [30] N. Shockey, H. Joseph, and M. Hagemann, "BOAI 15 Survey
Report," April 12, 2018
[9] K. Fiedler, and N.Schwarz, "Questionable research practices
revisited," Social Psychological and Personality Science, vol.7, [31] B.A. Nosek, G. Alter, G.C. Banks, D. Borsboom, S. Bowman, S.J.
pp. 45-52, 2016 Breckler, ... & M. Contestabile, "Promoting an open research
culture", Science, vol 348, pp. 1422-1425, 2015.
[10] L. K. John, G. Loewenstein, and D. Prelec, "Measuring the
prevalence of questionable research practices with incentives for [32] M. C. Kidwell, L. B. Lazarevic, E. Baranski, T. E., Hardwicke, S.
truth telling," Psychological Science, vol.23, pp. 524–532, 2012 Piechowski, L-S. Falkenberg, C. Kennett, A. Slowik, C.
Sonnleitner, C. Hess-Holden, T. M. Errington, S. Fiedler, and B.
[11] B. A. Nosek, J. R. Spies, and M. Motyl, "Scientific Utopia: II. A. Nosek, "Badges to acknowledge open practices: A simple, low-
Restructuring incentives and practices to promote truth over cost, effective method for increasing transparency," PLoS Biology,
publishability,"Perspectives on Psychological Science, vol. 7, pp. vol. 14, pp. 1-15, 2016
615-631, 2012

15

View publication stats