CORRESPONDENCE L I N K T O O R I G I N A L A RT I C L E

results that are published are hard to repro-

Believe it or not: how much can we duce. However, there is an imbalance between
this apparently widespread impression and its

rely on published data on potential

public recognition (for example, see REFS 2,3),
and the surprisingly few scientific publica-
tions dealing with this topic. Indeed, to our
drug targets? knowledge, so far there has been no published
in-depth, systematic analysis that compares
reproduced results with published results for
Florian Prinz, Thomas Schlange and Khusru Asadullah wet-lab experiments related to target identifica-
tion and validation.
A recent report by Arrowsmith noted that the to ‘feasible/marketable’, and the financial costs Early research in the pharmaceutical indus-
success rates for new development projects in of pursuing a full-blown drug discovery and try, with a dedicated budget and scientists who
Phase II trials have fallen from 28% to 18% in development programme for a particular tar- mainly work on target validation to increase
recent years, with insufficient efficacy being get could ultimately be hundreds of millions of the confidence in a project, provides a unique
the most frequent reason for failure (Phase II Euros. Even in the earlier stages, investments opportunity to generate a broad data set on the
failures: 2008–2010. Nature Rev. Drug Discov. in activities such as high-throughput screen- reproducibility of published data. To substanti-
10, 328–329 (2011))1. This indicates the limi- ing programmes are substantial, and thus the ate our incidental observations that published
tations of the predictivity of disease models validity of published data on potential targets reports are frequently not reproducible with
and also that the validity of the targets being is crucial for companies when deciding to start quantitative data, we performed an analysis
investigated is frequently questionable, which novel projects. of our early (target identification and valida-
is a crucial issue to address if success rates in To mitigate some of the risks of such invest- tion) in-house projects in our strategic research
clinical trials are to be improved. ments ultimately being wasted, most phar- fields of oncology, women’s health and cardio-
Candidate drug targets in industry are maceutical companies run in-house target vascular diseases that were performed over the
derived from various sources, including in- validation programmes. However, validation past 4 years (FIG. 1a). We distributed a ques-
house target identification campaigns, in- projects that were started in our company tionnaire to all involved scientists from target
licensing and public sourcing, in particular based on exciting published data have often discovery, and queried names, main relevant
based on reports published in the literature and resulted in disillusionment when key data published data (including citations), in-house
presented at conferences. During the transfer could not be reproduced. Talking to scien- data obtained and their relationship to the pub-
of projects from an academic to a company tists, both in academia and in industry, there lished data, the impact of the results obtained
setting, the focus changes from ‘interesting’ seems to be a general impression that many for the outcome of the projects, and the models

C 
 D 
 E 
 



 
 





 


 


1PEQNQI[ /QFGNCFCRVGFVQKPVGTPCNPGGFU +PEQPUKUVGPEKGU

9QOGPoUJGCNVJ .KVGTCVWTGFCVCVTCPUHGTTGFVQCPQVJGT 0QVCRRNKECDNG
%CTFKQXCUEWNCT KPFKECVKQP .KVGTCVWTGFCVCCTGKPNKPGYKVJKPJQWUGFCVC
0QVCRRNKECDNG /CKPFCVCUGVYCUTGRTQFWEKDNG
/QFGNTGRTQFWEGF 5QOGTGUWNVUYGTGTGRTQFWEKDNG
F
/QFGN /QFGNCFCRVGFVQKPVGTPCN .KVGTCVWTGFCVCVTCPUHGTTGF 0QV
TGRTQFWEGF PGGFU
EGNNNKPGCUUC[U VQCPQVJGTKPFKECVKQP CRRNKECDNG
+PJQWUGFCVCKPNKPGYKVJRWDNKUJGFTGUWNVU 
 
  

+PEQPUKUVGPEKGUVJCVNGFVQRTQLGEVVGTOKPCVKQP 
 
 
 

Figure 1 | Analysis of the reproducibility of published data in 67 in- of each of the following outcomes is shown: data were completely in line
house projects. a | This figure illustrates the distribution of projects within with published data; the main set was reproducible; some results (including
0CVWTG4GXKGYU^&TWI&KUEQXGT[
the oncology, women’s health and cardiovascular indications that were ana- the most relevant hypothesis) were reproducible; or the data showed incon-
lysed in this study. b | Several approaches were used to reproduce the pub- sistencies that led to project termination. ‘Not applicable’ refers to projects
lished data. Models were either exactly copied, adapted to internal needs that were almost exclusively based on in-house data, such as gene expres-
(for example, using other cell lines than those published, other assays and so sion analysis. The number of projects and the percentage of projects within
on) or the published data was transferred to models for another indication. this study (a– c) are indicated. d | A comparison of model usage in the repro-
‘Not applicable’ refers to projects in which general hypotheses could not be ducible and irreproducible projects is shown. The respective numbers of
verified. c | Relationship of published data to in-house data. The proportion projects and the percentages of the groups are indicated.

NATURE REVIEWS | DRUG DISCOVERY www.nature.com/reviews/drugdisc

© 2011 Macmillan Publishers Limited. All rights reserved

CORRESPONDENCE

that were used in the experiments and publica- There may be several reasons for the However, with reasonable efforts (sometimes
tions. The questionnaire can be obtained from observed lack of reproducibility. Among the equivalent of 3–4 full-time employees over
the authors. these, incorrect or inappropriate statistical 6–12 months), we have frequently been unable
We received input from 23 scientists (heads analysis of results or insufficient sample sizes, to reconfirm published data.
of laboratories) and collected data from 67 which result in potentially high numbers Our observations indicate that literature
projects, most of them (47) from the field of of irreproducible or even false results, have data on potential drug targets should be viewed
oncology. This analysis revealed that only in been discussed6. Among the more obvious with caution, and underline the importance
~20–25% of the projects were the relevant yet unquantifiable reasons, there is immense of confirmatory validation studies for phar-
published data completely in line with our in- competition among laboratories and a pres- maceutical companies and academia before
house findings (FIG. 1c). In almost two-thirds sure to publish. It is conceivable that this may larger investments are made in assay develop-
of the projects, there were inconsistencies sometimes result in negligence over the con- ment, high-throughput screening campaigns,
between published data and in-house data that trol or reporting of experimental conditions lead optimization and animal testing. Effective
either considerably prolonged the duration of (for example, a variation in cell-line stocks and target validation, however, should not just be
the target validation process or, in most cases, suppliers, or insufficient description of materi- confirmatory, but should complement the
resulted in termination of the projects because als and methods). There is also a bias towards knowledge on a particular target. An in-depth
the evidence that was generated for the thera- publishing positive results, as it is easier to get biological understanding of a target is required
peutic hypothesis was insufficient to justify positive results accepted in good journals. It and should contribute to a reduction in the
further investments into these projects. remains to be studied further whether there high attrition rates that are observed in early
We wondered whether heterogeneous are indeed hurdles to publishing results that clinical development.
experimental conditions could be an explana- contradict data from high-impact journals or Florian Prinz is at Target Research Berlin, Bayer
tion for the frequent inconsistencies (FIG. 1b). the currently established scientific opinion in HealthCare, Müllerstraße 178, 13342 Berlin,
Interestingly, a transfer of the models — for a given field, which could lead to the litera- Germany.
example, by changes in the cell lines or assay ture supporting a certain hypothesis even if Thomas Schlange is at Target Research Wuppertal,
formats — was not crucial for the discrepancies there are many (unpublished) data arguing Bayer HealthCare, Aprather Weg 18a, 42096
that were detected. Rather, either the results against it. One might speculate that the above Wuppertal, Germany.
were reproducible and showed transferabil- mentioned issues should be eliminated by the Khusru Asadullah is Vice President and Head of Target
ity in other models, or even a 1:1 reproduc- peer review system. However, reviewers have Discovery at Bayer HealthCare, Müllerstraße 178,
13342 Berlin, Germany.
tion of published experimental procedures no time and no resources to reproduce data
revealed inconsistencies between published and to dig deeply into the presented work. Correspondence to K.A. 
and in-house data (FIG.  1d). Furthermore, As a consequence, errors often remain unde- e-mail: [email protected]
1. Arrowsmith, J. Phase II failures: 2008–2010. Nature
despite the low numbers, there was no appar- tected7. Adding to this problem, many initially Rev. Drug Discov. 10, 328–329 (2011).
ent difference between the different research rejected papers will subsequently be published 2. Lehrer, J. The truth wears off: is there something
wrong with the scientific method? The New Yorker
fields. Surprisingly, even publications in pres- in other journals without substantial changes [online], http://www.newyorker.com/
tigious journals or from several independent or improvements8,9. reporting/2010/12/13/101213fa_fact_lehrer (2010).
3. Freeman, D. H. Lies, damned lies, and medical
groups did not ensure reproducibility. Indeed, We are aware that our data set — albeit science. The Atlantic [online], http://www.theatlantic.
our analysis revealed that the reproducibility quite large for wet-lab science — is still rather com/magazine/archive/2010/11/lies-damned-lies-and-
medical-science/8269 (2010).
of published data did not significantly corre- small and its statistical significance can be 4. Osherovich, L. Hedging against academic risk. SciBX
late with journal impact factors, the number questioned. We are also aware that our own 14 Apr 2011 (doi:10.1038/scibx.2011.416).
5. Barrows, N. J., Le Sommer, C., Garcia-Blanco, M. A. &
of publications on the respective target or the experimental results might also be irreproduc- Pearson, J. L. Factors affecting reproducibility
number of independent groups that authored ible in other laboratories. However, the aim of between genome-scale siRNA-based screens.
J. Biomol. Screen. 15, 735–747 (2010).
the publications. our target validation work is: first, to increase 6. Ioannidis, J. P. Why most published research findings
Our findings are mirrored by ‘gut feelings’ confidence in the biology of the targets with an are false. PLoS Med. 2, e124 (2005).
7. Schroter, S, et al. What errors do peer reviewers
expressed in personal communications with unbiased approach; second, to provide assays detect, and does training improve their ability to
scientists from academia or other companies, that need to be reliable during later stages such detect them? J. R. Soc. Med. 101, 507–514 (2008).
8. Nemery, B. What happens to the manuscripts that
as well as published observations. An unspo- as compound optimization; and third, to trans- have not been accepted for publication in
ken rule among early-stage venture capital fer these assays to various laboratories in other Occupational and Environmental Medicine? Occup.
Environ. Med. 58, 604–607 (2001).
firms that “at least 50% of published studies, departments in-house. With an average pro- 9. McDonald, R. J., Cloft, H. J. & Kallmes, D. F. Fate of
even those in top-tier academic journals, can’t ject duration of 6–12 months, numerous well- submitted manuscripts rejected from the American
Journal of Neuroradiology: outcomes and commentary.
be repeated with the same conclusions by an established cellular and in vivo models and Am. J. Neuroradiol. 28, 1430–1434 (2007).
industrial lab” has been recently reported (see several independent and often specialized lab-
Further information) and discussed4. The chal- oratories that are involved in the projects with Acknowledgements
We would like to thank B. Kreft and T. Zollner for their valua-
lenge of reproducibility — even under ideal highly qualified scientists who are dedicated ble contributions to this project, S. Schoepe for support in the
conditions — has also been highlighted, indi- to target discovery, we feel confident that our data analysis and S. Decker for support with bioinformatics
analysis of the results.
cating that even in an optimal setting (the same data are quite reliable. It is important, however,
laboratory, the same people, the same tools and to emphasize that we do not want to make the Competing interests statement
The authors are employees of Bayer Healthcare.
the same assays, with experiments separated by point that our experimental data are correct,
5 months), there were substantial variations, whereas data from other groups are ‘false’. We
FURTHER INFORMATION
as the intra- and interscreen reproducibility are not reporting fraud, but a lack of reproduc- Life Sci VC: Academic bias & biotech failures: http://
of two genome-scale small interfering RNA ibility. In fact, to our knowledge, none of the lifescivc.com/2011/03/academic-bias-biotech-failures/#0_
undefined,0_
screens was influenced by the methodology of studies that our internal projects were based ALL LINKS ARE ACTIVE IN THE ONLINE PDF
the analysis and ranged from 32–99% (REF. 5). on was retracted or suspected to be flawed.

NATURE REVIEWS | DRUG DISCOVERY www.nature.com/reviews/drugdisc

© 2011 Macmillan Publishers Limited. All rights reserved