Sali Structure 2002
Sali Structure 2002
Sali Structure 2002
292
heavily penalized, since in this case chances of having Livermore, California 94551
3
models in common with other groups are lower. One Sanger Centre
example: in Sali and colleagues’ scheme, group 526 Wellcome Trust Genome Campus
achieves a higher ranking than group 384. Both groups Cambridgeshire, CB10 1SA
predicted four targets. The first selected very “popular” United Kingdom
4
ones and obtained results comparable to the average, Department of Biochemical Sciences
and the second achieved outstanding results on a set “A. Rossi Fanelli”
of targets that a significant fraction of the predicting University of Rome “La Sapienza”
groups (up to 40%) decided not to tackle and that were P.le Aldo Moro 5
very difficult to predict accurately. 00185 Rome
These criticisms of the Sali and coworkers scheme Italy
should not be interpreted as complacency on our part.
5
We do recognize that there is much room for improve- Correspondence: [email protected]
ment in the CASP criteria. In particular, we appreciate
Received: December 8, 2001
the point that it would be useful to attempt to attach
Revised: February 11, 2002
statistical significance to all of the CASP rankings. Much Accepted: February 11, 2002
time and energy is wasted in arguing over the signifi-
cance of rankings, distracting from the more important References
aspects of the results. A reliable way of assigning signifi-
cance might ameliorate these difficulties, and be fairer 1. Marti-Renom, M.A., Madhusudhan, M.S., Fiser, A., Rost, B., and
Sali, A. (2002). Structure 10, this issue, 435–440.
to some predictors.
2. Moult, J., Fidelis, K., Zemla, A., and Hubbard, T. (2001). Proteins
There is one final point where we do agree with the Suppl. 5, 2–7.
Sali and coworkers position. There is no doubt that 3. Tramontano, A.L., Leplae, R., and Morea,V. (2001). Proteins
large-scale bench marking, such as LiveBench [8] and Suppl. 5, 22–38.
EVA (http://cubic.bioc.columbia.edu/eva) will play an in- 4. Lesk, A.M., Lo Conte, L., and Hubbard, T.J. (2001). Proteins
creasing role in the assessment of structure modeling Suppl. 5, 98–118.
5. Sippl, M.J., Lackner, P., Domingues, F.S., Prlic, A., Malik, R.,
methods. The original form of the CASP experiment was
Andreeva, A., and Wiederstein, M. (2001). Proteins Suppl. 5,
designed in 1993, and was tailored to conditions that 55–67.
existed then. In particular, the process is built around 6. Zemla, A., Venclovas, C., Moult, J., and Fidelis, K. (1999). Pro-
the concept of collecting targets from the experimental teins Suppl. 3, 22–29.
community within a fixed time window. There are now 7. Venclovas, C., Zemla, A., Fidelis, K., and Moult, J. (1997). Pro-
many new opportunities for performing effective mea- teins Suppl. 1, 7–13.
8. Bujnicki, J.M., Elofsson, A., Fischer, D., and Rychlewski, L.
surements of performance of structure prediction meth-
(2001). Protein Sci. 10, 352–361.
ods. LiveBench and EVA were both discussed at the 9. Fischer, D., Elofsson, A., Rychlewski, L., Pazos, F., Valencia, A.,
CASP4 meeting, and we expect more emphasis on these Rost, B., Ortiz, A.R., and Dunbrack, R.L., Jr. (2001). Proteins
in CASP5. The CAFASP series of experiments [9], run Suppl. 5, 171–183.
in close collaboration with CASP, are providing an evalu- 10. Leplae, R.H., and Hubbard, T.J.P. (2002). Bioinformatics, in
ation of automatic prediction methods. We hope that press.
this will also be a feature of CASP5.
Other changes in the prediction assessment field are
underway. The sequence information for some of the
proteins “on hold” in the Protein Data Bank is now pub- PII S0969-2126(02)00729-3
lic. There is an international agreement to provide infor-
mation on proteins under study in structural genomics
projects, including progress in solving each structure
(http://www.nigms.nih.gov/news/reports/airlie_tasks.
Reply to Moult et al.
html). A “model database” equivalent to the Protein Data
Bank, but for computational models, will likely be estab-
lished. These are all valuable new sources of prediction Here we address in brief several criticisms of our paper
targets. New web-based services providing standard- offered by Moult et al. We do not bring up the many
ized evaluation of methods performance (http://predic- points of agreement already mentioned by Moult et al.,
tioncenter.llnl.gov/local/ace/ace.html) [10] will also have although they are greatly appreciated.
an impact. We look forward to the further evolution of the In our short paper, we did not aim to analyze the
CASP framework to incorporate these developments. state of comparative modeling, nor to propose specific
criteria for assessing the accuracy of comparative mod-
eling. Instead, we described how to assess the statistical
John Moult,1,5 Krzysztof Fidelis,2 Adam Zemla,2 significance of ranking of methods given a model quality
Tim Hubbard,3 and Anna Tramontano4 criterion. Applying the ranking method and one particu-
1
Center for Advanced Research in Biotechnology lar model quality criterion that has in fact been used
University of Maryland Biotechnology Institute previously, we illustrated difficulties with ranking of
Rockville, Maryland 20850 comparative modeling methods at CASP4. We sug-
2
Biology and Biotechnology Research Program gested that CASP use some measures of statistical sig-
Lawrence Livermore National Laboratory nificance for whatever model quality criteria are adopted.
Matters Arising
293