1. Critical Assessment of Metagenome Interpretation – a benchmark of computational metagenomics software
- Author
-
Alexander Sczyrba, Peter Hofmann, Peter Belmann, David Koslicki, Stefan Janssen, Johannes Dröge, Ivan Gregor, Stephan Majda, Jessika Fiedler, Eik Dahms, Andreas Bremges, Adrian Fritz, Ruben Garrido-Oter, Tue Sparholt Jørgensen, Nicole Shapiro, Philip D. Blood, Alexey Gurevich, Yang Bai, Dmitrij Turaev, Matthew Z. DeMaere, Rayan Chikhi, Niranjan Nagarajan, Christopher Quince, Fernando Meyer, Monika Balvoit, Lars Hestbjerg Hansen, Søren J. Sørensen, Burton K. H. Chia, Bertrand Denis, Jeff L. Froula, Zhong Wang, Robert Egan, Dongwan Don Kang, Jeffrey J. Cook, Charles Deltel, Michael Beckstette, Claire Lemaitre, Pierre Peterlongo, Guillaume Rizk, Dominique Lavenier, Yu-Wei Wu, Steven W. Singer, Chirag Jain, Marc Strous, Heiner Klingenberg, Peter Meinicke, Michael Barton, Thomas Lingner, Hsin-Hung Lin, Yu-Chieh Liao, Genivaldo Gueiros Z. Silva, Daniel A. Cuevas, Robert A. Edwards, Surya Saha, Vitor C. Piro, Bernhard Y. Renard, Mihai Pop, Hans-Peter Klenk, Markus Göker, Nikos C. Kyrpides, Tanja Woyke, Julia A. Vorholt, Paul Schulze-Lefert, Edward M. Rubin, Aaron E. Darling, Thomas Rattei, Alice C. McHardy, Center for Biotechnology (CeBiTec), Universität Bielefeld = Bielefeld University, Technische Fakultät, Universität Bielefeld, Algorithmische Bioinformatik [Düsseldorf], Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf], Computational Biology of Infection Research [Braunschweig], Helmholtz Centre for Infection Research (HZI), Braunschweig Integrated Centre of Systems Biology [Braunschweig] (BRICS), Technische Universität Braunschweig = Technical University of Braunschweig [Braunschweig]-Helmholtz Centre for Infection Research (HZI), Department of Mathematics [Corvallis, Oregon], Oregon State University (OSU), Department of Computer Science and Engineering [Univ California San Diego] (CSE - UC San Diego), University of California [San Diego] (UC San Diego), University of California (UC)-University of California (UC), Department of Pediatrics [Univ California San Diego] (UC San Diego), School of Medicine [Univ California San Diego] (UC San Diego), University of California (UC)-University of California (UC)-University of California [San Diego] (UC San Diego), Max Planck Institute for Informatics [Saarbrücken], Faculty of Biology [Essen], Universität Duisburg-Essen = University of Duisburg-Essen [Essen], German Center for Infection Research - partner site Hannover-Braunschweig (DZIF), Department of Plant Microbe Interactions, Max Planck Institute for Plant Breeding Research (MPIPZ), Cluster of Excellence on Plant Sciences (CEPLAS), Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf]-Max Planck Institute for Plant Breeding Research (MPIPZ)-Universität zu Köln = University of Cologne, Department of Environmental Science [Roskilde] (ENVS), Aarhus University [Aarhus], Section of Microbiology [Copenhagen], Department of Biology [Copenhagen], Faculty of Science [Copenhagen], University of Copenhagen = Københavns Universitet (UCPH)-University of Copenhagen = Københavns Universitet (UCPH)-Faculty of Science [Copenhagen], University of Copenhagen = Københavns Universitet (UCPH)-University of Copenhagen = Københavns Universitet (UCPH), Department of Science and Environment [Roskilde], Roskilde University, DOE Joint Genome Institute [Walnut Creek], Pittsburgh Supercomputing Center (PSC), Center for Algorithmic Biotechnology [Saint Petersburg], Institute of Translational Biomedicine [Saint-Petersburg], Saint Petersburg University (SPBU)-Saint Petersburg University (SPBU), Centre of Excellence for Plant and Microbial Sciences (CEPAMS), John Innes Centre [Norwich], Biotechnology and Biological Sciences Research Council (BBSRC)-Biotechnology and Biological Sciences Research Council (BBSRC)-Chinese Academy of Agricultural Sciences (CAAS), Department of Microbiology and Ecosystem Science [Vienna], University of Vienna [Vienna], iThree Institute, University of Technology Sydney (UTS), Bioinformatics and Sequence Analysis (BONSAI), Université de Lille, Sciences et Technologies-Inria Lille - Nord Europe, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centre National de la Recherche Scientifique (CNRS), Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS), Department of Computational and Systems Biology [Singapore], Genome Institute of Singapore (GIS), Department of Microbiology and Infection [Coventry], Warwick Medical School, University of Warwick [Coventry]-University of Warwick [Coventry], Intel Corporation [Hillsboro], Intel Corporation [USA], Scalable, Optimized and Parallel Algorithms for Genomics (GenScale), Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-GESTION DES DONNÉES ET DE LA CONNAISSANCE (IRISA-D7), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT), Department of Molecular Infection Biology [Braunschweig], Joint BioEnergy Institute [Emeryville], Graduate Institute of Biomedical Informatics [Taipei], Taipei Medical University, Biological Systems and Engineering [LBNL Berkeley], Lawrence Berkeley National Laboratory [Berkeley] (LBNL), Max planck Institute for Biology of Ageing [Cologne], Energy Engineering and Geomicrobiology [Calgary], University of Calgary, Institute of Microbiology and Genetics [Göttingen], Georg-August-University = Georg-August-Universität Göttingen, University Medical Center Göttingen (UMG), Institute of Population Health Sciences [Taiwan], National Health Research Institutes [Taiwan] (NHRI), San Diego State University (SDSU), Boyce Thompson Institute [Ithaca], Robert Koch Institute [Berlin] (RKI), Ministry of Education [Brazil], Center for Bioinformatics and Computational Biology [Maryland] (CBCB), University of Maryland [College Park], University of Maryland System-University of Maryland System, School of Biology [Newcastle upon Tyne], Newcastle University [Newcastle], Leibniz-Institut DSMZ-Deutsche Sammlung von Mikroorganismen und Zellkulturen GmbH / Leibniz Institute DSMZ-German Collection of Microorganisms and Cell Cultures (DSMZ), biological sciences department [Jeddah], King Abdulaziz University, Institute of Microbiology [Zurich], Eidgenössische Technische Hochschule - Swiss Federal Institute of Technology [Zürich] (ETH Zürich), Department of Computer Science and Engineering [San Diego] (CSE-UCSD), University of California-University of California, Department of Pediatrics [san Diego], UC San Diego School of Medicine, Universität Duisburg-Essen [Essen], Universität zu Köln-Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf]-Max Planck Institute for Plant Breeding Research (MPIPZ), University of Copenhagen = Københavns Universitet (KU)-University of Copenhagen = Københavns Universitet (KU)-Faculty of Science [Copenhagen], University of Copenhagen = Københavns Universitet (KU)-University of Copenhagen = Københavns Universitet (KU), John Innes Centre [Norwich]-Chinese Academy of Agricultural Sciences (CAAS), Centre National de la Recherche Scientifique (CNRS)-Centre de Recherche en Informatique, Signal et Automatique de Lille - UMR 9189 (CRIStAL), Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Centrale Lille-Université de Lille-Centre National de la Recherche Scientifique (CNRS)-Université de Lille, Sciences et Technologies-Inria Lille - Nord Europe, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-CentraleSupélec-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Georg-August-University [Göttingen], Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes 1 (UR1), and Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique)
- Subjects
0303 health sciences ,Biological data ,business.industry ,Benchmarking ,Biology ,Data science ,Genome ,03 medical and health sciences ,0302 clinical medicine ,Software ,Metagenomics ,Profiling (information science) ,Critical assessment ,Taxonomic rank ,[INFO.INFO-BI]Computer Science [cs]/Bioinformatics [q-bio.QM] ,business ,030217 neurology & neurosurgery ,030304 developmental biology - Abstract
In metagenome analysis, computational methods for assembly, taxonomic profiling and binning are key components facilitating downstream biological data interpretation. However, a lack of consensus about benchmarking datasets and evaluation metrics complicates proper performance assessment. The Critical Assessment of Metagenome Interpretation (CAMI) challenge has engaged the global developer community to benchmark their programs on datasets of unprecedented complexity and realism. Benchmark metagenomes were generated from ~700 newly sequenced microorganisms and ~600 novel viruses and plasmids, including genomes with varying degrees of relatedness to each other and to publicly available ones and representing common experimental setups. Across all datasets, assembly and genome binning programs performed well for species represented by individual genomes, while performance was substantially affected by the presence of related strains. Taxonomic profiling and binning programs were proficient at high taxonomic ranks, with a notable performance decrease below the family level. Parameter settings substantially impacted performances, underscoring the importance of program reproducibility. While highlighting current challenges in computational metagenomics, the CAMI results provide a roadmap for software selection to answer specific research questions.
- Published
- 2017
- Full Text
- View/download PDF