1. Variability in the analysis of a single neuroimaging dataset by many teams
- Author
-
Richard C. Reynolds, Jeremy Hogeveen, Joseph T. McGuire, Vittorio Iacovella, Gregory R. Samanez-Larkin, Cemal Koba, Paolo Avesani, Mikella A. Green, Margaret L. Schlichting, Claudio Toro-Serey, Tristan Glatard, Phui Cheng Lim, Emily A. Yearling, Simon B. Eickhoff, Katherine L. Bottenhorn, Susanne Weis, Andrew Erhart, Doris Pischedda, Mauricio R. Delgado, William A. Cunningham, Robert Langner, Matthew B. Wall, Amr Eed, Jorge Moll, Jeffrey B. Dennison, Wouter D. Weeda, Annabel B. Losecaat Vermeer, Anthony C. Juliano, Felix Hoffstaedter, Julia A. Camilleri, Nadège Bault, Cristian Buc Calderon, Xu Zhang, Gustav Tinghög, Vuong Truong, Leah Bakst, Stephan Heunis, Bertrand Thirion, Colin F. Camerer, Bradley C. Love, Tom Verguts, Luca Cecchetti, Patricia A. Reuter-Lorenz, Camille Maumet, Gabrielle Herman, Giacomo Handjaras, Michalis Kassinopoulos, Anthony Romyn, Xiangzhen Kong, Michael Notter, Nina Lauharatanahirun, Claire Donnat, Norberto Malpica, Jelle J. Goeman, Kamalaker Dadi, Michael Mack, Joke Durnez, Alberto De Luca, Margaret A. Sheridan, Shruti Ray, Colin Hawco, Enrico Glerean, J. Paul Hamilton, Qiang Shen, Sebastian Bobadilla-Suarez, Andrea Leo, David Wisniewski, Ayse Ilkay Isik, Lei Zhang, Bharat B. Biswal, Rotem Botvinik-Nezer, Flora Li, Bronson Harry, Andrew Jahn, Elise Lesage, Jean M. Vettel, Marco Barilari, Adrian I. Onicas, Susan Holmes, Khoi Vo, Aahana Bajracharya, Matthew Hughes, Rui Yuan, Nuri Erkut Kucukboyaci, Remi Gau, Sangsuk Yoon, Hongmi Lee, Alexandru D. Iordan, John Thorp, Russell A. Poldrack, Anna Dreber, Joshua E. Zosky, Dylan M. Nielson, Adriana S. Méndez Leal, Erin W. Dickie, Roeland Hancock, Tiago Bortolini, Kaustubh R. Patil, Monica Y.C. Li, Kristin N. Meyer, Fu Shiguang, Kenny Skagerlund, Juan Carlos de la Torre, Jennifer A. Silvers, R. Alison Adcock, Gustav Nilsonne, Zachary J. Cole, Emilio Sanz-Morales, Charles P. Davis, Michael Joseph, Sangil Lee, Tom Johnstone, Peder M. Isager, Jonathan E. Peelle, Emiliano Ricciardi, Scott A. Huettel, G. Matthew Fricke, Leonardo Tozzi, Brice A. Kuhl, Sagana Vijayarajah, Alexander Bowring, Jeanette A. Mumford, Blazej M. Baczkowski, Shabnam Hakimi, Sebastian Kupek, Michael Kirchler, Yanina Prystauka, Niall W. Duncan, Anna van 't Veer, Emily G. Brudner, Jamil P. Bhanji, Sarah M. Tashjian, Senne Braem, Paolo Papale, Simon R. Steinkamp, Georgios D. Mitsis, Kenneth S. L. Yuen, Magnus Johannesson, David J. White, Alec Smith, Krzysztof J. Gorgolewski, Alexander Leemans, Robert W. Cox, Emanuele Olivetti, Luca Turella, Juergen Huber, Tom Schonberg, David V. Smith, Lysia Demetriou, Peter Sokol-Hessner, Juergen Dukart, Glad Mihai, Taylor Salo, Jean-Baptiste Poline, Angela R. Laird, Sergej A.E. Golowin, Sarah Genon, Adriana Galván, Claus Lamm, Joseph W. Kable, Roni Iwanir, Theo Marins, Matthew R. Johnson, Anais Rodriguez-Thompson, Benjamin Meyer, Stefan Czoschke, Loreen Tisdall, Douglas H. Schultz, Evan N. Lintz, Derek Beaton, Peer Herholz, Olivia Guest, Jenny R. Rieck, Laura Fontanesi, Ekaterina Dobryakova, Alexandre Pérez, Olivier Collignon, Xin Di, Ruud Berkers, Sheryl B. Ball, Carlos González-García, Schuyler W. Liphardt, Edna C. Cieslik, Roland G. Benoit, João F. Guassi Moreira, Helena Melero, Jaime J. Castrellon, Hayley R. Brooks, Chuan-Peng Hu, Felix Holzmeister, Cheryl L. Grady, Thomas E. Nichols, Alba Xifra-Porxas, Steven H. Tompson, Kelsey McDonald, Julia Beitner, Timothy R. Koscik, Tel Aviv University (TAU), Dartmouth College [Hanover], Leopold Franzens Universität Innsbruck - University of Innsbruck, California Institute of Technology (CALTECH), Stockholm School of Economics (SSE), University of Wisconsin-Madison, Duke University [Durham], Fondazione Bruno Kessler [Trento, Italy] (FBK), University of Trento [Trento], Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig] (IMPNSC), Max-Planck-Gesellschaft, Washington University in Saint Louis (WUSTL), Boston University [Boston] (BU), Virginia Tech [Blacksburg], Université Catholique de Louvain = Catholic University of Louvain (UCL), Plymouth University, Rotman Research Institute at the Baycrest Centre (RRI), University of Amsterdam [Amsterdam] (UvA), Goethe-University Frankfurt am Main, Rutgers University [Newark], Rutgers University System (Rutgers), New Jersey Institute of Technology [Newark] (NJIT), University of Electronic Science and Technology of China [Chengdu] (UESTC), University College of London [London] (UCL), D'Or Institute for Research and Education [Rio de Janeiro], Florida International University [Miami] (FIU), University of Oxford, Vrije Universiteit Brussel (VUB), Universiteit Gent = Ghent University (UGENT), University of Denver, Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf], Jülich Research Centre, School for Advanced Studies Lucca (IMT), University of Nebraska–Lincoln, University of Nebraska System, National Institutes of Health [Bethesda] (NIH), University of Toronto, Modelling brain structure, function and variability based on high-field MRI data (PARIETAL), Service NEUROSPIN (NEUROSPIN), Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Lancashire Teaching Hospital, University of Connecticut (UCONN), University Medical Center [Utrecht], Imperial College London, Temple University [Philadelphia], Pennsylvania Commonwealth System of Higher Education (PCSHE), Kessler Foundation [East Hanover], Stanford University, Taipei Medical University, TMU-ShuangHo Hospital [New Taipei City], Instituto de Neurociencias de Alicante, University of Basel (Unibas), The University of New Mexico [Albuquerque], Zhejiang University of Technology, University of California [Los Angeles] (UCLA), University of California (UC), Université catholique de Lille (UCL), Concordia University [Montreal], Aalto University, Leiden University Medical Center (LUMC), Universiteit Leiden, Linköping University (LIU), Western Sydney University, Centre for Addiction and Mental Health [Toronto] (CAMH), McGill University = Université McGill [Montréal, Canada], Eindhoven University of Technology [Eindhoven] (TU/e), Epilepsy Centre Kempenhaeghe [Heeze], Leibniz-Institut für Resilienzforschung [Mainz] (LIR), Swinburne University of Technology (Hawthorn campus), University of Michigan [Ann Arbor], University of Michigan System, Max Planck Institute for Empirical Aesthetics [Frankfurt am Main], University of Pennsylvania, Max Planck Institute for Psycholinguistics, Carver College of Medicine, University of Iowa, Rutgers New Jersey Medical School (NJMS), University of Oregon [Eugene], University of Vienna [Vienna], Johns Hopkins University (JHU), Nanjing University (NJU), Haskins Laboratories, The Alan Turing Institute, Universidad Rey Juan Carlos [Madrid] (URJC), Neuroimagerie: méthodes et applications (Empenn), Institut National de la Santé et de la Recherche Médicale (INSERM)-Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT), University of North Carolina [Chapel Hill] (UNC), University of North Carolina System (UNC), Technische Universität Dresden = Dresden University of Technology (TU Dresden), Karolinska Institutet [Stockholm], Stockholm University, Université de Lausanne = University of Lausanne (UNIL), Netherlands Institute for Neuroscience (NIN), Royal Netherlands Academy of Arts and Sciences (KNAW), Charité - UniversitätsMedizin = Charité - University Hospital [Berlin], Humboldt University Of Berlin, Technical University of Berlin / Technische Universität Berlin (TU), University of California [Berkeley] (UC Berkeley), Columbia University [New York], Human Research and Engineering Directorate [Aberdeen Proving Ground], Stanford School of Medicine [Stanford], Stanford Medicine, Stanford University-Stanford University, University of California [Santa Barbara] (UC Santa Barbara), University of Dayton, Maumet, Camille, Technische Universität Berlin (TU), Leiden University, European Commission, Fundación 'la Caixa', Tel Aviv University [Tel Aviv], University of Innsbruck, University of Electronic Science and Technology of China (UESTC), University of Oxford [Oxford], Universiteit Gent = Ghent University [Belgium] (UGENT), University of Nebraska [Lincoln], Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Service NEUROSPIN (NEUROSPIN), Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay, University of California, University of Pennsylvania [Philadelphia], Empenn, Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-CentraleSupélec-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Lausanne (UNIL), Humboldt-Universität zu Berlin, University of California [Berkeley], University of California [Santa Barbara] (UCSB), Signal Processing Systems, and Human Technology Interaction
- Subjects
Data Analysis ,Male ,Scientific community ,Computer science ,Decision ,Datasets as Topic ,Variation (game tree) ,Field (computer science) ,0302 clinical medicine ,Multidisciplinary approach ,Data Science/methods ,Models ,Meta-Analysis as Topic ,0303 health sciences ,Multidisciplinary ,medicine.diagnostic_test ,Brain ,Magnetic Resonance Imaging ,Research Personnel ,Datasets as Topic/statistics & numerical data ,Neurological ,Female ,ddc:500 ,Research Personnel/organization & administration ,Models, Neurological ,BF ,Article ,03 medical and health sciences ,Neuroimaging ,Functional neuroimaging ,Human behaviour ,medicine ,Humans ,030304 developmental biology ,Statistical hypothesis testing ,Flexibility (engineering) ,[SCCO.NEUR]Cognitive science/Neuroscience ,Functional Neuroimaging ,[SCCO.NEUR] Cognitive science/Neuroscience ,Data Science ,Reproducibility of Results ,Brain/diagnostic imaging ,Pipeline (software) ,Data science ,Workflow ,Logistic Models ,Functional magnetic resonance imaging ,Decision making ,030217 neurology & neurosurgery ,Software - Abstract
Botvinik-Nezer, Rotem et al., Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed., European Union’s Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Skłodowska-Curie grant agreement no. 665501. A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa. S.B.E. was supported by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2).
- Published
- 2020