51. Reproducibility in the absence of selective reporting : An illustration from large-scale brain asymmetry research
- Author
-
Kong, Xiang-Zhen, Francks, Clyde, Allen, Nicholas B., Heslenfeld, Dirk, Hester, Robert, Hibar, Derrek Paul, Ho, Beng-Choon, Ho, Tiffany C., Hoekstra, Pieter J., Holst, Ruth J., Hoogman, Martine, Høvik, Marie F., Howells, Fleur M., Ames, David, Hugdahl, Kenneth, Huyser, Chaim, Ingvar, Martin, Ishikawa, Akari, James, Anthony, Jahanshad, Neda, Jernigan, Terry L., Jönsson, Erik G, Kaleda, Vasily, Kelly, Clare, Andreassen, Ole A., Kerich, Michael, Keshavan, Matcheri S., Khadka, Sabin, Kircher, Tilo, Kohls, Gregor, Konrad, Kerstin, Korucuoglu, Ozlem, Krämer, Bernd, Krug, Axel, Kuntsi, Jonna, Vasquez, Alejandro Arias, Kwon, Jun Soo, Lambregts-Rommelse, Nanda, Landén, Mikael, Lázaro, Luisa, Lebedeva, Irina, Lenroot, Rhoshel, Lesch, Klaus-Peter, Li, Qinqin, Lim, Kelvin O., Liu, Jia, Armstrong, Nicola J., Lochner, Christine, London, Edythe D., Lorenzetti, Valentina, Luciano, Michelle, Luijten, Maartje, Lundervold, Astri J., Mackey, Scott, MacMaster, Frank P., Maingault, Sophie, Malpas, Charles B., Asherson, Phil, Malt, Ulrik F., Mataix-Cols, David, Martin-Santos, Rocio, Mayer, Andrew R., McCarthy, Hazel, Medland, Sarah, Metha, Mitul, Mitchell, Philip B., Mueller, Bryon A., Maniega, Susana Muñoz, Bergo, Felipe, Mazoyer, Bernard, McDonald, Colm, McLellan, Quinn, McMahon, Katie L., McPhilemy, Genevieve, Momenan, Reza, Morales, Angelica M., Narayanaswamy, Janardhanan C., Moreira, José Carlos Vasques, Nerland, Stener, Bastin, Mark E., Nestor, Liam, Newman, Erik, Nigg, Joel T., Nordvik, Jan Egil, Novotny, Stephanie, Weiss, Eileen Oberwelland, O'Gorman, Ruth L., Oosterlaan, Jaap, Oranje, Bob, Orr, Catherine, Batalla, Albert, Overs, Bronwyn, Paloyelis, Yannis, Pauli, Paul, Paulus, Martin, Plessen, Kerstin Jessica, Polier, Georg G., Pomarol-Clotet, Edith, Portella, Maria J., Qiu, Jiang, Radua, Joaquim, Bauer, Jochen, Ramos-Quiroga, Josep Antoni, Reddy, Y. C. Janardhan, Reif, Andreas, Roberts, Gloria, Rosa, Pedro, Rubia, Katya, Sacchet, Matthew D., Sachdev, Perminder S., Salvador, Raymond, Schmaal, Lianne, Baune, Bernhard T, Schulte-Rüther, Martin, Schweren, Lizanne, Seitz, Jochen, Serpa, Mauricio Henriques, Shaw, Philip, Shumskaya, Elena, Silk, Timothy J., Simmons, Alan N., Simulionyte, Egle, Sinha, Rajita, Baur-Streubel, Ramona, Sjoerds, Zsuzsika, Smelror, Runar Elle, Soliva, Joan Carlos, Solowij, Nadia, Souza-Duran, Fabio Luisde, Sponheim, Scott R., Stein, Dan J., Stein, Elliot A., Stevens, Michael, Strike, Lachlan T., Biederman, Joseph, Sudre, Gustavo, Sui, Jing, Tamm, Leanne, Temmingh, Hendrik S., Thoma, Robert J., Tomyshev, Alexander, Tronchin, Giulia, Turner, Jessica, Uhlmann, Anne, Erp, Theo G. M., Blaine, Sara K., Heuvel, Odile A., Meer, Dennis, Eijk, Liza, Vance, Alasdair, Veer, Ilya M., Veltman, Dick J., Venkatasubramanian, Ganesan, Vilarroya, Oscar, Vives-Gilabert, Yolanda, Voineskos, Aristotle N, Boedhoe, Premika, Völzke, Henry, Vuletic, Daniella, Walitza, Susanne, Walter, Henrik, Walton, Esther, Wardlaw, Joanna M., Wen, Wei, Westlye, Lars T., Whelan, Christopher D., White, Tonya, Bøen, Erlend, Wiers, Reinout W., Wright, Margaret J., Wittfeld, Katharina, Yang, Tony T., Yasuda, Clarissa L., Yoncheva, Yuliya, Yücel, Murat, Yun, Je-Yeon, Zanetti, Marcus Vinicius, Zhen, Zonglei, Bose, Anushree, Zhu, Xing-xing, Ziegler, Georg C., Zubicaray, Greig I., Zwiers, Marcel, Project, Karolinska Schizophrenia, Glahn, David C., Crivello, Fabrice, Fisher, Simon E., Thompson, Paul M., Bralten, Janita, Farde, Lars, Flyckt, Lena, Engberg, Göran, Erhardt, Sophie, Fatouros-Bergman, Helena, Cervenka, Simon, Schwieler, Lilly, Piehl, Fredrik, Agartz, Ingrid, Collste, Karin, Brandeis, Daniel, Victorsson, Pauliina, Malmqvist, Anna, Hedberg, Mikael, Orhan, Funda, Sellgren, Carl, Brem, Silvia, Mathias, Samuel R., Brodaty, Henry, Yüksel, Dilara, Brooks, Samantha J., Buitelaar, Jan, Bürger, Christian, Bülow, Robin, Calhoun, Vince, Calvo, Anna, Canales-Rodríguez, Erick Jorge, Cannon, Dara M., Guadalupe, Tulio, Caparelli, Elisabeth C., Castellanos, Francisco X., Cendes, Fernando, Chaim-Avancini, Tiffany Moukbel, Chantiluke, Kaylita, Chen, Qun-lin, Chen, Xiayu, Cheng, Yuqi, Christakou, Anastasia, Clark, Vincent P., Abé, Christoph, Coghill, David, Connolly, Colm G., Conzelmann, Annette, Córdova-Palomera, Aldo, Cousijn, Janna, Crow, Tim, Cubillo, Ana, Dannlowski, Udo, Bruttopilo, Sara Ambrosino, Zeeuw, Patrick, Deary, Ian J., Demeter, Damion V., Di Martino, Adriana, Dickie, Erin W, Dietsche, Bruno, Doan, Nhat Trung, Doherty, Colin P., Doyle, Alysa, Durston, Sarah, Earl, Eric, Akudjedu, Theophilus N., Ehrlich, Stefan, Ekman, Carl Johan, Elvsåshagen, Torbjørn, Epstein, Jeffery N., Fair, Damien A., Faraone, Stephen V., Fernández, Guillén, Flint, Claas, Filho, Geraldo Busatto, Förster, Katharina, Aleman, Andre, Fouche, Jean-Paul, Foxe, John J., Frodl, Thomas, Fuentes-Claramonte, Paola, Fullerton, Janice M., Garavan, Hugh, Santos Garcia, Danielle, Gotlib, Ian H., Goudriaan, Anna E., Grabe, Hans Jörgen, Alhusaini, Saud, Groenewold, Nynke A., Grotegerd, Dominik, Gruber, Oliver, Gurholt, Tiril, Haavik, Jan, Hahn, Tim, Hansell, Narelle K., Harris, Mathew A., Hartman, Catharina A., Carmen Valdés Hernández, Maria, Alhusaini, Saud, Del Carmen Valdés Hernández, Maria, Heslenfeld, Dirk, Hester, Robert, Hibar, Derrek Paul, Ho, Beng-Choon, Ho, Tiffany C., Hoekstra, Pieter J., van Holst, Ruth J., Hoogman, Martine, Høvik, Marie F., Allen, Nicholas B., Howells, Fleur M., Hugdahl, Kenneth, Huyser, Chaim, Ingvar, Martin, Ishikawa, Akari, James, Anthony, Jahanshad, Neda, Jernigan, Terry L., Jönsson, Erik G., Kaleda, Vasily, Ames, David, Kelly, Clare, Kerich, Michael, Keshavan, Matcheri S., Khadka, Sabin, Kircher, Tilo, Kohls, Gregor, Konrad, Kerstin, Korucuoglu, Ozlem, Krämer, Bernd, Krug, Axel, Andreassen, Ole A., Kuntsi, Jonna, Kwon, Jun Soo, Lambregts-Rommelse, Nanda, Landén, Mikael, Lázaro, Luisa, Lebedeva, Irina, Lenroot, Rhoshel, Lesch, Klaus-Peter, Li, Qinqin, Lim, Kelvin O., Vasquez, Alejandro Arias, Liu, Jia, Lochner, Christine, London, Edythe D., Lorenzetti, Valentina, Luciano, Michelle, Luijten, Maartje, Lundervold, Astri J., Mackey, Scott, MacMaster, Frank P., Maingault, Sophie, Armstrong, Nicola J., Malpas, Charles B., Malt, Ulrik F., Mataix-Cols, David, Martin-Santos, Rocio, Mayer, Andrew R., McCarthy, Hazel, Medland, Sarah, Metha, Mitul, Mitchell, Philip B., Mueller, Bryon A., Asherson, Phil, Maniega, Susana Muñoz, Mazoyer, Bernard, McDonald, Colm, McLellan, Quinn, McMahon, Katie L., McPhilemy, Genevieve, Momenan, Reza, Morales, Angelica M., Narayanaswamy, Janardhanan C., Moreira, José Carlos Vasques, Bergo, Felipe, Nerland, Stener, Nestor, Liam, Newman, Erik, Nigg, Joel T., Nordvik, Jan Egil, Novotny, Stephanie, Weiss, Eileen Oberwelland, O'Gorman, Ruth L., Oosterlaan, Jaap, Oranje, Bob, Bastin, Mark E., Orr, Catherine, Overs, Bronwyn, Paloyelis, Yannis, Pauli, Paul, Paulus, Martin, Plessen, Kerstin Jessica, von Polier, Georg G., Pomarol-Clotet, Edith, Portella, Maria J., Qiu, Jiang, Batalla, Albert, Radua, Joaquim, Ramos-Quiroga, Josep Antoni, Reddy, Y. C. Janardhan, Reif, Andreas, Roberts, Gloria, Rosa, Pedro, Rubia, Katya, Sacchet, Matthew D., Sachdev, Perminder S., Salvador, Raymond, Bauer, Jochen, Schmaal, Lianne, Schulte-Rüther, Martin, Schweren, Lizanne, Seitz, Jochen, Serpa, Mauricio Henriques, Shaw, Philip, Shumskaya, Elena, Silk, Timothy J., Simmons, Alan N., Simulionyte, Egle, Baune, Bernhard T., Sinha, Rajita, Sjoerds, Zsuzsika, Smelror, Runar Elle, Soliva, Joan Carlos, Solowij, Nadia, Souza-Duran, Fabio Luisde, Sponheim, Scott R., Stein, Dan J., Stein, Elliot A., Stevens, Michael, Baur-Streubel, Ramona, Strike, Lachlan T., Sudre, Gustavo, Sui, Jing, Tamm, Leanne, Temmingh, Hendrik S., Thoma, Robert J., Tomyshev, Alexander, Tronchin, Giulia, Turner, Jessica, Uhlmann, Anne, Biederman, Joseph, van Erp, Theo G. M., van den Heuvel, Odile A., van der Meer, Dennis, van Eijk, Liza, Vance, Alasdair, Veer, Ilya M., Veltman, Dick J., Venkatasubramanian, Ganesan, Vilarroya, Oscar, Vives-Gilabert, Yolanda, Blaine, Sara K., Voineskos, Aristotle N., Völzke, Henry, Vuletic, Daniella, Walitza, Susanne, Walter, Henrik, Walton, Esther, Wardlaw, Joanna M., Wen, Wei, Westlye, Lars T., Whelan, Christopher D., Boedhoe, Premika, White, Tonya, Wiers, Reinout W., Wright, Margaret J., Wittfeld, Katharina, Yang, Tony T., Yasuda, Clarissa L., Yoncheva, Yuliya, Yücel, Murat, Yun, Je-Yeon, Zanetti, Marcus Vinicius, Bøen, Erlend, Zhen, Zonglei, Zhu, Xing-Xing, Ziegler, Georg C., de Zubicaray, Greig I., Zwiers, Marcel, Project, Karolinska Schizophrenia, Glahn, David C., Crivello, Fabrice, Fisher, Simon E., Thompson, Paul M., Bose, Anushree, Francks, Clyde, Farde, Lars, Flyckt, Lena, Engberg, Göran, Erhardt, Sophie, Fatouros-Bergman, Helena, Cervenka, Simon, Schwieler, Lilly, Piehl, Fredrik, Agartz, Ingrid, Bralten, Janita, Collste, Karin, Victorsson, Pauliina, Malmqvist, Anna, Hedberg, Mikael, Orhan, Funda, Sellgren, Carl, Brandeis, Daniel, Kong, Xiang-Zhen, Brem, Silvia, Brodaty, Henry, Yüksel, Dilara, Brooks, Samantha J., Buitelaar, Jan, Bürger, Christian, Bülow, Robin, Calhoun, Vince, Calvo, Anna, Canales-Rodríguez, Erick Jorge, Mathias, Samuel R., Cannon, Dara M., Caparelli, Elisabeth C., Castellanos, Francisco X., Cendes, Fernando, Chaim-Avancini, Tiffany Moukbel, Chantiluke, Kaylita, Chen, Qun-Lin, Chen, Xiayu, Cheng, Yuqi, Christakou, Anastasia, Guadalupe, Tulio, Clark, Vincent P., Coghill, David, Connolly, Colm G., Conzelmann, Annette, Córdova-Palomera, Aldo, Cousijn, Janna, Crow, Tim, Cubillo, Ana, Dannlowski, Udo, de Bruttopilo, Sara Ambrosino, Abé, Christoph, de Zeeuw, Patrick, Deary, Ian J., Demeter, Damion V., Di Martino, Adriana, Dickie, Erin W., Dietsche, Bruno, Doan, Nhat Trung, Doherty, Colin P., Doyle, Alysa, Durston, Sarah, Earl, Eric, Ehrlich, Stefan, Ekman, Carl Johan, Elvsåshagen, Torbjørn, Epstein, Jeffery N., Fair, Damien A., Faraone, Stephen V., Fernández, Guillén, Flint, Claas, Filho, Geraldo Busatto, Akudjedu, Theophilus N., Förster, Katharina, Fouche, Jean-Paul, Foxe, John J., Frodl, Thomas, Fuentes-Claramonte, Paola, Fullerton, Janice M., Garavan, Hugh, do Santos Garcia, Danielle, Gotlib, Ian H., Goudriaan, Anna E., Aleman, Andre, Grabe, Hans Jörgen, Groenewold, Nynke A., Grotegerd, Dominik, Gruber, Oliver, Gurholt, Tiril, Haavik, Jan, Hahn, Tim, Hansell, Narelle K., Harris, Mathew A., Hartman, Catharina A., Ontwikkelingspsychologie (Psychologie, FMG), Adult Psychiatry, APH - Mental Health, ANS - Compulsivity, Impulsivity & Attention, Child and Adolescent Psychiatry & Psychosocial Care, Child Psychiatry, ANS - Cellular & Molecular Mechanisms, Laboratory Genetic Metabolic Diseases, Paediatrics, General Paediatrics, ARD - Amsterdam Reproduction and Development, Paediatric Pulmonology, Graduate School, and APH - Digital Health
- Subjects
P‐hacking ,Datasets as Topic ,Publication bias ,0302 clinical medicine ,130 000 Cognitive Neurology & Memory ,Statistics ,team science ,Brain asymmetry ,Multicenter Studies as Topic ,Cervell ,Research Articles ,Cerebral Cortex ,Radiological and Ultrasound Technology ,P-hacking ,05 social sciences ,Brain ,Cerebral cortex ,Middle Aged ,Magnetic Resonance Imaging ,Discoveries in science ,Escorça cerebral ,Neurology ,Biaix de publicació ,Anatomy ,Psychology ,Research Article ,Neuroinformatics ,Adult ,Adolescent ,Neuroimaging ,Descobriments científics ,050105 experimental psychology ,Statistical power ,03 medical and health sciences ,Young Adult ,Magnetic resonance imaging ,Imatges per ressonància magnètica ,multisite collaboration ,Humans ,0501 psychology and cognitive sciences ,Radiology, Nuclear Medicine and imaging ,ddc:610 ,reproducibility ,Aged ,publication bias ,Reproducibility ,Neurodevelopmental disorders Donders Center for Medical Neuroscience [Radboudumc 7] ,Perspective (graphical) ,Reproducibility of Results ,Brain Cortical Thickness ,Research data ,Sample size determination ,Dades de recerca ,Neurology (clinical) ,Scale (map) ,Developmental Psychopathology ,030217 neurology & neurosurgery - Abstract
The problem of poor reproducibility of scientific findings has received much attention over recent years, in a variety of fields including psychology and neuroscience. The problem has been partly attributed to publication bias and unwanted practices such as p‐hacking. Low statistical power in individual studies is also understood to be an important factor. In a recent multisite collaborative study, we mapped brain anatomical left–right asymmetries for regional measures of surface area and cortical thickness, in 99 MRI datasets from around the world, for a total of over 17,000 participants. In the present study, we revisited these hemispheric effects from the perspective of reproducibility. Within each dataset, we considered that an effect had been reproduced when it matched the meta‐analytic effect from the 98 other datasets, in terms of effect direction and significance threshold. In this sense, the results within each dataset were viewed as coming from separate studies in an “ideal publishing environment,” that is, free from selective reporting and p hacking. We found an average reproducibility rate of 63.2% (SD = 22.9%, min = 22.2%, max = 97.0%). As expected, reproducibility was higher for larger effects and in larger datasets. Reproducibility was not obviously related to the age of participants, scanner field strength, FreeSurfer software version, cortical regional measurement reliability, or regional size. These findings constitute an empirical illustration of reproducibility in the absence of publication bias or p hacking, when assessing realistic biological effects in heterogeneous neuroscience data, and given typically‐used sample sizes., Region‐wise effect sizes and reproducibility rates of hemispheric asymmetry effects. In general, effects with higher effect sizes showed higher reproducibility, given the same conditions (e.g., sample size and data heterogeneity).
- Published
- 2022
- Full Text
- View/download PDF