Author: "Magnus Johannesson" - Searchworks@Jio Institute Digital Library Search Results

Your search keyword '"Magnus Johannesson"' showing total 345 results

Start Over Author "Magnus Johannesson"

345 results on '"Magnus Johannesson"'

1. Can large language models help predict results from a complex behavioural science study?

Author: Steffen Lippert, Anna Dreber, Magnus Johannesson, Warren Tierney, Wilson Cyrus-Lai, Eric Luis Uhlmann, and Thomas Pfeiffer
Subjects: forecasting, large language models, meta-research, Science
Abstract: We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate—at scale and minimal cost—which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human–AI collaboration in science.
Published: 2024
Full Text: View/download PDF

2. Subjective evidence evaluation survey for many-analysts studies

Author: Alexandra Sarafoglou, Suzanne Hoogeveen, Don van den Bergh, Balazs Aczel, Casper J. Albers, Tim Althoff, Rotem Botvinik-Nezer, Niko A. Busch, Andrea M. Cataldo, Berna Devezer, Noah N. N. van Dongen, Anna Dreber, Eiko I. Fried, Rink Hoekstra, Sabine Hoffman, Felix Holzmeister, Jürgen Huber, Nick Huntington-Klein, John Ioannidis, Magnus Johannesson, Michael Kirchler, Eric Loken, Jan-Francois Mangin, Dora Matzke, Albert J. Menkveld, Gustav Nilsonne, Don van Ravenzwaaij, Martin Schweinsberg, Hannah Schulz-Kuempel, David R. Shanks, Daniel J. Simons, Barbara A. Spellman, Andrea H. Stoevenbelt, Barnabas Szaszi, Darinka Trübutschek, Francis Tuerlinckx, Eric L. Uhlmann, Wolf Vanpaemel, Jelte Wicherts, and Eric-Jan Wagenmakers
Subjects: open science, team science, scientific transparency, metascience, crowdsourcing analysis, Science
Abstract: Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.
Published: 2024
Full Text: View/download PDF

3. Promoting Reproducibility and Replicability in Political Science

Author: Abel Brodeur, Kevin Esterling, Jörg Ankel-Peters, Natália S. Bueno, Scott Desposato, Anna Dreber, Federica Genovese, Donald P. Green, Matthew Hepplewhite, Fernando Hoces de la Guardia, Magnus Johannesson, Andreas Kotsadam, Edward Miguel, Yamil R. Velez, and Lauren Young
Subjects: Political science
Abstract: This article reviews and summarizes current reproduction and replication practices in political science. We first provide definitions for reproducibility and replicability. We then review data availability policies for 28 leading political science journals and present the results from a survey of editors about their willingness to publish comments and replications. We discuss new initiatives that seek to promote and generate high-quality reproductions and replications. Finally, we make the case for standards and practices that may help increase data availability, reproducibility, and replicability in political science.
Published: 2024
Full Text: View/download PDF

4. Forecasting the publication and citation outcomes of COVID-19 preprints

Author: Michael Gordon, Michael Bishop, Yiling Chen, Anna Dreber, Brandon Goldfedder, Felix Holzmeister, Magnus Johannesson, Yang Liu, Louisa Tran, Charles Twardy, Juntao Wang, and Thomas Pfeiffer
Subjects: preprinting, forecasting, science policy, Science
Abstract: Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.
Published: 2022
Full Text: View/download PDF

5. Consensus-based guidance for conducting and reporting multi-analyst studies

Author: Balazs Aczel, Barnabas Szaszi, Gustav Nilsonne, Olmo R van den Akker, Casper J Albers, Marcel ALM van Assen, Jojanneke A Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-Nezer, Laura F Bringmann, Niko A Busch, Emmanuel Caruyer, Andrea M Cataldo, Nelson Cowan, Andrew Delios, Noah NN van Dongen, Chris Donkin, Johnny B van Doorn, Anna Dreber, Gilles Dutilh, Gary F Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J Jonas, Alexander T Kindel, Michael Kirchler, Yoram K Kunkels, D Stephen Lindsay, Jean-Francois Mangin, Dora Matzke, Marcus R Munafò, Ben R Newell, Brian A Nosek, Russell A Poldrack, Don van Ravenzwaaij, Jörg Rieskamp, Matthew J Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J Simons, Barbara A Spellman, Samuel St-Jean, Jeffrey J Starns, Eric Luis Uhlmann, Jelte Wicherts, and Eric-Jan Wagenmakers
Subjects: multi-analyst, metascience, statistical practice, science forum, expert consensus, analytical variability, Medicine, Science, Biology (General), QH301-705.5
Abstract: Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.
Published: 2021
Full Text: View/download PDF

6. Associations of autozygosity with a broad range of human phenotypes

Author: David W Clark, Yukinori Okada, Kristjan H S Moore, Dan Mason, Nicola Pirastu, Ilaria Gandin, Hannele Mattsson, Catriona L K Barnes, Kuang Lin, Jing Hua Zhao, Patrick Deelen, Rebecca Rohde, Claudia Schurmann, Xiuqing Guo, Franco Giulianini, Weihua Zhang, Carolina Medina-Gomez, Robert Karlsson, Yanchun Bao, Traci M Bartz, Clemens Baumbach, Ginevra Biino, Matthew J Bixley, Marco Brumat, Jin-Fang Chai, Tanguy Corre, Diana L Cousminer, Annelot M Dekker, David A Eccles, Kristel R van Eijk, Christian Fuchsberger, He Gao, Marine Germain, Scott D Gordon, Hugoline G de Haan, Sarah E Harris, Edith Hofer, Alicia Huerta-Chagoya, Catherine Igartua, Iris E Jansen, Yucheng Jia, Tim Kacprowski, Torgny Karlsson, Marcus E Kleber, Shengchao Alfred Li, Ruifang Li-Gao, Anubha Mahajan, Koichi Matsuda, Karina Meidtner, Weihua Meng, May E Montasser, Peter J van der Most, Matthias Munz, Teresa Nutile, Teemu Palviainen, Gauri Prasad, Rashmi B Prasad, Tallapragada Divya Sri Priyanka, Federica Rizzi, Erika Salvi, Bishwa R Sapkota, Daniel Shriner, Line Skotte, Melissa C Smart, Albert Vernon Smith, Ashley van der Spek, Cassandra N Spracklen, Rona J Strawbridge, Salman M Tajuddin, Stella Trompet, Constance Turman, Niek Verweij, Clara Viberti, Lihua Wang, Helen R Warren, Robyn E Wootton, Lisa R Yanek, Jie Yao, Noha A Yousri, Wei Zhao, Adebowale A Adeyemo, Saima Afaq, Carlos Alberto Aguilar-Salinas, Masato Akiyama, Matthew L Albert, Matthew A Allison, Maris Alver, Tin Aung, Fereidoun Azizi, Amy R Bentley, Heiner Boeing, Eric Boerwinkle, Judith B Borja, Gert J de Borst, Erwin P Bottinger, Linda Broer, Harry Campbell, Stephen Chanock, Miao-Li Chee, Guanjie Chen, Yii-Der I Chen, Zhengming Chen, Yen-Feng Chiu, Massimiliano Cocca, Francis S Collins, Maria Pina Concas, Janie Corley, Giovanni Cugliari, Rob M van Dam, Anna Damulina, Maryam S Daneshpour, Felix R Day, Graciela E Delgado, Klodian Dhana, Alexander S F Doney, Marcus Dörr, Ayo P Doumatey, Nduna Dzimiri, S Sunna Ebenesersdóttir, Joshua Elliott, Paul Elliott, Ralf Ewert, Janine F Felix, Krista Fischer, Barry I Freedman, Giorgia Girotto, Anuj Goel, Martin Gögele, Mark O Goodarzi, Mariaelisa Graff, Einat Granot-Hershkovitz, Francine Grodstein, Simonetta Guarrera, Daniel F Gudbjartsson, Kamran Guity, Bjarni Gunnarsson, Yu Guo, Saskia P Hagenaars, Christopher A Haiman, Avner Halevy, Tamara B Harris, Mehdi Hedayati, David A van Heel, Makoto Hirata, Imo Höfer, Chao Agnes Hsiung, Jinyan Huang, Yi-Jen Hung, M Arfan Ikram, Anuradha Jagadeesan, Pekka Jousilahti, Yoichiro Kamatani, Masahiro Kanai, Nicola D Kerrison, Thorsten Kessler, Kay-Tee Khaw, Chiea Chuen Khor, Dominique P V de Kleijn, Woon-Puay Koh, Ivana Kolcic, Peter Kraft, Bernhard K Krämer, Zoltán Kutalik, Johanna Kuusisto, Claudia Langenberg, Lenore J Launer, Deborah A Lawlor, I-Te Lee, Wen-Jane Lee, Markus M Lerch, Liming Li, Jianjun Liu, Marie Loh, Stephanie J London, Stephanie Loomis, Yingchang Lu, Jian’an Luan, Reedik Mägi, Ani W Manichaikul, Paolo Manunta, Gísli Másson, Nana Matoba, Xue W Mei, Christa Meisinger, Thomas Meitinger, Massimo Mezzavilla, Lili Milani, Iona Y Millwood, Yukihide Momozawa, Amy Moore, Pierre-Emmanuel Morange, Hortensia Moreno-Macías, Trevor A Mori, Alanna C Morrison, Taulant Muka, Yoshinori Murakami, Alison D Murray, Renée de Mutsert, Josyf C Mychaleckyj, Mike A Nalls, Matthias Nauck, Matt J Neville, Ilja M Nolte, Ken K Ong, Lorena Orozco, Sandosh Padmanabhan, Gunnar Pálsson, James S Pankow, Cristian Pattaro, Alison Pattie, Ozren Polasek, Neil Poulter, Peter P Pramstaller, Lluis Quintana-Murci, Katri Räikkönen, Sarju Ralhan, Dabeeru C Rao, Wouter van Rheenen, Stephen S Rich, Paul M Ridker, Cornelius A Rietveld, Antonietta Robino, Frank J A van Rooij, Daniela Ruggiero, Yasaman Saba, Charumathi Sabanayagam, Maria Sabater-Lleal, Cinzia Felicita Sala, Veikko Salomaa, Kevin Sandow, Helena Schmidt, Laura J Scott, William R Scott, Bahareh Sedaghati-Khayat, Bengt Sennblad, Jessica van Setten, Peter J Sever, Wayne H-H Sheu, Yuan Shi, Smeeta Shrestha, Sharvari Rahul Shukla, Jon K Sigurdsson, Timo Tonis Sikka, Jai Rup Singh, Blair H Smith, Alena Stančáková, Alice Stanton, John M Starr, Lilja Stefansdottir, Leon Straker, Patrick Sulem, Gardar Sveinbjornsson, Morris A Swertz, Adele M Taylor, Kent D Taylor, Natalie Terzikhan, Yih-Chung Tham, Gudmar Thorleifsson, Unnur Thorsteinsdottir, Annika Tillander, Russell P Tracy, Teresa Tusié-Luna, Ioanna Tzoulaki, Simona Vaccargiu, Jagadish Vangipurapu, Jan H Veldink, Veronique Vitart, Uwe Völker, Eero Vuoksimaa, Salma M Wakil, Melanie Waldenberger, Gurpreet S Wander, Ya Xing Wang, Nicholas J Wareham, Sarah Wild, Chittaranjan S Yajnik, Jian-Min Yuan, Lingyao Zeng, Liang Zhang, Jie Zhou, Najaf Amin, Folkert W Asselbergs, Stephan J L Bakker, Diane M Becker, Benjamin Lehne, David A Bennett, Leonard H van den Berg, Sonja I Berndt, Dwaipayan Bharadwaj, Lawrence F Bielak, Murielle Bochud, Mike Boehnke, Claude Bouchard, Jonathan P Bradfield, Jennifer A Brody, Archie Campbell, Shai Carmi, Mark J Caulfield, David Cesarini, John C Chambers, Giriraj Ratan Chandak, Ching-Yu Cheng, Marina Ciullo, Marilyn Cornelis, Daniele Cusi, George Davey Smith, Ian J Deary, Rajkumar Dorajoo, Cornelia M van Duijn, David Ellinghaus, Jeanette Erdmann, Johan G Eriksson, Evangelos Evangelou, Michele K Evans, Jessica D Faul, Bjarke Feenstra, Mary Feitosa, Sylvain Foisy, Andre Franke, Yechiel Friedlander, Paolo Gasparini, Christian Gieger, Clicerio Gonzalez, Philippe Goyette, Struan F A Grant, Lyn R Griffiths, Leif Groop, Vilmundur Gudnason, Ulf Gyllensten, Hakon Hakonarson, Anders Hamsten, Pim van der Harst, Chew-Kiat Heng, Andrew A Hicks, Hagit Hochner, Heikki Huikuri, Steven C Hunt, Vincent W V Jaddoe, Philip L De Jager, Magnus Johannesson, Åsa Johansson, Jost B Jonas, J Wouter Jukema, Juhani Junttila, Jaakko Kaprio, Sharon L. R. Kardia, Fredrik Karpe, Meena Kumari, Markku Laakso, Sander W van der Laan, Jari Lahti, Matthias Laudes, Rodney A Lea, Wolfgang Lieb, Thomas Lumley, Nicholas G Martin, Winfried März, Giuseppe Matullo, Mark I McCarthy, Sarah E Medland, Tony R Merriman, Andres Metspalu, Brian F Meyer, Karen L Mohlke, Grant W Montgomery, Dennis Mook-Kanamori, Patricia B Munroe, Kari E North, Dale R Nyholt, Jeffery R O’connell, Carole Ober, Albertine J Oldehinkel, Walter Palmas, Colin Palmer, Gerard G Pasterkamp, Etienne Patin, Craig E Pennell, Louis Perusse, Patricia A Peyser, Mario Pirastu, Tinca J. C. Polderman, David J Porteous, Danielle Posthuma, Bruce M Psaty, John D Rioux, Fernando Rivadeneira, Charles Rotimi, Jerome I Rotter, Igor Rudan, Hester M Den Ruijter, Dharambir K Sanghera, Naveed Sattar, Reinhold Schmidt, Matthias B Schulze, Heribert Schunkert, Robert A Scott, Alan R Shuldiner, Xueling Sim, Neil Small, Jennifer A Smith, Nona Sotoodehnia, E-Shyong Tai, Alexander Teumer, Nicholas J Timpson, Daniela Toniolo, David-Alexandre Tregouet, Tiinamaija Tuomi, Peter Vollenweider, Carol A Wang, David R Weir, John B Whitfield, Cisca Wijmenga, Tien-Yin Wong, John Wright, Jingyun Yang, Lei Yu, Babette S Zemel, Alan B Zonderman, Markus Perola, Patrik K. E. Magnusson, André G Uitterlinden, Jaspal S Kooner, Daniel I Chasman, Ruth J. F. Loos, Nora Franceschini, Lude Franke, Chris S Haley, Caroline Hayward, Robin G Walters, John R. B. Perry, Tōnu Esko, Agnar Helgason, Kari Stefansson, Peter K Joshi, Michiaki Kubo, and James F Wilson
Subjects: Science
Abstract: Inbreeding depression has been observed in many different species, but in humans a systematic analysis has been difficult so far. Here, analysing more than 1.3 million individuals, the authors show that a genomic inbreeding coefficient (FROH) is associated with disadvantageous outcomes in 32 out of 100 traits tested.
Published: 2019
Full Text: View/download PDF

7. Using prediction markets to predict the outcomes in the Defense Advanced Research Projects Agency's next-generation social science programme

Author: Domenico Viganola, Grant Buckles, Yiling Chen, Pablo Diego-Rosell, Magnus Johannesson, Brian A. Nosek, Thomas Pfeiffer, Adam Siegel, and Anna Dreber
Subjects: prediction markets, peer beliefs, hypothesis, Science
Abstract: There is evidence that prediction markets are useful tools to aggregate information on researchers' beliefs about scientific results including the outcome of replications. In this study, we use prediction markets to forecast the results of novel experimental designs that test established theories. We set up prediction markets for hypotheses tested in the Defense Advanced Research Projects Agency's (DARPA) Next Generation Social Science (NGS2) programme. Researchers were invited to bet on whether 22 hypotheses would be supported or not. We define support as a test result in the same direction as hypothesized, with a Bayes factor of at least 10 (i.e. a likelihood of the observed data being consistent with the tested hypothesis that is at least 10 times greater compared with the null hypothesis). In addition to betting on this binary outcome, we asked participants to bet on the expected effect size (in Cohen's d) for each hypothesis. Our goal was to recruit at least 50 participants that signed up to participate in these markets. While this was the case, only 39 participants ended up actually trading. Participants also completed a survey on both the binary result and the effect size. We find that neither prediction markets nor surveys performed well in predicting outcomes for NGS2.
Published: 2021
Full Text: View/download PDF

8. Predicting replicability-Analysis of survey and prediction market data from large-scale forecasting projects.

Author: Michael Gordon, Domenico Viganola, Anna Dreber, Magnus Johannesson, and Thomas Pfeiffer
Subjects: Medicine, Science
Abstract: The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets.
Published: 2021
Full Text: View/download PDF

9. Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme

Author: Michael Gordon, Domenico Viganola, Michael Bishop, Yiling Chen, Anna Dreber, Brandon Goldfedder, Felix Holzmeister, Magnus Johannesson, Yang Liu, Charles Twardy, Juntao Wang, and Thomas Pfeiffer
Subjects: science policy, replication, forecasting, Science
Abstract: The Defense Advanced Research Projects Agency (DARPA) programme ‘Systematizing Confidence in Open Research and Evidence' (SCORE) aims to generate confidence scores for a large number of research claims from empirical studies in the social and behavioural sciences. The confidence scores will provide a quantitative assessment of how likely a claim will hold up in an independent replication. To create the scores, we follow earlier approaches and use prediction markets and surveys to forecast replication outcomes. Based on an initial set of forecasts for the overall replication rate in SCORE and its dependence on the academic discipline and the time of publication, we show that participants expect replication rates to increase over time. Moreover, they expect replication rates to differ between fields, with the highest replication rate in economics (average survey response 58%), and the lowest in psychology and in education (average survey response of 42% for both fields). These results reveal insights into the academic community's views of the replication crisis, including for research fields for which no large-scale replication studies have been undertaken yet.
Published: 2020
Full Text: View/download PDF

10. Principles of Good Practice for Decision Analytic Modeling in Health-Care Evaluation: Report of the ISPOR Task Force on Good Research Practices - Modeling Studies

Author: M. C. Weinstein, Bernie O’Brien, John Hornberger, Joseph Jackson, Magnus Johannesson, Chris Mccabe, Bryan R. Luce, A. V. Pavlysh, and A. S. Kolbin
Subjects: математическое моделирование, оценка технологий здравоохранения, принятие решений, валидация моделей, анализ чувствительности, mathematical modeling, health technology assessment, decision-making, model validation, sensitivity analysis, Medical technology, R855-855.5, Pharmacy and materia medica, RS1-441
Abstract: Principles of Good Practice for Decision Analytic Modeling in Health-Care Evaluation: Report of the ISPOR Task Force on Good Research Practices - Modeling Studies Abstract. Objectives: Mathematical modeling is used widely in economic evaluations of pharmaceuticals and other healthcare technologies. Users of models in government and the private sector need to be able to evaluate the quality of models according to scientific criteria of good practice. This report describes the consensus of a task force convened to provide modelers with guidelines for conducting and reporting modeling studies. Methods: The task force was appointed with the advice and consent of the Board of Directors of ISPOR. Members were experienced developers or users of models, worked in academia and industry, and came from several countries in North America and Europe. The task force met on three occasions, conducted frequent correspondence and exchanges of drafts by electronic mail, and solicited comments on three drafts from a core group of external reviewers and more broadly from the membership of ISPOR. Results: Criteria for assessing the quality of models fell into three areas: model structure, data used as inputs to models, and model validation. Several major themes cut across these areas. Models and their results should be represented as aids to decision making, not as statements of scientific fact; therefore, it is inappropriate to demand that models be validated prospectively before use. However, model assumptions regarding causal structure and parameter estimates should be continually assessed against data, and models should be revised accordingly. Structural assumptions and parameter estimates should be reported clearly and explicitly, and opportunities for users to appreciate the conditional relationship between inputs and outputs should be provided through sensitivity analyses. Conclusions: Model-based evaluations are a valuable resource for health-care decision makers. It is the responsibility of model developers to conduct modeling studies according to the best practicable standards of quality and to communicate results with adequate disclosure of assumptions and with the caveat that conclusions are conditional upon the assumptions and data on which the model is built.
Published: 2018

11. Predicting the replicability of social science lab experiments.

Author: Adam Altmejd, Anna Dreber, Eskil Forsell, Juergen Huber, Taisuke Imai, Magnus Johannesson, Michael Kirchler, Gideon Nave, and Colin Camerer
Subjects: Medicine, Science
Abstract: We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative.
Published: 2019
Full Text: View/download PDF

12. Meta-GWAS Accuracy and Power (MetaGAP) Calculator Shows that Hiding Heritability Is Partially Due to Imperfect Genetic Correlations across Studies.

Author: Ronald de Vlaming, Aysu Okbay, Cornelius A Rietveld, Magnus Johannesson, Patrik K E Magnusson, André G Uitterlinden, Frank J A van Rooij, Albert Hofman, Patrick J F Groenen, A Roy Thurik, and Philipp D Koellinger
Subjects: Genetics, QH426-470
Abstract: Large-scale genome-wide association results are typically obtained from a fixed-effects meta-analysis of GWAS summary statistics from multiple studies spanning different regions and/or time periods. This approach averages the estimated effects of genetic variants across studies. In case genetic effects are heterogeneous across studies, the statistical power of a GWAS and the predictive accuracy of polygenic scores are attenuated, contributing to the so-called 'missing heritability'. Here, we describe the online Meta-GWAS Accuracy and Power (MetaGAP) calculator (available at www.devlaming.eu) which quantifies this attenuation based on a novel multi-study framework. By means of simulation studies, we show that under a wide range of genetic architectures, the statistical power and predictive accuracy provided by this calculator are accurate. We compare the predictions from the MetaGAP calculator with actual results obtained in the GWAS literature. Specifically, we use genomic-relatedness-matrix restricted maximum likelihood to estimate the SNP heritability and cross-study genetic correlation of height, BMI, years of education, and self-rated health in three large samples. These estimates are used as input parameters for the MetaGAP calculator. Results from the calculator suggest that cross-study heterogeneity has led to attenuation of statistical power and predictive accuracy in recent large-scale GWAS efforts on these traits (e.g., for years of education, we estimate a relative loss of 51-62% in the number of genome-wide significant loci and a relative loss in polygenic score R2 of 36-38%). Hence, cross-study heterogeneity contributes to the missing heritability.
Published: 2017
Full Text: View/download PDF

13. Intuition and Moral Decision-Making - The Effect of Time Pressure and Cognitive Load on Moral Judgment and Altruistic Behavior.

Author: Gustav Tinghög, David Andersson, Caroline Bonn, Magnus Johannesson, Michael Kirchler, Lina Koppel, and Daniel Västfjäll
Subjects: Medicine, Science
Abstract: Do individuals intuitively favor certain moral actions over others? This study explores the role of intuitive thinking-induced by time pressure and cognitive load-in moral judgment and behavior. We conduct experiments in three different countries (Sweden, Austria, and the United States) involving over 1,400 subjects. All subjects responded to four trolley type dilemmas and four dictator games involving different charitable causes. Decisions were made under time pressure/time delay or while experiencing cognitive load or control. Overall we find converging evidence that intuitive states do not influence moral decisions. Neither time-pressure nor cognitive load had any effect on moral judgments or altruistic behavior. Thus we find no supporting evidence for the claim that intuitive moral judgments and dictator game giving differ from more reflectively taken decisions. Across all samples and decision tasks men were more likely to make utilitarian moral judgments and act selfishly compared to women, providing further evidence that there are robust gender differences in moral decision-making. However, there were no significant interactions between gender and the treatment manipulations of intuitive versus reflective decision-making.
Published: 2016
Full Text: View/download PDF

14. Using prediction markets to forecast research evaluations

Author: Marcus R. Munafo, Thomas Pfeiffer, Adam Altmejd, Emma Heikensten, Johan Almenberg, Alexander Bird, Yiling Chen, Brad Wilson, Magnus Johannesson, and Anna Dreber
Subjects: research evaluation, prediction market, research excellence framework, Science
Abstract: The 2014 Research Excellence Framework (REF2014) was conducted to assess the quality of research carried out at higher education institutions in the UK over a 6 year period. However, the process was criticized for being expensive and bureaucratic, and it was argued that similar information could be obtained more simply from various existing metrics. We were interested in whether a prediction market on the outcome of REF2014 for 33 chemistry departments in the UK would provide information similar to that obtained during the REF2014 process. Prediction markets have become increasingly popular as a means of capturing what is colloquially known as the ‘wisdom of crowds’, and enable individuals to trade ‘bets’ on whether a specific outcome will occur or not. These have been shown to be successful at predicting various outcomes in a number of domains (e.g. sport, entertainment and politics), but have rarely been tested against outcomes based on expert judgements such as those that formed the basis of REF2014.
Published: 2015
Full Text: View/download PDF

15. The molecular genetic architecture of self-employment.

Author: Matthijs J H M van der Loos, Cornelius A Rietveld, Niina Eklund, Philipp D Koellinger, Fernando Rivadeneira, Gonçalo R Abecasis, Georgina A Ankra-Badu, Sebastian E Baumeister, Daniel J Benjamin, Reiner Biffar, Stefan Blankenberg, Dorret I Boomsma, David Cesarini, Francesco Cucca, Eco J C de Geus, George Dedoussis, Panos Deloukas, Maria Dimitriou, Guðny Eiriksdottir, Johan Eriksson, Christian Gieger, Vilmundur Gudnason, Birgit Höhne, Rolf Holle, Jouke-Jan Hottenga, Aaron Isaacs, Marjo-Riitta Järvelin, Magnus Johannesson, Marika Kaakinen, Mika Kähönen, Stavroula Kanoni, Maarit A Laaksonen, Jari Lahti, Lenore J Launer, Terho Lehtimäki, Marisa Loitfelder, Patrik K E Magnusson, Silvia Naitza, Ben A Oostra, Markus Perola, Katja Petrovic, Lydia Quaye, Olli Raitakari, Samuli Ripatti, Paul Scheet, David Schlessinger, Carsten O Schmidt, Helena Schmidt, Reinhold Schmidt, Andrea Senft, Albert V Smith, Timothy D Spector, Ida Surakka, Rauli Svento, Antonio Terracciano, Emmi Tikkanen, Cornelia M van Duijn, Jorma Viikari, Henry Völzke, H-Erich Wichmann, Philipp S Wild, Sara M Willems, Gonneke Willemsen, Frank J A van Rooij, Patrick J F Groenen, André G Uitterlinden, Albert Hofman, and A Roy Thurik
Subjects: Medicine, Science
Abstract: Economic variables such as income, education, and occupation are known to affect mortality and morbidity, such as cardiovascular disease, and have also been shown to be partly heritable. However, very little is known about which genes influence economic variables, although these genes may have both a direct and an indirect effect on health. We report results from the first large-scale collaboration that studies the molecular genetic architecture of an economic variable-entrepreneurship-that was operationalized using self-employment, a widely-available proxy. Our results suggest that common SNPs when considered jointly explain about half of the narrow-sense heritability of self-employment estimated in twin data (σ(g)(2)/σ(P)(2) = 25%, h(2) = 55%). However, a meta-analysis of genome-wide association studies across sixteen studies comprising 50,627 participants did not identify genome-wide significant SNPs. 58 SNPs with p
Published: 2013
Full Text: View/download PDF

16. Genetic and environmental influences on the relationship between flow proneness, locus of control and behavioral inhibition.

Author: Miriam A Mosing, Nancy L Pedersen, David Cesarini, Magnus Johannesson, Patrik K E Magnusson, Jeanne Nakamura, Guy Madison, and Fredrik Ullén
Subjects: Medicine, Science
Abstract: Flow is a psychological state of high but subjectively effortless attention that typically occurs during active performance of challenging tasks and is accompanied by a sense of automaticity, high control, low self-awareness, and enjoyment. Flow proneness is associated with traits and behaviors related to low neuroticism such as emotional stability, conscientiousness, active coping, self-esteem and life satisfaction. Little is known about the genetic architecture of flow proneness, behavioral inhibition and locus of control--traits also associated with neuroticism--and their interrelation. Here, we hypothesized that individuals low in behavioral inhibition and with an internal locus of control would be more likely to experience flow and explored the genetic and environmental architecture of the relationship between the three variables. Behavioral inhibition and locus of control was measured in a large population sample of 3,375 full twin pairs and 4,527 single twins, about 26% of whom also scored the flow proneness questionnaire. Findings revealed significant but relatively low correlations between the three traits and moderate heritability estimates of .41, .45, and .30 for flow proneness, behavioral inhibition, and locus of control, respectively, with some indication of non-additive genetic influences. For behavioral inhibition we found significant sex differences in heritability, with females showing a higher estimate including significant non-additive genetic influences, while in males the entire heritability was due to additive genetic variance. We also found a mainly genetically mediated relationship between the three traits, suggesting that individuals who are genetically predisposed to experience flow, show less behavioral inhibition (less anxious) and feel that they are in control of their own destiny (internal locus of control). We discuss that some of the genes underlying this relationship may include those influencing the function of dopaminergic neural systems.
Published: 2012
Full Text: View/download PDF

17. Limbic justice--amygdala involvement in immediate rejection in the Ultimatum Game.

Author: Katarina Gospic, Erik Mohlin, Peter Fransson, Predrag Petrovic, Magnus Johannesson, and Martin Ingvar
Subjects: Biology (General), QH301-705.5
Abstract: Imaging studies have revealed a putative neural account of emotional bias in decision making. However, it has been difficult in previous studies to identify the causal role of the different sub-regions involved in decision making. The Ultimatum Game (UG) is a game to study the punishment of norm-violating behavior. In a previous influential paper on UG it was suggested that frontal insular cortex has a pivotal role in the rejection response. This view has not been reconciled with a vast literature that attributes a crucial role in emotional decision making to a subcortical structure (i.e., amygdala). In this study we propose an anatomy-informed model that may join these views. We also present a design that detects the functional anatomical response to unfair proposals in a subcortical network that mediates rapid reactive responses. We used a functional MRI paradigm to study the early components of decision making and challenged our paradigm with the introduction of a pharmacological intervention to perturb the elicited behavioral and neural response. Benzodiazepine treatment decreased the rejection rate (from 37.6% to 19.0%) concomitantly with a diminished amygdala response to unfair proposals, and this in spite of an unchanged feeling of unfairness and unchanged insular response. In the control group, rejection was directly linked to an increase in amygdala activity. These results allow a functional anatomical detection of the early neural components of rejection associated with the initial reactive emotional response. Thus, the act of immediate rejection seems to be mediated by the limbic system and is not solely driven by cortical processes, as previously suggested. Our results also prompt an ethical discussion as we demonstrated that a commonly used drug influences core functions in the human brain that underlie individual autonomy and economic decision making.
Published: 2011
Full Text: View/download PDF

18. No association between oxytocin receptor (OXTR) gene polymorphisms and experimentally elicited social preferences.

Author: Coren L Apicella, David Cesarini, Magnus Johannesson, Christopher T Dawes, Paul Lichtenstein, Björn Wallace, Jonathan Beauchamp, and Lars Westberg
Subjects: Medicine, Science
Abstract: BackgroundOxytocin (OXT) has been implicated in a suite of complex social behaviors including observed choices in economic laboratory experiments. However, actual studies of associations between oxytocin receptor (OXTR) gene variants and experimentally elicited social preferences are rare.Methodology/principal findingsWe test hypotheses of associations between social preferences, as measured by behavior in two economic games, and 9 single nucleotide polymorphisms (SNPs) of the OXTR gene in a sample of Swedish twins (n = 684). Two standard economic games, the dictator game and the trust game, both involving real monetary consequences, were used to elicit such preferences. After correction for multiple hypothesis testing, we found no significant associations between any of the 9 single nucleotide polymorphisms (SNPs) and behavior in either of the games.ConclusionWe were unable to replicate the most significant association reported in previous research between the amount donated in a dictator game and an OXTR genetic variant.
Published: 2010
Full Text: View/download PDF

19. Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study.

Author: Juntao Wang 0004, Jonathan Lei, Anna Dreber, Michael Gordon, Magnus Johannesson, Thomas Pfeiffer 0003, and Yiling Chen 0001
Published: 2022
Full Text: View/download PDF

20. Hormonal Contraceptives Do Not Impact Economic Preferences: Evidence from a Randomized Trial.

Author: Eva Ranehill, Niklas Zethraeus, Liselott Blomberg, Bo von Schoultz, Angelica Lindén Hirschberg, Magnus Johannesson, and Anna Dreber
Published: 2018
Full Text: View/download PDF

21. Does Gender Diversity Promote Nonconformity?

Author: Makan Amini, Mathias Ekström, Tore Ellingsen, Magnus Johannesson, and Fredrik Strömsten
Published: 2017
Full Text: View/download PDF

22. Non-Standard Errors

Author: UCL - SSH/LIDAM/LFIN - Louvain Finance, Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Sebastian Neusüss, Michael Razen, Utz Weitzel, Hasse, Jean-Baptiste, UCL - SSH/LIDAM/LFIN - Louvain Finance, Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Sebastian Neusüss, Michael Razen, Utz Weitzel, and Hasse, Jean-Baptiste
Abstract: In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence- generating process (EGP). We claim that EGP variation across researchers adds uncertainty: Non-standard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for better reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
Published: 2023

23. #EEGManyLabs

Author: Mahnaz Arvaneh, Guillaume Dumas, Christopher S.Y. Benwell, Magnus Johannesson, Cecile D. Ladouceur, José A. Hinojosa, Annmarie MacNamara, Muthuraman Muthuraman, Manuela Ruzzoli, Bradley N. Jack, Cyril Pernet, Christian Beste, Faisal Mushtaq, Joel S. Snyder, Sarah M. Sass, Christian K. Tamnes, Amy R. Bland, Florian Bublatzky, Benedikt V. Ehinger, Lauren B. Neal, Robin Vloeberghs, Guiomar Niso, Nicolas Langer, Yuri G. Pavlov, Anna Dreber, Artur Czeszumski, David Luque, Edelyn Verona, Emmanuelle Tognoli, Damian Cruse, Peter E. Clayson, Hamid Karimi-Rouzbahani, Evgenii Kalenkovich, Xun He, Heinrich René Liesefeld, Gilles Pourtois, Liad Mudrik, Magdalena Senderecka, Dominik Welke, Giorgio Ganis, Rhiannon Jones, Louisa Kulke, Sebastian Ocklenburg, Niko A. Busch, Christoph Huber-Huber, Gustav Nilsonne, Layla Kouara, Laura Kaltwasser, Alexandre Schaefer, Daniel E. Bradford, Jan R. Wessel, Peter König, Andreas Keil, Ilya Zakharov, Michael Inzlicht, Marieke K. van Vugt, Stefan Appelhoff, Nika Adamian, Robert Oostenveld, and Artificial Intelligence
Subjects: DYNAMICS, Open science, PREDICTION, NEUROSCIENCES, Social Sciences, REPRODUCIBILITY OF RESULTS, ELECTROENCEPHALOGRAM, Field (computer science), cognitive neuroscience, Cognition, MAGNITUDE, 0302 clinical medicine, REPRODUCIBILITY, open science, Psychology, EEG, Psychology, Experimental, 05 social sciences, HUMAN, Electroencephalography, HUMANS, Test (assessment), Neuropsychology and Physiological Psychology, NEUROSCIENCE, RELIABILITY, ELECTROENCEPHALOGRAPHY, Life Sciences & Biomedicine, Behavioral Sciences, ERP, replication, HUMAN EXPERIMENT, Process (engineering), Cognitive Neuroscience, Experimental and Cognitive Psychology, Cognitive neuroscience, 050105 experimental psychology, 03 medical and health sciences, FORECASTING, Humans, 0501 psychology and cognitive sciences, ARTICLE, Protocol (science), Science & Technology, Data collection, Neurosciences, ATTENTION, Reproducibility of Results, MOTIVATION, Data science, CONTROLLED STUDY, SAMPLE-SIZE, COGNITION, many labs, Neurosciences & Neurology, REFLECTS, 030217 neurology & neurosurgery
Abstract: There is growing awareness across the neuroscience community that the replicability of findings about the relationship between brain activity and cognitive phenomena can be improved by conducting studies with high statistical power that adhere to well-defined and standardised analysis pipelines. Inspired by recent efforts from the psychological sciences, and with the desire to examine some of the foundational findings using electroencephalog-raphy (EEG), we have launched #EEGManyLabs, a large-scale international collaborative replication effort. Since its discovery in the early 20th century, EEG has had a profound in-fluence on our understanding of human cognition, but there is limited evidence on the replicability of some of the most highly cited discoveries. After a systematic search and se-lection process, we have identified 27 of the most influential and continually cited studies in the field. We plan to directly test the replicability of key findings from 20 of these studies in teams of at least three independent laboratories. The design and protocol of each replication effort will be submitted as a Registered Report and peer-reviewed prior to data collection. Prediction markets, open to all EEG researchers, will be used as a forecasting tool to examine which findings the community expects to replicate. This project will update our confidence in some of the most influential EEG findings and generate a large open access database that can be used to inform future research practices. Finally, through this international effort, we hope to create a cultural shift towards inclusive, high-powered multi-laboratory collaborations. (c) 2021 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
Published: 2021

24. Non-Standard Errors

Author: Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Sebastian Neusüss, Michael Razen, Utz Weitzel, Hasse, Jean-Baptiste, and UCL - SSH/LIDAM/LFIN - Louvain Finance
Abstract: In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence- generating process (EGP). We claim that EGP variation across researchers adds uncertainty: Non-standard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for better reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.
Published: 2023

25. Using Genes to Explore the Relationship of Cognitive and Non-Cognitive Skills with Education and Labor Market Outcomes

Author: Thomas Buser, Rafael Ahlskog, Magnus Johannesson, Philipp Koellinger, and Sven Oskarsson
Subjects: History, Polymers and Plastics, Business and International Management, Industrial and Manufacturing Engineering
Published: 2023
Full Text: View/download PDF

26. Are women more generous than men? A meta-analysis

Author: Magnus Johannesson, David Bilén, and Anna Dreber
Subjects: Dictator game, Sample size determination, Meta-analysis, Statistical significance, Statistics, Percentage point, Sample (statistics), Mathematics
Abstract: We perform a meta analysis of gender differences in the standard windfall gains dictator game (DG) by collecting raw data from 53 studies with 117 conditions, giving us 15,016 unique individual observations. We find that women on average give 4 percentage points more than men (Cohen’s $$d=0.16$$ d = 0.16 ), and that this difference decreases to $$3.1\%$$ 3.1 % points (Cohen’s $$d=0.13$$ d = 0.13 ) if we exclude studies where dictators can only give all or nothing. The gender difference is larger if the recipient in the DG is a charity, compared to the standard DG with an anonymous individual as the recipient (a 10.9 versus a $$2.3\%$$ 2.3 % points gender difference). These effect sizes imply that many individual studies on gender differences are underpowered; the median power in our sample of standard DG studies is only $$9\%$$ 9 % to detect the meta-analytic gender difference at the $$5\%$$ 5 % significance level. Moving forward on this topic, sample sizes should thus be substantially larger than what has been the norm in the past.
Published: 2021
Full Text: View/download PDF

27. Comprehension in Economic Games

Author: Lina Koppel, David Andersson, Magnus Johannesson, Eirik Strømland, and Gustav Tinghög
Abstract: Most disciplines rely on economic games to measure prosocial behavior in controlled experimental settings. However, participants’ comprehension of these games might be lower than desirable, which complicates interpretation of results. We here assess subject comprehension of the payoff structure of five standard economic games commonly used to study prosocial behavior: the Dictator Game, Ultimatum Game, Trust Game, Public Goods Game, and Prisoner’s Dilemma. Participants were recruited from two online platforms: Prolific (n = 528) and CloudResearch (using the CloudResearch MTurk toolkit; n = 540). The Trust Game had the highest level of misunderstanding (70%), followed by the Public Goods Game and the Prisoner’s Dilemma (each at 52%), the Ultimatum Game (27%), and the Dictator Game (24%). Study platform was a significant predictor of misunderstanding in the Dictator Game, Ultimatum Game, and Public Goods Game, with greater misunderstanding on Prolific than CloudResearch. Incentivizing the comprehension questions had no significant impact on misunderstanding in any of the games. The only variable that significantly predicted misunderstanding across all games was numeracy, which was associated with lower misunderstanding. Finally, we found suggestive evidence in exploratory analyses that misunderstanding predicts greater contributions in the Public Goods Game (in line with previous studies) and in the Dictator Game, increased the likelihood to choose the option that maximizes total payoff in the Prisoner’s Dilemma and reduced back transfers in the Trust Game. These findings suggest that misunderstanding may be an important factor in explaining prosocial behavior and that reliance on standard one-shot games may lead researchers to overestimate the importance of social preferences.
Published: 2022
Full Text: View/download PDF

28. Delirium and fever – a rare but dangerous cause Case report

Author: Jon Magnus Johannesson, Hronn Hardardottir, Bjarni Gudmundsson, and Gunnar Gudmundsson
Subjects: Male, medicine.medical_specialty, Fever, business.industry, Delirium, General Medicine, Emergency department, Middle Aged, medicine.disease, Diagnosis, Differential, Neuroleptic malignant syndrome, medicine, Humans, medicine.symptom, Differential diagnosis, Emergency Service, Hospital, Intensive care medicine, business
Abstract: Fever complicated by delirium is a common problem in emergency departments and inpatient wards. There are various possible causes that must all be contemplated. Infections are prominent causes due to acuteness and severity, but when generating a differential diagnosis, other causes may be life-threatening and require swift diagnosis and management. We present here a case of a 58 year old man presenting at the emergency department with fever and delirium. After comprehensive history-taking and examination, alongside targeted testing, the correct diagnosis was ascertained, leading to appropriate treatment.
Published: 2021
Full Text: View/download PDF

29. The Behavioral Genetics of Behavioral Anomalies.

Author: David Cesarini, Magnus Johannesson, Patrik K. E. Magnusson, and Björn Wallace
Published: 2012
Full Text: View/download PDF

30. Does parametric fMRI analysis with SPM yield valid results? - An empirical study of 1484 rest datasets.

Author: Anders Eklund 0002, Mats T. Andersson, Camilla Josephson, Magnus Johannesson, and Hans Knutsson
Published: 2012
Full Text: View/download PDF

31. Social framing effects: Preferences or beliefs?

Author: Tore Ellingsen, Magnus Johannesson, Johanna Mollerstrom, and Sara Munkhammar
Published: 2012
Full Text: View/download PDF

32. Testing guilt aversion.

Author: Tore Ellingsen, Magnus Johannesson, Sigve Tjøtta, and Gaute Torsvik
Published: 2010
Full Text: View/download PDF

33. 2D:4D and Self-Employment: A Preregistered Replication Study in a Large General Population Sample

Author: Magnus Johannesson, Frank M. Fossen, Anna Dreber, and Levent Neyse
Subjects: Economics and Econometrics, Digit ratio, hormones, Second finger, Population sample, 05 social sciences, Fourth finger, Testosterone (patch), entrepreneurship, Biology, self-employment, 03 medical and health sciences, 0302 clinical medicine, In utero, testosterone, 0502 economics and business, Replication (statistics), ddc:330, 050207 economics, Business and International Management, Proxy (statistics), digit ratio, 030217 neurology & neurosurgery, Demography
Abstract: The 2D: 4D digit ratio, the ratio of the length of the second finger to the length of the fourth finger, is often considered a proxy for testosterone exposure in utero. A recent study reported, among other things, an association between the left-hand 2D:4D and self-employment in a sample of 974 adults. In this preregistered study, we replicate the 2D:4D results on a sample of more than 2100 adults from the German Socioeconomic Panel-Innovation Sample (SOEP-IS). We find no statistically significant associations between 2D:4D and self-employment.
Published: 2021
Full Text: View/download PDF

34. Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Author: Lena F. Aeschbach, Balazs Aczel, Maria Vlachou, Blair Saunders, Jennifer A. Joy-Gaba, Ailsa E. Millen, Christopher R. Chartier, Danielle J. Kellier, Carlo Chiorri, Damian Pieńkosz, Tiago Jessé Souza de Lima, Sean Hughes, Carmel A. Levitan, Luca Andrighetto, Mallory C. Kidwell, Domenico Viganola, Sebastiaan Pessers, Sue Kraus, Claudia Chloe Brumbaugh, John E. Edlund, Ernest Baskin, Anna Fedor, Brett Mercier, Michał J. Białek, Sean Coary, Antonia M. Ciunci, Bence E. Bakos, Jon Grahe, Sabina Kołodziej, Radomir Belopavlović, Emilian Pękala, William J. Chopik, Rosanna E. Guadagno, Don A. Moore, Florian Brühlmann, Gideon Nave, Katarzyna Idzikowska, Rachel L. Shubella, Ryan J. Walker, Orsolya Szöke, Mathias Kauff, Ana Orlić, Sara Steegen, Hans IJzerman, Katarzyna Kuchno, Mitchell M. Metzger, Heather M. Claypool, Michael J. Wood, Samuel Lincoln Bezerra Lins, Michael C. Frank, Benjamin Dering, Iris Žeželj, Erica Baranski, Sophia C. Weissgerber, Timothy Razza, Leanne Boucher, Magnus Johannesson, R. Weylin Sternglanz, Yiling Chen, Maya B. Mathur, Christian Nunnally, Jonathan Ravid, Charles R. Ebersole, Lauren Skorb, Kurt Schuepfer, Łukasz Markiewicz, Thomas Schultze, Katherine S. Corker, Thomas Pfeiffer, Darko Stojilović, Oliver Christ, Kayla Ashbaugh, Alan Jern, Caio Ambrosio Lage, Filipe Falcão, Austin Lee Nichols, Peter Babincak, Mauro Giacomantonio, Sean C. Rife, Rafał Muda, Lacy E. Krueger, Jeremy K. Miller, Juliette Richetin, Martin Corley, Venus Meyet, W. Matthew Collins, Luana Elayne Cunha de Souza, Lynda A. R. Stein, Christopher Day, Erica Casini, Astrid Schütz, Ann-Kathrin Torka, Anna Dreber, Diane-Jo Bart-Plange, Steffen R. Giessner, Holly Arrow, Przemysław Sawicki, Joachim Hüffmeier, Ian R. Ferguson, Anna Dalla Rosa, Natasha Tidwell, Hause Lin, Matthew R. Penner, Boban Petrović, Bojana Bodroža, Janos Salamon, Josiah P. J. King, Mark Zrubka, Diane B. V. Bonfiglio, Stefan Schulz-Hardt, Emily Fryberger, Gabriel Baník, David Zealley, Amanda M. Kimbrough, Ewa Hałasa, William Jiménez-Leal, Angelo Panno, Karolina Krasuska, Michael Inzlicht, Jack Arnal, Madhavi Menon, Jia E. Loy, Vanessa S. Kolb, Nicholas G. Bloxsom, Michael H. Bernstein, Máire B. Ford, Grecia Kessinger, Marija V. Čolić, Wolf Vanpaemel, Barnabas Szaszi, Carly tocco, Nick Buttrick, Emanuele Preti, Andres Montealegre, Brian A. Nosek, Katarzyna Gawryluk, Kaylis Hase Rudy, Leigh Ann Vaughn, Anna Palinkas, Rúben Silva, Daniel Wolf, Sarah A. Novak, Aaron L. Wichman, Manuela Thomae, Adam Siegel, Ivana Pedović, Eleanor V. Langford, Kathleen Schmidt, Daniel Storage, Attila Szuts, Ljiljana B. Lazarević, Paul G. Curran, Rias A. Hilliard, Alexander Garinther, Joshua K. Hartshorne, Ani N. Shabazian, Tiago Ramos, Peter Szecsi, Hugh Rabagliati, Kimberly P. Parks, Lily Feinberg, Dylan Manfredi, Ivan Ropovik, Katrin Rentzsch, Michelangelo Vianello, Barbara Sioma, Marton Kovacs, Francis Tuerlinckx, Peter J. B. Hancock, Bradford J. Wiggins, Gavin Brent Sullivan, Danka Purić, Laboratoire Inter-universitaire de Psychologie : Personnalité, Cognition, Changement Social (LIP-PC2S), Université Pierre Mendès France - Grenoble 2 (UPMF)-Université Savoie Mont Blanc (USMB [Université de Savoie] [Université de Chambéry]), Department of Organisation and Personnel Management, Human Resource Excellence, Ebersole, C, Mathur, M, Baranski, E, Bart-Plange, D, Buttrick, N, Chartier, C, Corker, K, Corley, M, Hartshorne, J, Ijzerman, H, Lazarević, L, Rabagliati, H, Ropovik, I, Aczel, B, Aeschbach, L, Andrighetto, L, Arnal, J, Arrow, H, Babincak, P, Bakos, B, Baník, G, Baskin, E, Belopavlović, R, Bernstein, M, Białek, M, Bloxsom, N, Bodroža, B, Bonfiglio, D, Boucher, L, Brühlmann, F, Brumbaugh, C, Casini, E, Chen, Y, Chiorri, C, Chopik, W, Christ, O, Ciunci, A, Claypool, H, Coary, S, Čolić, M, Collins, W, Curran, P, Day, C, Dering, B, Dreber, A, Edlund, J, Falcão, F, Fedor, A, Feinberg, L, Ferguson, I, Ford, M, Frank, M, Fryberger, E, Garinther, A, Gawryluk, K, Ashbaugh, K, Giacomantonio, M, Giessner, S, Grahe, J, Guadagno, R, Hałasa, E, Hancock, P, Hilliard, R, Hüffmeier, J, Hughes, S, Idzikowska, K, Inzlicht, M, Jern, A, Jiménez-Leal, W, Johannesson, M, Joy-Gaba, J, Kauff, M, Kellier, D, Kessinger, G, Kidwell, M, Kimbrough, A, King, J, Kolb, V, Kołodziej, S, Kovacs, M, Krasuska, K, Kraus, S, Krueger, L, Kuchno, K, Lage, C, Langford, E, Levitan, C, de Lima, T, Lin, H, Lins, S, Loy, J, Manfredi, D, Markiewicz, Ł, Menon, M, Mercier, B, Metzger, M, Meyet, V, Millen, A, Miller, J, Montealegre, A, Moore, D, Muda, R, Nave, G, Nichols, A, Novak, S, Nunnally, C, Orlić, A, Palinkas, A, Panno, A, Parks, K, Pedović, I, Pękala, E, Penner, M, Pessers, S, Petrović, B, Pfeiffer, T, Pieńkosz, D, Preti, E, Purić, D, Ramos, T, Ravid, J, Razza, T, Rentzsch, K, Richetin, J, Rife, S, Rosa, A, Rudy, K, Salamon, J, Saunders, B, Sawicki, P, Schmidt, K, Schuepfer, K, Schultze, T, Schulz-Hardt, S, Schütz, A, Shabazian, A, Shubella, R, Siegel, A, Silva, R, Sioma, B, Skorb, L, de Souza, L, Steegen, S, Stein, L, Sternglanz, R, Stojilović, D, Storage, D, Sullivan, G, Szaszi, B, Szecsi, P, Szöke, O, Szuts, A, Thomae, M, Tidwell, N, Tocco, C, Torka, A, Tuerlinckx, F, Vanpaemel, W, Vaughn, L, Vianello, M, Viganola, D, Vlachou, M, Walker, R, Weissgerber, S, Wichman, A, Wiggins, B, Wolf, D, Wood, M, Zealley, D, Žeželj, I, Zrubka, M, Nosek, B, and Faculdade de Psicologia e de Ciências da Educação
Subjects: replication, metascience, Registered Reports, biology, media_common.quotation_subject, Curran, 05 social sciences, [SHS.PSY]Humanities and Social Sciences/Psychology, open data, Art history, 050109 social psychology, Art, biology.organism_classification, preregistered, 050105 experimental psychology, Attila, [STAT.ML]Statistics [stat]/Machine Learning [stat.ML], 0501 psychology and cognitive sciences, reproducibility, [STAT.ME]Statistics [stat]/Methodology [stat.ME], General Psychology, media_common
Abstract: Additional co-authors: Ivan Ropovik, Balazs Aczel, Lena F. Aeschbach, Luca Andrighetto, Jack D. Arnal, Holly Arrow, Peter Babincak, Bence E. Bakos, Gabriel Banik, Ernest Baskin, Radomir Belopavlovic, Michael H. Bernstein, Michal Bialek, Nicholas G. Bloxsom, Bojana Bodroža, Diane B. V. Bonfiglio, Leanne Boucher, Florian Bruhlmann, Claudia C. Brumbaugh, Erica Casini, Yiling Chen, Carlo Chiorri, William J. Chopik, Oliver Christ, Antonia M. Ciunci, Heather M. Claypool, Sean Coary, Marija V. Cˇolic, W. Matthew Collins, Paul G. Curran, Chris R. Day, Anna Dreber, John E. Edlund, Filipe Falcao, Anna Fedor, Lily Feinberg, Ian R. Ferguson, Maire Ford, Michael C. Frank, Emily Fryberger, Alexander Garinther, Katarzyna Gawryluk, Kayla Ashbaugh, Mauro Giacomantonio, Steffen R. Giessner, Jon E. Grahe, Rosanna E. Guadagno, Ewa Halasa, Rias A. Hilliard, Joachim Huffmeier, Sean Hughes, Katarzyna Idzikowska, Michael Inzlicht, Alan Jern, William Jimenez-Leal, Magnus Johannesson, Jennifer A. Joy-Gaba, Mathias Kauff, Danielle J. Kellier, Grecia Kessinger, Mallory C. Kidwell, Amanda M. Kimbrough, Josiah P. J. King, Vanessa S. Kolb, Sabina Kolodziej, Marton Kovacs, Karolina Krasuska, Sue Kraus, Lacy E. Krueger, Katarzyna Kuchno, Caio Ambrosio Lage, Eleanor V. Langford, Carmel A. Levitan, Tiago Jesse Souza de Lima, Hause Lin, Samuel Lins, Jia E. Loy, Dylan Manfredi, Łukasz Markiewicz, Madhavi Menon, Brett Mercier, Mitchell Metzger, Venus Meyet, Jeremy K. Miller, Andres Montealegre, Don A. Moore, Rafal Muda, Gideon Nave, Austin Lee Nichols, Sarah A. Novak, Christian Nunnally, Ana Orlic, Anna Palinkas, Angelo Panno, Kimberly P. Parks, Ivana Pedovic, Emilian Pekala, Matthew R. Penner, Sebastiaan Pessers, Boban Petrovic, Thomas Pfeiffer, Damian Pienkosz, Emanuele Preti, Danka Puric, Tiago Ramos, Jonathan Ravid, Timothy S. Razza, Katrin Rentzsch, Juliette Richetin, Sean C. Rife, Anna Dalla Rosa, Kaylis Hase Rudy, Janos Salamon, Blair Saunders, Przemyslaw Sawicki, Kathleen Schmidt, Kurt Schuepfer, Thomas Schultze, Stefan Schulz-Hardt, Astrid Schutz, Ani N. Shabazian, Rachel L. Shubella, Adam Siegel, Ruben Silva, Barbara Sioma, Lauren Skorb, Luana Elayne Cunha de Souza, Sara Steegen, L. A. R. Stein, R. Weylin Sternglanz, Darko Stojilovic, Daniel Storage, Gavin Brent Sullivan, Barnabas Szaszi, Peter Szecsi, Orsolya Szoke, Attila Szuts, Manuela Thomae, Natasha D. Tidwell, Carly Tocco, Ann-Kathrin Torka, Francis Tuerlinckx, Wolf Vanpaemel, Leigh Ann Vaughn, Michelangelo Vianello, Domenico Viganola, Maria Vlachou, Ryan J. Walker, Sophia C. Weissgerber, Aaron L. Wichman, Bradford J. Wiggins, Daniel Wolf, Michael J. Wood, David Zealley, Iris Žeželj, Mark Zrubka, and Brian A. Nosek
Published: 2020
Full Text: View/download PDF

35. Author Correction: Comparing meta-analyses and preregistered multiple-laboratory replication projects

Author: Amanda Kvarven, Magnus Johannesson, and Eirik Strømland
Subjects: Behavioral Neuroscience, Social Psychology, Replication (statistics), Experimental and Cognitive Psychology, Computational biology, Psychology
Published: 2020
Full Text: View/download PDF

36. Using prediction markets to estimate ratings of academic research quality in a mock Research Excellence Framework exercise

Author: Jacqueline Thompson, Anna Dreber, Tom R Gaunt, Michael Gordon, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Matthew Lyon, Ian Penton-Voak, Thomas Pfeiffer, and Marcus Robert Munafo
Subjects: MetaArXiv|Social and Behavioral Sciences, bepress|Social and Behavioral Sciences
Abstract: Background: The Research Excellence Framework (REF) in the United Kingdom is a system to assess the quality of research, requiring higher education institutions to select a sample of research outputs to submit for assessment. We tested whether prediction markets, in a novel application, could help institutions decide which research outputs to submit to the REF. Methods: Prediction markets are tools used to aggregate knowledge from a group who trade on the outcomes of future events. We ran six different prediction markets across three institutions in four different academic fields, asking participants to predict which papers in their department would score above a certain threshold in the REF.Results: Despite low participation in some markets, overall the prediction markets predicted mock REF outcomes with greater than 70% accuracy. We also compared the results of one market to random forests and logistic regression models trained to predict REF scores using bibliometrics, and found they had similar accuracy with prediction markets and high correlation with final market price, suggesting trades were partly informed by publication metrics. Conclusions: Our study provides a proof of concept that prediction markets can be used to forecast REF ratings, and that they capture similar information to other methods. Due to their pragmatic limitations, prediction markets are unlikely to entirely replace other methods of REF selection or research assessment, but they may offer a useful addition to REF selection and may be valuable in training contexts.
Published: 2022

37. Examining the generalizability of research findings from archival data

Author: Delios, A, Giulia Clemente, E, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Luis Uhlmann, E, Zogmaister, C, Andrew Delios, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Eric Luis Uhlmann, Cristina Zogmaister, Delios, A, Giulia Clemente, E, Wu, T, Tan, H, Wang, Y, Gordon, M, Viganola, D, Chen, Z, Dreber, A, Johannesson, M, Pfeiffer, T, Luis Uhlmann, E, Zogmaister, C, Andrew Delios, Elena Giulia Clemente, Tao Wu, Hongbin Tan, Yong Wang, Michael Gordon, Domenico Viganola, Zhaowei Chen, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Eric Luis Uhlmann, and Cristina Zogmaister
Abstract: This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.
Published: 2022

38. Sunk costs and fairness in incomplete information bargaining.

Author: Tore Ellingsen and Magnus Johannesson
Published: 2005
Full Text: View/download PDF

39. Cognitive Reflection and 2D:4D: Evidence from a Large Population Sample

Author: Levent Neyse, Frank M. Fossen, Magnus Johannesson, and Anna Dreber
Subjects: Organizational Behavior and Human Resource Management, Economics and Econometrics, History, Polymers and Plastics, Business and International Management, Industrial and Manufacturing Engineering
Published: 2022
Full Text: View/download PDF

40. Non-Standard Errors

Author: Albert Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Michael Kirchler, Tinbergen Institute, and Finance
Published: 2021

41. Author response: Consensus-based guidance for conducting and reporting multi-analyst studies

Author: Jeffrey J. Starns, Yoram K. Kunkels, Gustav Nilsonne, Balazs Aczel, Felix Holzmeister, Jörg Rieskamp, Andrea M Cataldo, Casper J. Albers, Ben R. Newell, Brian A Nosek, Raphael Silberzahn, Jean-François Mangin, Morton Ann Gernsbacher, Gary F. Egan, Alexandra Sarafoglou, Barnabas Szaszi, Matthew J. Salganik, Martin Schweinsberg, Alexander T. Kindel, Niko A. Busch, Daniel J. Simons, Emmanuel Caruyer, Eric-Jan Wagenmakers, Russell A. Poldrack, Dora Matzke, Noah van Dongen, Anna Dreber, Nelson Cowan, Barbara A. Spellman, Chris Donkin, Gilles Dutilh, Johnny van Doorn, Don van Ravenzwaaij, Marcus R. Munafò, Marcel A.L.M. van Assen, Rink Hoekstra, Juergen Huber, Samuel St-Jean, Laura F. Bringmann, Udo Boehm, Daniel J. Benjamin, Tom Schonberg, Eric Luis Uhlmann, D. Stephen Lindsay, Michael Kirchler, David R. Shanks, Jojanneke A. Bastiaansen, Kai J. Jonas, Andrew Delios, Olmo van den Akker, Jelte M. Wicherts, Sabine Hoffmann, Rotem Botvinik-Nezer, and Magnus Johannesson
Published: 2021
Full Text: View/download PDF

42. Comparing meta-analyses and preregistered multiple-laboratory replication projects

Author: Amanda Kvarven, Eirik Strømland, and Magnus Johannesson
Subjects: 0303 health sciences, Social Psychology, Experimental Replication, Statistics as Topic, Reproducibility of Results, Data interpretation, Experimental and Cognitive Psychology, Publication bias, 03 medical and health sciences, Behavioral Neuroscience, Experimental Laboratories, 0302 clinical medicine, Bias, Meta-Analysis as Topic, Data Interpretation, Statistical, Meta-analysis, Replication (statistics), Statistics, Humans, 030217 neurology & neurosurgery, Behavioral Research, 030304 developmental biology, Research evidence
Abstract: Many researchers rely on meta-analysis to summarize research evidence. However, there is a concern that publication bias and selective reporting may lead to biased meta-analytic effect sizes. We compare the results of meta-analyses to large-scale preregistered replications in psychology carried out at multiple laboratories. The multiple-laboratory replications provide precisely estimated effect sizes that do not suffer from publication bias or selective reporting. We searched the literature and identified 15 meta-analyses on the same topics as multiple-laboratory replications. We find that meta-analytic effect sizes are significantly different from replication effect sizes for 12 out of the 15 meta-replication pairs. These differences are systematic and, on average, meta-analytic effect sizes are almost three times as large as replication effect sizes. We also implement three methods of correcting meta-analysis for bias, but these methods do not substantively improve the meta-analytic results. Kvarven, Stromland and Johannesson compare meta-analyses to multiple-laboratory replication projects and find that meta-analyses overestimate effect sizes by a factor of almost three. Commonly used methods of adjusting for publication bias do not substantively improve results.
Published: 2019
Full Text: View/download PDF

43. The digit ratio (2D:4D) and economic preferences: no robust associations in a sample of 330 women

Author: Anna Dreber, Bo von Schoultz, Angelica Lindén Hirschberg, Eva Ranehill, Magnus Johannesson, Niklas Zethraeus, Elle Parslow, and Liselott Blomberg
Subjects: 2D:4D, Digit ratio, Multiple hypotheses, Sample (statistics), Economic preferences, 050105 experimental psychology, Experiment, 03 medical and health sciences, 0302 clinical medicine, Dictator game, C91, 0502 economics and business, Testosterone, 0501 psychology and cognitive sciences, 050207 economics, Association (psychology), 05 social sciences, Replication Paper, Replicate, Regression, Test (assessment), Sample size determination, D03, Psychology, 030217 neurology & neurosurgery, Demography
Abstract: Many studies report on the association between 2D:4D, a putative marker for prenatal testosterone exposure, and economic preferences. However, most of these studies have limited sample sizes and test multiple hypotheses (without preregistration). In this study we mainly replicate the common specifications found in the literature for the association between the 2D:4D ratio and risk taking, the willingness to compete, and dictator game giving separately. In a sample of 330 women we find no robust associations between any of these economic preferences and 2D:4D. We find no evidence of a statistically significant relation for 16 of the 18 total regressions we run. The two regression specifications which are statistically significant have not previously been reported and the associations are not in the expected direction, and therefore they are unlikely to represent a real effect. Electronic supplementary material The online version of this article (10.1007/s40881-019-00076-y) contains supplementary material, which is available to authorized users.
Published: 2019
Full Text: View/download PDF

44. Data from: Are replication rates the same across academic fields? community forecasts from the DARPA SCORE program

Author: Brandon Goldfedder, Michael P. Bishop, Juntao Wang, Yang Liu, Felix Holzmeister, Yiling Chen, Domenico Viganola, Anna Dreber, Thomas Pfeiffer, Magnus Johannesson, Michael Gordon, and Charles Twardy
Subjects: Computer science, Replication (statistics), Data science
Published: 2021
Full Text: View/download PDF

45. Resource Profile and User Guide of the Polygenic Index Repository

Author: Magnus Johannesson, Hariharan Jayashankar, Avshalom Caspi, David Laibson, Matt McGue, Sven Oskarsson, Alexander I. Young, Nancy Wang, Jeremy Freese, David A. Hinds, William G. Iacono, Andrew Steptoe, Lili Milani, Casper A.P. Burik, Aysu Okbay, Grant Goldman, K. Paige Harden, Jonathan P. Beauchamp, Elliot M. Tucker-Drob, Rafael Ahlskog, Peter M. Visscher, Michael Bennett, Aaron Kleinman, Patrick Turley, Travis T. Mallard, Philipp Koellinger, Olesya Ajnakina, David Cesarini, Tõnu Esko, Michelle N. Meyer, Daniel J. Benjamin, Joel Becker, Benjamin Williams, Kathleen Mullan Harris, Terrie E. Moffitt, Richie Poulton, Richard Karlsson Linnér, Pamela Herd, David L. Corcoran, Patrik K. E. Magnusson, Karen Sugden, and Daniel W. Belsky
Subjects: Measure (data warehouse), Resource (project management), Computer science, Estimator, Latent variable, Data mining, Python (programming language), Construct (philosophy), computer.software_genre, Proxy (statistics), computer, Biobank, computer.programming_language
Abstract: Polygenic indexes (PGIs) are DNA-based predictors. Their value for research in many scientific disciplines is rapidly growing. As a resource for researchers, we used a consistent methodology to construct PGIs for 47 phenotypes in 11 datasets. To maximize the PGIs’ prediction accuracies, we constructed them using genome-wide association studies—some of which are novel—from multiple data sources, including 23andMe and UK Biobank. We present a theoretical framework to help interpret analyses involving PGIs. A key insight is that a PGI can be understood as an unbiased but noisy measure of a latent variable we call the “additive SNP factor.” Regressions in which the true regressor is the additive SNP factor but the PGI is used as its proxy therefore suffer from errors-in-variables bias. We derive an estimator that corrects for the bias, illustrate the correction, and make a Python tool for implementing it publicly available.
Published: 2021
Full Text: View/download PDF

46. Systematizing Confidence in Open Research and Evidence (SCORE)

Author: Nazanin Alipourfard, Beatrix Arendt, Daniel M. Benjamin, Noam Benkler, Michael Metcalf Bishop, Mark Burstein, Martin Bush, James Caverlee, Yiling Chen, Chae Clark, Anna Dreber, Timothy M. Errington, Fiona Fidler, Nicholas William Fox, Aaron Frank, Hannah Fraser, Scott Friedman, Ben Gelman, James Gentile, C Lee Giles, Michael B Gordon, Reed Gordon-Sarney, Christopher Griffin, Timothy Gulden, Krystal Hahn, Robert Hartman, Felix Holzmeister, Xia Ben Hu, Magnus Johannesson, Lee Kezar, Melissa Kline Struhl, Ugur Kuter, Anthony M. Kwasnica, Dong-Ho Lee, Kristina Lerman, Yang Liu, Zachary Loomas, Brianna Luis, Ian Magnusson, Olivia Miske, Fallon Mody, Fred Morstatter, Brian A. Nosek, Elan Simon Parsons, David Pennock, Thomas Pfeiffer, Jay Pujara, Sarah Rajtmajer, Xiang Ren, Abel Salinas, Ravi Kiran Selvam, Frank Shipman, Priya Silverstein, Amber Sprenger, Anna Ms Squicciarini, Steve Stratman, Kexuan Sun, Saatvik Tikoo, Charles Richard Twardy, Andrew Tyner, Domenico Viganola, Juntao Wang, David Peter Wilkinson, Bonnie Wintle, and Jian Wu
Subjects: SocArXiv|Social and Behavioral Sciences|Organization Development, bepress|Social and Behavioral Sciences|Psychology, bepress|Social and Behavioral Sciences|Economics, SocArXiv|Social and Behavioral Sciences|Economics, SocArXiv|Social and Behavioral Sciences|Political Science, SocArXiv|Social and Behavioral Sciences|Leadership Studies, bepress|Social and Behavioral Sciences|Political Science, bepress|Social and Behavioral Sciences|Sociology, SocArXiv|Social and Behavioral Sciences|Sociology, bepress|Social and Behavioral Sciences|Organization Development, SocArXiv|Social and Behavioral Sciences|Legal Studies, SocArXiv|Social and Behavioral Sciences|Public Affairs, Public Policy and Public Administration, bepress|Social and Behavioral Sciences, bepress|Social and Behavioral Sciences|Public Affairs, Public Policy and Public Administration, SocArXiv|Social and Behavioral Sciences, SocArXiv|Social and Behavioral Sciences|Psychology, bepress|Social and Behavioral Sciences|Leadership Studies, bepress|Social and Behavioral Sciences|Legal Studies
Abstract: Assessing the credibility of research claims is a central, continuous, and laborious part of the scientific process. Credibility assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts. Such assessments can require substantial time and effort. Research progress could be accelerated if there were rapid, scalable, accurate credibility indicators to guide attention and resource allocation for further assessment. The SCORE program is creating and validating algorithms to provide confidence scores for research claims at scale. To investigate the viability of scalable tools, teams are creating: a database of claims from papers in the social and behavioral sciences; expert and machine generated estimates of credibility; and, evidence of reproducibility, robustness, and replicability to validate the estimates. Beyond the primary research objective, the data and artifacts generated from this program will be openly shared and provide an unprecedented opportunity to examine research credibility and evidence.
Published: 2021

47. 2D:4D does not predict economic preferences: Evidence from a large, representative sample

Author: Levent Neyse, Anna Dreber, and Magnus Johannesson
Subjects: Economic behavior, Organizational Behavior and Human Resource Management, Economics and Econometrics, Digit ratio, media_common.quotation_subject, Sample (statistics), Prenatal hormones, Altruism, 03 medical and health sciences, 0302 clinical medicine, 0502 economics and business, ddc:330, Preference elicitation, Testosterone, 050207 economics, Association (psychology), Proxy (statistics), Socioeconomic status, Reciprocity (cultural anthropology), media_common, 05 social sciences, Small sample, D03, D9, Psychology, 030217 neurology & neurosurgery, D87, Demography
Abstract: The digit ratio (2D:4D) is considered a proxy for testosterone exposure in utero, and there has been a recent surge of studies testing whether 2D:4D is associated with economic preferences. Although the results are not conclusive, previous studies have reported statistically significant correlations between 2D:4D and risk taking, altruism, positive reciprocity, negative reciprocity and trust. However, most previous studies have small sample sizes gathered from university students and there is also no consensus on the type of analysis (e.g., which hand to analyze or subgroup to focus on) or the preference elicitation method. We present results from a pre-registered large sample study testing if 2D:4D is associated with economic preferences. Data were collected in a representative sample of adults in the German Socioeconomic Panel-Innovation Sample (SOEP-IS), in a sample of about 3450 respondents (about 5 times larger than the previously largest study in this field). Using experimentally validated survey questions, we find no statistically significant association between 2D:4D and economic preferences in the largest study to this date on the topic.
Published: 2021

48. Data from: Are replication rates the same across academic fields? community forecasts from the DARPA SCORE program

Author: Michael, Gordon,, primary, Thomas, Pfeiffer,, additional, Domenico, Viganola,, additional, Michael, Bishop,, additional, Yiling, Chen,, additional, Anna, Dreber,, additional, Brandon, Goldfedder,, additional, Felix, Holzmeister,, additional, Magnus, Johannesson,, additional, Yang, Liu,, additional, Charles, Twardy,, additional, and Juntao, Wang,, additional
Published: 2021
Full Text: View/download PDF

49. Variability in the analysis of a single neuroimaging dataset by many teams

Author: Richard C. Reynolds, Jeremy Hogeveen, Joseph T. McGuire, Vittorio Iacovella, Gregory R. Samanez-Larkin, Cemal Koba, Paolo Avesani, Mikella A. Green, Margaret L. Schlichting, Claudio Toro-Serey, Tristan Glatard, Phui Cheng Lim, Emily A. Yearling, Simon B. Eickhoff, Katherine L. Bottenhorn, Susanne Weis, Andrew Erhart, Doris Pischedda, Mauricio R. Delgado, William A. Cunningham, Robert Langner, Matthew B. Wall, Amr Eed, Jorge Moll, Jeffrey B. Dennison, Wouter D. Weeda, Annabel B. Losecaat Vermeer, Anthony C. Juliano, Felix Hoffstaedter, Julia A. Camilleri, Nadège Bault, Cristian Buc Calderon, Xu Zhang, Gustav Tinghög, Vuong Truong, Leah Bakst, Stephan Heunis, Bertrand Thirion, Colin F. Camerer, Bradley C. Love, Tom Verguts, Luca Cecchetti, Patricia A. Reuter-Lorenz, Camille Maumet, Gabrielle Herman, Giacomo Handjaras, Michalis Kassinopoulos, Anthony Romyn, Xiangzhen Kong, Michael Notter, Nina Lauharatanahirun, Claire Donnat, Norberto Malpica, Jelle J. Goeman, Kamalaker Dadi, Michael Mack, Joke Durnez, Alberto De Luca, Margaret A. Sheridan, Shruti Ray, Colin Hawco, Enrico Glerean, J. Paul Hamilton, Qiang Shen, Sebastian Bobadilla-Suarez, Andrea Leo, David Wisniewski, Ayse Ilkay Isik, Lei Zhang, Bharat B. Biswal, Rotem Botvinik-Nezer, Flora Li, Bronson Harry, Andrew Jahn, Elise Lesage, Jean M. Vettel, Marco Barilari, Adrian I. Onicas, Susan Holmes, Khoi Vo, Aahana Bajracharya, Matthew Hughes, Rui Yuan, Nuri Erkut Kucukboyaci, Remi Gau, Sangsuk Yoon, Hongmi Lee, Alexandru D. Iordan, John Thorp, Russell A. Poldrack, Anna Dreber, Joshua E. Zosky, Dylan M. Nielson, Adriana S. Méndez Leal, Erin W. Dickie, Roeland Hancock, Tiago Bortolini, Kaustubh R. Patil, Monica Y.C. Li, Kristin N. Meyer, Fu Shiguang, Kenny Skagerlund, Juan Carlos de la Torre, Jennifer A. Silvers, R. Alison Adcock, Gustav Nilsonne, Zachary J. Cole, Emilio Sanz-Morales, Charles P. Davis, Michael Joseph, Sangil Lee, Tom Johnstone, Peder M. Isager, Jonathan E. Peelle, Emiliano Ricciardi, Scott A. Huettel, G. Matthew Fricke, Leonardo Tozzi, Brice A. Kuhl, Sagana Vijayarajah, Alexander Bowring, Jeanette A. Mumford, Blazej M. Baczkowski, Shabnam Hakimi, Sebastian Kupek, Michael Kirchler, Yanina Prystauka, Niall W. Duncan, Anna van 't Veer, Emily G. Brudner, Jamil P. Bhanji, Sarah M. Tashjian, Senne Braem, Paolo Papale, Simon R. Steinkamp, Georgios D. Mitsis, Kenneth S. L. Yuen, Magnus Johannesson, David J. White, Alec Smith, Krzysztof J. Gorgolewski, Alexander Leemans, Robert W. Cox, Emanuele Olivetti, Luca Turella, Juergen Huber, Tom Schonberg, David V. Smith, Lysia Demetriou, Peter Sokol-Hessner, Juergen Dukart, Glad Mihai, Taylor Salo, Jean-Baptiste Poline, Angela R. Laird, Sergej A.E. Golowin, Sarah Genon, Adriana Galván, Claus Lamm, Joseph W. Kable, Roni Iwanir, Theo Marins, Matthew R. Johnson, Anais Rodriguez-Thompson, Benjamin Meyer, Stefan Czoschke, Loreen Tisdall, Douglas H. Schultz, Evan N. Lintz, Derek Beaton, Peer Herholz, Olivia Guest, Jenny R. Rieck, Laura Fontanesi, Ekaterina Dobryakova, Alexandre Pérez, Olivier Collignon, Xin Di, Ruud Berkers, Sheryl B. Ball, Carlos González-García, Schuyler W. Liphardt, Edna C. Cieslik, Roland G. Benoit, João F. Guassi Moreira, Helena Melero, Jaime J. Castrellon, Hayley R. Brooks, Chuan-Peng Hu, Felix Holzmeister, Cheryl L. Grady, Thomas E. Nichols, Alba Xifra-Porxas, Steven H. Tompson, Kelsey McDonald, Julia Beitner, Timothy R. Koscik, Tel Aviv University (TAU), Dartmouth College [Hanover], Leopold Franzens Universität Innsbruck - University of Innsbruck, California Institute of Technology (CALTECH), Stockholm School of Economics (SSE), University of Wisconsin-Madison, Duke University [Durham], Fondazione Bruno Kessler [Trento, Italy] (FBK), University of Trento [Trento], Max Planck Institute for Human Cognitive and Brain Sciences [Leipzig] (IMPNSC), Max-Planck-Gesellschaft, Washington University in Saint Louis (WUSTL), Boston University [Boston] (BU), Virginia Tech [Blacksburg], Université Catholique de Louvain = Catholic University of Louvain (UCL), Plymouth University, Rotman Research Institute at the Baycrest Centre (RRI), University of Amsterdam [Amsterdam] (UvA), Goethe-University Frankfurt am Main, Rutgers University [Newark], Rutgers University System (Rutgers), New Jersey Institute of Technology [Newark] (NJIT), University of Electronic Science and Technology of China [Chengdu] (UESTC), University College of London [London] (UCL), D'Or Institute for Research and Education [Rio de Janeiro], Florida International University [Miami] (FIU), University of Oxford, Vrije Universiteit Brussel (VUB), Universiteit Gent = Ghent University (UGENT), University of Denver, Heinrich Heine Universität Düsseldorf = Heinrich Heine University [Düsseldorf], Jülich Research Centre, School for Advanced Studies Lucca (IMT), University of Nebraska–Lincoln, University of Nebraska System, National Institutes of Health [Bethesda] (NIH), University of Toronto, Modelling brain structure, function and variability based on high-field MRI data (PARIETAL), Service NEUROSPIN (NEUROSPIN), Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay-Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria), Lancashire Teaching Hospital, University of Connecticut (UCONN), University Medical Center [Utrecht], Imperial College London, Temple University [Philadelphia], Pennsylvania Commonwealth System of Higher Education (PCSHE), Kessler Foundation [East Hanover], Stanford University, Taipei Medical University, TMU-ShuangHo Hospital [New Taipei City], Instituto de Neurociencias de Alicante, University of Basel (Unibas), The University of New Mexico [Albuquerque], Zhejiang University of Technology, University of California [Los Angeles] (UCLA), University of California (UC), Université catholique de Lille (UCL), Concordia University [Montreal], Aalto University, Leiden University Medical Center (LUMC), Universiteit Leiden, Linköping University (LIU), Western Sydney University, Centre for Addiction and Mental Health [Toronto] (CAMH), McGill University = Université McGill [Montréal, Canada], Eindhoven University of Technology [Eindhoven] (TU/e), Epilepsy Centre Kempenhaeghe [Heeze], Leibniz-Institut für Resilienzforschung [Mainz] (LIR), Swinburne University of Technology (Hawthorn campus), University of Michigan [Ann Arbor], University of Michigan System, Max Planck Institute for Empirical Aesthetics [Frankfurt am Main], University of Pennsylvania, Max Planck Institute for Psycholinguistics, Carver College of Medicine, University of Iowa, Rutgers New Jersey Medical School (NJMS), University of Oregon [Eugene], University of Vienna [Vienna], Johns Hopkins University (JHU), Nanjing University (NJU), Haskins Laboratories, The Alan Turing Institute, Universidad Rey Juan Carlos [Madrid] (URJC), Neuroimagerie: méthodes et applications (Empenn), Institut National de la Santé et de la Recherche Médicale (INSERM)-Inria Rennes – Bretagne Atlantique, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-SIGNAUX ET IMAGES NUMÉRIQUES, ROBOTIQUE (IRISA-D5), Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT), University of North Carolina [Chapel Hill] (UNC), University of North Carolina System (UNC), Technische Universität Dresden = Dresden University of Technology (TU Dresden), Karolinska Institutet [Stockholm], Stockholm University, Université de Lausanne = University of Lausanne (UNIL), Netherlands Institute for Neuroscience (NIN), Royal Netherlands Academy of Arts and Sciences (KNAW), Charité - UniversitätsMedizin = Charité - University Hospital [Berlin], Humboldt University Of Berlin, Technical University of Berlin / Technische Universität Berlin (TU), University of California [Berkeley] (UC Berkeley), Columbia University [New York], Human Research and Engineering Directorate [Aberdeen Proving Ground], Stanford School of Medicine [Stanford], Stanford Medicine, Stanford University-Stanford University, University of California [Santa Barbara] (UC Santa Barbara), University of Dayton, Maumet, Camille, Technische Universität Berlin (TU), Leiden University, European Commission, Fundación 'la Caixa', Tel Aviv University [Tel Aviv], University of Innsbruck, University of Electronic Science and Technology of China (UESTC), University of Oxford [Oxford], Universiteit Gent = Ghent University [Belgium] (UGENT), University of Nebraska [Lincoln], Inria Saclay - Ile de France, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Service NEUROSPIN (NEUROSPIN), Direction de Recherche Fondamentale (CEA) (DRF (CEA)), Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Commissariat à l'énergie atomique et aux énergies alternatives (CEA)-Université Paris-Saclay, University of California, University of Pennsylvania [Philadelphia], Empenn, Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National de Recherche en Informatique et en Automatique (Inria)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Rennes (UNIV-RENNES)-CentraleSupélec-IMT Atlantique Bretagne-Pays de la Loire (IMT Atlantique), Institut Mines-Télécom [Paris] (IMT)-Institut Mines-Télécom [Paris] (IMT)-Université de Bretagne Sud (UBS)-Institut National des Sciences Appliquées - Rennes (INSA Rennes), Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-Institut National des Sciences Appliquées (INSA)-Université de Rennes (UNIV-RENNES)-École normale supérieure - Rennes (ENS Rennes)-Centre National de la Recherche Scientifique (CNRS)-Université de Rennes 1 (UR1), Université de Lausanne (UNIL), Humboldt-Universität zu Berlin, University of California [Berkeley], University of California [Santa Barbara] (UCSB), Signal Processing Systems, and Human Technology Interaction
Subjects: Data Analysis, Male, Scientific community, Computer science, Decision, Datasets as Topic, Variation (game tree), Field (computer science), 0302 clinical medicine, Multidisciplinary approach, Data Science/methods, Models, Meta-Analysis as Topic, 0303 health sciences, Multidisciplinary, medicine.diagnostic_test, Brain, Magnetic Resonance Imaging, Research Personnel, Datasets as Topic/statistics & numerical data, Neurological, Female, ddc:500, Research Personnel/organization & administration, Models, Neurological, BF, Article, 03 medical and health sciences, Neuroimaging, Functional neuroimaging, Human behaviour, medicine, Humans, 030304 developmental biology, Statistical hypothesis testing, Flexibility (engineering), [SCCO.NEUR]Cognitive science/Neuroscience, Functional Neuroimaging, [SCCO.NEUR] Cognitive science/Neuroscience, Data Science, Reproducibility of Results, Brain/diagnostic imaging, Pipeline (software), Data science, Workflow, Logistic Models, Functional magnetic resonance imaging, Decision making, 030217 neurology & neurosurgery, Software
Abstract: Botvinik-Nezer, Rotem et al., Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed., European Union’s Horizon 2020 research and innovation programme (https://ec.europa.eu/programmes/horizon2020/en) under the Marie Skłodowska-Curie grant agreement no. 665501. A. Eed was supported by a predoctoral fellowship La Caixa-Severo Ochoa from Obra Social La Caixa. S.B.E. was supported by the European Union’s Horizon 2020 Research and Innovation Programme under grant agreement no. 785907 (HBP SGA2).
Published: 2020
Full Text: View/download PDF

50. Crowdsourcing hypothesis tests:Making transparent how design choices shape research results

Author: Landy, Justin F., Miaolei (Liam) Jia, Isabel, Ding, Domenico, Viganola, Warren, Tierney, Anna, Dreber, Magnus, Johannesson, Thomas, Pfeiffer, Ebersole, Charles R., Gronau, Quentin F., Alexander, Ly, Don van den Bergh, Maarten, Marsman, Eric-Jan, Wagenmakers, Bartels, Daniel M., Bauman, Christopher W., William, Brady, Felix, Cheung, Andrei, Cimpian, Simone, Dohle, Brent Donnellan, M., Adam, Hahn, Michael, Hall, William, Jiménez-Leal, Johnson, David J., Lucas, Richard E., Benoît, Monin, Andres, Montealegre, Elizabeth, Mullen, Jun, Pang, Jennifer, Ray, Reinero, Diego A., Jesse, Reynolds, Walter, Sowden, Daniel, Storage, Runkun, Su, Tworek, Christina M., Van Bavel, Jay J., Daniel, Walco, Julian, Wills, Xiaobing, Xu, Kai Chi Yam, Xiaoyu, Yang, Martin, Schweinsberg, Molly, Urwitz, Matúš, Adamkovič, Ravin, Alaei, Albers, Casper J., Aurélien, Allard, Anderson, Ian A., Andreychik, Michael R., Peter, Babinčák, Baker, Bradley J., Gabriel, Baník, Ernest, Baskin, Jozef, Bavolar, Berkers, Ruud M. W. J., Michał, Białek, Joel, Blanke, Johannes, Breuer, Ambra, Brizi, Brown, Stephanie E. V., Florian, Brühlmann, Hendrik, Bruns, Leigh, Caldwell, Jean-François, Campourcy, Chan, Eugene Y., Yen-Ping, Chang, Cheung, Benjamin Y., Alycia, Chin, Cho, Kit W., Simon, Columbus, Paul, Conway, Corretti, Conrad A., Craig, Adam W., Curran, Paul G., Danvers, Alexander F., Dawson, Ian G. J., Day, Martin V., Erik, Dietl, Doerflinger, Johannes T., Alice, Dominici, Vilius, Dranseika, Edelsbrunner, Peter A., Edlund, John E., Matthew, Fisher, Anna, Fung, Oliver, Genschow, Timo, Gnambs, Goldberg, Matthew H., Lorenz, Graf-Vlachy, Hafenbrack, Andrew C., Sebastian, Hafenbrädl, Andree, Hartanto, Heck, Patrick R., Heffner, Joseph P., Joseph, Hilgard, Felix, Holzmeister, Horchak, Oleksandr V., Huang, Tina S. -T., Joachim, Hüffmeier, Sean, Hughes, Ian, Hussey, Roland, Imhoff, Bastian, Jaeger, Konrad, Jamro, Johnson, Samuel G. B., Andrew, Jones, Lucas, Keller, Olga, Kombeiz, Krueger, Lacy E., Anthony, Lantian, Laplante, Justin P., Lazarevic, Ljiljana B., Jonathan, Leclerc, Nicole, Legate, Leonhardt, James M., Leung, Desmond W., Levitan, Carmel A., Hause, Lin, Qinglan, Liu, Marco Tullio Liuzza, Locke, Kenneth D., Albert L., Ly, Maceacheron, Melanie D., Madan, Christopher R., Harry, Manley, Silvia, Mari, Marcel, Martončik, Mclean, Scott L., Jonathon, Mcphetres, Mercier, Brett G., Corinna, Michels, Mullarkey, Michael C., Musser, Erica D., Ladislas, Nalborczyk, Gustav, Nilsonne, Otis, Nicholas G., Otner, Sarah M. G., Otto, Philipp E., Oscar, Oviedo-Trespalacios, Mariola Paruzel- Czachura, Francesco, Pellegrini, Pereira, Vitor M. D., Hannah, Perfecto, Gerit, Pfuhl, Phillips, Mark H., Ori, Plonsky, Pozzi, Maura, Purić, Danka B., Brett, Raymond-Barker, Redman, David E., Reynolds, Caleb J., Ivan, Ropovik, Lukas, Röseler, Ruessmann, Janna K., Ryan, William H., Nika, Sablaturova, Schuepfer, Kurt J., Astrid, Schütz, Miroslav, Sirota, Matthias, Stefan, Stocks, Eric L., Strosser, Garrett L., Suchow, Jordan W., Anna, Szabelska, Tey, Kian-Siong S., Leonid, Tiokhin, Jais, Troian, Till, Utesch, Alejandro, Vásquez-Echeverría, Leigh Ann Vaughn, Mark, Verschoor, Bettina von Helversen, Pascal, Wallisch, Weissgerber, Sophia C., Wichman, Aaron L., Woike, Jan K., Iris, Žeželj, Zickfeld, Janis H., Yeonsin, Ahn, Blaettchen, Philippe F., Kang, Xi, Yoo Jin Lee, Parker, Philip M., Parker, Paul A., Song, Jamie S., May-Anne, Very, Lynn, Wong, Uhlmann, Eric L., Psychometrics and Statistics, The Crowdsourcing Hypothesis Tests Collaboration [Member of the MPIB: Jan K. Woike], Laboratoire Parisien de Psychologie Sociale (LAPPS), Université Paris Nanterre (UPN)-Université Paris 8 Vincennes-Saint-Denis (UP8), Human Technology Interaction, Psychologische Methodenleer (Psychologie, FMG), Psychology Other Research (FMG), Université Paris 8 Vincennes-Saint-Denis (UP8)-Université Paris Nanterre (UPN), Organizational Psychology, Department of Social Psychology, Landy, J, Jia, M, Ding, I, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, C, Gronau, Q, Ly, A, van den Bergh, D, Marsman, M, Derks, K, Wagenmakers, E, Proctor, A, Bartels, D, Bauman, C, Brady, W, Cheung, F, Cimpian, A, Dohle, S, Donnellan, M, Hahn, A, Hall, M, Jiménez-Leal, W, Johnson, D, Lucas, R, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, D, Reynolds, J, Sowden, W, Storage, D, Su, R, Tworek, C, Van Bavel, J, Walco, D, Wills, J, Xu, X, Yam, K, Yang, X, Cunningham, W, Schweinsberg, M, Urwitz, M, The Crowdsourcing Hypothesis Tests, C, Uhlmann, E, Mari, S, and Imperial College London
Subjects: Research design, 1ST OFFERS, 1702 Cognitive Sciences, Social Sciences, 050109 social psychology, Ciências Sociais::Psicologia [Domínio/Área Científica], CONCEPTUAL REPLICATIONS, Random Allocation, Empirical research, Crowdsourcing Hypothesis Tests Collaboration, Psychology, research robustness, Psychology(all), General Psychology, ComputingMilieux_MISCELLANEOUS, Marketing, 05 social sciences, SCIENCE, Settore M-PSI/05 - PSICOLOGIA SOCIALE, scientific transparency, Research Design, VDP::Samfunnsvitenskap: 200::Psykologi: 260, [SCCO.PSYC]Cognitive science/Psychology, Crowdsourcing, Cognitive Sciences, crowdsourcing, Cognitive psychology, Adult, replication, Conceptual replications, Social Psychology, Implicit cognition, VDP::Social science: 200::Psychology: 260, Bayesian probability, forecasting, stimulus sampling, INDIVIDUAL-DIFFERENCES, Scientific transparency, Consistency (negotiation), History and Philosophy of Science, Psychology, Multidisciplinary, IMPLICIT, Humans, conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency, 0501 psychology and cognitive sciences, ATTITUDES, 1505 Marketing, METAANALYSIS, M-PSI/05 - PSICOLOGIA SOCIALE, Statistical hypothesis testing, CONSEQUENCES, business.industry, Crowdsourced testing, SOCIAL-PSYCHOLOGY, M-PSI/03 - PSICOMETRIA, 1701 Psychology, REPLICABILITY, Research robustness, business, Forecasting
Abstract: ©American Psychological Association, 2020. This paper is not the copy of record and may not exactly replicate the authoritative document published in the APA journal. Please do not copy or cite without author's permission. The final article is available, upon publication, at: https://doi.apa.org/doi/10.1037/bul0000220 To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = 0.37 to 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim.
Published: 2020
Full Text: View/download PDF

Catalog

Books, media, physical & digital resources

See catalog results

Searchworks

Select search scope, currently: Articles Catalog books, media & more in Jio Institute collections Articles journal articles & other e-resources

Search

Search Constraints

Refine your results

Search Limiters

Topic

Publication Year Range

Language

Category

Publication Type

Journal

Database

Publisher

345 results on '"Magnus Johannesson"'

Search Results

Catalog

Select search scope, currently: Articles

Catalog

books, media & more in Jio Institute collections

Articles

journal articles & other e-resources