Search

Your search keyword '"Dreber, Anna"' showing total 461 results
Start Over Author "Dreber, Anna"
461 results on '"Dreber, Anna"'

1. Promoting Reproducibility and Replicability in Political Science

Author

Brodeur, Abel, Esterling, Kevin, Ankel-Peters, Jörg, Bueno, Natália S, Desposato, Scott, Dreber, Anna, Genovese, Federica, Green, Donald P, Hepplewhite, Matthew, de la Guardia, Fernando Hoces, Johannesson, Magnus, Kotsadam, Andreas, Miguel, Edward, Velez, Yamil R, and Young, Lauren

Subjects
Political Science, Human Society, Reproducibility, replicability, political science, Political science
Abstract

This article reviews and summarizes current reproduction and replication practices in political science. We first provide definitions for reproducibility and replicability. We then review data availability policies for 28 leading political science journals and present the results from a survey of editors about their willingness to publish comments and replications. We discuss new initiatives that seek to promote and generate high-quality reproductions and replications. Finally, we make the case for standards and practices that may help increase data availability, reproducibility, and replicability in political science.

Published
2024

2. Reproduction and replication at scale

Author

Brodeur, Abel, Dreber, Anna, Hoces de la Guardia, Fernando, and Miguel, Edward

Subjects
Biomedical and Clinical Sciences, Health Sciences, Psychology, Humans, Reproduction, Models, Biological, Biomedical and clinical sciences, Health sciences
Published
2024

3. On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions

Author

Schaerer, Michael, du Plessis, Christilene, Nguyen, My Hoang Bao, van Aert, Robbie CM, Tiokhin, Leo, Lakens, Daniël, Clemente, Elena Giulia, Pfeiffer, Thomas, Dreber, Anna, Johannesson, Magnus, Clark, Cory J, Collaboration, Gender Audits Forecasting, and Uhlmann, Eric Luis

Subjects
Commerce, Management, Tourism and Services, Psychology, Gender Equality, Psychology and Cognitive Sciences, Social Psychology, Commerce, management, tourism and services
Abstract

A preregistered meta-analysis, including 244 effect sizes from 85 field audits and 361,645 individual job applications, tested for gender bias in hiring practices in female-stereotypical and gender-balanced as well as male-stereotypical jobs from 1976 to 2020. A “red team” of independent experts was recruited to increase the rigor and robustness of our meta-analytic approach. A forecasting survey further examined whether laypeople (n = 499 nationally representative adults) and scientists (n = 312) could predict the results. Forecasters correctly anticipated reductions in discrimination against female candidates over time. However, both scientists and laypeople overestimated the continuation of bias against female candidates. Instead, selection bias in favor of male over female candidates was eliminated and, if anything, slightly reversed in sign starting in 2009 for mixed-gender and male-stereotypical jobs in our sample. Forecasters further failed to anticipate that discrimination against male candidates for stereotypically female jobs would remain stable across the decades.

Published
2023

4. Replication games: how to make reproducibility research more systematic

Author

Brodeur, Abel, Dreber, Anna, Hoces de la Guardia, Fernando, and Miguel, Edward

Subjects
Economics, Applied Economics, Psychology, Reproducibility of Results, Research, Research Design, Research data, Research management, Scientific community, Sociology, General Science & Technology
Published
2023

5. Can laypeople predict the replicability of social science studies without expert intervention: an exploratory study

Author

Wang, Juntao, Lei, Jonathan, Dreber, Anna, Gordon, Michael, Johannesson, Magnus, Pfeiffer, Thomas, and Chen, Yiling

Subjects
Computer Science - Human-Computer Interaction
Abstract

The low replication rate of published studies has long concerned the social science community, making understanding the replicability a critical problem. Several studies have shown that relevant research communities can make predictions about the replicability of individual studies with above-chance accuracy. Follow-up work further indicates that laypeople can also achieve above-chance accuracy in predicting replicability when experts interpret the studies into short descriptions that are more accessible for laypeople. The involvement of scarce expert resources may make these methods expensive from financial and time perspectives. In this work, we explored whether laypeople can predict the replicability of social science studies without expert intervention. We presented laypeople with raw materials truncated from published social science papers and elicited their answers to questions related to the paper. Our results suggested that laypeople were engaged in this technical task, providing reasonable and self-contained answers. The majority of them also demonstrated a good understanding of the material. However, the solicited information had limited predictive power on the actual replication outcomes. We further discuss several lessons we learned compared to the approach with expert intervention to inspire future works.

Published
2022

6. Examining the generalizability of research findings from archival data.

Author

Delios, Andrew, Clemente, Elena, Wu, Tao, Tan, Hongbin, Wang, Yong, Gordon, Michael, Viganola, Domenico, Chen, Zhaowei, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, and Uhlmann, Eric

Subjects
archival data, context sensitivity, generalizability, reproducibility, research reliability
Abstract

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Published
2022

7. Predicting replicability -- analysis of survey and prediction market data from large-scale forecasting projects

Author

Gordon, Michael, Viganola, Domenico, Dreber, Anna, Johannesson, Magnus, and Pfeiffer, Thomas

Subjects
Statistics - Applications
Abstract

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n=103). Both the prediction market prices and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p < .001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package pooledmaRket and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets., Comment: 19 Pages

Published
2021
Full Text
View/download PDF

8. Consensus-based guidance for conducting and reporting multi-analyst studies

Author

Aczel, Balazs, Szaszi, Barnabas, Nilsonne, Gustav, van den Akker, Olmo R, Albers, Casper J, van Assen, Marcel ALM, Bastiaansen, Jojanneke A, Benjamin, Daniel, Boehm, Udo, Botvinik-Nezer, Rotem, Bringmann, Laura F, Busch, Niko A, Caruyer, Emmanuel, Cataldo, Andrea M, Cowan, Nelson, Delios, Andrew, van Dongen, Noah NN, Donkin, Chris, van Doorn, Johnny B, Dreber, Anna, Dutilh, Gilles, Egan, Gary F, Gernsbacher, Morton Ann, Hoekstra, Rink, Hoffmann, Sabine, Holzmeister, Felix, Huber, Juergen, Johannesson, Magnus, Jonas, Kai J, Kindel, Alexander T, Kirchler, Michael, Kunkels, Yoram K, Lindsay, D Stephen, Mangin, Jean-Francois, Matzke, Dora, Munafò, Marcus R, Newell, Ben R, Nosek, Brian A, Poldrack, Russell A, van Ravenzwaaij, Don, Rieskamp, Jörg, Salganik, Matthew J, Sarafoglou, Alexandra, Schonberg, Tom, Schweinsberg, Martin, Shanks, David, Silberzahn, Raphael, Simons, Daniel J, Spellman, Barbara A, St-Jean, Samuel, Starns, Jeffrey J, Uhlmann, Eric Luis, Wicherts, Jelte, and Wagenmakers, Eric-Jan

Subjects
Biological Sciences, Biomedical and Clinical Sciences, Health Sciences, Consensus, Data Analysis, Datasets as Topic, Research, multi-analyst, metascience, statistical practice, science forum, expert consensus, analytical variability, None, medicine, neuroscience, none, Biochemistry and Cell Biology, Biological sciences, Biomedical and clinical sciences, Health sciences
Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Published
2021

10. Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Author

Ebersole, Charles R, Mathur, Maya B, Baranski, Erica, Bart-Plange, Diane-Jo, Buttrick, Nicholas R, Chartier, Christopher R, Corker, Katherine S, Corley, Martin, Hartshorne, Joshua K, IJzerman, Hans, Lazarević, Ljiljana B, Rabagliati, Hugh, Ropovik, Ivan, Aczel, Balazs, Aeschbach, Lena F, Andrighetto, Luca, Arnal, Jack D, Arrow, Holly, Babincak, Peter, Bakos, Bence E, Baník, Gabriel, Baskin, Ernest, Belopavlović, Radomir, Bernstein, Michael H, Białek, Michał, Bloxsom, Nicholas G, Bodroža, Bojana, Bonfiglio, Diane BV, Boucher, Leanne, Brühlmann, Florian, Brumbaugh, Claudia C, Casini, Erica, Chen, Yiling, Chiorri, Carlo, Chopik, William J, Christ, Oliver, Ciunci, Antonia M, Claypool, Heather M, Coary, Sean, Čolić, Marija V, Collins, W Matthew, Curran, Paul G, Day, Chris R, Dering, Benjamin, Dreber, Anna, Edlund, John E, Falcão, Filipe, Fedor, Anna, Feinberg, Lily, Ferguson, Ian R, Ford, Máire, Frank, Michael C, Fryberger, Emily, Garinther, Alexander, Gawryluk, Katarzyna, Ashbaugh, Kayla, Giacomantonio, Mauro, Giessner, Steffen R, Grahe, Jon E, Guadagno, Rosanna E, Hałasa, Ewa, Hancock, Peter JB, Hilliard, Rias A, Hüffmeier, Joachim, Hughes, Sean, Idzikowska, Katarzyna, Inzlicht, Michael, Jern, Alan, Jiménez-Leal, William, Johannesson, Magnus, Joy-Gaba, Jennifer A, Kauff, Mathias, Kellier, Danielle J, Kessinger, Grecia, Kidwell, Mallory C, Kimbrough, Amanda M, King, Josiah PJ, Kolb, Vanessa S, Kołodziej, Sabina, Kovacs, Marton, Krasuska, Karolina, Kraus, Sue, Krueger, Lacy E, Kuchno, Katarzyna, Lage, Caio Ambrosio, Langford, Eleanor V, Levitan, Carmel A, de Lima, Tiago Jessé Souza, Lin, Hause, Lins, Samuel, Loy, Jia E, Manfredi, Dylan, Markiewicz, Łukasz, Menon, Madhavi, Mercier, Brett, Metzger, Mitchell, Meyet, Venus, Millen, Ailsa E, Miller, Jeremy K, and Montealegre, Andres

Subjects
Psychology, Cognitive and Computational Psychology, Applied and Developmental Psychology, Behavioral and Social Science, replication, reproducibility, metascience, peer review, Registered Reports, open data, preregistered, Applied and developmental psychology, Cognitive and computational psychology
Abstract

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect (p

Published
2020

11. Variability in the analysis of a single neuroimaging dataset by many teams

Author

Botvinik-Nezer, Rotem, Holzmeister, Felix, Camerer, Colin F, Dreber, Anna, Huber, Juergen, Johannesson, Magnus, Kirchler, Michael, Iwanir, Roni, Mumford, Jeanette A, Adcock, R Alison, Avesani, Paolo, Baczkowski, Blazej M, Bajracharya, Aahana, Bakst, Leah, Ball, Sheryl, Barilari, Marco, Bault, Nadège, Beaton, Derek, Beitner, Julia, Benoit, Roland G, Berkers, Ruud MWJ, Bhanji, Jamil P, Biswal, Bharat B, Bobadilla-Suarez, Sebastian, Bortolini, Tiago, Bottenhorn, Katherine L, Bowring, Alexander, Braem, Senne, Brooks, Hayley R, Brudner, Emily G, Calderon, Cristian B, Camilleri, Julia A, Castrellon, Jaime J, Cecchetti, Luca, Cieslik, Edna C, Cole, Zachary J, Collignon, Olivier, Cox, Robert W, Cunningham, William A, Czoschke, Stefan, Dadi, Kamalaker, Davis, Charles P, Luca, Alberto De, Delgado, Mauricio R, Demetriou, Lysia, Dennison, Jeffrey B, Di, Xin, Dickie, Erin W, Dobryakova, Ekaterina, Donnat, Claire L, Dukart, Juergen, Duncan, Niall W, Durnez, Joke, Eed, Amr, Eickhoff, Simon B, Erhart, Andrew, Fontanesi, Laura, Fricke, G Matthew, Fu, Shiguang, Galván, Adriana, Gau, Remi, Genon, Sarah, Glatard, Tristan, Glerean, Enrico, Goeman, Jelle J, Golowin, Sergej AE, González-García, Carlos, Gorgolewski, Krzysztof J, Grady, Cheryl L, Green, Mikella A, Guassi Moreira, João F, Guest, Olivia, Hakimi, Shabnam, Hamilton, J Paul, Hancock, Roeland, Handjaras, Giacomo, Harry, Bronson B, Hawco, Colin, Herholz, Peer, Herman, Gabrielle, Heunis, Stephan, Hoffstaedter, Felix, Hogeveen, Jeremy, Holmes, Susan, Hu, Chuan-Peng, Huettel, Scott A, Hughes, Matthew E, Iacovella, Vittorio, Iordan, Alexandru D, Isager, Peder M, Isik, Ayse I, Jahn, Andrew, Johnson, Matthew R, Johnstone, Tom, Joseph, Michael JE, Juliano, Anthony C, Kable, Joseph W, Kassinopoulos, Michalis, Koba, Cemal, and Kong, Xiang-Zhen

Subjects
Generic health relevance, Brain, Data Analysis, Data Science, Datasets as Topic, Female, Functional Neuroimaging, Humans, Logistic Models, Magnetic Resonance Imaging, Male, Meta-Analysis as Topic, Models, Neurological, Reproducibility of Results, Research Personnel, Software, General Science & Technology
Abstract

Data analysis workflows in many scientific domains have become increasingly complex and flexible. Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses1. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology. Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset2-5. Our findings show that analytical flexibility can have substantial effects on scientific conclusions, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to analytical variability are discussed.

Published
2020

12. Crowdsourcing hypothesis tests: Making transparent how design choices shape research results.

Author

Landy, Justin F, Jia, Miaolei Liam, Ding, Isabel L, Viganola, Domenico, Tierney, Warren, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Ebersole, Charles R, Gronau, Quentin F, Ly, Alexander, van den Bergh, Don, Marsman, Maarten, Derks, Koen, Wagenmakers, Eric-Jan, Proctor, Andrew, Bartels, Daniel M, Bauman, Christopher W, Brady, William J, Cheung, Felix, Cimpian, Andrei, Dohle, Simone, Donnellan, M Brent, Hahn, Adam, Hall, Michael P, Jiménez-Leal, William, Johnson, David J, Lucas, Richard E, Monin, Benoît, Montealegre, Andres, Mullen, Elizabeth, Pang, Jun, Ray, Jennifer, Reinero, Diego A, Reynolds, Jesse, Sowden, Walter, Storage, Daniel, Su, Runkun, Tworek, Christina M, Van Bavel, Jay J, Walco, Daniel, Wills, Julian, Xu, Xiaobing, Yam, Kai Chi, Yang, Xiaoyu, Cunningham, William A, Schweinsberg, Martin, Urwitz, Molly, The Crowdsourcing Hypothesis Tests Collaboration, and Uhlmann, Eric L

Subjects
conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency, Crowdsourcing Hypothesis Tests Collaboration, Crowdsourcing, stimulus sampling, replication, Psychology, Cognitive Sciences, Marketing, Social Psychology
Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Published
2020

13. Nonstandard Errors

Author

MENKVELD, ALBERT J., primary, DREBER, ANNA, additional, HOLZMEISTER, FELIX, additional, HUBER, JUERGEN, additional, JOHANNESSON, MAGNUS, additional, KIRCHLER, MICHAEL, additional, NEUSÜß, SEBASTIAN, additional, RAZEN, MICHAEL, additional, WEITZEL, UTZ, additional, ABAD‐DÍAZ, DAVID, additional, ABUDY, MENACHEM (MENI), additional, ADRIAN, TOBIAS, additional, AIT‐SAHALIA, YACINE, additional, AKMANSOY, OLIVIER, additional, ALCOCK, JAMIE T., additional, ALEXEEV, VITALI, additional, ALOOSH, ARASH, additional, AMATO, LIVIA, additional, AMAYA, DIEGO, additional, ANGEL, JAMES J., additional, AVETIKIAN, ALEJANDRO T., additional, BACH, AMADEUS, additional, BAIDOO, EDWIN, additional, BAKALLI, GAETAN, additional, BAO, LI, additional, BARBON, ANDREA, additional, BASHCHENKO, OKSANA, additional, BINDRA, PARAMPREET C., additional, BJØNNES, GEIR H., additional, BLACK, JEFFREY R., additional, BLACK, BERNARD S., additional, BOGOEV, DIMITAR, additional, CORREA, SANTIAGO BOHORQUEZ, additional, BONDARENKO, OLEG, additional, BOS, CHARLES S., additional, BOSCH‐ROSA, CIRIL, additional, BOURI, ELIE, additional, BROWNLEES, CHRISTIAN, additional, CALAMIA, ANNA, additional, CAO, VIET NGA, additional, CAPELLE‐BLANCARD, GUNTHER, additional, ROMERO, LAURA M. CAPERA, additional, CAPORIN, MASSIMILIANO, additional, CARRION, ALLEN, additional, CASKURLU, TOLGA, additional, CHAKRABARTY, BIDISHA, additional, CHEN, JIAN, additional, CHERNOV, MIKHAIL, additional, CHEUNG, WILLIAM, additional, CHINCARINI, LUDWIG B., additional, CHORDIA, TARUN, additional, CHOW, SHEUNG‐CHI, additional, CLAPHAM, BENJAMIN, additional, COLLIARD, JEAN‐EDOUARD, additional, COMERTON‐FORDE, CAROLE, additional, CURRAN, EDWARD, additional, DAO, THONG, additional, DARE, WALE, additional, DAVIES, RYAN J., additional, BLASIS, RICCARDO DE, additional, NARD, GIANLUCA F. DE, additional, DECLERCK, FANY, additional, DEEV, OLEG, additional, DEGRYSE, HANS, additional, DEKU, SOLOMON Y., additional, DESAGRE, CHRISTOPHE, additional, DIJK, MATHIJS A. VAN, additional, DIM, CHUKWUMA, additional, DIMPFL, THOMAS, additional, DONG, YUN JIANG, additional, DRUMMOND, PHILIP A., additional, DUDDA, TOM, additional, DUEVSKI, TEODOR, additional, DUMITRESCU, ARIADNA, additional, DYAKOV, TEODOR, additional, DYHRBERG, ANNE HAUBO, additional, DZIELIŃSKI, MICHAŁ, additional, EKSI, ASLI, additional, KALAK, IZIDIN EL, additional, ELLEN, SASKIA TER, additional, EUGSTER, NICOLAS, additional, EVANS, MARTIN D. D., additional, FARRELL, MICHAEL, additional, FELEZ‐VINAS, ESTER, additional, FERRARA, GERARDO, additional, FERROUHI, EL MEHDI, additional, FLORI, ANDREA, additional, FLUHARTY‐JAIDEE, JONATHAN T., additional, FOLEY, SEAN D. V., additional, FONG, KINGSLEY Y. L., additional, FOUCAULT, THIERRY, additional, FRANUS, TATIANA, additional, FRANZONI, FRANCESCO, additional, FRIJNS, BART, additional, FRÖMMEL, MICHAEL, additional, FU, SERVANNA M., additional, FÜLLBRUNN, SASCHA C., additional, GAN, BAOQING, additional, GAO, GE, additional, GEHRIG, THOMAS P., additional, GEMAYEL, ROLAND, additional, GERRITSEN, DIRK, additional, GIL‐BAZO, JAVIER, additional, GILDER, DUDLEY, additional, GLOSTEN, LAWRENCE R., additional, GOMEZ, THOMAS, additional, GORBENKO, ARSENY, additional, GRAMMIG, JOACHIM, additional, GRÉGOIRE, VINCENT, additional, GÜÇBILMEZ, UFUK, additional, HAGSTRÖMER, BJÖRN, additional, HAMBUCKERS, JULIEN, additional, HAPNES, ERIK, additional, HARRIS, JEFFREY H., additional, HARRIS, LAWRENCE, additional, HARTMANN, SIMON, additional, HASSE, JEAN‐BAPTISTE, additional, HAUTSCH, NIKOLAUS, additional, HE, XUE‐ZHONG (TONY), additional, HEATH, DAVIDSON, additional, HEDIGER, SIMON, additional, HENDERSHOTT, TERRENCE, additional, HIBBERT, ANN MARIE, additional, HJALMARSSON, ERIK, additional, HOELSCHER, SETH A., additional, HOFFMANN, PETER, additional, HOLDEN, CRAIG W., additional, HORENSTEIN, ALEX R., additional, HUANG, WENQIAN, additional, HUANG, DA, additional, HURLIN, CHRISTOPHE, additional, ILCZUK, KONRAD, additional, IVASHCHENKO, ALEXEY, additional, IYER, SUBRAMANIAN R., additional, JAHANSHAHLOO, HOSSEIN, additional, JALKH, NAJI, additional, JONES, CHARLES M., additional, JURKATIS, SIMON, additional, JYLHÄ, PETRI, additional, KAECK, ANDREAS T., additional, KAISER, GABRIEL, additional, KARAM, ARZÉ, additional, KARMAZIENE, EGLE, additional, KASSNER, BERNHARD, additional, KAUSTIA, MARKKU, additional, KAZAK, EKATERINA, additional, KEARNEY, FEARGHAL, additional, KERVEL, VINCENT VAN, additional, KHAN, SAAD A., additional, KHOMYN, MARTA K., additional, KLEIN, TONY, additional, KLEIN, OLGA, additional, KLOS, ALEXANDER, additional, KOETTER, MICHAEL, additional, KOLOKOLOV, ALEKSEY, additional, KORAJCZYK, ROBERT A., additional, KOZHAN, ROMAN, additional, KRAHNEN, JAN P., additional, KUHLE, PAUL, additional, KWAN, AMY, additional, LAJAUNIE, QUENTIN, additional, LAM, F. Y. ERIC C., additional, LAMBERT, MARIE, additional, LANGLOIS, HUGUES, additional, LAUSEN, JENS, additional, LAUTER, TOBIAS, additional, LEIPPOLD, MARKUS, additional, LEVIN, VLADIMIR, additional, LI, YIJIE, additional, LI, HUI, additional, LIEW, CHEE YOONG, additional, LINDNER, THOMAS, additional, LINTON, OLIVER, additional, LIU, JIACHENG, additional, LIU, ANQI, additional, LLORENTE, GUILLERMO, additional, LOF, MATTHIJS, additional, LOHR, ARIEL, additional, LONGSTAFF, FRANCIS, additional, LOPEZ‐LIRA, ALEJANDRO, additional, MANKAD, SHAWN, additional, MANO, NICOLA, additional, MARCHAL, ALEXIS, additional, MARTINEAU, CHARLES, additional, MAZZOLA, FRANCESCO, additional, MELOSO, DEBRAH, additional, MI, MICHAEL G., additional, MIHET, ROXANA, additional, MOHAN, VIJAY, additional, MOINAS, SOPHIE, additional, MOORE, DAVID, additional, MU, LIANGYI, additional, MURAVYEV, DMITRIY, additional, MURPHY, DERMOT, additional, NESZVEDA, GABOR, additional, NEUMEIER, CHRISTIAN, additional, NIELSSON, ULF, additional, NIMALENDRAN, MAHENDRARAJAH, additional, NOLTE, SVEN, additional, NORDEN, LARS L., additional, O'NEILL, PETER, additional, OBAID, KHALED, additional, ØDEGAARD, BERNT A., additional, ÖSTBERG, PER, additional, PAGNOTTA, EMILIANO, additional, PAINTER, MARCUS, additional, PALAN, STEFAN, additional, PALIT, IMON J., additional, PARK, ANDREAS, additional, PASCUAL, ROBERTO, additional, PASQUARIELLO, PAOLO, additional, PASTOR, LUBOS, additional, PATEL, VINAY, additional, PATTON, ANDREW J., additional, PEARSON, NEIL D., additional, PELIZZON, LORIANA, additional, PELLI, MICHELE, additional, PELSTER, MATTHIAS, additional, PÉRIGNON, CHRISTOPHE, additional, PFIFFER, CAMERON, additional, PHILIP, RICHARD, additional, PLÍHAL, TOMÁŠ, additional, PRAKASH, PUNEET, additional, PRESS, OLIVER‐ALEXANDER, additional, PRODROMOU, TINA, additional, PROKOPCZUK, MARCEL, additional, PUTNINS, TALIS, additional, QIAN, YA, additional, RAIZADA, GAURAV, additional, RAKOWSKI, DAVID, additional, RANALDO, ANGELO, additional, REGIS, LUCA, additional, REITZ, STEFAN, additional, RENAULT, THOMAS, additional, RENJIE, REX W., additional, RENO, ROBERTO, additional, RIDDIOUGH, STEVEN J., additional, RINNE, KALLE, additional, RINTAMÄKI, PAUL, additional, RIORDAN, RYAN, additional, RITTMANNSBERGER, THOMAS, additional, LONGARELA, IÑAKI RODRÍGUEZ, additional, ROESCH, DOMINIK, additional, ROGNONE, LAVINIA, additional, ROSEMAN, BRIAN, additional, ROŞU, IOANID, additional, ROY, SAURABH, additional, RUDOLF, NICOLAS, additional, RUSH, STEPHEN R., additional, RZAYEV, KHALADDIN, additional, RZEŹNIK, ALEKSANDRA A., additional, SANFORD, ANTHONY, additional, SANKARAN, HARIKUMAR, additional, SARKAR, ASANI, additional, SARNO, LUCIO, additional, SCAILLET, OLIVIER, additional, SCHARNOWSKI, STEFAN, additional, SCHENK‐HOPPÉ, KLAUS R., additional, SCHERTLER, ANDREA, additional, SCHNEIDER, MICHAEL, additional, SCHROEDER, FLORIAN, additional, SCHÜRHOFF, NORMAN, additional, SCHUSTER, PHILIPP, additional, SCHWARZ, MARCO A., additional, SEASHOLES, MARK S., additional, SEEGER, NORMAN J., additional, SHACHAR, OR, additional, SHKILKO, ANDRIY, additional, SHUI, JESSICA, additional, SIKIC, MARIO, additional, SIMION, GIORGIA, additional, SMALES, LEE A., additional, SÖDERLIND, PAUL, additional, SOJLI, ELVIRA, additional, SOKOLOV, KONSTANTIN, additional, SÖNKSEN, JANTJE, additional, SPOKEVICIUTE, LAIMA, additional, STEFANOVA, DENITSA, additional, SUBRAHMANYAM, MARTI G., additional, SZASZI, BARNABAS, additional, TALAVERA, OLEKSANDR, additional, TANG, YUEHUA, additional, TAYLOR, NICK, additional, THAM, WING WAH, additional, THEISSEN, ERIK, additional, THIMME, JULIAN, additional, TONKS, IAN, additional, TRAN, HAI, additional, TRAPIN, LUCA, additional, TROLLE, ANDERS B., additional, VADUVA, M. ANDREEA, additional, VALENTE, GIORGIO, additional, NESS, ROBERT A. VAN, additional, VASQUEZ, AURELIO, additional, VEROUSIS, THANOS, additional, VERWIJMEREN, PATRICK, additional, VILHELMSSON, ANDERS, additional, VILKOV, GRIGORY, additional, VLADIMIROV, VLADIMIR, additional, VOGEL, SEBASTIAN, additional, VOIGT, STEFAN, additional, WAGNER, WOLF, additional, WALTHER, THOMAS, additional, WEISS, PATRICK, additional, WEL, MICHEL VAN DER, additional, WERNER, INGRID M., additional, WESTERHOLM, P. JOAKIM, additional, WESTHEIDE, CHRISTIAN, additional, WIKA, HANS C., additional, WIPPLINGER, EVERT, additional, WOLF, MICHAEL, additional, WOLFF, CHRISTIAN C. P., additional, WOLK, LEONARD, additional, WONG, WING‐KEUNG, additional, WRAMPELMEYER, JAN, additional, WU, ZHEN‐XING, additional, XIA, SHUO, additional, XIU, DACHENG, additional, XU, KE, additional, XU, CAIHONG, additional, YADAV, PRADEEP K., additional, YAGÜE, JOSÉ, additional, YAN, CHENG, additional, YANG, ANTTI, additional, YOO, WOONGSUN, additional, YU, WENJIA, additional, YU, YIHE, additional, YU, SHIHAO, additional, YUESHEN, BART Z., additional, YUFEROVA, DARYA, additional, ZAMOJSKI, MARCIN, additional, ZAREEI, ABALFAZL, additional, ZEISBERGER, STEFAN M., additional, ZHANG, LU, additional, ZHANG, S. SARAH, additional, ZHANG, XIAOYU, additional, ZHAO, LU, additional, ZHONG, ZHUO, additional, ZHOU, Z. IVY, additional, ZHOU, CHEN, additional, ZHU, XINGYU S., additional, ZOICAN, MARIUS, additional, and ZWINKELS, REMCO, additional

Published
2024
Full Text
View/download PDF

16. Heterogeneity in effect size estimates.

Author

Holzmeister, Felix, Johannesson, Magnus, Böhm, Robert, Dreber, Anna, Huber, Jürgen, and Kirchler, Michael

Subjects
PATH analysis (Statistics), ERROR rates, HETEROGENEITY, EXPERIMENTAL design, EMPIRICAL research
Abstract

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

17. A framework for evaluating reproducibility and replicability in economics.

Author

Dreber, Anna and Johannesson, Magnus

Subjects
*MEASUREMENT
Abstract

We propose a framework for evaluating reproducibility and replicability in economics. Reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We further divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability. In addition to this typology we propose indicators to measure the degree of reproducibility and replicability in both individual studies and for a group of studies. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

18. Selective reporting of placebo tests in top economics journals.

Author

Dreber, Anna, Johannesson, Magnus, and Yang, Yifan

Subjects
*PLACEBOS, *NULL hypothesis
Abstract

Placebo tests provide incentives to underreport statistically significant tests, a form of reversed p‐hacking. We test for such underreporting in 11 top economics journals between 2009 and 2021 based on a pre‐registered analysis plan. If the null hypothesis is true in all tests, 2.5% of them should be significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

19. Redefine statistical significance.

Author

Benjamin, Daniel J, Berger, James O, Johannesson, Magnus, Nosek, Brian A, Wagenmakers, E-J, Berk, Richard, Bollen, Kenneth A, Brembs, Björn, Brown, Lawrence, Camerer, Colin, Cesarini, David, Chambers, Christopher D, Clyde, Merlise, Cook, Thomas D, De Boeck, Paul, Dienes, Zoltan, Dreber, Anna, Easwaran, Kenny, Efferson, Charles, Fehr, Ernst, Fidler, Fiona, Field, Andy P, Forster, Malcolm, George, Edward I, Gonzalez, Richard, Goodman, Steven, Green, Edwin, Green, Donald P, Greenwald, Anthony G, Hadfield, Jarrod D, Hedges, Larry V, Held, Leonhard, Hua Ho, Teck, Hoijtink, Herbert, Hruschka, Daniel J, Imai, Kosuke, Imbens, Guido, Ioannidis, John PA, Jeon, Minjeong, Jones, James Holland, Kirchler, Michael, Laibson, David, List, John, Little, Roderick, Lupia, Arthur, Machery, Edouard, Maxwell, Scott E, McCarthy, Michael, Moore, Don A, Morgan, Stephen L, Munafó, Marcus, Nakagawa, Shinichi, Nyhan, Brendan, Parker, Timothy H, Pericchi, Luis, Perugini, Marco, Rouder, Jeff, Rousseau, Judith, Savalei, Victoria, Schönbrodt, Felix D, Sellke, Thomas, Sinclair, Betsy, Tingley, Dustin, Van Zandt, Trisha, Vazire, Simine, Watts, Duncan J, Winship, Christopher, Wolpert, Robert L, Xie, Yu, Young, Cristobal, Zinman, Jonathan, and Johnson, Valen E

Subjects
Data Interpretation, Statistical, Models, Statistical, Reproducibility of Results, Scholarly Communication
Abstract

We propose to change the default P-value threshold for statistical significance for claims of new discoveries from 0.05 to 0.005.

Published
2018

20. #EEGManyLabs: Investigating the replicability of influential EEG experiments

Author

Pavlov, Yuri G., Adamian, Nika, Appelhoff, Stefan, Arvaneh, Mahnaz, Benwell, Christopher S.Y., Beste, Christian, Bland, Amy R., Bradford, Daniel E., Bublatzky, Florian, Busch, Niko A., Clayson, Peter E., Cruse, Damian, Czeszumski, Artur, Dreber, Anna, Dumas, Guillaume, Ehinger, Benedikt, Ganis, Giorgio, He, Xun, Hinojosa, José A., Huber-Huber, Christoph, Inzlicht, Michael, Jack, Bradley N., Johannesson, Magnus, Jones, Rhiannon, Kalenkovich, Evgenii, Kaltwasser, Laura, Karimi-Rouzbahani, Hamid, Keil, Andreas, König, Peter, Kouara, Layla, Kulke, Louisa, Ladouceur, Cecile D., Langer, Nicolas, Liesefeld, Heinrich R., Luque, David, MacNamara, Annmarie, Mudrik, Liad, Muthuraman, Muthuraman, Neal, Lauren B., Nilsonne, Gustav, Niso, Guiomar, Ocklenburg, Sebastian, Oostenveld, Robert, Pernet, Cyril R., Pourtois, Gilles, Ruzzoli, Manuela, Sass, Sarah M., Schaefer, Alexandre, Senderecka, Magdalena, Snyder, Joel S., Tamnes, Christian K., Tognoli, Emmanuelle, van Vugt, Marieke K., Verona, Edelyn, Vloeberghs, Robin, Welke, Dominik, Wessel, Jan R., Zakharov, Ilya, and Mushtaq, Faisal

Published
2021
Full Text
View/download PDF

21. Subjective Evidence Evaluation Survey For Multi-Analyst Studies

Author

Sarafoglou, Alexandra, primary, Hoogeveen, Suzanne, additional, van den Bergh, Don, additional, Aczel, Balazs, additional, Albers, Casper J, additional, Althoff, Tim, additional, Botvinik-Nezer, Rotem, additional, Busch, Niko, additional, Cataldo, Andrea Michael, additional, Devezer, Berna, additional, van Dongen, Noah N'Djaye Nikolai, additional, Dreber, Anna, additional, Fried, Eiko I, additional, Hoekstra, Rink, additional, Hoffmann, Sabine, additional, Holzmeister, Felix, additional, Huber, Juergen, additional, Huntington-Klein, Nick, additional, Ioannidis, John P.A., additional, Johannesson, Magnus, additional, Kirchler, Michael, additional, Loken, Eric, additional, Mangin, Jan-Francois, additional, Matzke, Dora, additional, Menkveld, Albert J., additional, Nilsonne, Gustav, additional, van Ravenzwaaij, Don, additional, Schweinsberg, Martin, additional, Schulz-Kümpel, Hannah, additional, Shanks, David, additional, Simons, Daniel J., additional, Spellman, Barbara A., additional, Stoevenbelt, Andrea Helena, additional, Szaszi, Barnabas, additional, Trübutschek, Darinka, additional, tuerlinckx, francis, additional, Uhlmann, Eric Luis, additional, vanpaemel, wolf, additional, Wicherts, Jelte M., additional, and Wagenmakers, Eric-Jan, additional

Published
2024
Full Text
View/download PDF

26. A creative destruction approach to replication: Implicit work and sex morality across cultures

Author

Tierney, Warren, Hardy, Jay, III, Ebersole, Charles R., Viganola, Domenico, Clemente, Elena Giulia, Gordon, Michael, Hoogeveen, Suzanne, Haaf, Julia, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, Huang, Jason L., Vaughn, Leigh Ann, DeMarree, Kenneth, Igou, Eric R., Chapman, Hanah, Gantman, Ana, Vanaman, Matthew, Wylie, Jordan, Storbeck, Justin, Andreychik, Michael R., McPhetres, Jon, and Uhlmann, Eric Luis

Published
2021
Full Text
View/download PDF

27. Creative destruction in science

Author

Tierney, Warren, Hardy, Jay H., III, Ebersole, Charles R., Leavitt, Keith, Viganola, Domenico, Clemente, Elena Giulia, Gordon, Michael, Dreber, Anna, Johannesson, Magnus, Pfeiffer, Thomas, and Uhlmann, Eric Luis

Published
2020
Full Text
View/download PDF

28. On the trajectory of discrimination: A meta-analysis and forecasting survey capturing 44 years of field experiments on gender and hiring decisions

Author

Schaerer, Michael, primary, du Plessis, Christilene, additional, Nguyen, My Hoang Bao, additional, van Aert, Robbie C.M., additional, Tiokhin, Leo, additional, Lakens, Daniël, additional, Giulia Clemente, Elena, additional, Pfeiffer, Thomas, additional, Dreber, Anna, additional, Johannesson, Magnus, additional, Clark, Cory J., additional, and Luis Uhlmann, Eric, additional

Published
2023
Full Text
View/download PDF

36. Competition and moral behavior:A meta-analysis of forty-five crowd-sourced experimental designs

Author

Huber, Christoph, Dreber, Anna, Huber, Jürgen, Johannesson, Magnus, Kirchler, Michael, Weitzel, Utz, Abellán, Miguel, Adayeva, Xeniya, Ay, Fehime Ceren, Barron, Kai, Berry, Zachariah, Bönte, Werner, Brütt, Katharina, Bulutay, Muhammed, Campos-Mercade, Pol, Cardella, Eric, Claassen, Maria Almudena, Cornelissen, Gert, Dawson, Ian G. J., Delnoij, Joyce, Demiral, Elif E., Dimant, Eugen, Doerflinger, Johannes Theodor, Dold, Malte, Emery, Cécile, Fiala, Lenka, Fiedler, Susann, Freddi, Eleonora, Fries, Tilman, Gasiorowska, Agata, Glogowsky, Ulrich, M. Gorny, Paul, Gretton, Jeremy David, Grohmann, Antonia, Hafenbrädl, Sebastian, Handgraaf, Michel, Hanoch, Yaniv, Hart, Einav, Hennig, Max, Hudja, Stanton, Hütter, Mandy, Hyndman, Kyle, Ioannidis, Konstantinos, Isler, Ozan, Jeworrek, Sabrina, Jolles, Daniel, Juanchich, Marie, KC, Raghabendra Pratap, Khadjavi, Menusch, Kugler, Tamar, Li, Shuwen, Lucas, Brian, Mak, Vincent, Mechtel, Mario, Merkle, Christoph, Meyers, Ethan Andrew, Mollerstrom, Johanna, Nesterov, Alexander, Neyse, Levent, Nieken, Petra, Nussberger, Anne-Marie, Palumbo, Helena, Peters, Kim, Pirrone, Angelo, Qin, Xiangdong, Rahal, Rima Maria, Rau, Holger, Rincke, Johannes, Ronzani, Piero, Roth, Yefim, Saral, Ali Seyhun, Schmitz, Jan, Schneider, Florian, Schram, Arthur, Schudy, Simeon, Schweitzer, Maurice E., Schwieren, Christiane, Scopelliti, Irene, Sirota, Miroslav, Sonnemans, Joep, Soraperra, Ivan, Spantig, Lisa, Steimanis, Ivo, Steinmetz, Janina, Suetens, Sigrid, Theodoropoulou, Andriana, Urbig, Diemo, Vorlaufer, Tobias, Waibel, Joschka, Woods, Daniel, Yakobi, Ofir, Yilmaz, Onurcan, Zaleskiewicz, Tomasz, Zeisberger, Stefan, Holzmeister, Felix, Finance, Ethics, Governance and Society, Tinbergen Institute, Spatial Economics, Amsterdam Sustainability Institute, School of Business and Economics, Huber, Christoph [0000-0001-5820-571X], Dreber, Anna [0000-0003-3989-9941], Huber, Jürgen [0000-0003-0073-0321], Johannesson, Magnus [0000-0001-8759-6393], Kirchler, Michael [0000-0002-5416-2545], Weitzel, Utz [0000-0003-0493-9333], Abellán, Miguel [0000-0002-0159-8505], Adayeva, Xeniya [0000-0003-3761-8763], Ay, Fehime Ceren [0000-0003-4444-2268], Barron, Kai [0000-0001-7259-4184], Berry, Zachariah [0000-0002-0827-6437], Brütt, Katharina [0000-0002-3003-0652], Claassen, Maria Almudena [0000-0001-7411-7330], Cornelissen, Gert [0000-0002-0300-4942], Dawson, Ian GJ [0000-0003-0555-9682], Delnoij, Joyce [0000-0002-0627-5318], Dimant, Eugen [0000-0002-5228-6966], Doerflinger, Johannes Theodor [0000-0002-0228-7725], Dold, Malte [0000-0002-1626-9878], Emery, Cécile [0000-0002-7272-1144], Fiala, Lenka [0000-0003-0216-7050], Fiedler, Susann [0000-0001-9337-2142], Freddi, Eleonora [0000-0002-1794-5188], Gasiorowska, Agata [0000-0002-3354-1095], Glogowsky, Ulrich [0000-0002-0781-1119], M Gorny, Paul [0000-0003-3737-2695], Gretton, Jeremy David [0000-0002-2417-4035], Hafenbrädl, Sebastian [0000-0002-5148-766X], Handgraaf, Michel [0000-0003-0809-5393], Hanoch, Yaniv [0000-0001-9453-4588], Hart, Einav [0000-0003-0641-2359], Hütter, Mandy [0000-0002-0952-3831], Hyndman, Kyle [0000-0003-3666-8734], Ioannidis, Konstantinos [0000-0003-2858-4688], Isler, Ozan [0000-0002-4638-2230], Jeworrek, Sabrina [0000-0001-7967-9383], Jolles, Daniel [0000-0003-1277-0793], Juanchich, Marie [0000-0003-0241-9529], Kc, Raghabendra Pratap [0000-0001-5627-5570], Khadjavi, Menusch [0000-0002-8152-3369], Kugler, Tamar [0000-0002-7211-9858], Li, Shuwen [0000-0002-8822-4185], Lucas, Brian [0000-0003-3569-3030], Mak, Vincent [0000-0002-4690-0819], Merkle, Christoph [0000-0002-5141-4639], Meyers, Ethan Andrew [0000-0001-6171-6780], Mollerstrom, Johanna [0000-0002-8097-2058], Nesterov, Alexander [0000-0002-9143-2938], Neyse, Levent [0000-0001-6005-579X], Nieken, Petra [0000-0001-6157-6749], Nussberger, Anne-Marie [0000-0002-1805-9399], Palumbo, Helena [0000-0003-1978-3386], Pirrone, Angelo [0000-0001-5984-7853], Qin, Xiangdong [0000-0002-4471-8730], Rahal, Rima Maria [0000-0002-1404-0471], Ronzani, Piero [0000-0002-8211-6028], Roth, Yefim [0000-0003-2732-0496], Saral, Ali Seyhun [0000-0002-5523-5355], Schmitz, Jan [0000-0002-5410-9322], Schneider, Florian [0000-0002-8406-2756], Schram, Arthur [0000-0002-1767-4998], Schudy, Simeon [0000-0002-4409-0576], Schweitzer, Maurice E [0000-0003-4795-4816], Schwieren, Christiane [0000-0002-5266-3725], Scopelliti, Irene [0000-0001-6712-5332], Sirota, Miroslav [0000-0003-2117-9532], Sonnemans, Joep [0000-0001-7545-7409], Soraperra, Ivan [0000-0002-4596-3885], Spantig, Lisa [0000-0003-0776-3863], Steimanis, Ivo [0000-0002-8550-4675], Steinmetz, Janina [0000-0003-3299-4858], Suetens, Sigrid [0000-0001-6168-6059], Theodoropoulou, Andriana [0000-0003-3100-543X], Vorlaufer, Tobias [0000-0002-1586-5715], Yakobi, Ofir [0000-0002-9253-7483], Yilmaz, Onurcan [0000-0002-6094-7162], Zaleskiewicz, Tomasz [0000-0002-4232-6151], Zeisberger, Stefan [0000-0002-8412-622X], Holzmeister, Felix [0000-0001-9606-0427], and Apollo - University of Cambridge Repository

Subjects
metascience, experimental design, Postdoc Directie - ENR, BF Psychology, Economics, WASS, moral behavior, Urban Economics, H Social Sciences, ddc:330, ddc:500, Institute for Management Research, competition, generalizability
Abstract

Proceedings of the National Academy of Sciences of the United States of America 120(23), e2215572120 (2023). doi:10.1073/pnas.2215572120, Published by National Acad. of Sciences, Washington, DC

Published
2023

38. Competition and moral behavior: A meta-analysis of forty-five crowd-sourced experimental designs

Author

Huber, Christoph, primary, Dreber, Anna, additional, Huber, Jürgen, additional, Johannesson, Magnus, additional, Kirchler, Michael, additional, Weitzel, Utz, additional, Abellán, Miguel, additional, Adayeva, Xeniya, additional, Ay, Fehime Ceren, additional, Barron, Kai, additional, Berry, Zachariah, additional, Bönte, Werner, additional, Brütt, Katharina, additional, Bulutay, Muhammed, additional, Campos-Mercade, Pol, additional, Cardella, Eric, additional, Claassen, Maria Almudena, additional, Cornelissen, Gert, additional, Dawson, Ian G. J., additional, Delnoij, Joyce, additional, Demiral, Elif E., additional, Dimant, Eugen, additional, Doerflinger, Johannes Theodor, additional, Dold, Malte, additional, Emery, Cécile, additional, Fiala, Lenka, additional, Fiedler, Susann, additional, Freddi, Eleonora, additional, Fries, Tilman, additional, Gasiorowska, Agata, additional, Glogowsky, Ulrich, additional, M. Gorny, Paul, additional, Gretton, Jeremy David, additional, Grohmann, Antonia, additional, Hafenbrädl, Sebastian, additional, Handgraaf, Michel, additional, Hanoch, Yaniv, additional, Hart, Einav, additional, Hennig, Max, additional, Hudja, Stanton, additional, Hütter, Mandy, additional, Hyndman, Kyle, additional, Ioannidis, Konstantinos, additional, Isler, Ozan, additional, Jeworrek, Sabrina, additional, Jolles, Daniel, additional, Juanchich, Marie, additional, KC, Raghabendra Pratap, additional, Khadjavi, Menusch, additional, Kugler, Tamar, additional, Li, Shuwen, additional, Lucas, Brian, additional, Mak, Vincent, additional, Mechtel, Mario, additional, Merkle, Christoph, additional, Meyers, Ethan Andrew, additional, Mollerstrom, Johanna, additional, Nesterov, Alexander, additional, Neyse, Levent, additional, Nieken, Petra, additional, Nussberger, Anne-Marie, additional, Palumbo, Helena, additional, Peters, Kim, additional, Pirrone, Angelo, additional, Qin, Xiangdong, additional, Rahal, Rima Maria, additional, Rau, Holger, additional, Rincke, Johannes, additional, Ronzani, Piero, additional, Roth, Yefim, additional, Saral, Ali Seyhun, additional, Schmitz, Jan, additional, Schneider, Florian, additional, Schram, Arthur, additional, Schudy, Simeon, additional, Schweitzer, Maurice E., additional, Schwieren, Christiane, additional, Scopelliti, Irene, additional, Sirota, Miroslav, additional, Sonnemans, Joep, additional, Soraperra, Ivan, additional, Spantig, Lisa, additional, Steimanis, Ivo, additional, Steinmetz, Janina, additional, Suetens, Sigrid, additional, Theodoropoulou, Andriana, additional, Urbig, Diemo, additional, Vorlaufer, Tobias, additional, Waibel, Joschka, additional, Woods, Daniel, additional, Yakobi, Ofir, additional, Yilmaz, Onurcan, additional, Zaleskiewicz, Tomasz, additional, Zeisberger, Stefan, additional, and Holzmeister, Felix, additional

Published
2023
Full Text
View/download PDF

40. Non-Standard Errors

Author

Menkveld, Albert J., Dreber, Anna, Declerck, Fany, and Moinas, Sophie

Subjects
B- ECONOMIE ET FINANCE
Abstract

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in estimates of population parameters. In science, evidence is gener-ated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: Non-standard errors (NSEs). We study NSEs by letting 164 teams test the same hypotheses on the same data. NSEs turn out to be sizable, but smaller for better reproducible or higher rated research. Adding peer-review stages reduces NSEs. We further find that this type of uncertainty is underestimated by participants.

Published
2023

41. Competition and moral behavior: a meta-analysis of forty-five crowd-sourced experimental designs

Author

Huber, Christoph, Dreber, Anna, Huber, Jurgen, Dawson, Ian, and Hanoch, Yaniv

Abstract

Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity—variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity—estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs—indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis.

Published
2023

43. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

Author

Camerer, Colin F., Dreber, Anna, Holzmeister, Felix, Ho, Teck-Hua, Huber, Jürgen, Johannesson, Magnus, Kirchler, Michael, Nave, Gideon, Nosek, Brian A., Pfeiffer, Thomas, Altmejd, Adam, Buttrick, Nick, Chan, Taizan, Chen, Yiling, Forsell, Eskil, Gampa, Anup, Heikensten, Emma, Hummer, Lily, Imai, Taisuke, Isaksson, Siri, Manfredi, Dylan, Rose, Julia, Wagenmakers, Eric-Jan, and Wu, Hang

Published
2018
Full Text
View/download PDF

47. Evaluating replicability of laboratory experiments in economics

Author

Camerer, Colin F., Dreber, Anna, Forsell, Eskil, Ho, Teck-Hua, Huber, Jürgen, Johannesson, Magnus, Kirchler, Michael, Almenberg, Johan, Altmejd, Adam, Chan, Taizan, Heikensten, Emma, Holzmeister, Felix, Imai, Taisuke, Isaksson, Siri, Nave, Gideon, Pfeiffer, Thomas, Razen, Michael, and Wu, Hang

Published
2016

48. Selective Reporting of Placebo Tests in Top Economics Journals

Author

Dreber, Anna, Johannesson, Magnus, and Yang, Yifan

Subjects
ddc:330
Abstract

Placebo tests, where a null result is used to support the validity of the research design, is common in economics. Such tests provide an incentive to underreport statistically significant tests, a form of reversed p-hacking. Based on a pre-registered analysis plan, we test for such underreporting in all papers meeting our inclusion criteria (n=377) published in 11 top economics journals between 2009-2021. If the null hypothesis is true in all tests, 2.5% of them should be statistically significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). Our results provide strong evidence of selective underreporting of statistically significant placebo tests in top economics journals.

Published
2023

49. A framework for evaluating reproducibility and replicability in economics

Author

Dreber, Anna and Johannesson, Magnus

Subjects
ddc:330
Abstract

A fundamental question to the scientific enterprise is to what extent published scientific findings are credible. This question is related to the reproducibility and replicability of scientific findings where reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We provide a framework for evaluating reproducibility and replicability in economics and divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability, and we propose indicators to be reported for each type.

Published
2023

50. Non-Standard Errors

Author

Menkfeld, Albert J., Dreber, Anna, Holzmeister, Felix, Huber, Juergen, Johannesson, Magnus, Kirchler, Michael, Neusüss, Sebastian, Razen, Michael, Weitzel, Utz, Abad-Díaz, David, Abudy, Menachem, Adrian, Tobias, Ait-Sahalia, Yacine, Akmansoy, Olivier, Alcock, Jamie T., Alexeev, Vitali, Aloosh, Arash, Amato, Livia, Amaya, Diego, Angel, James J., Avetikian, Alejandro T., Bach, Amadeus, Baidoo, Edwin, Bakalli, Gaetan, Bao, Li, Barbon, Andrea, Bashchenko, Oksana, Bindra, Parampreet C., Bjønnes, Geir H., Black, Jeffrey R., Black, Bernard S., Bogoev, Dimitar, Bohorquez Correa, Santiago, Bondarenko, Oleg, Bos, Charles S., Bosch-Rosa, Ciril, Bouri, Elie, Brownlees, Christian, Calamia, Anna, Cao, Viet Nga, Capelle-Blancard, Gunther, Capera Romero, Laura M., Caporin, Massimiliano, Carrion, Allen, Caskurlu, Tolga, Chakrabarty, Bidisha, Chen, Jian, Chernov, Mikhail, Cheung, William, Chincarini, Ludwig B., Chordia, Tarun, Chow, Sheung-Chi, Clapham, Benjamin, Colliard, Jean-Edouard, Comerton-Forde, Carole, Curran, Edward, Dao, Thong, Dare, Wale, Davies, Ryan J., De Blasis, Riccardo, De Nard, Gianluca F., Declerck, Fany, Deev, Oleg, Degryse, Hans, Deku, Solomon Y., Desagre, Christophe, van Dijk, Mathijs A., Dim, Chukwuma, Dimpfl, Thomas, Dong, Yun Jiang, Drummond, Philip A., Dudda, Tom, Duevski, Teodor, Dumitrescu, Ariadna, Dyakov, Teodor, Dyhrberg, Anne Haubo, Dzielinski, Michał, Eksi, Asli, El Kalak, Izidin, ter Ellen, Saskia, Eugster, Nicolas, Evans, Martin D. D., Farrell, Michael, Felez-Vinas, Ester, Ferrara, Gerardo, Ferrouhi, El Mehdi, Flori, Andrea, Fluharty, Jonathan T., Foley, Sean D. V., Fong, Kingsley Y. L., Foucault, Thierry, Franus, Tatiana, Franzoni, Francesco, Frijns, Bart, Frömmel, Michael, Fu, Servanna M., Füllbrunn, Sascha C., Gan, Baoqing, Gao, Ge, Gehrig, Thomas P., Gemayel, Roland, Gerritsen, Dirk, Gil-Bazo, Javier, Gilder, Dudley, Glosten, Lawrence R., Gomez, Thomas, Gorbenko, Arseny, Grammig, Joachim, Grégoire, Vincent, Güçbilmez, Ufuk, Hagströmer, Björn, Hambuckers, Julien, Hapnes, Erik, Harris, Jeffrey H., Harris, Lawrence, Hartmann, Simon, Hasse, Jean-Baptiste, Hautsch, Nikolaus, He, Xue-Zhong (Tony), Heath, Davidson, Hediger, Simon, Hendershott, Terrence, Hibbert, Ann Marie, Hjalmarsson, Erik, Hoelscher, Seth, Hoffmann, Peter, Holden, Craig W., Horenstein, Alex R., Huang, Wenqian, Huang, Da, Hurlin, Christophe, Ilczuk, Konrad, Ivashchenko, Alexey, Iyer, Subramanian R., Jahanshahloo, Hossein, Jalkh, Naji P., Jones, Charles M., Jurkatis, Simon, Jylhä, Petri, Kaeck, Andreas T., Kaiser, Gabriel, Karam, Arzé, Karmaziene, Egle, Kassner, Bernhard, Kaustia, Markku, Kazak, Ekaterina, Kearney, Fearghal, Kervel, Vincent van, Khan, Saad A., Khomyn, Marta K., Klein, Tony, Klein, Olga, Klos, Alexander, Koetter, Michael, Kolokolov, Aleksey, Korajczyk, Robert A., Kozhan, Roman, Krahnen, Jan P., Kuhle, Paul, Kwan, Amy, Lajaunie, Quentin, Lam, F. Y. Eric C., Lambert, Marie, Langlois, Hugues, Lausen, Jens, Lauter, Tobias, Leippold, Markus, Levin, Vladimir, Li, Yijie, Li, Hui, Liew, Chee Yoong, Lindner, Thomas, Linton, Oliver, Liu, Jiacheng, Liu, Anqi, Llorente, Guillermo, Lof, Matthijs, Lohr, Ariel, Longstaff, Francis, Lopez-Lira, Alejandro, Mankad, Shawn, Mano, Nicola, Marchal, Alexis, Martineau, Charles, Mazzola, Francesco, Meloso, Debrah, Mi, Michael G., Mihet, Roxana, Mohan, Vijay, Moinas, Sophie, Moore, David, Mu, Liangyi, Muravyev, Dmitriy, Murphy, Dermot, Neszveda, Gabor, Neumeier, Christian, Nielsson, Ulf, Nimalendran, Mahendrarajah, Nolte, Sven, Norden, Lars L., O'Neill, Peter W., Obaid, Khaled, Ødegaard, Bernt A., Östberg, Per, Pagnotta, Emiliano, Painter, Marcus, Palan, Stefan, Palit, Imon J., Park, Andreas, Pascual, Roberto, Pasquariello, Paolo, Pastor, Lubos, Patel, Vinay, Patton, Andrew J., Pearson, Neil D., Pelizzon, Loriana, Pelli, Michele, Pelster, Matthias, Pérignon, Christophe, Pfiffer, Cameron, Philip, Richard, Plíhal, Tomáš, Prakash, Puneet, Press, Oliver-Alexander, Prodromou, Tina, Prokopczuk, Marcel, Putnins, Talis, Qian, Ya, Raizada, Gaurav, Rakowski, David, Ranaldo, Angelo, Regis, Luca, Reitz, Stefan, Renault, Thomas, Renjie, Rex W., Reno, Roberto, Riddiough, Steven J., Rinne, Kalle, Rintamäki, Paul J., Riordan, Ryan, Rittmannsberger, Thomas, Rodríguez Longarela, Iñaki, Roesch, Dominik, Rognone, Lavinia, Roseman, Brian, Rosu, Ioanid, Roy, Saurabh, Rudolf, Nicolas, Rush, Stephen R., Rzayev, Khaladdin, Rzeznik, Aleksandra A., Sanford, Anthony, Sankaran, Harikumar, Sarkar, Asani, Sarno, Lucio, Scaillet, Olivier, Scharnowski, Stefan, Schenk-Hoppé, Klaus R., Schertler, Andrea, Schneider, Michael, Schroeder, Florian, Schürhoff, Norman, Schuster, Philipp, Schwarz, Marco A., Seasholes, Mark S., Seeger, Norman J., Shachar, Or, Shkilko, Andriy, Shui, Jessica, Sikic, Mario, Simion, Giorgia, Smales, Lee A., Söderlind, Paul, Sojli, Elvira, Sokolov, Konstantin, Sönksen, Jantje, Spokeviciute, Laima, Stefanova, Denitsa, Subrahmanyam, Marti G., Szaszi, Barnabas, Talavera, Oleksandr, Tang, Yuehua, Taylor, Nick, Tham, Wing Wah, Theissen, Erik, Thimme, Julian, Tonks, Ian, Tran, Hai, Trapin, Luca, Trolle, Anders B., Vaduva, M. Andreea, Valente, Giorgio, Van Ness, Robert A., Vasquez, Aurelio, Verousis, Thanos, Verwijmeren, Patrick, Vilhelmsson, Anders, Vilkov, Grigory, Vladimirov, Vladimir, Vogel, Sebastian, Voigt, Stefan, Wagner, Wolf, Walther, Thomas, Weiss, Patrick, van der Wel, Michel, Werner, Ingrid M., Westerholm, Joakim, Westheide, Christian, Wika, Hans C., Wipplinger, Evert, Wolf, Michael, Wolff, Christian C. P., Wolk, Leonard, Wong, Wink-Keung, Wrampelmeyer, Jan, Wu, Zhen-Xing, Xia, Shuo, Xiu, Dacheng, Xu, Ke, Xu, Caihong, Yadav, Pradeep K., Yagüe, José, Yan, Cheng, Yang, Antti, Yoo, Woongsun, Yu, Wenjia, Yu, Yihe, Yu, Shihao, Yueshen, Bart Z., Yuferova, Darya, Zamojski, Marcin, Zareei, Abalfazl, Zeisberger, Stefan M., Zhang, Lu, Zhang, S. Sarah, Zhang, Xiaoyu, Zhao, Lu, Zhong, Zhuo, Zhou, Zeyang (Ivy), Zhou, Chen, Zhu, Xingyu S., Zoican, Marius, Zwinkels, Remco, Finance, and Tinbergen Institute

Subjects
jel:G14, jel:G1, ddc:330, jel:C12
Abstract

In statistics, samples are drawn from a population in a data- generating process (DGP). Standard errors measure the uncer- tainty in sample estimates of population parameters. In sci- ence, evidence is generated to test hypotheses in an evidence- generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sam- ple. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Published
2023
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results