Search

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate-at scale and minimal cost-which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human-AI collaboration in science., Competing Interests: We declare we have no competing interests., (© 2024 The Author(s).)

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study., Competing Interests: We declare we have no competing interests., (© 2024 The Authors.)

Automatic preferences can influence a decision maker's choice before any relevant or meaningful information is available. We account for this element of human cognition in a computational model of problem solving that involves active trial and error and show that automatic biases are not just a beneficial or detrimental property: they are a tool that, if properly managed over time, can give rise to superior performance. In particular, automatic preferences are beneficial early on and detrimental at later stages. What is more, additional value can be generated by a timely rebiasing , i.e., a calculated reversal of the initial automatic preference. Remarkably, rebiasing can dominate not only debiasing (i.e., eliminating the bias) but also continuously unbiased decision making. This research contributes to the debate on the adaptiveness of automatic and intuitive biases, which has centered primarily on one-shot controlled laboratory experiments, by simulating outcomes across extended time spans. We also illustrate the value of the novel intervention of adopting the opposite automatic preference-something organizations can readily achieve by changing key decision makers-as opposed to attempting to correct for or simply accepting the ubiquity of such biases., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2022 Korniychuk and Uhlmann.)

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research., Competing Interests: BA, BS, GN, Ov, CA, Mv, JB, DB, UB, RB, LB, NB, EC, AC, NC, AD, Nv, CD, Jv, AD, GD, GE, MG, RH, SH, FH, JH, MJ, KJ, AK, MK, YK, DL, JM, DM, MM, BN, RP, Dv, JR, MS, AS, TS, MS, DS, RS, DS, BS, SS, JS, EU, JW, EW No competing interests declared, BN Executive Director of the Center for Open Science, a non-profit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research, (© 2021, Aczel et al.)

The present investigation modeled the emergence and persistence of intergroup bias and discrimination in artificial societies. Initial unfair prejudices held by members of a dominant group elicit confirmatory behavior (diminished cooperation) from members of a subordinate group via a self-fulfilling prophecy. Further, when individual learning is tempered by conformity to peers, inaccurate beliefs about the stigmatized subordinate group persist long-term. Even completely replacing dominant group members with enlightened individuals through generational change is inadequate to break the cycle of intergroup distrust and non-collaboration. The longer the enlightenment of a society is delayed, the more intergroup trust is irretrievably lost.

High-stakes team competitions can present a social dilemma in which participants must choose between concentrating on their personal performance and assisting teammates as a means of achieving group objectives. We find that despite the seemingly strong group incentive to win the NBA title, cooperative play actually diminishes during playoff games, negatively affecting team performance. Thus team cooperation decreases in the very high stakes contexts in which it is most important to perform well together. Highlighting the mixed incentives that underlie selfish play, personal scoring is rewarded with more lucrative future contracts, whereas assisting teammates to score is associated with reduced pay due to lost opportunities for personal scoring. A combination of misaligned incentives and psychological biases in performance evaluation bring out the "I" in "team" when cooperation is most critical.

Three studies examined the relationships among anger, gender, and status conferral. As in prior research, men who expressed anger in a professional context were conferred higher status than men who expressed sadness. However, both male and female evaluators conferred lower status on angry female professionals than on angry male professionals. This was the case regardless of the actual occupational rank of the target, such that both a female trainee and a female CEO were given lower status if they expressed anger than if they did not. Whereas women's emotional reactions were attributed to internal characteristics (e.g., "she is an angry person,""she is out of control"), men's emotional reactions were attributed to external circumstances. Providing an external attribution for the target person's anger eliminated the gender bias. Theoretical implications and practical applications are discussed.