Search

This crowdsourced project introduces a collaborative approach to improving the reproducibility of scientific research, in which findings are replicated in qualified independent laboratories before (rather than after) they are published. Our goal is to establish a non-adversarial replication process with highly informative final results. To illustrate the Pre-Publication Independent Replication (PPIR) approach, 25 research groups conducted replications of all ten moral judgment effects which the last author and his collaborators had “in the pipeline” as of August 2014. Six findings replicated according to all replication criteria, one finding replicated but with a significantly smaller effect size than the original, one finding replicated consistently in the original culture but not outside of it, and two findings failed to find support. In total, 40% of the original findings failed at least one major replication criterion. Potential ways to implement and incentivize pre-publication independent replication on a large scale are discussed.

We tested whether large language models (LLMs) can help predict results from a complex behavioural science experiment. In study 1, we investigated the performance of the widely used LLMs GPT-3.5 and GPT-4 in forecasting the empirical findings of a large-scale experimental study of emotions, gender, and social perceptions. We found that GPT-4, but not GPT-3.5, matched the performance of a cohort of 119 human experts, with correlations of 0.89 (GPT-4), 0.07 (GPT-3.5) and 0.87 (human experts) between aggregated forecasts and realized effect sizes. In study 2, providing participants from a university subject pool the opportunity to query a GPT-4 powered chatbot significantly increased the accuracy of their forecasts. Results indicate promise for artificial intelligence (AI) to help anticipate-at scale and minimal cost-which claims about human behaviour will find empirical support and which ones will not. Our discussion focuses on avenues for human-AI collaboration in science., Competing Interests: We declare we have no competing interests., (© 2024 The Author(s).)

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study., Competing Interests: We declare we have no competing interests., (© 2024 The Authors.)

Demonstrating the limitations of the one-at-a-time approach, crowd initiatives reveal the surprisingly powerful role of analytic and design choices in shaping scientific results. At the same time, cross-cultural variability in effects is far below the levels initially expected. This highlights the value of "medium" science, leveraging diverse stimulus sets and extensive robustness checks to achieve integrative tests of competing theories.

Contradicting our earlier claims of American moral exceptionalism, recent self-replication evidence from our laboratory indicates that implicit puritanism characterizes the judgments of people across cultures. Implicit cultural evolution may lag behind explicit change, such that differences between traditional and non-traditional cultures are greater at a deliberative than an intuitive level. Not too deep down, perhaps we are all implicit puritans.

Automatic preferences can influence a decision maker's choice before any relevant or meaningful information is available. We account for this element of human cognition in a computational model of problem solving that involves active trial and error and show that automatic biases are not just a beneficial or detrimental property: they are a tool that, if properly managed over time, can give rise to superior performance. In particular, automatic preferences are beneficial early on and detrimental at later stages. What is more, additional value can be generated by a timely rebiasing , i.e., a calculated reversal of the initial automatic preference. Remarkably, rebiasing can dominate not only debiasing (i.e., eliminating the bias) but also continuously unbiased decision making. This research contributes to the debate on the adaptiveness of automatic and intuitive biases, which has centered primarily on one-shot controlled laboratory experiments, by simulating outcomes across extended time spans. We also illustrate the value of the novel intervention of adopting the opposite automatic preference-something organizations can readily achieve by changing key decision makers-as opposed to attempting to correct for or simply accepting the ubiquity of such biases., Competing Interests: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest., (Copyright © 2022 Korniychuk and Uhlmann.)

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability-for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples.

By organizing crowds of scientists to independently tackle the same research questions, we can collectively overcome the generalizability crisis. Strategies to draw inferences from a heterogeneous set of research approaches include aggregation, for instance, meta-analyzing the effect sizes obtained by different investigators, and parsing, attempting to identify theoretically meaningful moderators that explain the variability in results.

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research., Competing Interests: BA, BS, GN, Ov, CA, Mv, JB, DB, UB, RB, LB, NB, EC, AC, NC, AD, Nv, CD, Jv, AD, GD, GE, MG, RH, SH, FH, JH, MJ, KJ, AK, MK, YK, DL, JM, DM, MM, BN, RP, Dv, JR, MS, AS, TS, MS, DS, RS, DS, BS, SS, JS, EU, JW, EW No competing interests declared, BN Executive Director of the Center for Open Science, a non-profit technology and culture change organization with a mission to increase openness, integrity, and reproducibility of research, (© 2021, Aczel et al.)

Science is often perceived to be a self-correcting enterprise. In principle, the assessment of scientific claims is supposed to proceed in a cumulative fashion, with the reigning theories of the day progressively approximating truth more accurately over time. In practice, however, cumulative self-correction tends to proceed less efficiently than one might naively suppose. Far from evaluating new evidence dispassionately and infallibly, individual scientists often cling stubbornly to prior findings. Here we explore the dynamics of scientific self-correction at an individual rather than collective level. In 13 written statements, researchers from diverse branches of psychology share why and how they have lost confidence in one of their own published findings. We qualitatively characterize these disclosures and explore their implications. A cross-disciplinary survey suggests that such loss-of-confidence sentiments are surprisingly common among members of the broader scientific population yet rarely become part of the public record. We argue that removing barriers to self-correction at the individual level is imperative if the scientific community as a whole is to achieve the ideal of efficient self-correction.

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Critical aspects of the "rationality of rationalizations" thesis are open empirical questions. These include the frequency with which past behavior determines attitudes (as opposed to attitudes causing future behaviors), the extent to which post hoc justifications take on a life of their own and shape future actions, and whether rationalizers experience benefits in well-being, social influence, performance, or other desirable outcomes.

Previous research has identified many positive outcomes resulting from a deeply held moral identity, while overlooking potential negative social consequences for the moral individual. Drawing from Benign Violation Theory, we explore the tension between moral identity and humor, and the downstream workplace consequence of such tension. Consistent with our hypotheses, compared with participants in the control condition, participants whose moral identities were situationally activated (Study 1a) or chronically accessible (Study 1b) were less likely to appreciate humor and generate jokes others found funny (Study 2), especially humor that involved benign moral violations. We also found that participants with a strong moral identity do not generally compensate for their lack of humor by telling more jokes that do not involve moral violations (Study 3). Additional field studies demonstrated that employees (Study 4) and leaders (Study 5) with strong moral identities and who display ethical leadership are perceived as less humorous by their coworkers and subordinates, and to the extent that this is the case are less liked in the workplace. Study 5 further demonstrated two competing mediating pathways-leaders with strong moral identities are perceived as less humorous but also as more trustworthy, with differentiated effects on interpersonal liking. Although having moral employees and leaders can come with many benefits, our research shows that there can be offsetting costs associated with an internalized moral identity: reduced humor and subsequent likability in the workplace. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Most scientific research is conducted by small teams of investigators who together formulate hypotheses, collect data, conduct analyses, and report novel findings. These teams operate independently as vertically integrated silos. Here we argue that scientific research that is horizontally distributed can provide substantial complementary value, aiming to maximize available resources, promote inclusiveness and transparency, and increase rigor and reliability. This alternative approach enables researchers to tackle ambitious projects that would not be possible under the standard model. Crowdsourced scientific initiatives vary in the degree of communication between project members from largely independent work curated by a coordination team to crowd collaboration on shared activities. The potential benefits and challenges of large-scale collaboration span the entire research process: ideation, study design, data collection, data analysis, reporting, and peer review. Complementing traditional small science with crowdsourced approaches can accelerate the progress of science and improve the quality of scientific research.

The present investigation modeled the emergence and persistence of intergroup bias and discrimination in artificial societies. Initial unfair prejudices held by members of a dominant group elicit confirmatory behavior (diminished cooperation) from members of a subordinate group via a self-fulfilling prophecy. Further, when individual learning is tempered by conformity to peers, inaccurate beliefs about the stigmatized subordinate group persist long-term. Even completely replacing dominant group members with enlightened individuals through generational change is inadequate to break the cycle of intergroup distrust and non-collaboration. The longer the enlightenment of a society is delayed, the more intergroup trust is irretrievably lost.

The widespread replication of research findings in independent laboratories prior to publication is suggested as a complement to traditional replication approaches. The pre-publication independent replication approach further addresses three key concerns from replication skeptics by systematically taking context into account, reducing reputational costs for original authors and replicators, and increasing the theoretical value of failed replications.

Both normative theories of ethics in philosophy and contemporary models of moral judgment in psychology have focused almost exclusively on the permissibility of acts, in particular whether acts should be judged on the basis of their material outcomes (consequentialist ethics) or on the basis of rules, duties, and obligations (deontological ethics). However, a longstanding third perspective on morality, virtue ethics, may offer a richer descriptive account of a wide range of lay moral judgments. Building on this ethical tradition, we offer a person-centered account of moral judgment, which focuses on individuals as the unit of analysis for moral evaluations rather than on acts. Because social perceivers are fundamentally motivated to acquire information about the moral character of others, features of an act that seem most informative of character often hold more weight than either the consequences of the act or whether a moral rule has been broken. This approach, we argue, can account for numerous empirical findings that are either not predicted by current theories of moral psychology or are simply categorized as biases or irrational quirks in the way individuals make moral judgments., (© The Author(s) 2014.)

Recent experimental evidence indicates that intuitions about inherence and system justification are distinct psychological processes, and that the inherence heuristic supplies important explanatory frameworks that are accepted or rejected based on their consistency with one's motivation to justify the system.

High-stakes team competitions can present a social dilemma in which participants must choose between concentrating on their personal performance and assisting teammates as a means of achieving group objectives. We find that despite the seemingly strong group incentive to win the NBA title, cooperative play actually diminishes during playoff games, negatively affecting team performance. Thus team cooperation decreases in the very high stakes contexts in which it is most important to perform well together. Highlighting the mixed incentives that underlie selfish play, personal scoring is rewarded with more lucrative future contracts, whereas assisting teammates to score is associated with reduced pay due to lost opportunities for personal scoring. A combination of misaligned incentives and psychological biases in performance evaluation bring out the "I" in "team" when cooperation is most critical.

Newell & Shanks (N&S) argue that when awareness measures are more reliable and valid, greater evidence of awareness of supposedly unconscious influences is revealed. A related issue is that unconsciousness is typically the null hypothesis that evidence of awareness will not emerge. As it is difficult to conclude the null, it is also difficult to conclude a lack of conscious awareness.

We apply Bentley et al.'s theoretical framework to better understand gender discrimination in online labor markets. Although such settings are designed to encourage employer behavior in the northwest corner of Homo economicus, actual online hiring decisions tend to drift southeast into a "confirmation bias plus weak feedback loops" pattern of discrimination based on inaccurate social stereotypes.

People have a fundamental motive to view their social system as just, fair, and good and will engage in a number of strategies to rationalize the status quo (Jost & Banaji, 1994). We propose that one way in which individuals may "justify the system" is through endorsement of essentialist explanations, which attribute group differences to deep, essential causes. We suggest that system-justifying motives lead to greater endorsement of essentialist explanations because those explanations portray group differences as immutable. Study 1 employed an established system threat manipulation. We found that activating system-justifying motives increases both male and female participants' endorsement of essentialist explanations for gender differences and that this effect is mediated by beliefs in immutability. In Study 2, we used a goal contagion manipulation and found that both male and female participants primed with a system-justifying goal are significantly more likely to agree with essentialist explanations for gender differences. Study 3 demonstrated that providing an opportunity to explicitly reject a system threat (an alternative means of satisfying the goal to defend the system) attenuates system threat effects on endorsement of essentialist explanations, further suggesting that this process is motivated. Finally, Studies 4a and 4b dissociated the type of cause (biological vs. social) from whether group differences are portrayed as mutable versus immutable and found that system threat increases endorsement of immutable explanations, independent of the type of cause., (PsycINFO Database Record (c) 2013 APA, all rights reserved)

In the field study reported here (N = 222,924), we found that Germans with noble-sounding surnames, such as Kaiser ("emperor"), König ("king"), and Fürst ("prince"), more frequently hold managerial positions than Germans with last names that either refer to common everyday occupations, such as Koch ("cook"), Bauer ("farmer"), and Becker/Bäcker ("baker"), or do not refer to any social role. This phenomenon occurs despite the fact that noble-sounding surnames never indicated that the person actually held a noble title. Because of basic properties of associative cognition, the status linked to a name may spill over to its bearer and influence his or her occupational outcomes.

Due to basic processes of psychological essentialism and contagion, one particular token of monetary currency is not always interchangeable with another piece of currency of equal economic value. When money loses its physical form it is perceived as "not quite the same" money (i.e., to have partly lost the original essence that distinguished it from other monetary tokens), diminishing its intuitive link with its original owner. Participants were less likely to recommend stolen or lost money be returned when it had been subsequently deposited in an electronic bank account, as opposed to retaining its original physical form (Studies 1a and 1b). Conversely, an intuitive sense of ownership is enhanced through physical contact with a piece of hard currency. Participants felt the piece of currency a person had originally lost should be returned to him rather than another piece of currency of equivalent value, even when they did not believe he would be able to tell the difference and considered distinguishing it from other money illogical. This effect was reduced when the currency had been sterilized, wiping it clean of all physical traces of its previous owner (Studies 2a, 2b, and 3)., (Copyright © 2013 Elsevier B.V. All rights reserved.)

Three studies demonstrate that morally praiseworthy behavior can signal negative information about an agent's character. In particular, consequentialist decisions such as sacrificing one life to save an even greater number of lives can lead to unfavorable character evaluations, even when they are viewed as the preferred course of action. In Study 1, throwing a dying man overboard to prevent a lifeboat from sinking was perceived as the morally correct course of action, but led to negative aspersions about the motivations and personal character of individuals who carried out such an act. In Studies 2 and 3, a hospital administrator who decided not to fund an expensive operation to save a child (instead buying needed hospital equipment) was seen as making a pragmatic and morally praiseworthy decision, but also as deficient in empathy and moral character., (Copyright © 2012 Elsevier B.V. All rights reserved.)

McCullough et al.’s functionalist model of revenge is highly compatible with the person-centered approach to moral judgment, which emphasizes the adaptive manner in which social perceivers derive character information from moral acts. Evidence includes act–person dissociations in which an act is seen as less immoral than a comparison act, yet as a clearer indicator of poor moral character.

Three empirical studies document the intuitive spillover of moral taint from a person who engages in immoral acts to another individual who is related by ties of blood kinship. In Study 1, participants were more likely to recommend that the biological grandchild of a wrongdoer, compared to a non-biological grandchild, help the descendants of his grandfather's victims. In Study 2, participants were more willing to hold two long-lost identical twins in custody for a crime committed by one twin than to hold two perfect look-alikes for a crime committed by one look-alike. Study 3 provides direct evidence that spillover effects based on blood kinship are manifested in an intuitive sense of moral taint., (Copyright © 2012 Elsevier B.V. All rights reserved.)

This review of 122 research reports (184 independent samples, 14,900 subjects) found average r = .274 for prediction of behavioral, judgment, and physiological measures by Implicit Association Test (IAT) measures. Parallel explicit (i.e., self-report) measures, available in 156 of these samples (13,068 subjects), also predicted effectively (average r = .361), but with much greater variability of effect size. Predictive validity of self-report was impaired for socially sensitive topics, for which impression management may distort self-report responses. For 32 samples with criterion measures involving Black-White interracial behavior, predictive validity of IAT measures significantly exceeded that of self-report measures. Both IAT and self-report measures displayed incremental validity, with each measure predicting criterion variance beyond that predicted by the other. The more highly IAT and self-report measures were intercorrelated, the greater was the predictive validity of each., ((PsycINFO Database Record (c) 2009 APA, all rights reserved).)

Three studies examined the relationships among anger, gender, and status conferral. As in prior research, men who expressed anger in a professional context were conferred higher status than men who expressed sadness. However, both male and female evaluators conferred lower status on angry female professionals than on angry male professionals. This was the case regardless of the actual occupational rank of the target, such that both a female trainee and a female CEO were given lower status if they expressed anger than if they did not. Whereas women's emotional reactions were attributed to internal characteristics (e.g., "she is an angry person,""she is out of control"), men's emotional reactions were attributed to external circumstances. Providing an external attribution for the target person's anger eliminated the gender bias. Theoretical implications and practical applications are discussed.