Search

We propose a framework for evaluating reproducibility and replicability in economics. Reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We further divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability. In addition to this typology we propose indicators to measure the degree of reproducibility and replicability in both individual studies and for a group of studies. [ABSTRACT FROM AUTHOR]

Placebo tests provide incentives to underreport statistically significant tests, a form of reversed p‐hacking. We test for such underreporting in 11 top economics journals between 2009 and 2021 based on a pre‐registered analysis plan. If the null hypothesis is true in all tests, 2.5% of them should be significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). [ABSTRACT FROM AUTHOR]

Does competition affect moral behavior? This fundamental question has been debated among leading scholars for centuries, and more recently, it has been tested in experimental studies yielding a body of rather inconclusive empirical evidence. A potential source of ambivalent empirical results on the same hypothesis is design heterogeneity--variation in true effect sizes across various reasonable experimental research protocols. To provide further evidence on whether competition affects moral behavior and to examine whether the generalizability of a single experimental study is jeopardized by design heterogeneity, we invited independent research teams to contribute experimental designs to a crowd-sourced project. In a large-scale online data collection, 18,123 experimental participants were randomly allocated to 45 randomly selected experimental designs out of 95 submitted designs. We find a small adverse effect of competition on moral behavior in a meta-analysis of the pooled data. The crowd-sourced design of our study allows for a clean identification and estimation of the variation in effect sizes above and beyond what could be expected due to sampling variance. We find substantial design heterogeneity--estimated to be about 1.6 times as large as the average standard error of effect size estimates of the 45 research designs--indicating that the informativeness and generalizability of results based on a single experimental design are limited. Drawing strong conclusions about the underlying hypotheses in the presence of substantive design heterogeneity requires moving toward much larger data collections on various experimental designs testing the same hypothesis. [ABSTRACT FROM AUTHOR]

We measure how accurately replication of experimental results can be predicted by black-box statistical models. With data from four large-scale replication projects in experimental psychology and economics, and techniques from machine learning, we train predictive models and study which variables drive predictable replication. The models predicts binary replication with a cross-validated accuracy rate of 70% (AUC of 0.77) and estimates of relative effect sizes with a Spearman ρ of 0.38. The accuracy level is similar to market-aggregated beliefs of peer scientists [1, 2]. The predictive power is validated in a pre-registered out of sample test of the outcome of [3], where 71% (AUC of 0.73) of replications are predicted correctly and effect size correlations amount to ρ = 0.25. Basic features such as the sample and effect sizes in original papers, and whether reported effects are single-variable main effects or two-variable interactions, are predictive of successful replication. The models presented in this paper are simple tools to produce cheap, prognostic replicability metrics. These models could be useful in institutionalizing the process of evaluation of new findings and guiding resources to those direct replications that are likely to be most informative. [ABSTRACT FROM AUTHOR]

• We randomize 648 participants of both genders in the lab to advertising images. • The images contain either half-dressed women, fully dressed women, or no women. • We study the effects on risk taking, willingness to compete and math performance. • We find no treatment effects on any outcome measure for women. • There is some evidence of men taking more risk after viewing half-dressed women. Images of half-dressed women are ubiquitous in advertising and popular culture. Yet little is known about the potential impacts of such images on economic decision making. We randomize 648 participants of both genders to advertising images including either women in bikini or underwear, fully dressed women, or no women, and examine the effects on risk taking, willingness to compete and math performance in a lab experiment. We find no treatment effects on any outcome measure for women. For men, our results indicate that men take more risk after having been exposed to images of half-dressed women compared to no women. [ABSTRACT FROM AUTHOR]

We let participants indicate their intended action in a repeated game experiment where actions are implemented with errors. Even though communication is cheap talk, we find that the majority of messages were honest (although the majority of participants lied at least occasionally). As a result, communication has a positive effect on cooperation when the payoff matrix makes the returns to cooperation high; when the payoff matrix gives a lower return to cooperation, communication reduces overall cooperation. These results suggest that cheap talk communication can promote cooperation in repeated games, but only when there is already a self-interested motivation to cooperate. [ABSTRACT FROM AUTHOR]

Objective: To determine whether there is a causal effect of oral contraceptive (OC) treatment on general well-being and depressed mood in healthy women.Design: Double-blind, randomized, and placebo-controlled trial.Setting: University hospital.Patient(s): Three hundred and forty healthy women aged 18-35 years randomized to treatment, of whom 332 completed the data collection at follow-up evaluation.Intervention(s): A combined OC (150 μg levonorgestrel and 30 μg ethinylestradiol) or placebo for 3 months of treatment.Main Outcome Measure(s): Primary outcome measures: global score of Psychological General Well-Being Index (PGWBI) and the Beck Depression Inventory (BDI); secondary outcome measures: six separate dimensions of the PGWBI.Result(s): The OC treatment statistically significantly decreased general well-being compared with placebo -4.12 (95% CI, -7.18 to -1.06). Furthermore, OC decreased the following PGWBI dimensions compared with placebo: positive well-being -3.90 (95% CI, -7.78 to -0.01), self-control -6.63 (95% CI, -11.20 to -2.06), and vitality -6.84 (95% CI, -10.80 to -2.88). The effect of OC on depressive symptoms and on the PGWBI dimension depressed mood were not statistically significant.Conclusion(s): This study demonstrates a statistically significant reduction in general well-being by a first-choice OC in comparison with placebo in healthy women. We found no statistically significant effects on depressive symptoms. A reduction in general well-being should be of clinical importance. [ABSTRACT FROM AUTHOR]

The replicability of some scientific findings has recently been called into question.To contribute data about replicability in economics, we replicated 18 studies published in the American Economic Review and the Quarterly Journal of Economics between 2011 and 2014. All of these replications followed predefined analysis plans that were made publicly available beforehand, and they all have a statistical power of at least 90% to detect the original effect size at the 5% significance level.We found a significant effect in the same direction as in the original study for 11 replications (61%); on average, the replicated effect size is 66% of the original. The replicability rate varies between 67% and 78% for four additional replicability indicators, including a prediction market measure of peer beliefs. [ABSTRACT FROM AUTHOR]

Concerns about a lack of reproducibility of statistically significant results have recently been raised in many fields, and it has been argued that this lack comes at substantial economic costs. We here report the results from prediction markets set up to quantify the reproducibility of 44 studies published in prominent psychology journals and replicated in the Reproducibility Project: Psychology. The prediction markets predict the outcomes of the replications well and outperform a survey of market participants' individual forecasts. This shows that prediction markets are a promising tool for assessing the reproducibility of published scientific results. The prediction markets also allow us to estimate probabilities for the hypotheses being true at different testing stages, which provides valuable information regarding the temporal dynamics of scientific discovery. We find that the hypotheses being tested in psychology typically have low prior probabilities of being true (median, 9%) and that a "statistically significant" finding needs to be confirmed in a well-powered replication to have a high probability of being true. We argue that prediction markets could be used to obtain speedy information about reproducibility at low cost and could potentially even be used to determine which studies to replicate to optimally allocate limited resources into replications. [ABSTRACT FROM AUTHOR]

We look at gender differences in competitiveness, risk preferences and altruism in a large sample of children and adolescents aged 7–16 in Armenia. Post-Soviet Armenia has few formal barriers to gender equality but is also characterized by a patrilineal kinship system and traditional gender roles. In contrast to research conducted in Western countries, we find that girls increase their performance more than boys in response to competition in a running task. We find no gender differences in the other three tasks we explore: skipping rope, a mathematical task, and a verbal task. We also find no difference in the willingness to compete in either the mathematical or the verbal task. In line with previous research, we find that boys are less altruistic and more risk taking than girls, and that the latter gap appears around the age of puberty. [ABSTRACT FROM AUTHOR]

The authors comment on a study by D. R. Carney and colleagues on the robustness of power posing and its effect on hormones such as testosterone and cortisol and risk tolerance in a large sample of men and women. Topics discussed include the assessment of behavioral tasks such as risk taking in the loss domain and willingness to compete in the study and the study's findings which suggest a significant effect of power posing on self-reported feelings of power but has no effect on hormone levels.

In this article we compare cooperation among Colombian and Swedish children aged 9–12. We illustrate the dynamics of the prisoner's dilemma in a new task that is easily understood by children and performed during a physical education class. We find no robust evidence of a difference in cooperation between Colombia and Sweden overall. However, Colombian girls cooperate less than Swedish girls. We also find indications that girls in Colombia are less cooperative than boys. Finally, there is also a tendency for children to be more cooperative with boys than with girls on average. [ABSTRACT FROM AUTHOR]

Highlights: [•] We explore the role of altruism and inequity aversion in supporting cooperation in repeated games. [•] We include various proxies of social preferences, including dictator game giving. [•] Cooperation in repeated games appears to be mainly motivated by payoff maximization. [ABSTRACT FROM AUTHOR]

Summary: While baseline testosterone has recently been implicated in risk-taking in men, less is known about the effects of changing levels of testosterone on financial risk. Here we attempt to influence testosterone in men by having them win or lose money in a chance-based competition against another male opponent. We employ two treatments where we vary the amount of money at stake so that we can directly compare winners to losers who earn the same amount, thereby abstracting from income effects. We find that men who experience a greater increase in bioactive testosterone take on more risk, an association that remains when controlling for whether the participant won the competition. In fact, whether subjects won the competition did not predict future risk. These results suggest that testosterone change, and thus individual differences in testosterone reactivity, rather than the act of winning or losing, influence financial risk-taking. [Copyright &y& Elsevier]

Highlights: [•] We experimentally explore cooperation in a common pool resource game. [•] We test how cooperation varies with the level at which votes are aggregated. [•] Individual extractions and voting in subgroups lead to resource overextraction. [•] Voting on globally binding rules achieves the social optimum. [Copyright &y& Elsevier]

Highlights: [•] We use a large international panel dataset on chess competitions. [•] Photos of 626 expert chess players are rated for attractiveness in a large survey. [•] Male players choose riskier strategies when playing against attractive female opponents. [•] Playing riskier against attractive women does not improve performance. [•] Women's strategies are not affected by the attractiveness of the opponent. [Copyright &y& Elsevier]

Abstract: We explore gender differences in preferences for competition and risk among children aged 9–12 in Colombia and Sweden, two countries differing in gender equality according to macro indices. We include four types of tasks that vary in gender stereotyping when looking at competitiveness: running, skipping rope, math and word search. We find that boys and girls are equally competitive in all tasks and all measures in Colombia. Unlike the consistent results in Colombia, the results in Sweden are mixed, with some indication of girls being more competitive than boys in some tasks in terms of performance change, whereas boys are more likely to choose to compete in general. Boys in both countries are more risk taking than girls, with a smaller gender gap in Sweden. [Copyright &y& Elsevier]

Abstract: To test the relationship between androgen exposure, dopaminergic reward and sensation-seeking, we compared variation in salivary testosterone (T), 2D:4D digit ratio, facial masculinity, Zuckerman''s sensation-seeking scale (SSS) and the D4 dopamine receptor (DRD4) genes from 98 young men, between the ages of 18 and 23years. In univariate analyses, both salivary T and facial masculinity were significantly correlated with the SSS boredom susceptibility subscale, while the presence of the 7-repeat allele (7R+) in the dopamine receptor D4 gene was associated with the SSS thrill and adventure-seeking and overall sensation-seeking. Neither left nor right 2D:4D digit ratio was associated with any sensation-seeking scale. In multivariate models, salivary T and facial masculinity were significant predictors of SSS boredom susceptibility, while 7R+ was a significant predictor of SSS thrill and adventure-seeking. For overall SSS, both 7R+ and salivary T were significant predictors. There was no significant interaction of 7R+ and androgen exposure for SSS or any of the SSS subscales. These results add to earlier reports of an association between T and sensation-seeking. In addition, our results also indicate that genetic variation in DRD4 is independently associated with SSS sensation-seeking. [Copyright &y& Elsevier]

The public goods game is the classic laboratory paradigm for studying collective action problems. Each participant chooses how much to contribute to a common pool that returns benefits to all participants equally. The ideal outcome occurs if everybody contributes the maximum amount, but the self-interested strategy is not to contribute anything. Most previous studies have found punishment to be more effective than reward for maintaining cooperation in public goods games. The typical design of these studies, however, represses future consequences for today's actions. In an experimental setting, we compare public goods games followed by punishment, reward, or both in the setting of truly repeated games, in which player identities persist from round to round. We show that reward is as effective as punishment for maintaining public cooperation and leads to higher total earnings. Moreover, when both options are available, reward leads to increased contributions and payoff, whereas punishment has no effect on contributions and leads to lower payoff. We conclude that reward outperforms punishment in repeated public goods games and that human cooperation in such repeated settings is best supported by positive interactions with others. [ABSTRACT FROM AUTHOR]

How do people respond to others' accidental behaviors? Reward and punishment for an accident might depend on the actor's intentions, or instead on the unintended outcomes she brings about. Yet, existing paradigms in experimental economics do not include the possibility of accidental monetary allocations. We explore the balance of outcomes and intentions in a two-player economic game where monetary allocations are made with a ''trembling hand'': that is, intentions and outcomes are sometimes mismatched. Player 1 allocates $10 between herself and Player 2 by rolling one of three dice. One die has a high probability of a selfish outcome, another has a high probability of a fair outcome, and the third has a high probability of a generous outcome. Based on Player 1's choice of die, Player 2 can infer her intentions. However, any of the three die can yield any of the three possible outcomes. Player 2 is given the opportunity to respond to Player 1's allocation by adding to or subtracting from Player 1's payoff. We find that Player 2's responses are influenced substantially by the accidental outcome of Player 1's roll of the die. Comparison to control conditions suggests that in contexts where the allocation is at least partially under the control of Player 1, Player 2 will punish Player 1 accountable for unintentional negative outcomes. In addition, Player 2's responses are influenced by Player 1's intention. However, Player 2 tends to modulate his responses substantially more for selfish intentions than for generous intentions. This novel economic game provides new insight into the psychological mechanisms underlying social preferences for fairness and retribution. [ABSTRACT FROM AUTHOR]

Indicators of past wealth –‘old money’– typically bestow prestige and contribute to high status. Nobility, a culturally determined, hereditary status marker, might act as such an indicator, and thereby serve as a vehicle for the cultural transmission of economic standing. As an institution, nobility is an anachronism. Status, however, plays an important role in most societies, making individuals allocate valuable resources to status-enhancing activities. This suggests that even though nobility no longer entails formal privileges, it may continue to be coveted as a status marker. We examine the relative performance of nobility in the marriage market. Data on Swedish marriages provide an opportunity to test the hypothesis that the probability of hypergamy (marrying ‘up’) in terms of wealth increases when an individual belongs to the nobility. Our main finding is a significantly higher probability of hypergamy for members of the nobility, controlling for own wealth and other covariates. This ‘nobility premium’ is sizeable. The effect is statistically significant and robust to a number of different measures of hypergamy. This finding has implications for the intergenerational transmission of inequality, and for the longevity of the institution of nobility itself. [ABSTRACT FROM AUTHOR]

A key aspect of human behaviour is cooperation. We tend to help others even if costs are involved. We are more likely to help when the costs are small and the benefits for the other person significant. Cooperation leads to a tension between what is best for the individual and what is best for the group. A group does better if everyone cooperates, but each individual is tempted to defect. Recently there has been much interest in exploring the effect of costly punishment on human cooperation. Costly punishment means paying a cost for another individual to incur a cost. It has been suggested that costly punishment promotes cooperation even in non-repeated games and without any possibility of reputation effects. But most of our interactions are repeated and reputation is always at stake. Thus, if costly punishment is important in promoting cooperation, it must do so in a repeated setting. We have performed experiments in which, in each round of a repeated game, people choose between cooperation, defection and costly punishment. In control experiments, people could only cooperate or defect. Here we show that the option of costly punishment increases the amount of cooperation but not the average payoff of the group. Furthermore, there is a strong negative correlation between total payoff and use of costly punishment. Those people who gain the highest total payoff tend not to use costly punishment: winners don’t punish. This suggests that costly punishment behaviour is maladaptive in cooperation games and might have evolved for other reasons. [ABSTRACT FROM AUTHOR]

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies. [ABSTRACT FROM AUTHOR]

The digit ratio (2D:4D) is considered a proxy for testosterone exposure in utero, and there has been a recent surge of studies testing whether 2D:4D is associated with economic preferences. Although the results are not conclusive, previous studies have reported statistically significant correlations between 2D:4D and risk taking, altruism, positive reciprocity, negative reciprocity and trust. However, most previous studies have small sample sizes gathered from university students and there is also no consensus on the type of analysis (e.g., which hand to analyze or subgroup to focus on) or the preference elicitation method. We present results from a pre-registered large sample study testing if 2D:4D is associated with economic preferences. Data were collected in a representative sample of adults in the German Socioeconomic Panel-Innovation Sample (SOEP-IS), in a sample of about 3450 respondents (about 5 times larger than the previously largest study in this field). Using experimentally validated survey questions, we find no statistically significant association between 2D:4D and economic preferences in the largest study to this date on the topic. [ABSTRACT FROM AUTHOR]

The reproducibility of published research has become an important topic in science policy. A number of large-scale replication projects have been conducted to gauge the overall reproducibility in specific academic fields. Here, we present an analysis of data from four studies which sought to forecast the outcomes of replication projects in the social and behavioural sciences, using human experts who participated in prediction markets and answered surveys. Because the number of findings replicated and predicted in each individual study was small, pooling the data offers an opportunity to evaluate hypotheses regarding the performance of prediction markets and surveys at a higher power. In total, peer beliefs were elicited for the replication outcomes of 103 published findings. We find there is information within the scientific community about the replicability of scientific findings, and that both surveys and prediction markets can be used to elicit and aggregate this information. Our results show prediction markets can determine the outcomes of direct replications with 73% accuracy (n = 103). Both the prediction market prices, and the average survey responses are correlated with outcomes (0.581 and 0.564 respectively, both p <.001). We also found a significant relationship between p-values of the original findings and replication outcomes. The dataset is made available through the R package "pooledmaRket" and can be used to further study community beliefs towards replications outcomes as elicited in the surveys and prediction markets. [ABSTRACT FROM AUTHOR]

A response by David G. Rand and colleagues to their article "Positive Interactions Promote Public Cooperation" in the September 4, 2009 issue is presented.

The article comments on a game developed by researcher Milinski Marotzke and colleagues to investigate the willingness of people to gamble in order to save the climate. The researchers found that people must be well informed about the risk of climate change to gain their cooperation. People should learn how to cooperate and avoid a wasteful lifestyle.

Bosch-Domènech et al. (2014) reported a negative association between 2D:4D, a suggested marker of prenatal testosterone exposure, and the Cognitive Reflection Test (CRT) in a sample of 623 university students. In this pre-registered study, we test if we can replicate their findings in a general population sample of over 2,500 individuals from Germany. We find no statistically significant association between 2D:4D and the CRT in any of our primary hypothesis tests, or in any of our pre-registered exploratory analyses and robustness tests. The evidence is strong (based on the 99.5% confidence intervals in all three primary hypothesis tests) against effect sizes in the hypothesized direction larger than 0.075 CRT units (0.073 of the CRT standard deviation) for a one standard deviation change in 2D:4D. [ABSTRACT FROM AUTHOR]

Guala argues that there is a mismatch between most laboratory experiments on costly punishment and behavior in the field. In the lab, experimental designs typically suppress retaliation. The same is true for most theoretical models of the co-evolution of costly punishment and cooperation, which a priori exclude the possibility of defectors punishing cooperators. [ABSTRACT FROM PUBLISHER]

This initiative examined systematically the extent to which a large set of archival research findings generalizes across contexts. We repeated the key analyses for 29 original strategic management effects in the same context (direct reproduction) as well as in 52 novel time periods and geographies; 45% of the reproductions returned results matching the original reports together with 55% of tests in different spans of years and 40% of tests in novel geographies. Some original findings were associated with multiple new tests. Reproducibility was the best predictor of generalizability—for the findings that proved directly reproducible, 84% emerged in other available time periods and 57% emerged in other geographies. Overall, only limited empirical evidence emerged for context sensitivity. In a forecasting survey, independent scientists were able to anticipate which effects would find support in tests in new samples. [ABSTRACT FROM AUTHOR]

We examine cooperation in repeated interactions where intended actions are implemented with noise but intentions are perfectly observable. Observable intentions lead to more cooperation compared to control games where intentions are unobserved, allowing subjects to reach similar cooperation levels as in games without noise. Most subjects condition exclusively on intentions, and use simpler, lower-memory strategies compared to games where intentions are unobservable. When the returns to cooperation are high, some subjects are tolerant, using good outcomes to forgive attempted defections; when the returns to cooperation are low, some subjects are punitive, using bad outcomes to punish accidental defections. [ABSTRACT FROM AUTHOR]

Scientists and policymakers seek to choose effective interventions that promote preventative health measures. We evaluated whether academics, behavioral science practitioners, and laypeople (N = 1034) were able to forecast the effectiveness of seven different messages compared to a baseline message for Republicans and Democrats separately. These messages were designed to nudge mask-wearing attitudes, intentions, and behaviors. When examining predictions across political parties, forecasters predicted larger effects than those observed for Democrats compared to Republicans and made more accurate predictions for Republicans compared to Democrats. These results are partly driven by a lack of nudge effects on Democrats, as reported in Gelfand et al. (J Exp Soc Psychol, 2021). Academics and practitioners made more accurate predictions compared to laypeople. Although forecasters' predictions were correlated with the nudge interventions, all groups overestimated the observed results. We discuss potential reasons for why the forecasts did not perform better and how more accurate forecasts of behavioral intervention outcomes could potentially provide insight that can help save resources and increase the efficacy of interventions. [ABSTRACT FROM AUTHOR]

Replication—an important, uncommon, and misunderstood practice—is gaining appreciation in psychology. Achieving replicability is important for making research progress. If findings are not replicable, then prediction and theory development are stifled. If findings are replicable, then interrogation of their meaning and validity can advance knowledge. Assessing replicability can be productive for generating and testing hypotheses by actively confronting current understandings to identify weaknesses and spur innovation. For psychology, the 2010s might be characterized as a decade of active confrontation. Systematic and multi-site replication projects assessed current understandings and observed surprising failures to replicate many published findings. Replication efforts highlighted sociocultural challenges such as disincentives to conduct replications and a tendency to frame replication as a personal attack rather than a healthy scientific practice, and they raised awareness that replication contributes to self-correction. Nevertheless, innovation in doing and understanding replication and its cousins, reproducibility and robustness, has positioned psychology to improve research practices and accelerate progress. [ABSTRACT FROM AUTHOR]

People often favor members of their own group, while discriminating against members of other groups. Such in-group favoritism has been shown to play an important role in human cooperation. However, in the face of changing conflicts and shifting alliances, it is essential for group identities to be flexible. Using the dictator game from behavioral economics, we demonstrate the remodeling of group identities among supporters of Democratic presidential candidates Barack Obama and Hillary Clinton. After Clinton's concession in June 2008, Democrats were more generous toward supporters of their own preferred candidate than to supporters of the other Democratic candidate. The bias observed in June persisted into August, and disappeared only in early September after the Democratic National Convention. We also observe a strong gender effect, with bias both appearing and subsiding among men only. This experimental study illustrates a dynamic change in bias, tracking the realignment of real world conflict lines and public efforts to reconstitute group identity. The change in salient group identity we describe here likely contributed to the victory of Barack Obama in the 2008 presidential election. [ABSTRACT FROM AUTHOR]

Background: In a recent controversial essay, published by JPA Ioannidis in PLoS Medicine, it has been argued that in some research fields, most of the published findings are false. Based on theoretical reasoning it can be shown that small effect sizes, error-prone tests, low priors of the tested hypotheses and biases in the evaluation and publication of research findings increase the fraction of false positives. These findings raise concerns about the reliability of research. However, they are based on a very simple scenario of scientific research, where single tests are used to evaluate independent hypotheses. Methodology/Principal Findings: In this study, we present computer simulations and experimental approaches for analyzing more realistic scenarios. In these scenarios, research tasks are solved sequentially, i.e. subsequent tests can be chosen depending on previous results. We investigate simple sequential testing and scenarios where only a selected subset of results can be published and used for future rounds of test choice. Results from computer simulations indicate that for the tasks analyzed in this study, the fraction of false among the positive findings declines over several rounds of testing if the most informative tests are performed. Our experiments show that human subjects frequently perform the most informative tests, leading to a decline of false positives as expected from the simulations. Conclusions/Significance: For the research tasks studied here, findings tend to become more reliable over time. We also find that the performance in those experimental settings where not all performed tests could be published turned out to be surprisingly inefficient. Our results may help optimize existing procedures used in the practice of scientific research and provide guidance for the development of novel forms of scholarly communication. [ABSTRACT FROM AUTHOR]

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research. [ABSTRACT FROM AUTHOR]

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = −0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. Public Significance Statement: Research in the social sciences often has implications for public policies as well as individual decisions—for good reason, the robustness of research findings is therefore of widespread interest both inside and outside academia. Yet, even findings that directly replicate—emerging again when the same methodology is repeated—may not always prove conceptually robust to different methodological approaches. The present initiative suggests that crowdsourcing study designs using many research teams can help reveal the conceptual robustness of the effects, better informing the public about the state of the empirical evidence. [ABSTRACT FROM AUTHOR]

Abstract The present investigation provides the first systematic empirical tests for the role of politics in academic research. In a large sample of scientific abstracts from the field of social psychology, we find both evaluative differences , such that conservatives are described more negatively than liberals, and explanatory differences , such that conservatism is more likely to be the focus of explanation than liberalism. In light of the ongoing debate about politicized science, a forecasting survey permitted scientists to state a priori empirical predictions about the results, and then change their beliefs in light of the evidence. Participating scientists accurately predicted the direction of both the evaluative and explanatory differences, but at the same time significantly overestimated both effect sizes. Scientists also updated their broader beliefs about political bias in response to the empirical results, providing a model for addressing divisive scientific controversies across fields. Highlights • In scientific abstracts, conservatives are described more negatively than liberals. • In research abstracts, conservatism is also more often explained than liberalism. • In a forecasting survey, scientists overestimated both effects. • Forecasters updated their beliefs about politics in science in light of the results. [ABSTRACT FROM AUTHOR]

The 7 repeat (7R) allele of the dopamine receptor D4 gene has been associated with attention deficit hyperactivity disorder and risk taking. On the cross-population scale, 7R allele frequencies have been shown to be higher in populations with more of a history of long-term migrations. It has also been shown that the 7R allele is associated with individuals having multiple ancestries. Here, we conduct a replication of this latter finding with two independent samples. Measures of subjects’ ancestry are used to examine past reproductive bonds. The individuals’ history of interracial/ancestral dating and their feelings about this are also assessed. Tentative support for an association between multiple ancestries and the 7R allele was found. These results are dependent upon the method of questioning subjects about their ancestries, with only finer-scale measures of ancestry being associated with 7R. Interracial dating and feelings about interracial pairing were not related to the presence of the 7R allele. This study provides continued support for a role for the 7R allele in migration and/or mate choice patterns. However, replications and extensions of this study are needed and the way ancestry/race is assessed must be carefully considered. [ABSTRACT FROM PUBLISHER]

How can we maximize what is learned from a replication study? In the creative destruction approach to replication, the original hypothesis is compared not only to the null hypothesis, but also to predictions derived from multiple alternative theoretical accounts of the phenomenon. To this end, new populations and measures are included in the design in addition to the original ones, to help determine which theory best accounts for the results across multiple key outcomes and contexts. The present pre-registered empirical project compared the Implicit Puritanism account of intuitive work and sex morality to theories positing regional, religious, and social class differences; explicit rather than implicit cultural differences in values; self-expression vs. survival values as a key cultural fault line; the general moralization of work; and false positive effects. Contradicting Implicit Puritanism's core theoretical claim of a distinct American work morality, a number of targeted findings replicated across multiple comparison cultures, whereas several failed to replicate in all samples and were identified as likely false positives. No support emerged for theories predicting regional variability and specific individual-differences moderators (religious affiliation, religiosity, and education level). Overall, the results provide evidence that work is intuitively moralized across cultures. • This "creative destruction" replication initiative added new measures and populations to four original study designs. • The theory of Implicit Puritanism was competed against seven alternative accounts of work morality. • A number of original findings replicated across multiple cultures, whereas two were identified as likely false positives. • The best-fitting model suggests work is intuitively moralized across cultures. [ABSTRACT FROM AUTHOR]