Search

Proceedings of the National Academy of Sciences of the United States of America 120(23), e2215572120 (2023). doi:10.1073/pnas.2215572120, Published by National Acad. of Sciences, Washington, DC

Placebo tests, where a null result is used to support the validity of the research design, is common in economics. Such tests provide an incentive to underreport statistically significant tests, a form of reversed p-hacking. Based on a pre-registered analysis plan, we test for such underreporting in all papers meeting our inclusion criteria (n=377) published in 11 top economics journals between 2009-2021. If the null hypothesis is true in all tests, 2.5% of them should be statistically significant at the 5% level with an effect in the same direction as the main test (and 5% in total). The actual fraction of statistically significant placebo tests with an effect in the same direction is 1.29% (95% CI [0.83, 1.63]), and the overall fraction of statistically significant placebo tests is 3.10% (95% CI [2.2, 4.0]). Our results provide strong evidence of selective underreporting of statistically significant placebo tests in top economics journals.

A fundamental question to the scientific enterprise is to what extent published scientific findings are credible. This question is related to the reproducibility and replicability of scientific findings where reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We provide a framework for evaluating reproducibility and replicability in economics and divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability, and we propose indicators to be reported for each type.

In statistics, samples are drawn from a population in a data- generating process (DGP). Standard errors measure the uncer- tainty in sample estimates of population parameters. In sci- ence, evidence is generated to test hypotheses in an evidence- generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sam- ple. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Bosch-Domènech et al. (2014) reported a negative association between 2D:4D, a suggested marker of prenatal testosterone exposure, and the Cognitive Reflection Test (CRT) in a sample of 623 university students. In this pre-registered study, we test whether we can replicate their findings in a general population sample of over 2,500 individuals from Germany. We find no statistically significant association between 2D:4D and the CRT in any of our primary hypothesis tests, or in any of our pre-registered exploratory analyses and robustness tests. The evidence is strong (based on the 99.5% confidence intervals in all three primary hypotheses tests) against effect sizes in the hypothesized direction larger than 0.075 CRT units (0.073 of the CRT standard deviation) for a one standard deviation change in 2D:4D.

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

The 2D:4D digit ratio, the ratio of the length of the 2nd digit to the length of the 4th digit, is often considered a proxy for testosterone exposure in utero. A recent study by Nicolaou et al. (2018) reported an association between the left-hand 2D:4D and self-employment (in a sample of about 1,000 adults). In this pre-registered study we replicate these results on a new and larger sample of about 2,600 adults from the German Socioeconomic Panel-Innovation Sample (SOEP-IS). We find no statistically significant associations between 2D:4D and self-employment and thus cannot confirm the findings of Nicolaou et al. (2018) for left-hand 2D:4D. Our estimated 99.5% confidence intervals are within an about 2 percentage points change in self-employment for a one standard deviation change in the 2D:4D when we pool results for men and women (the association does not differ significantly between men and women). Even larger studies are needed to rule out smaller effect sizes.

Recent large-scale replications of social science experiments provide important information on the reliability of experimental research. Unfortunately, there exist no mechanisms to ensure replications are done. We propose such a mechanism: journal-based replication, in which the publishing journal contracts for a replication between acceptance and publication. We discuss what we learned from a proof-of-concept journal-based replication at the Journal of Public Economics. Our experience indicates that journal-based replication would be relatively straightforward to implement for laboratory experiments.

Women are often less willing than men to compete, even in tasks where there is no gender gap in performance. Also, many people experience competitive contexts as stressful and previous research has documented that men and women sometimes react differently to acute stressors. We use two laboratory experiments to investigate whether factors related to stress can help explain the gender gap in competitiveness. Experiment 1 studies whether stress responses (measured with salivary cortisol and through self-assessment) to taking part in a mandatory competition predict individual willingness to participate in a voluntary competition. We find that while the mandatory competition does increase stress levels, there is no gender difference in this reaction. Cortisol response does not predict willingness to compete for men but is positively and significantly correlated with choosing to enter the voluntary competition for women. In Experiment 2 we exogenously induce stress using the cold-pressor task. We find no causal effect of stress on competitiveness for the sample as a whole and only tentative evidence of a positive effect for women. In summary, even though there are some gender differences in the relation between stress responses and the decision to enter a competition or not, these cannot explain the general gender gap in willingness to compete that is generally found in the literature and which we replicate.

We compare how children aged 9-12 in Colombia and Sweden cooperate in a Prisoner's Dilemma. We introduce a new measurement device for cooperation that can be easily understood by children. There is some evidence of more cooperation in Sweden than in Colombia. Girls in Colombia are less cooperative than boys, whereas our results indicate the opposite in Sweden. Girls are in general more cooperative with boys than with girls. Relating cooperation to competitiveness, this appears to be task and country dependent.

Many previous experiments document that behavior in multi-person settings responds to the name of the game and the labeling of strategies. Usually these studies cannot tell whether frames affect preferences or beliefs. In this Dictator game study, we investigate whether social framing effects are also present when only one of the subjects makes a decision, in which case the frame may only affect preferences. We find that behavior is insensitive to social framing.

We look at gender differences in preferences for altruism, risk and competition in math and word search among adolescents in Sweden. We find that girls are more altruistic and less risk taking than boys. We find no gender gap in performance change when comparing performance under non-competition with performance under competition. Boys and girls are equally likely to choose to compete in word search, but boys are significantly more likely to choose to compete in math. However, this gender gap diminishes and becomes non significant when we control for relative performance beliefs, indicating that some of the gender gap in our sample is not due to competition preferences per se.

We explore the relationship between attractiveness and risk taking in chess. We use a large international panel dataset on chess competitions which includes a control for the players' skill in chess. This data is combined with results from a survey on an online labor market where participants were asked to rate the photos of 626 expert chess players according to attractiveness. Our results suggest that male chess players choose significantly riskier strategies when playing against an attractive female opponent, even though this does not improve their performance. Women's strategies are not affected by the attractiveness of the opponent.

We designed an experiment that examines how knowledge about the price of a good, and the time at which the information is received, affects how the good is experienced. The good in question was wine, and the price was either high or low. Our results suggest that hosts offering wine to guests can safely reveal the price: much is gained if the wine is expensive, and little is lost if it is cheap. Disclosing the high price before tasting the wine produces considerably higher ratings, although only from women. Disclosing the low price, by contrast, does not result in lower ratings. Our finding supports the notion that price not only serves to clear markets, it also serves as a marketing tool; it influences expectations that in turn shape a consumer's experience. In addition, our results suggest that men and women respond differently to attribute information concerning wine.

Recent studies find that women are less competitive than men. This gender difference in competitiveness has been suggested as one possible explanation for why men occupy the majority of top positions in many sectors. In this study we explore competitiveness in children, with the premise that both culture and gendered stereotypes regarding the task at hand may influence competitive behavior. A related field experiment on Israeli children shows that only boys react to competition by running faster when competing in a race. We here test if there is a gender gap in running among 7-10 year old Swedish children. We also introduce two female sports, skipping rope and dancing, to see if competitiveness is task dependent. We find no gender difference in reaction to competition in any task; boys and girls compete equally. If gender equality matters for competitiveness, this result may be explained by cultural factors, since Sweden scores higher on gender equality indices than Israel.

Individuals who are unaware of the price do not derive more enjoyment from more expensive wine. In a sample of more than 6,000 blind tastings, we find that the correlation between price and overall rating is small and negative, suggesting that individuals on average enjoy more expensive wines slightly less. For individuals with wine training, however, we find indications of a positive, or at any rate non-negative, correlation. Our results are robust to the inclusion of individual fixed effects, and are not driven by outliers: when omitting the top and bottom deciles of the price distribution, our qualitative results are strengthened, and the statistical significance is improved even further. Our results indicate that both the prices of wines and wine recommendations by experts may be poor guides for non-expert wine consumers.

We explore gender differences in preferences for competition and risk among children aged 9-12 in Colombia and Sweden, two countries differing in gender equality according to macro indices. We include four types of tasks that vary in gender stereotyping when looking at competitiveness: running, skipping rope, math and word search. We find that boys and girls are equally competitive in all tasks and all measures in Colombia. Unlike the consistent results in Colombia, the results in Sweden are mixed, with some indication of girls being more competitive than boys in some tasks in terms of performance change, whereas boys are more likely to choose to compete in general. Boys in both countries are more risk taking than girls, with a smaller gender gap in Sweden.