Search

This is a pre-analysis plan for the Facebook 2020 Election Research Project like-minded information experiment. The results from the study outlined in this PAP will be published in a paper forthcoming in Nature, July 28, 2023.

Pre-analysis plan for the Facebook 2020 Election Research Project. The results from the study outlined in this PAP will be published in a paper forthcoming in Science, July 28, 2023.

In observational data studies in economics trying to estimate causal effects (typically using instrumental variables, difference-in-differences, or regression discontinuity methods) it has become common to carry out so called placebo tests (e.g. Almond et al. 2015). In such placebo tests, the same test as in the main hypothesis test is carried out on a time period or situation where the estimated effect is expected to be zero (i.e. the null hypothesis is expected to be true). An example can be using an outcome where there should be no effect or applying a regression discontinuity test on another time period than that for the studied discontinuity. A failure to reject the null hypothesis in the placebo test is interpreted as supporting the validity of the research design to identify causal effects. In placebo tests researchers face different incentives than in regular hypothesis tests. In regular hypothesis tests researchers have an incentive to engage in “p-hacking” and selectively report statistically significant findings. In placebo tests, researchers have an incentive to report null results and thus have an incentive to engage in a form of “reverse p-hacking” (selectively only reporting placebo tests that cannot reject the null hypothesis). In this study we will test if statistically significant placebo tests are selectively underreported. If the null hypothesis is true in placebo tests, the false positive probability of these tests should equal the significance threshold used (i.e. if the tests are carried out at the 5% significance level, 5% of the placebo tests should report a statistically significant finding). This implies that 5% of published placebo tests should be significant at the 5% level if there is no selective reporting. If less than 5% of placebo tests in published papers have a two-sided p-value below 0.05, this provides evidence that placebo tests are selectively reported. As placebo tests that yield a significant effect in the opposite direction of the main hypothesis test are often interpreted as supporting the validity of the research design (see e.g. Ananyev & Guriev 2019 and Bahar & Rapoport 2018); the incentives to underreport statistically significant placebo tests is strongest for placebo tests that yield a significant effect in the same direction as the main hypothesis test. In our primary hypothesis test below we therefore test if the fraction of significant placebo tests with an effect in the same direction as the main hypothesis differ from 2.5% (the expected fraction if true null hypotheses are tested). Note that it is possible that placebo tests are not valid (i.e. do not test true null hypotheses), in which case the fraction of published placebo tests that are significant at the 5% level should be larger than 5% if there is no selective reporting (and larger than 2.5% in our primary hypothesis test). This would bias the results against our hypothesis of selective reporting of placebo tests.

Pre-analysis plans for the Facebook 2020 Election Research Project. The results from the studies outlined in this PAP will be published in two papers forthcoming in Science, July 28, 2023.