Search

Replication is an important “credibility control” mechanism for clarifying the reliability of published findings. However, replication is costly, and it is infeasible to replicate everything. Accurate, fast, lower cost alternatives such as eliciting predictions from experts or novices could accelerate credibility assessment and improve allocation of replication resources for important and uncertain findings. We elicited judgments from experts and novices on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using a new interactive structured elicitation protocol and we conducted 35 new replications. Participants’ average estimates were similar to the observed replication rate of 60%. After interacting with their peers, novices updated both their estimates and confidence in their judgements significantly more than experts and their accuracy improved more between elicitation rounds. Experts’ average accuracy was 0.54 (95% CI: [0.454, 0.628]) after interaction and they correctly classified 55% of claims; novices’ average accuracy was 0.55 (95% CI: [0.455, 0.628]), correctly classifying 61% of claims. The difference in accuracy between experts and novices was not significant and their judgments on the full set of claims were strongly correlated (r=.48). These results are consistent with prior investigations eliciting predictions about the replicability of published findings in established areas of research and suggest that expertise may not be required for credibility assessment of some research findings.

This is a replication attempt on the following claims from O’Brien et al. (2015): RQ1: Latent class analysis Claim ID: m6qkxx Claim 4 text: In Table 4, the insignificant LMR test from the four-class model suggests the three-class model best fits the data. [Table 4, Number of Classes: 3, p < .001; Number of Classes: 4, p = .762] RQ2: Science Knowledge Claim ID: g33kw7 Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Members of the traditional class have significantly lower scores than do the other groups for all science literacy and attitude items. For example, 47 percent of the traditional class, compared to 92…percent of the modern…groups…correctly answered that radioactivity occurs naturally. [Table 2, Science Knowledge (scientifically correct answer equals one, else equals zero), All radioactivity is man-made? Traditional: Conditional Mean = .466, Significantly different from modern, p < .05 (two-tailed t-test), Modern: Conditional Mean = .918] Claim ID: mq2jv3 Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Members of the modern category are knowledgeable about science…Unlike other perspectives, large majorities of moderns responded that the universe began with a big bang…(p < .05). [Table 2, Science Knowledge (scientifically correct answer equals one, else equals zero), The Universe began with a huge explosion? Modern: Conditional Mean = .678, Significantly different from traditional, p < .05 (two-tailed t-test), Traditional: Conditional Mean = .212] Claim ID: vdy6zp Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Despite their generally favorable outlook on science, members of the post-secular category were substantially and significantly less likely than the other classes to respond that the universe began with a big bang (6 percent)… [Table 2, Science Knowledge (scientifically correct answer equals one, else equals zero), The Universe began with a huge explosion? Post-Secular: Conditional Mean = .058, Significantly different from traditional, p < .05 (two-tailed t-test), Significantly different from modern, p < .05 (two-tailed t-test), Traditional: Conditional Mean = .212, Modern: Conditional Mean = .678] Claim ID: kr452q Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Despite their generally favorable outlook on science, members of the post-secular category were substantially and significantly less likely than the other classes to respond that…humans evolved from other animals (3 percent). [Table 2, Science Knowledge (scientifically correct answer equals one, else equals zero), Human beings developed from earlier species of animals? Post-Secular: Conditional Mean = .032, Significantly different from traditional, p < .05 (two-tailed t-test), Significantly different from modern, p < .05 (two-tailed t-test), Traditional: Conditional Mean = .330, Modern: Conditional Mean = .877] RQ3: Scientific attitudes Claim ID: 9dv7o7 Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Members of the modern category are knowledgeable about science and the most appreciative of its uses… [Table 2, Science Attitudes, Do the benefits of scientific research outweigh the harmful results (0 = harm strongly outweighs benefits; 2 = harm and benefits about equal; 4 = benefits strongly outweigh harm)? Modern: Conditional Mean = 3.553, Significantly different from traditional, p < .05 (two-tailed t-test), Traditional: Conditional Mean = 2.606] Claim ID: 8123nd Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…The post-secular perspective, held by 21 percent of respondents, is characterized by less favorable attitudes about science than those of the modern class, but their attitudes toward science are significantly more favorable than those of the traditional class. [Table 2, Science Attitudes, Scientific research that advances the frontiers of knowledge is necessary and should be supported by the federal government (1 = strongly disagree, 4 = strongly agree). Post-Secular: Conditional Mean = 3.095, Significantly different from traditional, p < .05 (two-tailed t-test), Significantly different from modern, p < 0.05 (two-tailed t-test), Traditional: Conditional Mean = 3.008, Modern: Conditional Mean = 3.436] RQ4: Religious indicators Claim ID: gwk6zj Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…the traditional class…religious affiliation strength is…significantly higher than that of the modern class. [Table 2, Religion Indicators, Strength of religious affiliation (1 = none, 4 = very strong). Traditional: Conditional Mean = 2.679, Significantly different from modern, p < .05 (two-tailed t-test), Modern: Conditional Mean = 2.058] Claim ID: rdyrvd Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…post-seculars’ mean religious affiliation strength is significantly higher than that of each other latent class (p < .05). [Table 2, Religion Indicators, Strength of religious affiliation (1 = none, 4 = very strong). Post-Secular: Conditional Mean = 3.336, Significantly different from traditional, p < .05 (two-tailed t-test), Significantly different from modern, p < .05 (two-tailed t-test), Traditional: Conditional Mean = 2.679, Modern: Conditional Mean = 2.058] Claim ID: 472vko Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…Members of the modern category are…the least religious…only 3 percent of the modern group responded that the Bible is the literal word of God. [Table 2, Religion Indicators, Bible is the actual word of God (0 = no, 1 = yes)? Modern: Conditional Mean = .030, Significantly different from Traditional, p < .05 (two-tailed t-test), Traditional: Conditional Mean = .458] Claim ID: n5y8v1 Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…In contrast to moderns…48 percent of post-seculars reported that the Bible is the literal word of God… [Table 2, Religion Indicators, Bible is the actual word of God (0 = no, 1 = yes)? Post-Secular: Conditional Mean = .481, Significantly different from modern, p < .05 (two-tailed t-test), Modern: Conditional Mean = .030] Claim ID: d74pk1 Claim 4 text: The final three columns in Table 2 contain conditional means and proportions of manifest variables…In contrast to moderns…48 percent of post-seculars reported that the Bible is the literal word of God and none reported that the Bible is a book of myths and fables. [Table 2, Religion Indicators, Bible is a book of myths and fables (0 = no, 1 = yes)? Post-Secular: Conditional Mean =

This is for an attempted push-button replication of Cleave, Nikiforakis, and Slonim (2013) https://link.springer.com/article/10.1007/s10683-012-9342-8

The claim selected for replication from Lee & Mas (2012) is as follows: “The paper conducts this analysis to examine the heterogeneity in the stock market reaction to election outcomes and to determine whether there is a discontinuous relationship between cumulative abnormal returns and the vote share at the 50% threshold. This reflects the following statement from the paper's abstract: "We find a negative relationship between the cumulative abnormal returns and the vote share in support of the union, allowing us to reconcile these seemingly contradictory findings." Formal statistical inference are conducted in Table V. In Column (1) the paper regresses CAR(0,24) on a dummy for whether the union won the election. CAR corresponds to event i between months T1 and T2. The dependent variable is ”CAR(0,24)” and the independent variable is “Union Victory”. Consistent with earlier analyses, we find that union victories are associated with CARs that are 12.1 percentage points lower than firms with union losses (t-ratio = − 3.5). [Coefficient of “Union Victory” is -0.121 with robust standard errors of 0.035.]”

This is a Direct Replication attempt on the claim 4 claim: The findings also supported the next hypothesis, H4: a positive relation will be found between openness to experiences and leadership experiences. A significant (p < .01) positive relation was indeed found here (beta = .34). from Popper et al (2009). The statistical evidence is: b = .34, p < .01

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “Table 2, columns 1...report coefficients from the linear regression of BMI percentile (pBMI) on family size indicators and varying sets of controls. The unadjusted coefficients in Column 1 show that compared to only-children, children with...three-or-more siblings have...3.9 percentile points lower BMI...[Table 2, BMI percentile (1), Three or more siblings = -3.933, robust standard error = 1.182, significant at 1%]“ from Datar (2017). The statistical evidence is: Table 2, BMI percentile (1), Three or more siblings = -3.933, robust standard error = 1.182, significant at 1%.

The claim selected from Da & Warachka (2009) is that higher earnings betas imply higher returns. This reflects the following statement from the paper's abstract: "From 1984 to 2005, 55.1% of the return variation across book-to-market, size, and long-term return reversal portfolios is captured by their analyst earnings betas." The claim is tested with cross-sectional regression. See Panel B Fama-MacBeth regression on 30 test portfolios. Panel Panel B contains the results of the cross-sectional regression in Eq. (17), which is conducted on 30 book-to-market, size, and past long-term return portfolios. The dependent variable is “r_(t+delta)^i - rf_t”. “r_(t+delta)^I” represents the realized return for a particular portfolio while “rf_t” denotes the risk-free rate over the same monthly horizon. The focal independent variable is “beta_(CF)^I”, the earnings betas. It’s coefficient in Eq. (17) is “lambda_1”. The results from this cross-sectional regression in panel B of Table 2 indicate that the r-squared from Eq. (17) equals 55.1%. The “lambda_1” estimate is positive (63 bp [book-to-price ratio] per month), with a t-value of 2.05, indicating that higher earnings betas imply higher returns. [The estimate of “lambda_1” is 0.0063.]

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “In terms of the overall importance of schools (columns 3...Table 5)...for...math...we find that...in countries that are poor and unequal, schools overall matter more. [Table 5, Variance Component of the Intercept, 3, Math, Country measure, External, Gini index = .63, p

The claim selected for replication from Baxter et al. (2015) is that becoming a parent leads [men] to more strongly support a traditional division of labor in relation to the care of children. This reflects the following statement from the paper's abstract: "Key findings indicate that attitudes become more traditional after individuals experience the birth of their first child, with both men and women becoming more likely to support mothering as women’s most important role in life." The authors model the relationships between their explanatory variables and each of their outcome variables using fixed-effects regression models. The selected test result concerns the Item 2 outcome. The specification of the regression model can be gleaned from equations 1-3, and Table 3 under the heading “Item 2”. The focal estimand is the coefficient in the “Male and had 1st birth” term. Among men, significant coefficients on the parenthood variables point unambiguously toward more traditional attitudes. This is apparent for attitudinal item 2: after they become fathers, men are less likely to agree that men and women in dual-earner couples should share housework and childcare equally (b = .11, p < .1).

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “The results also suggest…systematic underestimation of literacy skills among Black children…Table 3…Model 2. [Table 3, Fall, Model 2, Black, Estimate = -.13, p < .01]” from Ready and Wright (2011). The statistical evidence is: Table 3, Fall, Model 2, Black, Estimate = -.13, p < .01.

The claim selected for replication from Kuo & Raley (2016) is as follows: “Union transition patterns among cohabitors in the recent cohabiting cohort is significantly contingent on cohabitors’ educational attainment. Specifically, educational disparities in the probabilities of transitioning to marriage have become statistically significant among the 2005–2010 cohabiting cohort due to drastic drops in the probabilities of transitioning to marriage among all non-college-educated cohabitors in the recent cohabiting cohort. This reflects the following statement from the paper's abstract: ‘With data on two cohabiting cohorts from the NSFG 1995 and 2006–2010, the current study finds that marriage rates among cohabitors have declined steeply among those with no college degree, resulting in growing educational disparities over time.’ Data for the analysis are from the National Study of Family Growth 1995 (NSFG Cycle5) and 2006–2010 (NSFG 2006–2010). The authors convert data into person-month data sets, with the first month indicating the month when cohabitation started and the last month signaling the month when cohabitation was ended either by marriage or separation or the month when the interview was conducted for those whose first premarital cohabitation remained intact at the time of interview (i.e., censoring). Then, the authors estimate multivariate discrete-time event history models with multinomial logistic regression to investigate how educational disparities in the odds of transitioning to marriage and the odds of separation, as opposed to staying cohabiting, differ between the two cohabiting cohorts. The specification of the model can be gleaned from Table 2 Model 1. The focal test result concerns the location of the estimated coefficient in the Some College x Period interaction term in the multinomial equation contrasting getting married vs. staying intact. The reference categories are College + and 1990-1995 respectively. The result was a statistically significant coefficient in the Some College x Period interaction term, b = -.725, SE = .27, p < .01.”

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “The dotted line represents the overall effect of parental SES on first-union timing for all European countries. Figure 1A shows an overall negative effect of parental SES on the timing of first union for women (b = -0.171, P < 0.01). Thus, the higher the SES of parents, the later women enter their first co-residential union.” from Brons et al (2017). The statistical evidence is: Figure 1A, b = -0.171, P < 0.01.

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “Regarding the effect of societal discrimination against young girls, holding constant other socioeconomic and cultural factors, and given the mean number of unwanted daughters within the household (0.80), our estimates support that increases in societal discrimination are associated with 0.045 increases in the expected number of daughters’ deaths (p < 0.001).” from Echávarri and Husillos (2016). The statistical evidence is: Table 2, Column (3), societal discrimination (D) estimate = 0.045, p < 0.001.

This is a Data Analytic Replication (DAR) attempt on the claim 4 “For LM learners who were iFEP, the period between spring of kindergarten and fall of first grade was a time of particularly rapid growth. This group’s K–1 rate of growth was significantly faster than that of native English speakers (an equivalent of .144 additional theta score points per year, p < .0001), allowing them to substantially narrow the gap with native English speakers to only .02 theta score points (or an effect size of .04 in spring of first grade SDs) over the course of the first year of school.” The statistical evidence is: Table 3, Model 2, K-1 rate of growth, LM-iFEP = 0.144 additional theta score points per year, p < .0001.

This is a Data Analytic Replication (DAR) attempt on the claim 4 “Table 1 summarizes the baseline results focused on Support redistribution...Models 2...directly test hypotheses...suggesting that the economic nonintegration measures particularly diminish the degree to which Foreign-born percent has a negative effect. This can be seen by the significant interaction terms in model 2 for interaction with Gap in unemployment (hypothesis 1)...[Table 1, Column (2), Foreign-born × Gap = –0.021 (0.008), p < 0.05]”, from Burgoon (2014). The statistical evidence is: Table 1, Column (2), Foreign-born × Gap = –0.021 (0.008), p < 0.05.

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “As expected, overall job satisfaction makes an employee less likely to leave across the board: as job satisfaction increases, employees are less likely to intend to leave their agency for another within the federal government...[Table 2, Leaving Agency, Job satisfaction = –0.444, SE = 0.0163, significant at p < .01, two tail test]” from Pitts et al. (2011). The statistical evidence is: Table 2, Leaving Agency, Job satisfaction = –0.444, SE = 0.0163, significant at p < .01.

This is a Data Analytic Replication (DAR) attempt on the Claim 4 "Conversely, the percentage living in nuclear-family households increased from 35.7 % in 1992–1993 to 43.5 % in 2005–2006. All these changes are statistically significant. Overall, in this roughly 14-year period, the percentage of young women living in nuclear households rose by almost 8 %. [Table 1, 1992–1993 vs 2005–2006, Total = p < .001]" From Allendorf (2013). The statistical evidence is: Table 1, 1992–1993 vs 2005–2006, Total = p < .001.

Attempting a reproduction of Wilfahrt M. (2018)' analysis.

This is a Data Analytic Replication (DAR) attempt on the Claim 4 “In Table 4, multilevel, multinomial models jointly estimate men’s likelihood of being gender-typed, egalitarian or pragmatic, rather than conforming selectively to the task hierarchy…As hypothesized for time availability, a man whose wife works longer hours is significantly more likely to choose egalitarianism over conformity. [Table 4, Reference: conforming, Egalitarian, Time availability, Wife’s work hours = 1.011, p < 0.001 (two-tailed)]“ from Tai and Treas (2013). The statistical evidence is: Table 4, Reference: conforming, Egalitarian, Time availability, Wife’s work hours = 1.011, p < 0.001 (two-tailed).

The claim selected from Stambaugh et al. (2012) is that there is a positive relation between the profitability of each long-short spread and investor sentiment. This reflects the following statement from the paper's abstract: "This study explores the role of investor sentiment in a broad set of anomalies in cross-sectional stock returns. First, each anomaly is stronger (its long-short strategy is more profitable) following high levels of sentiment." Table 4 reports regressions of long-short returns on the lagged sentiment index. The dependent variable in each regression is the long-short return spread corresponding to a given anomaly. The focal independent variable is “S_t” (with the coefficient “b_hat”), the level of the investor-sentiment index of Baker and Wurgler (2006). The slope coefficients for the spread of [gross profitability is] positive in both Tables 4 and 5. [In Table 4, the coefficient for “b_hat” is 0.32 with t-stat of 1.81.]

This is a Data Analytic Replication (DAR) attempt on the claim 4 “...(Table 3a). Child race/ethnicity provided mixed results where being Hispanic relative to white decreased externalizing behaviors. [Table 3a, High Poverty Estimate, SE, Race (Ref = White), Hispanic = -0.19, standard error = 0.10, p < 0.05]“ from Humphrey and Root (2017). The statistical evidence is: Table 3a, High Poverty Estimate, SE, Race (Ref = White), Hispanic = -0.19, standard error = 0.10, p < 0.05.

This is a direct replication attempt on the claim 4 claim: In support of H1, we found a significant two-way interaction between tie strength and complaining on participants’ intended loyalty (F(1, 188) = 8.32, p < .01). from Umashankar et al (2017), Study 2. The statistical evidence is: F(1, 188) = 8.32, p < .01

This is a direct replication attempt on the claim 4 claim: Consistent with our predictions, participants who read that Efficient Corp corrected (vs. prevented) a mistake perceived that its actions...required more effort (M_Prevention: 5.83, SD_Prevention: 1.99; M_Correction: 6.38, SD_Correction: 1.87; t(198) = 2.02, p = .045; Cohen’s d = .286; 95% CI: .020 to .553). from Kupor et al (2018), Study 3. The statistical evidence is: M_Prevention: 5.83, SD_Prevention: 1.99; M_Correction: 6.38, SD_Correction: 1.87; t(198) = 2.02, p = .045; Cohen’s d = .286; 95% CI: .020 to .553)

This is a direct replication attempt on the claim 4 claim: The mean number of extreme responses was M = 3.1 (SE = .26) for strongly (dis)agree and M = 4.4 (SE = .33) for completely (dis)agree, and the difference in means is significant (t(216) = 3.07, p < .05). This result is inconsistent with the intensity hypothesis but supports the familiarity hypothesis. That is, respondents are more likely to choose the extremes of the rating scale when the associated category labels are more familiar rather than less intense. from Weijters et al (2013). The statistical evidence is: M = 3.1 (SE = .26) for strongly (dis)agree and M = 4.4 (SE = .33) for completely (dis)agree, and the difference in means is significant (t(216) = 3.07, p < .05).

This is a direct replication attempt on the claim 4 claim: Consistent with H1, the social-adjustive participants had a higher purchase intent than the value-expressive ones (M_Value Expressive = 2.48, M_Social Adjustive = 3.31; F(1, 130) = 8.91, p < .05, omega^2 = .05) (p. 252). from Wilcox et al (2009). The statistical evidence is: M_Value Expressive = 2.48, M_Social Adjustive = 3.31; F(1, 130) = 8.91, p < .05, omega^2 = .05

This is a direct replication attempt on the claim 4 claim: Consistent with our prediction, the mean flexibility in the continual-switch condition (M = 15.57, SD = 5.19) was significantly higher than in the discretionary-switch condition (M = 13.35, SD = 4.44), t(77)=2.05; p=0.04; d=0.47. from Lu et al (2017). The statistical evidence is: continual-switch condition (M = 15.57, SD = 5.19), discretionary-switch condition (M = 13.35, SD = 4.44), t(77)=2.05; p=0.04; d=0.47.

This is a direct replication attempt on the claim 4 claim: Fig. 2 shows the mean ratings for the four judgments we elicited, separately for each of the AA procedures. Ratings of support for the procedure yielded a significant difference between the two procedures: support for the Unidentifiable procedure was greater than for the Identifiable procedure (t(107) = 3.624, p < .001, eta-squared = .110). from Ritov et al (2014). The statistical evidence is: t(107) = 3.624, p < .001, eta-squared = .110

This is a direct replication attempt on the claim 4 claim: Political ideology predicted nostalgia, meaning that the more conservative participants were, the more nostalgic they were, b = .10, SE = .03, t(261) = 3.12, p = .002, R^2 = .04, 95% confidence interval (CI_b) [.03; .17]. from Lammers et al (2018). The statistical evidence is: b = .10, SE = .03, t(261) = 3.12, p = .002, R^2 = .04, 95% confidence interval (CI_b) [.03; .17]