Search

Many researchers have identified distribution shift as a likely contributor to the reproducibility crisis in behavioral and biomedical sciences. The idea is that if treatment effects vary across individual characteristics and experimental contexts, then studies conducted in different populations will estimate different average effects. This paper uses ``generalizability" methods to quantify how much of the effect size discrepancy between an original study and its replication can be explained by distribution shift on observed unit-level characteristics. More specifically, we decompose this discrepancy into ``components" attributable to sampling variability (including publication bias), observable distribution shifts, and residual factors. We compute this decomposition for several directly-replicated behavioral science experiments and find little evidence that observable distribution shifts contribute appreciably to non-replicability. In some cases, this is because there is too much statistical noise. In other cases, there is strong evidence that controlling for additional moderators is necessary for reliable replication.

In observational causal inference, exact covariate matching plays two statistical roles: (i) it effectively controls for bias due to measured confounding; (ii) it justifies assumption-free inference based on randomization tests. This paper shows that inexact covariate matching does not always play these same roles. We find that inexact matching often leaves behind statistically meaningful bias and that this bias renders standard randomization tests asymptotically invalid. We therefore recommend additional model-based covariate adjustment after inexact matching. In the framework of local misspecification, we prove that matching makes subsequent parametric analyses less sensitive to model selection or misspecification. We argue that gaining this robustness is the primary statistical role of inexact matching., Comment: To appear in Biometrika

We consider the problem of constructing bounds on the average treatment effect (ATE) when unmeasured confounders exist but have bounded influence. Specifically, we assume that omitted confounders could not change the odds of treatment for any unit by more than a fixed factor. We derive the sharp partial identification bounds implied by this assumption by leveraging distributionally robust optimization, and we propose estimators of these bounds with several novel robustness properties. The first is double sharpness: our estimators consistently estimate the sharp ATE bounds when one of two nuisance parameters is misspecified and achieve semiparametric efficiency when all nuisance parameters are suitably consistent. The second is double validity: even when most nuisance parameters are misspecified, our estimators still provide valid but possibly conservative bounds for the ATE and our Wald confidence intervals remain valid even when our estimators are not asymptotically normal. As a result, our estimators provide a highly credible method for sensitivity analysis of causal inferences.

Inverse propensity weighting (IPW) is a popular method for estimating treatment effects from observational data. However, its correctness relies on the untestable (and frequently implausible) assumption that all confounders have been measured. This paper introduces a robust sensitivity analysis for IPW that estimates the range of treatment effects compatible with a given amount of unobserved confounding. The estimated range converges to the narrowest possible interval (under the given assumptions) that must contain the true treatment effect. Our proposal is a refinement of the influential sensitivity analysis by Zhao, Small, and Bhattacharya (2019), which we show gives bounds that are too wide even asymptotically. This analysis is based on new partial identification results for Tan (2006)'s marginal sensitivity model., Comment: This is an original manuscript of an article published by Taylor & Francis in the Journal of the American Statistical Association in 2022, available online: https://doi.org/10.1080/01621459.2022.2069572

After performing a randomized experiment, researchers often use ordinary-least squares (OLS) regression to adjust for baseline covariates when estimating the average treatment effect. It is widely known that the resulting confidence interval is valid even if the linear model is misspecified. In this paper, we generalize that conclusion to covariate adjustment with nonlinear models. We introduce an intuitive way to use any "simple" nonlinear model to construct a covariate-adjusted confidence interval for the average treatment effect. The confidence interval derives its validity from randomization alone, and when nonlinear models fit the data better than linear models, it is narrower than the usual interval from OLS adjustment.