Randomized Controlled Studies and Alternative Designs in Outcome Studies: Challenges and Opportunities

This article reviews several decades of the author's meta-analytic and experimental research on the conditions under which nonrandomized experiments can approximate the results from randomized experiments (REs). Several studies make clear that we can expect accurate effect estimates from the regression discontinuity design, though its statistical power is lower, it estimates a different parameter than the RE, and its analysis is considerably more complex. For other nonrandomized designs, the picture is more complex. They may yield accurate estimates if they are prospectively designed to include comprehensive and reliable measurement of the process by which participants selected into conditions, if they use large sample sizes, and if they carefully select control groups that are from the same location and with the same substantive characteristics. By contrast, we have little good reason to think that nonrandomized experiments using archival data without comprehensive selection measures are likely to yield accurate effect estimates. (Contains 1 figure.)