Search

A randomized experiment that measures the impact of a social policy in a sample of the population reveals whether the policy will work on average with universal application. An experiment that includes only the subset of the population that volunteers for the intervention generates narrower “proof-of-concept” evidence of whether the policy can work for motivated individuals. Both forms of learning carry value, yet evaluations rarely combine the two designs. The U.S. Social Security Administration conducted an exception, the Benefit Offset National Demonstration (BOND). This article uses BOND to examine the statistical power implications and potential gains in policy learning—relative to costs—from combining volunteer and population-representative experiments. It finds that minimum detectable effects of volunteer experiments rise little when one adds a population-representative experiment, but those of a population-representative experiment double or quadruple with the addition of a volunteer experiment.

The Benefit Offset National Demonstration (BOND) tested a $1 reduction in benefits per $2 earnings increase above the level at which Social Security Disability Insurance benefits drop from full to zero under current law. BOND included a rare and large “population-representative” experiment: It applied the rule to a nationwide, random sample of beneficiaries. BOND also included a parallel “volunteer” experiment: The same offset rules applied to a nationwide random sample of volunteers. BOND’s dual experiments offer a unique opportunity to consider the merits and limitations of these two types of experiments. The findings provide a way to illustrate the limitations of volunteer experiments relative to population experiments when, as in BOND, the goal is to produce impact estimates that are valid for the entire population, not just would be volunteers. We also highlight and compare ethical issues, outreach challenges, and cost considerations for the two types of designs.

Evidence‐based policy at the local level requires predicting the impact of an intervention to inform whether it should be adopted. Increasingly, local policymakers have access to published research evaluating the effectiveness of policy interventions from national research clearinghouses that review and disseminate evidence from program evaluations. Through these evaluations, local policymakers have a wealth of evidence describing what works, but not necessarily where. Multisite evaluations may produce unbiased estimates of the average impact of an intervention in the study sample and still produce inaccurate predictions of the impact for localities outside the sample for two reasons: (1) the impact of the intervention may vary across localities, and (2) the evaluation estimate is subject to sampling error. Unfortunately, there is relatively little evidence on how much the impacts of policy interventions vary from one locality to another and almost no evidence on the implications of this variation for the accuracy with which the local impact of adopting an intervention can be predicted using findings from an evaluation in other localities. In this paper, we present a set of methods for quantifying the accuracy of the local predictions that can be obtained using the results of multisite randomized trials and for assessing the likelihood that prediction errors will lead to errors in local policy decisions. We demonstrate these methods using three evaluations of educational interventions, providing the first empirical evidence of the ability to use multisite evaluations to predict impacts in individual localities—i.e., the ability of “evidence‐based policy” to improve local policy.

Using exogenous characteristics to identify endogenous subgroups, the approach discussed in this method note creates symmetric subsets within treatment and control groups, allowing the analysis to take advantage of an experimental design. In order to maintain treatment–control symmetry, however, prior work has posited that it is necessary to use a prediction subsample, separate from the subsample used for impact estimation in order to prevent overfitting from affecting impact estimates. Doing so diminishes sample size—both for prediction and analysis—and so has costs. This article delves into this topic to consider the conditions under which overfitting occurs and to characterize the effects of overfitting in terms of bias and variance. It suggests a strategy for preserving the full sample size in all phases of the analysis. The research uses Monte Carlo simulation to directly measure overfitting, identify the circumstances that should concern us, and to explore possible recommended practices and future research implications.

To answer “what works?” questions about policy interventions based on an experimental design, Peck (2003) proposes to use baseline characteristics to symmetrically divide treatment and control group members into subgroups defined by endogenously determined postrandom assignment events. Symmetric prediction of these subgroups in both experimental arms ensures the internal validity of the subgroup impacts estimates but leaves the external validity of the findings in doubt. A final step of the procedure solves for impacts on actual subgroups using a system of equations that is underidentified without further assumptions. We address these assumptions by first extending the methodology to encompass three rather than two endogenous subgroups and then proposing plausible assumptions for deriving impacts for actual endogenous subgroups. We also consider how the first-stage prediction process can be structured to better support the accuracy of the assumptions.