Search

Although randomized experiments are lauded for their high internal validity, they have been criticized for the limited external validity of their results. This chapter describes research strategies for investigating how much nonrepresentative site selection may limit external validity and bias impact findings. The magnitude of external validity bias is potentially much larger than what is thought of as an acceptable level of internal validity bias. The chapter argues that external validity bias should always be investigated by the best available means and addressed directly when presenting evaluation results. These observations flag the importance of making external validity a priority in evaluation planning.

Program evaluators generally prefer to use the strongest design available to answer relevant impact questions, reserving analytic strategies for use only as necessary. Although the “simple” treatment versus control experimental design is well understood and widespread in its use, it is our contention that creativity in evaluation design can help answer more nuanced questions regarding what about a program is responsible for its impacts. In response, this chapter discusses several experimental evaluation designs that randomize individuals—including multiarmed, multistage, factorial, and blended designs—to permit estimating impacts for specific policy design features or program elements. We hope that recasting what are some long-standing but underused designs in a new light will motivate their increased use, where appropriate.