Reducing Threats to Validity by Design in a Nonrandomized Experiment of a School-Wide Prevention Model

This paper reviews literature on the rationale, challenges, and recommendations for choosing a nonequivalent comparison (NEC) group design when evaluating intervention effects. After reviewing frequently addressed threats to validity, the paper describes recommendations for strengthening the research design and how the recommendations were implemented in a current study. Next, tests were performed in order to investigate whether the added design components had contributed to increased internal validity by creating comparable cluster groups at baseline. In order to estimate the external validity or the potential generalizability of results, the statistical power of the study was calculated up front and representativeness of the participating schools in relation to other schools in Norway was examined. Baseline comparisons indicated that the addition of several design elements, such as the random invitation of multiple, stratified, and active comparison groups, may have reduced significant validity threats stemming from selection bias. The need for more accurate and reliable effect estimates in school-based evaluation research is discussed.