Search

Researchers are increasingly using comparative interrupted time series (CITS) designs to estimate the effects of programs and policies when randomized controlled trials are not feasible. In a simple interrupted time series design, researchers compare the pre-treatment values of a treatment group time series to post-treatment values in order to assess the impact of a treatment, without any comparison group to account for confounding factors. The CITS design is a version of the ITS design in which both a treatment and a comparison group are evaluated both before and after the onset of a treatment. A growing body of literature is employing a within study comparison (WSC) methodology to examine the validity of the CITS model. WSC studies empirically estimate the extent to which a given observational study reproduces the results of a randomized controlled trial (RCT) when both share the same treatment group, and represent a rigorous method of evaluating non-experimental designs using real data. A number of recent within-study comparisons have demonstrated that CITS can produce estimates that are comparable to those from a randomized controlled trial (RCT) in practice. In the St. Clair et al. (2014) application, the authors found that correspondence with the RCT was possible when the CITS model accounted for baseline trends, but that additional time points could actually increase bias when the pre-treatment trend was not modeled correctly. Examination of the pretreatment trends in this data set showed clearly that in at least one of the outcomes the treatment and comparison groups had different slopes in the pretreatment period, and as a result the "parallel trends" assumption often invoked in the difference-in-difference literature was clearly violated. This paper employs a within study comparison (WSC) methodology to examine the performance of two approaches: (1) a more flexible modeling approach, which employs year fixed-effects rather than trying to parametrically model the pretest trend; and (2) match treatment and comparison cases to reduce reliance on modeling the pretreatment trend. The paper then compares the approaches to the performance of the baseline mean and baseline slope models across three datasets. The purpose of this research is two-fold: (1) to examine what approach, if any, works in the unclear functional form case; and (2) to examine the relative superiority of the different approaches across the three datasets in terms of both bias reduction and precision. Tables and figures are appended.