Search

Interrupted Time Series (ITS) are a type of nonrandomized design commonly used to evaluate public health policy interventions, and the impact of exposures, at the population level. Meta-analysis may be used to combine results from ITS across studies (in the context of systematic reviews) or across sites within the same study. We aimed to examine the statistical approaches, methods, and completeness of reporting in reviews that meta-analyze results from ITS.Eight electronic databases were searched to identify reviews (published 2000-2019) that meta-analyzed at least two ITS. Characteristics of the included reviews, the statistical methods used to analyze the ITS and meta-analyze their results, effect measures, and risk of bias assessment tools were extracted.Of the 4213 identified records, 54 reviews were included. Nearly all reviews (94%) used two-stage meta-analysis, most commonly fitting a random effects model (69%). Among the 41 reviews that re-analyzed the ITS, linear regression (39%) and ARIMA (20%) were most commonly used; 38% adjusted for autocorrelation. The most common effect measure meta-analyzed was an immediate level-change (46/54). Reporting of the statistical methods and ITS characteristics was often incomplete.Improvement is needed in the conduct and reporting of reviews that meta-analyze results from ITS.

Background The Interrupted Time Series (ITS) is a robust design for evaluating public health and policy interventions or exposures when randomisation is infeasible. Several statistical methods are available for the analysis and meta-analysis of ITS studies. We sought to empirically compare available methods when applied to real-world ITS data. Methods We sourced ITS data from published meta-analyses to create an online data repository. Each dataset was re-analysed using two ITS estimation methods. The level- and slope-change effect estimates (and standard errors) were calculated and combined using fixed-effect and four random-effects meta-analysis methods. We examined differences in meta-analytic level- and slope-change estimates, their 95% confidence intervals, p-values, and estimates of heterogeneity across the statistical methods. Results Of 40 eligible meta-analyses, data from 17 meta-analyses including 283 ITS studies were obtained and analysed. We found that on average, the meta-analytic effect estimates, their standard errors and between-study variances were not sensitive to meta-analysis method choice, irrespective of the ITS analysis method. However, confidence interval widths and p-values for the meta-analytic effect estimates varied depending on the choice of confidence interval method and ITS analysis method. Conclusions The meta-analysis effect estimates, their standard errors and between-study variance estimates were minimally impacted by ITS analysis and meta-analysis method choice. However, the confidence interval widths and p-values could vary according to the statistical method, which may impact interpretations and conclusions of a meta-analysis. This empirical study, in conjunction with evidence from numerical simulation, allows for a more complete understanding of which methods should be used in different scenarios.

BackgroundInterrupted time series (ITS) are often meta-analysed to inform public health and policy decisions but examination of the statistical methods for ITS analysis and meta-analysis in this context is limited.MethodsWe simulated meta-analyses of ITS studies with continuous outcome data, analysed the studies using segmented linear regression with two estimation methods [ordinary least squares (OLS) and restricted maximum likelihood (REML)], and meta-analysed the immediate level- and slope-change effect estimates using fixed-effect and (multiple) random-effects meta-analysis methods. Simulation design parameters included varying series length; magnitude of lag-1 autocorrelation; magnitude of level- and slope-changes; number of included studies; and, effect size heterogeneity.ResultsAll meta-analysis methods yielded unbiased estimates of the interruption effects. All random effects meta-analysis methods yielded coverage close to the nominal level, irrespective of the ITS analysis method used and other design parameters. However, heterogeneity was frequently overestimated in scenarios where the ITS study standard errors were underestimated, which occurred for short series or when the ITS analysis method did not appropriately account for autocorrelation.ConclusionsThe performance of meta-analysis methods depends on the design and analysis of the included ITS studies. Although all random effects methods performed well in terms of coverage, irrespective of the ITS analysis method, we recommend the use of effect estimates calculated from ITS methods that adjust for autocorrelation when possible. Doing so will likely to lead to more accurate estimates of the heterogeneity variance.

Introduction Interrupted Time Series (ITS) studies may be used to assess the impact of an interruption, such as an intervention or exposure. The data from such studies are particularly amenable to visual display and, when clearly depicted, can readily show the short‐ and long‐term impact of an interruption. Further, well‐constructed graphs allow data to be extracted using digitizing software, which can facilitate their inclusion in systematic reviews and meta‐analyses. Aim We provide recommendations for graphing ITS data, examine the properties of plots presented in ITS studies, and provide examples employing our recommendations. Methods and results Graphing recommendations from seminal data visualization resources were adapted for use with ITS studies. The adapted recommendations cover plotting of data points, trend lines, interruptions, additional lines and general graph components. We assessed whether 217 graphs from recently published (2013‐2017) ITS studies met our recommendations and found that 130 graphs (60%) had clearly distinct data points, 100 (46%) had trend lines, and 161 (74%) had a clearly defined interruption. Accurate data extraction (requiring distinct points that align with axis tick marks and labels that allow the points to be interpreted) was possible in only 72 (33%) graphs. Conclusion We found that many ITS graphs did not meet our recommendations and could be improved with simple changes. Our proposed recommendations aim to achieve greater standardization and improvement in the display of ITS data, and facilitate re‐use of the data in systematic reviews and meta‐analyses., Highlights Application of data visualization recommendations can improve quality of interrupted time series graphs.Well‐designed graphs accurately depict time series data, any impact of the interruption, and the results of the analysis.Well‐designed graphs facilitate data extraction for use in systematic reviews and reproducibility.An assessment of graphs included in interrupted time series studies (published between 2013 and 2017) found that graphs often do not meet core graphing recommendations.

Objectives Interrupted time series (ITS) designs are frequently used in public health to examine whether an intervention or exposure has influenced health outcomes. Few reviews have been undertaken to examine the design characteristics, statistical methods, and completeness of reporting of published ITS studies. Study Design and Setting We used stratified random sampling to identify 200 ITS studies that evaluated public health interventions or exposures from PubMed (2013–2017). Study characteristics, details of statistical models and estimation methods used, effect metrics, and parameter estimates were extracted. From the 200 studies, 230 time series were examined. Results Common statistical methods used were linear regression (31%, 72/230) and autoregressive integrated moving average (19%, 43/230). In 17% (40/230) of the series, we could not determine the statistical method used. Autocorrelation was acknowledged in 63% (145/230) of the series. An estimate of the autocorrelation coefficient was given for only 1% of the series (3/230). Measures of precision were reported for 63% of effect measures (541/852). Conclusion Many aspects of the design, methods, analysis, and reporting of ITS studies can be improved, particularly description of the statistical methods and approaches to adjust for and estimate autocorrelation. More guidance on the conduct and reporting of ITS studies is needed to improve this study design.

BackgroundThe Interrupted Time Series (ITS) is a quasi-experimental design commonly used in public health to evaluate the impact of interventions or exposures. Multiple statistical methods are available to analyse data from ITS studies, but no empirical investigation has examined how the different methods compare when applied to real-world datasets.MethodsA random sample of 200 ITS studies identified in a previous methods review were included. Time series data from each of these studies was sought. Each dataset was re-analysed using six statistical methods. Point and confidence interval estimates for level and slope changes, standard errors,p-values and estimates of autocorrelation were compared between methods.ResultsFrom the 200 ITS studies, including 230 time series, 190 datasets were obtained. We found that the choice of statistical method can importantly affect the level and slope change point estimates, their standard errors, width of confidence intervals andp-values. Statistical significance (categorised at the 5% level) often differed across the pairwise comparisons of methods, ranging from 4 to 25% disagreement. Estimates of autocorrelation differed depending on the method used and the length of the series.ConclusionsThe choice of statistical method in ITS studies can lead to substantially different conclusions about the impact of the interruption. Pre-specification of the statistical method is encouraged, and naive conclusions based on statistical significance should be avoided.

Background: Systematic reviews underpin clinical practice and policies that guide healthcare decisions. A core component of many systematic reviews is meta-analysis, which is a statistical synthesis of results across studies. Errors in the conduct and interpretation of meta-analysis can lead to incorrect conclusions regarding the benefits and harms of interventions; and studies have shown that these errors are common. Enabling peer reviewers to better detect errors in meta-analysis through the use of a checklist provides an opportunity for these errors to be rectified before publication. To our knowledge, no such checklist exists. Objective: To develop and evaluate a checklist to detect errors in pairwise meta-analyses in systematic reviews of interventions. Methods: We will undertake a four-step process to develop the checklist. First, we will undertake a systematic review of studies that have evaluated errors in the conduct and interpretation of meta-analysis to generate a bank of items to consider for the checklist. Second, we will undertake a survey of systematic review methodologists and statisticians to seek their views on which items, of the bank of items generated in step 1, are most important to include in the checklist. Third, we will hold a virtual meeting to agree upon which items to include in the checklist. Fourth, before finalising the checklist, we will pilot with editors and peer reviewers of journals. Conclusion: The developed checklist is intended to help journal editors and peer reviewers identify errors in the application and interpretation of meta-analyses in systematic reviews. Fewer errors in the conduct and improved interpretation will lead to more accurate review findings and conclusions to inform clinical practice.

BackgroundInterrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures. However, examination of the performance of statistical methods for this design has received relatively little attention.MethodsWe simulated continuous data to compare the performance of a set of statistical methods under a range of scenarios which included different level and slope changes, varying lengths of series and magnitudes of lag-1 autocorrelation. We also examined the performance of the Durbin-Watson (DW) test for detecting autocorrelation.ResultsAll methods yielded unbiased estimates of the level and slope changes over all scenarios. The magnitude of autocorrelation was underestimated by all methods, however, restricted maximum likelihood (REML) yielded the least biased estimates. Underestimation of autocorrelation led to standard errors that were too small and coverage less than the nominal 95%. All methods performed better with longer time series, except for ordinary least squares (OLS) in the presence of autocorrelation and Newey-West for high values of autocorrelation. The DW test for the presence of autocorrelation performed poorly except for long series and large autocorrelation.ConclusionsFrom the methods evaluated, OLS was the preferred method in series with fewer than 12 points, while in longer series, REML was preferred. The DW test should not be relied upon to detect autocorrelation, except when the series is long. Care is needed when interpreting results from all methods, given confidence intervals will generally be too narrow. Further research is required to develop better performing methods for ITS, especially for short series.

Interrupted time series (ITS) studies are frequently used to evaluate the effects of population-level interventions or exposures. To our knowledge, no studies have compared the performance of different statistical methods for this design. We simulated data to compare the performance of a set of statistical methods under a range of scenarios which included different level and slope changes, varying lengths of series and magnitudes of autocorrelation. We also examined the performance of the Durbin-Watson (DW) test for detecting autocorrelation. All methods yielded unbiased estimates of the level and slope changes over all scenarios. The magnitude of autocorrelation was underestimated by all methods, however, restricted maximum likelihood (REML) yielded the least biased estimates. Underestimation of autocorrelation led to standard errors that were too small and coverage less than the nominal 95%. All methods performed better with longer time series, except for ordinary least squares (OLS) in the presence of autocorrelation and Newey-West for high values of autocorrelation. The DW test for the presence of autocorrelation performed poorly except for long series and large autocorrelation. From the methods evaluated, OLS was the preferred method in series with fewer than 12 points, while in longer series, REML was preferred. The DW test should not be relied upon to detect autocorrelation, except when the series is long. Care is needed when interpreting results from all methods, given confidence intervals will generally be too narrow. Further research is required to develop better performing methods for ITS, especially for short series.

Objectives: Debate regarding factors associated with persistent symptoms following mild traumatic brain injury continues. Nested within a trial aiming to change practice in emergency department management of mild traumatic brain injury, this study investigated the nature of persistent symptoms, work/ study outcomes, anxiety and quality of life and factors associated with persistent symptoms following injury, including the impact of receiving information about mild traumatic brain injuries in the emergency department.Methods: A total of 343 individuals with mild traumatic brain injury completed the Rivermead Post-Concussion Symptom Questionnaire, Hospital Anxiety Depression Scale - Anxiety Scale, and Quality of Life - Short Form an average 7 months post-injury.Results: Overall, 18.7% of participants reported 3 or more post-concussional symptoms, most commonly fatigue (17.2%) and forgetfulness (14.6%). Clinically significant anxiety was reported by 12.8%, and was significantly associated with symptom reporting, as were mental and physical quality of life scores. Significant predictors of post-concussional symptoms at follow-up were pre-injury psychological issues, experiencing loss of consciousness, and having no recall of receiving information about brain injury in the emergency department.Conclusion: This study confirms that loss of consciousness and pre-injury psychological issues are associated with persistent symptom reporting. Not receiving injury information in the emergency department may also negatively influence symptom reporting.

IntroductionAn interrupted time series (ITS) design is an important observational design used to examine the effects of an intervention or exposure. This design has particular utility in public health where it may be impracticable or infeasible to use a randomised trial to evaluate health system-wide policies, or examine the impact of exposures (such as earthquakes). There have been relatively few studies examining the design characteristics and statistical methods used to analyse ITS designs. Further, there is a lack of guidance to inform the design and analysis of ITS studies.This is the first study in a larger project that aims to provide tools and guidance for researchers in the design and analysis of ITS studies. The objectives of this study are to (1) examine and report the design characteristics and statistical methods used in a random sample of contemporary ITS studies examining public health interventions or exposures that impact on health-related outcomes, and (2) create a repository of time series data extracted from ITS studies. Results from this study will inform the remainder of the project which will investigate the performance of a range of commonly used statistical methods, and create a repository of input parameters required for sample size calculation.Methods and analysisWe will collate 200 ITS studies evaluating public health interventions or the impact of exposures. ITS studies will be identified from a search of the bibliometric database PubMed between the years 2013 and 2017, combined with stratified random sampling. From eligible studies, we will extract study characteristics, details of the statistical models and estimation methods, effect metrics and parameter estimates. Further, we will extract the time series data when available. We will use systematic review methods in the screening, application of inclusion and exclusion criteria, and extraction of data. Descriptive statistics will be used to summarise the data.Ethics and disseminationEthics approval is not required since information will only be extracted from published studies. Dissemination of the results will be through peer-reviewed publications and presentations at conferences. A repository of data extracted from the published ITS studies will be made publicly available.

Background Evidence-based guidelines for management of mild traumatic brain injury (mTBI) in the emergency department (ED) are now widely available; however, clinical practice remains inconsistent with these guidelines. A targeted, theory-informed implementation intervention (Neurotrauma Evidence Translation (NET) intervention) was designed to increase the uptake of three clinical practice recommendations regarding the management of patients who present to Australian EDs with mild head injuries. The intervention involved local stakeholder meetings, identification and training of nursing and medical local opinion leaders, train-the-trainer workshops and standardised education materials and interactive workshops delivered by the opinion leaders to others within their EDs during a 3 month period. This paper reports on the effects of this intervention. Methods EDs (clusters) were allocated to receive either access to a clinical practice guideline (control) or the implementation intervention, using minimisation, a method that allocates clusters to groups using an algorithm to minimise differences in predefined factors between the groups. We measured clinical practice outcomes at the patient level using chart audit. The primary outcome was appropriate screening for post-traumatic amnesia (PTA) using a validated tool until a perfect score was achieved (indicating absence of acute cognitive impairment) before the patient was discharged home. Secondary outcomes included appropriate CT scanning and the provision of written patient information upon discharge. Patient health outcomes (anxiety, primary outcome: Hospital Anxiety and Depression Scale) were also assessed using follow-up telephone interviews. Outcomes were assessed by independent auditors and interviewers, blinded to group allocation. Results Fourteen EDs were allocated to the intervention and 17 to the control condition; 1943 patients were included in the chart audit. At 2 months follow-up, patients attending intervention EDs (n = 893) compared with control EDs (n = 1050) were more likely to have been appropriately assessed for PTA (adjusted odds ratio (OR) 20.1, 95%CI 6.8 to 59.3; adjusted absolute risk difference (ARD) 14%, 95%CI 8 to 19). The odds of compliance with recommendations for CT scanning and provision of written patient discharge information were small (OR 1.2, 95%CI 0.8 to 1.6; ARD 3.2, 95%CI − 3.7 to 10 and OR 1.2, 95%CI 0.8 to 1.8; ARD 3.1, 95%CI − 3.0 to 9.3 respectively). A total of 343 patients at ten interventions and 14 control sites participated in follow-up interviews at 4.3 to 10.7 months post-ED presentation. The intervention had a small effect on anxiety levels (adjusted mean difference − 0.52, 95%CI − 1.34 to 0.30; scale 0–21, with higher scores indicating greater anxiety). Conclusions Our intervention was effective in improving the uptake of the PTA recommendation; however, it did not appreciably increase the uptake of the other two practice recommendations. Improved screening for PTA may be clinically important as it leads to appropriate periods of observation prior to safe discharge. The estimated intervention effect on anxiety was of limited clinical significance. We were not able to compare characteristics of EDs who declined trial participation with those of participating sites, which may limit the generalizability of the results. Trial registration Australian New Zealand Clinical Trials Registry (ACTRN12612001286831), date registered 12 December 2012. Electronic supplementary material The online version of this article (10.1186/s13012-018-0841-7) contains supplementary material, which is available to authorized users.

Background Network meta-analysis, a method to synthesise evidence from multiple treatments, has increased in popularity in the past decade. Two broad approaches are available to synthesise data across networks, namely, arm- and contrast-synthesis models, with a range of models that can be fitted within each. There has been recent debate about the validity of the arm-synthesis models, but to date, there has been limited empirical evaluation comparing results using the methods applied to a large number of networks. We aim to address this gap through the re-analysis of a large cohort of published networks of interventions using a range of network meta-analysis methods. Methods We will include a subset of networks from a database of network meta-analyses of randomised trials that have been identified and curated from the published literature. The subset of networks will include those where the primary outcome is binary, the number of events and participants are reported for each direct comparison, and there is no evidence of inconsistency in the network. We will re-analyse the networks using three contrast-synthesis methods and two arm-synthesis methods. We will compare the estimated treatment effects, their standard errors, treatment hierarchy based on the surface under the cumulative ranking (SUCRA) curve, the SUCRA value, and the between-trial heterogeneity variance across the network meta-analysis methods. We will investigate whether differences in the results are affected by network characteristics and baseline risk. Discussion The results of this study will inform whether, in practice, the choice of network meta-analysis method matters, and if it does, in what situations differences in the results between methods might arise. The results from this research might also inform future simulation studies.