Search

Objectives: To describe statistical tools available for assessing publication integrity of groups of randomized controlled trials (RCTs)., Study Design and Setting: Narrative review., Results: Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomization occurred. For continuous variables, the tools assess baseline means, baseline P values, and the occurrence of identical means and/or standard deviation. For categorical variables, they assess baseline P values, frequency counts for individual or all variables, numbers of trial participants randomized or withdrawing, and compare reported with independently calculated P values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission., Conclusion: Statistical tools are useful for the assessment of publication integrity of groups of RCTs., Competing Interests: Declaration of competing interest M.J.B. is a recipient of an HRC Clinical Practitioners Fellowship. The authors are independent of the HRC. The HRC had no role in design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. There are no competing interests for any other author., (Copyright © 2024 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: Randomized controlled trials (RCTs) with repeatedly measured continuous variables as primary outcomes are common. Although statistical methodologies for calculating sample sizes in such trials have been extensively investigated, their practical application remains unclear. This study aims to provide an overview of sample size calculation methods for different research questions (e.g., key time point treatment effect, treatment effect change over time) and evaluate the adequacy of current practices in trial design., Study Design and Setting: We conducted a comprehensive search of PubMed to identify RCTs published in core journals in 2019 that utilized repeatedly measured continuous variables as their primary outcomes. Data were extracted using a predefined questionnaire including general study characteristics, primary outcomes, detailed sample size calculation methods, and methods for analyzing the primary outcome. We re-estimated the sample size for trials that provided all relevant parameters., Results: A total of 168 RCTs were included, with a median of four repeated measurements (interquartile range 3-6) per outcome. In 48 (28.6%) trials, the primary outcome used for sample size calculation differed from the one used in defining the primary outcomes. There were 90 (53.6%) trials exhibited inconsistencies between the hypotheses specified for sample size calculation and those specified for primary analysis. The statistical methods used for sample size calculation in 158 (94.0%) trials did not align with those used for primary analysis. Additionally, only 6 (3.6%) trials accounted for the number of repeated measurements, and 7 (4.2%) trials considered the correlation among these measurements when calculating the sample size. Furthermore, of the 128 (76.2%) trials that considered loss to follow-up, 33 (25.8%) used an incorrect formula (i.e., N∗(1+lose rate) for sample size adjustment. In 53 (49.5%) out of 107 trials, the re-estimated sample size was larger than the reported sample size., Conclusion: The practice of sample size calculation for RCTs with repeatedly measured continuous variables as primary outcomes displayed significant deficiencies, with a notable proportion of trials failed to report essential parameters about repeated measurement required for sample size calculation. Our findings highlight the urgent need to use optimal sample size methods that align with the research hypothesis, primary analysis method, and the form of the primary outcome., Competing Interests: Declaration of competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: Studies from the first waves of the coronavirus disease 2019 (COVID-19) pandemic suggest that individuals from minority ethnicities are at an increased risk of worse outcomes. Concerns exist that this relationship is potentially driven by bias from analyzing hospitalized patients only. We investigate this relationship and the possible presence of bias., Study Design and Setting: Using data from South London hospitals across two COVID-19 waves (February 2020 - May 2021), the relationship between ethnicity and COVID-19 outcomes was examined using regression models. Three iterations of each model were completed: 1) an unadjusted analysis, 2) adjusting for covariates (medical history and deprivation), and 3) adjusting for covariates and bias induced by conditioning on hospitalization., Results: Among 3,133 patients, those who were Asian had a two-fold increased risk of death during the hospital stay that was consistent across the two COVID-19 waves and was not affected by correcting for conditioning on hospitalization. However, wave-specific effects demonstrate significant differences between ethnic groups until bias from using a hospitalized cohort was corrected for., Conclusion: Worsened COVID-19 outcomes in minority ethnicities may be minimized by correcting for bias induced by conditioning on hospitalization. Consideration of this bias should be a key component of study design., Competing Interests: Declaration of competing interest There are no conflicts of interest to declare., (Copyright © 2023 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: To assess the quality of reporting and accuracy of a priori estimates used in sample size calculations for cluster randomized trials (CRTs). Study Design and Setting: We reviewed 300 CRTs published between 2000 and 2008. The prevalence of reporting sample size elements from the 2004 CONSORT recommendations was evaluated and a priori estimates compared with those observed in the trial. Results: Of the 300 trials, 166 (55%) reported a sample size calculation. Only 36 of 166 (22%) reported all recommended descriptive elements. Elements specific to CRTs were the worst reported: a measure of within-cluster correlation was specified in only 58 of 166 (35%). Only 18 of 166 articles (11%) reported both a priori and observed within-cluster correlation values. Except in two cases, observed withincluster correlation values were either close to or less than a priori values. Conclusion: Even with the CONSORT extension for cluster randomization, the reporting of sample size elements specific to these trials remains below that necessary for transparent reporting. Journal editors and peer reviewers should implement stricter requirements for authors to follow CONSORT recommendations. Authors should report observed and a priori within-cluster correlation values to enable comparisons between these over a wider range of trials. [ABSTRACT FROM AUTHOR]

Objectives: To determine the number of independent variables that can be included in a linear regression model. Study Design and Setting: We used a series of Monte Carlo simulations to examine the impact of the number of subjects per variable (SPV) on the accuracy of estimated regression coefficients and standard errors, on the empirical coverage of estimated confidence intervals, and on the accuracy of the estimated R² of the fitted model. Results: A minimum of approximately two SPV tended to result in estimation of regression coefficients with relative bias of less than 10%. Furthermore, with this minimum number of SPV, the standard errors of the regression coefficients were accurately estimated and estimated confidence intervals had approximately the advertised coverage rates. A much higher number of SPV were necessary to minimize bias in estimating the model R², although adjusted R² estimates behaved well. The bias in estimating the model R² statistic was inversely proportional to the magnitude of the proportion of variation explained by the population regression model. Conclusion: Linear regression models require only two SPV for adequate estimation of regression coefficients, standard errors, and confidence intervals. [ABSTRACT FROM AUTHOR]

Background and Objectives: Comparing observed and expected distributions of baseline continuous variables in randomized controlled trials (RCTs) can be used to assess publication integrity. We explored whether baseline categorical variables could also be used., Methods: The observed and expected (binomial) distribution of all baseline categorical variables were compared in four sets of RCTs: two controls, and two with publication integrity concerns. We also compared baseline calculated and reported P-values., Results: The observed and expected distributions of baseline categorical variables were similar in the control datasets, both for frequency counts (and percentages) and for between-group differences in frequency counts. However, in both sets of RCTs with publication integrity concerns, about twice as many variables as expected had between-group differences in frequency counts of one or 2, and far fewer variables than expected had between-group differences of >4 (P < 0.001 for both datasets). Furthermore, about one in six reported P-values for baseline categorial variables differed by > 0.1 from the calculated P-value in trials with publication integrity concerns., Conclusion: Comparing the observed and expected distributions and reported and calculated P-values of baseline categorical variables may help in the assessment of publication integrity of a body of RCTs., (Copyright © 2022 Elsevier Inc. All rights reserved.)

Objectives: We aimed to review the extent to which analysis of randomized controlled trials (RCTs) of device interventions includes methods to handle compliance to the study intervention as described in the protocol., Study Design and Setting: We conducted a systematic review of the statistical methods used to handle compliance to a device intervention when estimating the effect of the device compared to another intervention in RCTs. We searched Embase, MEDLINE, PsychInfo, and the Cochrane Central Register of Controlled Trials. We sought to evaluate what methods were used and how using these methods impacted the estimate of the effect size., Results: One hundred fifty eight RCTs were identified for inclusion, of which only 21 (13%) described using a method to account for compliance to the device intervention, consisting of alternative analysis populations such as per-protocol, modified intention-to-treat, or as-treated, alongside a primary intention-to-treat analysis. No causal inference methods were used. Fourteen (9%) studies included compliance as a factor in the analysis and investigated its effect on outcomes., Conclusion: Although some studies consider methods to handle compliance, causal inference methods have not been well adopted in the analysis of device trials. An increased awareness of the applications of statistical methods to adjust for compliance is needed., (Copyright © 2022 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: To examine whether appropriate statistical methods were used in acupuncture randomized controlled trials (RCTs)., Study Design and Setting: We searched PubMed to identify acupuncture RCTs with continuous outcome as primary outcome published in the core clinical journals and complementary and alternative medicine (CAM) journals between January 2010 and December 2019 (10 years). We compared statistical characteristics of included trials published in core clinical journals and CAM journals., Results: We included 262 RCTs, including 46 published in core clinical journals and 216 in CAM journals. Of included RCTs, only 132 (50.4%) clearly predefined the primary outcome, 72 (27.5%) specified the use of intention to treat or modified intention to treat population for primary analysis. In the 167 trials reported missing participant data (MPD), 118 (70.7%) used suboptimal methods (e.g., complete case analysis) for dealing with MPD; 11 (6.6%) conducted sensitivity analysis regarding MPD. Among the 161 trials with repeated measures design, only 21 (13.0%) used advanced statistical models (e.g., mixed-effects models) for handling repeated-measure data in the primary analysis. In the 72 trials involving multiple acupuncturists, only 4 (5.6%) adjusted acupuncturist variable or considered the clustering by acupuncturist in analysis. Trials in core clinical journals were more likely to predefine primary outcome (78.3% vs. 44.4%, P < 0.001), use multiple imputations for handling MPD (40% vs. 1.5%, P < 0.001), and use statistically advanced methods for assessing treatment effect at a single time point (26.1% vs. 2%, P = 0.001)., Conclusion: The use of statistical methods among acupuncture RCTs is far from satisfactory. Our findings highlighted the need for researchers to carefully use the optimal statistical methods and for journal editors to strengthen the use of statistical methods., (Copyright © 2022 The Author(s). Published by Elsevier Inc. All rights reserved.)

Background and Objectives: As a result of the development of sophisticated techniques, such as multiple imputation, the interest in handling missing data in longitudinal studies has increased enormously in past years. Within the field of longitudinal data analysis, there is a current debate on whether it is necessary to use multiple imputations before performing a mixed-model analysis to analyze the longitudinal data. In the current study this necessity is evaluated. Study Design and Setting: The results of mixed-model analyses with and without multiple imputation were compared with each other. Four data sets with missing values were createddone data set with missing completely at random, two data sets with missing at random, and one data set with missing not at random). In all data sets, the relationship between a continuous outcome variable and two different covariates were analyzed: a time-independent dichotomous covariate and a time-dependent continuous covariate. Results: Although for all types of missing data, the results of the mixed-model analysis with or without multiple imputations were slightly different, they were not in favor of one of the two approaches. In addition, repeating the multiple imputations 100 times showed that the results of the mixed-model analysis with multiple imputation were quite unstable. Conclusion: It is not necessary to handle missing data using multiple imputations before performing a mixed-model analysis on longitudinal data. [ABSTRACT FROM AUTHOR]

Objectives: In the evaluation of cancer screening tests, cancer-free controls are often matched to cancer cases on factors such as sex and age. We assessed the potential merits and pitfalls of such matching using an example from colorectal cancer (CRC) screening. Study Design and Setting: We compared sex and age distribution of CRC cases and cancer-free people undergoing screening colonoscopy in Germany in 2006 and 2007. We assessed specificity by sex and age of two immunochemical fecal occult blood tests (iFOBTs) in a study among screening colonoscopy participants conducted in the same years, and we assessed the expected impact of matching by sex and age on the validity of specificity estimates at various cut points. Results: In the screening colonoscopy program, the proportion of men and mean age were 59.6% and 68.6 years among 10,324 CRC patients compared with 45.6% and 64.7 years, respectively, among 997,490 cancer-free participants. The specificity of the iFOBTs was higher among women than among men and decreased with age. Matching of cancer-free controls by age and sex would have led to the underestimation of specificity at all cut points assessed. Conclusion: In the evaluation of cancer screening tests, matching of controls may lead to biased estimates of specificity. [ABSTRACT FROM AUTHOR]

Abstract: Randomized controlled trials (RCTs) are the gold standard for evaluating new interventions. Different RCT designs apply depending on the patient population, clinical setting, and intervention being evaluated. A design that may help to generate evidence in some clinical areas where recruitment is a challenge is aggregated n-of-1 trials. N-of-1 trials are randomized, double-blind, and multiple crossover comparisons of an intervention and a control treatment. Methodologically robust n-of-1 trials provide an objective means of testing the effectiveness of treatments within individual participants. Aggregation of multiple cycle identically conducted n-of-1 trials yield a population estimate of effect, which potentially commensurate with that derived from other RCT designs. Trial participants contribute data for both intervention and control treatments creating matched data sets while using generally smaller sample sizes than conventional RCT trials. Careful choice of symptoms and medications are required for n-of-1 trials to be feasible. A validated and reliable outcome measure sensitive to change is still required. This article reviews the utility and limitations of aggregated n-of-1 trials to gather evidence in populations where conducting formal RCTs is difficult because of the low prevalence of the underlying condition or the clinical condition making recruitment and retention difficult. The article examines a proposed palliative care trial as a test case. [Copyright &y& Elsevier]

Abstract: Objective: Meta-analyses of continuous outcomes typically use mean differences (MDs) or standardized mean differences (SMDs) (MD in pooled standard deviation units). Ratio of means (RoM) is an alternative effect measure that performs comparably in simulation. We compared treatment effects and heterogeneity for RoM, MD, and SMD using empiric data. Study Design and Setting: From the Cochrane Database (2008, issue 1), we included systematic reviews reporting continuous outcomes, selected the meta-analysis with the most (and ≥five) trials, and calculated MD (where possible), SMD, and RoM. For each pair of effect measures, we compared P-values separately for treatment effect and heterogeneity and assessed asymmetry of discordant pairs (statistically significant result for only one of two measures). Results: Two hundred thirty-two of 5,053 reviews were included. Measures demonstrated similar treatment effects, with ≤6% discordant pairs and no asymmetry. A 0.5 SMD increase corresponded to 22 (95% confidence interval: 19, 24)% increase using RoM. There was less heterogeneity in RoM vs. MD (n =143, P =0.007), SMD vs. RoM (n =232, P =0.005), and SMD vs. MD (n =143, P =0.004). Comparing discordant pairs, fewer meta-analyses showed significant heterogeneity with SMD vs. RoM (P =0.04), consistent with the known bias of SMD. Conclusion: Empiric data from diverse meta-analyses demonstrate similar treatment effects and no large differences in heterogeneity of RoM compared with difference-based methods. [Copyright &y& Elsevier]

Abstract: Objective: Logistic regression models are frequently used in cohort studies to determine the association between treatment and dichotomous outcomes in the presence of confounding variables. In a logistic regression model, the association between exposure and outcome is measured using the odds ratio (OR). The OR can be difficult to interpret and only approximates the relative risk (RR) in certain restrictive settings. Several authors have suggested that for dichotomous outcomes, RRs, RR reductions, absolute risk reductions, and the number needed to treat (NNT) are more clinically meaningful measures of treatment effect. Study Design and Setting: We describe a method for deriving clinically meaningful measures of treatment effect from a logistic regression model. This method involves determining the probability of the outcome if each subject in the cohort was treated and if each subject was untreated. These probabilities are then averaged across the study cohort to determine the average probability of the outcome in the population if all subjects were treated and if they were untreated. Results: Risk differences, RRs, and NNTs were derived using a logistic regression model. Conclusions: Clinically meaningful measures of effect can be derived from a logistic regression model in a cohort study. These methods can also be used in randomized controlled trials when logistic regression is used to adjust for possible imbalance in prognostically important baseline covariates. [Copyright &y& Elsevier]

Objectives: To estimate the extent to which different methods of assessing response shift lead to different conclusions about its presence. Study Design and Setting: A reanalysis of a randomized clinical trial involving 190 persons poststroke to evaluate the effectiveness of a case management intervention aimed at assisting persons with stroke to make the transition from acute care to home. Results: Response shift was found to be nondifferential across groups and was therefore ruled out as an explanation for why the case management intervention showed no impact on patients' perceptions of their health. The results of a then-test did not show differential recalibration overtime. Factor analysis did not support reconceptualization response shift in either group but did find weak evidence for reprioritization response shift. An analysis of residuals indicated that approximately 50% of study subjects in each group experienced response shift that is probably close to reconceptualization. Conclusion: A framework for assessing response shift was proposed and investigators planning trials of interventions targeting patient-reported outcomes should build into the trial methods for response shift investigation. In trials of interventions likely to induce response shift, before concluding about intervention-induced change, response shift should be ruled out by using a combination of design and statistical approaches. [ABSTRACT FROM AUTHOR]

Background: In many epidemiologic longitudinal studies, the outcome variable has floor or ceiling effects. Although it is not correct, these variables are often treated as normally distributed continuous variables. Objectives: In this article, the performance of a relatively new statistical technique, longitudinal tobit analysis, is compared with a classical longitudinal data analysis technique (i.e., linear mixed models). Study Design and Setting: The analyses are performed on an example data set from rehabilitation research in which the outcome variable of interest (the Barthel index measured at on average 16.3 times) has typical floor and ceiling effects. For both the longitudinal tobit analysis and the linear mixed models an analysis with both a random intercept and a random slope were performed. Results: Based on model fit parameters, plots of the residuals and the mean of the squared residuals, the longitudinal tobit analysis with both a random intercept and a random slope performed best. In the tobit models, the estimation of the development over time revealed a steeper development compared with the linear mixed models. Conclusion: Although there are some computational difficulties, longitudinal tobit analysis provides a very nice solution for the longitudinal analysis of outcome variables with floor or ceiling effects. [ABSTRACT FROM AUTHOR]

Objectives: To assess what statistical methods are commonly used in high-impact clinical research and how they are presented in abstracts of articles published in high-impact medical journals.Study Design and Setting: A cross-sectional survey of abstracts of original articles published in July 2003 in four high-impact medical journals was conducted. The primary outcome was the distribution of statistical methods used in study results presented in the abstract of articles.Results: Seventy articles met inclusion criteria. One hundred twenty-five unique statistical method presentations were analyzed. Sixty-eight percent of statistical methods used summary statistics, and 27.2% used regression analysis. When summary statistics were used, clinical evidence was presented with a P-value or confidence interval (CI) in 51.8% of statistical methods compared to 72.5% when summary statistics were not used (P=0.0282). Clinical evidence was presented verbally in 7.1% of statistical methods when summary statistics were used and in 20.0% when summary statistics were not used (P=0.0323).Conclusions: Summary statistics are the most frequently used statistical method to generate high-impact clinical evidence presented in the abstract of a medical article. Evidence described by summary statistics is significantly associated with less frequent reporting of a P-value or CI, and less frequent verbal presentations. [ABSTRACT FROM AUTHOR]

Objective: Case-control studies of the effectiveness of a vaccine are useful to answer important questions, such as the effectiveness of a vaccine over time, that usually are not addressed by prelicensure clinical trials of the vaccine's efficacy. This report describes methodological issues related to design and analysis that were used to determine the effects of time since vaccination and age at the time of vaccination.Study Design and Setting: A matched case-control study of the effectiveness of varicella vaccine was conducted.Results: Sampling procedures and conditional logistic regression models including interaction terms are described.Conclusion: Use of these methods will allow investigators to assess the effects of a wide range of variables, such as time since vaccination and age at the time of vaccination, on the effectiveness of a vaccine. [ABSTRACT FROM AUTHOR]

Abstract: Objectives: To illustrate how multiple hypotheses testing can produce associations with no clinical plausibility. Study Design and Setting: We conducted a study of all 10,674,945 residents of Ontario aged between 18 and 100 years in 2000. Residents were randomly assigned to equally sized derivation and validation cohorts and classified according to their astrological sign. Using the derivation cohort, we searched through 223 of the most common diagnoses for hospitalization until we identified two for which subjects born under one astrological sign had a significantly higher probability of hospitalization compared to subjects born under the remaining signs combined (P <0.05). Results: We tested these 24 associations in the independent validation cohort. Residents born under Leo had a higher probability of gastrointestinal hemorrhage (P =0.0447), while Sagittarians had a higher probability of humerus fracture (P =0.0123) compared to all other signs combined. After adjusting the significance level to account for multiple comparisons, none of the identified associations remained significant in either the derivation or validation cohort. Conclusions: Our analyses illustrate how the testing of multiple, non-prespecified hypotheses increases the likelihood of detecting implausible associations. Our findings have important implications for the analysis and interpretation of clinical studies. [Copyright &y& Elsevier]

Abstract: Background and Objective: Period analysis provides more up-to-date estimates of cancer patient survival than traditional methods, but there is a trade-off between up-to-dateness and precision. Our objective was to compare the performance of period and complete analysis in terms of up-to-dateness and precision of survival estimates. Study Design and Setting: Five-year relative survival data actually observed for patients diagnosed with 1 of 20 common forms of cancer in Finland in 36 overlapping 5-year periods between 1958–1962 and 1993–1997 were compared with period estimates and various variants of complete estimates of 5-year relative survival potentially available during these periods. Results: At comparable levels of up-to-dateness, survival estimates from period analysis were more precise than survival estimates from complete analysis. At comparable levels of precision, period analysis provided more up-to-date survival estimates than did complete analysis. Conclusion: These results further encourage more widespread use of period analysis as a standard tool for up-to-date monitoring of cancer patient survival by population-based cancer registries. [Copyright &y& Elsevier]

Objective: Propensity score (PS) analyses attempt to control for confounding in nonexperimental studies by adjusting for the likelihood that a given patient is exposed. Such analyses have been proposed to address confounding by indication, but there is little empirical evidence that they achieve better control than conventional multivariate outcome modeling.Study Design and Methods: Using PubMed and Science Citation Index, we assessed the use of propensity scores over time and critically evaluated studies published through 2003.Results: Use of propensity scores increased from a total of 8 reports before 1998 to 71 in 2003. Most of the 177 published studies abstracted assessed medications (N=60) or surgical interventions (N=51), mainly in cardiology and cardiac surgery (N=90). Whether PS methods or conventional outcome models were used to control for confounding had little effect on results in those studies in which such comparison was possible. Only 9 of 69 studies (13%) had an effect estimate that differed by more than 20% from that obtained with a conventional outcome model in all PS analyses presented.Conclusions: Publication of results based on propensity score methods has increased dramatically, but there is little evidence that these methods yield substantially different estimates compared with conventional multivariable methods. [ABSTRACT FROM AUTHOR]

Abstract: Objective: To determine whether adjusting for confounder bias in observational studies using propensity scores gives different results than using traditional regression modeling. Methods: Medline and Embase were used to identify studies that described at least one association between an exposure and an outcome using both traditional regression and propensity score methods to control for confounding. From 43 studies, 78 exposure–outcome associations were found. Measures of the quality of propensity score implementation were determined. The statistical significance of each association using both analytical methods was compared. The odds or hazard ratios derived using both methods were compared quantitatively. Results: Statistical significance differed between regression and propensity score methods for only 8 of the associations (10%), κ=0.79 (95% CI=0.65–0.92). In all cases, the regression method gave a statistically significant association not observed with the propensity score method. The odds or hazard ratio derived using propensity scores was, on average, 6.4% closer to unity than that derived using traditional regression. Conclusions: Observational studies had similar results whether using traditional regression or propensity scores to adjust for confounding. Propensity scores gave slightly weaker associations; however, many of the reviewed studies did not implement propensity scores well. [Copyright &y& Elsevier]

Abstract: Objective: To compare the use of conventional statistical models with multilevel regression models in volume–outcome analyses of surgical procedures in an empirical case study. Study Design and Setting: Using conventional regression models and multilevel regression models, we estimated the effect of hospital volume and surgeon volume on 30-day mortality and length of postoperative hospital stay in persons who had an esophagectomy, pancreaticoduodenectomy, or major lung resection for cancer in Ontario, Canada, from 1994 to 1999. Results: The point estimates of volume–outcome associations were similar using either method; however, the 95% confidence intervals estimated by multilevel models were wider than those estimated by conventional models. A significant association between volume and mortality was identified in 2 of 18 (11%) comparisons using conventional analysis but in none of the 18 (0%) comparisons using multilevel analysis, and between volume and length of stay in 15 of 18 (83%) comparisons using conventional analysis and in 1 of 18 (6%) comparisons using multilevel analysis. Conclusion: Conventional and multilevel statistical models can yield substantially different results in the analysis of volume–outcome associations for surgical procedures. [Copyright &y& Elsevier]

Abstract: Background and objective: To review methods that seek to adjust for confounding in observational studies when assessing intended drug effects. Methods: We reviewed the statistical, economical and medical literature on the development, comparison and use of methods adjusting for confounding. Results: In addition to standard statistical techniques of (logistic) regression and Cox proportional hazards regression, alternative methods have been proposed to adjust for confounding in observational studies. A first group of methods focus on the main problem of nonrandomization by balancing treatment groups on observed covariates: selection, matching, stratification, multivariate confounder score, and propensity score methods, of which the latter can be combined with stratification or various matching methods. Another group of methods look for variables to be used like randomization in order to adjust also for unobserved covariates: instrumental variable methods, two-stage least squares, and grouped-treatment approach. Identifying these variables is difficult, however, and assumptions are strong. Sensitivity analyses are useful tools in assessing the robustness and plausibility of the estimated treatment effects to variations in assumptions about unmeasured confounders. Conclusion: In most studies regression-like techniques are routinely used for adjustment for confounding, although alternative methods are available. More complete empirical evaluations comparing these methods in different situations are needed. [Copyright &y& Elsevier]

Abstract: Background and objective: Clinical epidemiology and statistics skills of clinical researchers are often limited. We assessed methodological skills of medical researchers and identified factors associated with higher skill levels. Methods: In a cross-sectional mail survey at two Swiss teaching hospitals, participants (N=409) rated their ability to perform 26 research-related activities, such as identifying the research question, selecting a study design, computing the required sample size, performing data analysis, and reporting results. Results: The proportion of respondents who were able to perform a specific activity was 33.2% on average, ranging from 1.5% for “numerical statistics (bootstrap, simulation, cross-validation,…)” to 76.0% for “oral presentation of results.” The overall skill level (expressed as a percentage of the 26 activities) was associated with principal investigator experience (+8.7%), greater percentage of time devoted to research (+12.4% for near full-time versus no time commitment), years of research experience (+17.6% for 15–40 years versus 0 years), past number of clinical research projects (+18.0% for 15–230 projects versus 0–1 projects), and hours of formal methodological training (+32.6% for 200–1200 hours versus 0–9 hours). Conclusion: Self-reported methodological skills were generally modest. The most important covariates of skill levels were current time commitment to research, past experience, and formal training. [Copyright &y& Elsevier]

Conventional graphs often include error bars around the group means. Regardless of what these bars depict, they are uninformative as to whether a difference between the groups is statistically significant.This article suggests plotting the null bar or null zone: that is, the range or area in which the means of the two groups fall if the null hypothesis of no difference cannot be rejected. The bar and zone are simply derived from the confidence interval around the difference, obtained from the t-test. [Copyright &y& Elsevier]

Objectives: Interrupted Time Series (ITS) are a type of nonrandomized design commonly used to evaluate public health policy interventions, and the impact of exposures, at the population level. Meta-analysis may be used to combine results from ITS across studies (in the context of systematic reviews) or across sites within the same study. We aimed to examine the statistical approaches, methods, and completeness of reporting in reviews that meta-analyze results from ITS., Study Design and Settings: Eight electronic databases were searched to identify reviews (published 2000-2019) that meta-analyzed at least two ITS. Characteristics of the included reviews, the statistical methods used to analyze the ITS and meta-analyze their results, effect measures, and risk of bias assessment tools were extracted., Results: Of the 4213 identified records, 54 reviews were included. Nearly all reviews (94%) used two-stage meta-analysis, most commonly fitting a random effects model (69%). Among the 41 reviews that re-analyzed the ITS, linear regression (39%) and ARIMA (20%) were most commonly used; 38% adjusted for autocorrelation. The most common effect measure meta-analyzed was an immediate level-change (46/54). Reporting of the statistical methods and ITS characteristics was often incomplete., Conclusion: Improvement is needed in the conduct and reporting of reviews that meta-analyze results from ITS., (Copyright © 2022 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Systematic understanding is lacking regarding how current trials handle repeated measure data and the extent to which appropriate statistical methods are used for such data set. This study investigated the current practice of analyzing the repeated measure data among randomized controlled trials (RCTs)., Study Design and Setting: We searched the Core Clinical Journals indexed in PubMed for RCTs published in 2019 and included a continuous primary outcome with repeated measures. We randomly sampled RCTs from the eligible trials. Team of methods trained investigators screened studies for eligibility and collected data using the pilot-tested, standardized questionnaires. We thoroughly documented statistical analyses of the continuous primary outcome with repeated measures and particularly recorded how statistically advanced methods were used to handle these repeated measures., Results: In total, 200 trials were included. Of these trials, the mean number of repeated measures for the continuous primary outcome was 5.46 (SD = 3.4); 58 (29.0%) trials did not specify the time point of primary outcome in the method; 113 (56.5%) trials did not use statistically advanced methods for handling repeated measure data in the primary analyses. Among187 trials included the baseline values, 88 (47.1%) trials did not adjust for outcome value at baseline. Among 87 trials using statistically advanced methods, 49 (56.3%) did not specify correlation structure for model., Conclusions: The statistical analyses of repeatedly measured continuous outcomes in RCTs need substantial improvements. Careful planning of the primary outcome and the use of statistically advanced methods for analyzing data are warranted., (Copyright © 2021. Published by Elsevier Inc.)

Objective: To examine the proposition that identical summary statistics (mean and/or SD) in different randomized controlled trials (RCT) or clinical cohorts can be explained by common or homogeneous source populations., Study Design: We estimated the probability of identical summary data in studies with high proportions of identical summary statistics, in simulations, and in control datasets., Results: The probability of both an identical mean and an identical SD for a variable in separate RCT is low (<~3%), unless the variable is rounded to 1 significant figure. In two RCT with identical summary statistics for 16 of 39 shared variables, simulations indicated the probability of the observed matches was <1 in 100,000. In 34 clinical cohorts with publication integrity concerns, the proportion of summary statistics from variables reported in ≥10 studies that were identical in ≥2 cohorts were high (42% for means, 52% for SD, and 29% for both), and improbable based on simulations and comparisons to control datasets., Conclusions: The likelihood of multiple identical summary statistics within an individual RCT or across a body of RCT or cohort studies by the same research group is low, especially when both the mean, and the SD are identical, unless the variables are rounded to 1 significant figure., (Copyright © 2021. Published by Elsevier Inc.)

Objectives: Comparing observed and expected distributions of categorical outcome variables in randomized controlled trials (RCTs) has been previously used to assess publication integrity. We applied this technique to withdrawals from RCTs., Study Design and Setting: We compared the observed distribution of withdrawals with the expected binomial distribution in six sets of RCTs: four control sets and two sets with concerns about their publication integrity., Results: In the control data sets (n = 13, 115, 71, and 36 trials, respectively), the observed distributions of withdrawals were consistent with the expected distributions, both for the numbers of withdrawals per trial arm and for the differences in withdrawals between trial arms in two-arm RCTs. In contrast, in both sets of RCTs with concerns regarding publication integrity (n = 151 and 35 trials, respectively), there were striking differences between the observed and expected distributions of trial withdrawals. Two-arm RCTs from the two sets with publication integrity concerns were 2.6 (95% confidence interval 2.0-3.3) times more likely to have a difference of 0 or 1 withdrawals between trial arms than control RCTs (P < 0.001). Simulating a 50% higher rate of withdrawals in active treatment arms in the largest set of control RCTs still produced an observed distribution of withdrawals per trial arm consistent with the expected distribution., Conclusion: Comparing the observed and expected distribution of trial withdrawals may be a useful technique when considering publication integrity of a body of RCTs., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Objectives: The objective of this study was to examine how rescue medication is defined, reported, and accounted for in randomized controlled trials (RCTs) in eczema and asthma populations., Study Design and Setting: This is a systematic review of phase II/III RCTs evaluating monoclonal antibodies for treating chronic eczema or asthma. A search of EMBASE, MEDLINE, and the Cochrane Central Register of Controlled Trials was conducted to identify eligible RCTs., Results: Sixty published RCTs were identified, of which 60 (100%) allowed use of rescue medication but only 28 (47%) reported its use. Twenty-seven (45%) articles summarized rescue use by arm, with an average of 25% (95% CI (17%, 36%)) greater use in the placebo arm. Nine (15%) trials undertook an analysis that adjusted the primary treatment effect estimate for rescue medication use, but 8 of these used a suboptimal approach using single imputation, including 4 which used "last observation carried forward" after setting postrescue data to missing., Conclusion: Rescue medication use in eczema and asthma trials evaluating monoclonal antibodies is often permitted, but not routinely reported. There is evidence of imbalance in rescue use between arms, but few articles attempted to estimate a rescue-adjusted treatment effect. In trials that did, the methods used were suboptimal which could introduce bias., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Interrupted time series (ITS) designs are frequently used in public health to examine whether an intervention or exposure has influenced health outcomes. Few reviews have been undertaken to examine the design characteristics, statistical methods, and completeness of reporting of published ITS studies., Study Design and Setting: We used stratified random sampling to identify 200 ITS studies that evaluated public health interventions or exposures from PubMed (2013-2017). Study characteristics, details of statistical models and estimation methods used, effect metrics, and parameter estimates were extracted. From the 200 studies, 230 time series were examined., Results: Common statistical methods used were linear regression (31%, 72/230) and autoregressive integrated moving average (19%, 43/230). In 17% (40/230) of the series, we could not determine the statistical method used. Autocorrelation was acknowledged in 63% (145/230) of the series. An estimate of the autocorrelation coefficient was given for only 1% of the series (3/230). Measures of precision were reported for 63% of effect measures (541/852)., Conclusion: Many aspects of the design, methods, analysis, and reporting of ITS studies can be improved, particularly description of the statistical methods and approaches to adjust for and estimate autocorrelation. More guidance on the conduct and reporting of ITS studies is needed to improve this study design., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Objective: Article full texts are often inaccessible via the standard search engines of biomedical literature, such as PubMed and Embase, which are commonly used for systematic reviews. Excluding the full-text bodies from a literature search may result in a small or selective subset of articles being included in the review because of the limited information that is available in only title, abstract, and keywords. This article describes a comparison of search strategies based on a systematic literature review of all articles published in 5 top-ranked epidemiology journals between 2000 and 2017., Study Design and Setting: Based on a text-mining approach, we studied how nine different methodological topics were mentioned across text fields (title, abstract, keywords, and text body). The following methodological topics were studied: propensity score methods, inverse probability weighting, marginal structural modeling, multiple imputation, Kaplan-Meier estimation, number needed to treat, measurement error, randomized controlled trial, and latent class analysis., Results: In total, 31,641 Hypertext Markup Language (HTML) files were downloaded from the journals' websites. For all methodological topics and journals, at most 50% of articles with a mention of a topic in the text body also mentioned the topic in the title, abstract, or keywords. For several topics, a gradual decrease over calendar time was observed of reporting in the title, abstract, or keywords., Conclusion: Literature searches based on title, abstract, and keywords alone may not be sufficiently sensitive for studies of epidemiological research practice. This study also illustrates the potential value of full-text literature searches, provided there is accessibility of full-text bodies for literature searches., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: The objective of the study was to conduct a scoping review of the published literature on methods used to combine randomized and nonrandomized evidence (NRE) in network meta-analyses (NMAs) and their respective characteristics., Study Design and Setting: We conducted a scoping review using a list of NMAs which incorporated NRE that were identified from a previous review. All NMAs that included NRE in the analysis of main outcomes or sensitivity analyses were eligible for inclusion. Two reviewers independently screened studies for inclusion and performed data abstraction. Data analysis involved quantitative (frequencies and percentages) and qualitative (narrative synthesis) methods., Results: A total of 23 NMAs met the predefined inclusion criteria, of which 74% (n = 17) used naïve pooling, 0% used NRE as informative priors, 9% (n = 2) used the 3-level Bayesian hierarchical model, 9% (n = 2) used all methods, and 9% (n = 2) used other methods. Most NMAs were supplemented with additional analyses to investigate the effect estimates when only randomized evidence was included., Conclusion: Although most studies provided justification for the inclusion of NRE, transparent reporting of the method used to combine randomized evidence and NRE was unclear in most published networks. Most NMAs used naïve pooling for combining randomized evidence and NRE., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Objective: Study designs involving self-controlled or exposure-matched samples are commonly used to monitor postmarket vaccine and drug safety, and they use a subset of the available larger cohort. This article overviews group sequential methods designed for observational data safety monitoring that use the whole exposed and unexposed cohorts by implementing regression adjustment or weighting to control confounding., Methods: We summarize what is known about the performance of "whole cohort" methods in multisite health plan data networks such as the Sentinel System of the Food and Drug Administration, where outcomes are rare, individual-level patient data cannot be pooled across sites, site heterogeneity is large, and data are dynamically updated over time., Results: Group sequential estimation and testing methods that use regression or weighting can flexibly handle electronic health care data's unpredictability, including an uncertain rate of new product uptake, variable composition of the population over time, and data changes due to dynamic administrative updates. Regression and weighting methods generally have higher power, faster signal detection, and fewer practical challenges compared with some design-based confounder adjustment methods., Conclusion: Group sequential regression adjustment and weighting approaches are feasible and underused in practice. They leverage more information than designs that involved sampling and increase power to detect rare adverse effects without increasing bias., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Objective: Comparing observed and expected distributions of baseline variables in randomized controlled trials (RCTs) has been used to investigate possible research misconduct, although the validity of this approach has been questioned. We explored this technique and introduced a novel metric to compare P values from baseline variables between treatment arms., Study Design and Setting: We compared observed with expected distributions of baseline P values using a one-way chi-square test and by comparing the area under the curve (AUC) of the cumulative distribution function in 13 RCTs conducted by our group, two groups of RCTs known to contain fabricated data, and simulations., Results: In our 13 RCTs, the distribution of P values from baseline continuous variables was consistent with the expected theoretical uniform distribution (P = 0.19, difference from expected AUC -0.03, 95% confidence interval [-0.04, 0.04]). For categorical variables, the P value distribution was not uniform. The distributions of P values from RCTs with fabricated data were highly unusual and not consistent with the uniform distribution for continuous variables, nor with the expected distribution for categorical variables, nor with the distribution of P values in genuine RCTs., Conclusions: Assessing baseline P values in groups of RCTs can identify highly unusual distributions that might raise or reinforce concerns about randomization and data integrity., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Objectives: To investigate whether comparing observed with expected P-value distributions for baseline continuous variables in randomized controlled trials (RCTs) might be limited by randomization methods, normality and correlation of variables, or calculation of P-values from rounded summary statistics., Study Design and Setting: We assessed how each factor affects differences from expected for P-value distributions and area under the curve of the cumulative distribution function (AUC-CDF) of baseline P-values in 13 RCTs and in simulations., Results: The P-value distributions and AUC-CDF for variables with possible non-normal distribution and in simulations using eight different randomization methods were consistent with the theoretical uniform distribution and AUC-CDF, respectively, although stratification and minimization produced smaller-than-expected proportions of P-values <0.10. Seventy-seven percentage of 3,813 pairwise correlations between baseline variables in the 13 individual RCTs were between -0.2 and 0.2. P-value distribution and AUC-CDF remained consistent with the uniform distribution in simulations with incrementally increasing correlation strength. The P-value distributions calculated from rounded summary statistics were not uniform, but expected distributions could be empirically generated., Conclusions: Randomization methods, non-normality, and strength of correlation of baseline variables did not have important effects on baseline P-value distribution or AUC-CDF, but baseline P-values calculated from rounded summary statistics are non-uniformly distributed., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Objective: To undertake a methodological review of statistical methods used in randomized controlled trials (RCTs) for handling intervention nonadherence., Study Design and Setting: Bibliographic databases were searched using predefined search terms., Results: A substantive number of identified studies (56%) were excluded as they only used naive per protocol analysis for handling nonadherence. Our review included 58 articles published between 1991 and 2015. A total of 88 methodological applications were made by these studies. The two most used methods were complier average causal effect (56%) and instrumental variable (23%) predominantly with the use of maximum likelihood (ML) estimators. These alternative applications typically produced treatment effects greater than the intention-to-treat effect but as their standard errors were larger there was no statistical difference between the methods., Conclusion: A substantive proportion of RCTs rely on naive per protocol for handling nonadherence. Recent years have seen an increasing number of applications of more appropriate statistical methods, in particular complier average causal effect and instrumental variable methods. However, these later methods rely on strong underlying assumptions that may be vulnerable to violation. More empirical studies are needed that directly compare the usability and performance of different statistical methods for nonadherence in RCTs., (Copyright © 2018. Published by Elsevier Inc.)

Objective: To conduct (1) a systematic survey of the reporting quality of simulation studies dealing with how to handle missing participant data (MPD) in randomized control trials and (2) summarize the findings of these studies., Study Design and Setting: We included simulation studies comparing statistical methods dealing with continuous MPD in randomized controlled trials addressing bias, precision, coverage, accuracy, power, type-I error, and overall ranking. For the reporting of simulation studies, we adapted previously developed criteria for reporting quality and applied them to eligible studies., Results: Of 16,446 identified citations, the 60 eligible generally had important limitations in reporting, particularly in reporting simulation procedures. Of the 60 studies, 47 addressed ignorable and 32 addressed nonignorable data. For ignorable missing data, mixed model was most frequently the best on overall ranking (9 times best, 34.6% of times tested) and bias (10, 55.6%). Multiple imputation was also performed well. For nonignorable data, mixed model was most frequently the best on overall ranking (7, 46.7%) and bias (8, 57.1%). Mixed model performance varied on other criteria. Last observation carried forward (LOCF) was very seldom the best performing, and for nonignorable MPD frequently the worst., Conclusion: Simulation studies addressing methods to deal with MPD suffered from serious limitations. The mixed model approach was superior to other methods in terms of overall performance and bias. LOCF performed worst., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Objective: To systematically review methodological articles which focus on nonpublication of studies and to describe methods of detecting and/or quantifying and/or adjusting for dissemination in meta-analyses. To evaluate whether the methods have been applied to an empirical data set for which one can be reasonably confident that all studies conducted have been included., Study Design and Setting: We systematically searched Medline, the Cochrane Library, and Web of Science, for methodological articles that describe at least one method of detecting and/or quantifying and/or adjusting for dissemination bias in meta-analyses., Results: The literature search retrieved 2,224 records, of which we finally included 150 full-text articles. A great variety of methods to detect, quantify, or adjust for dissemination bias were described. Methods included graphical methods mainly based on funnel plot approaches, statistical methods, such as regression tests, selection models, sensitivity analyses, and a great number of more recent statistical approaches. Only few methods have been validated in empirical evaluations using unpublished studies obtained from regulators (Food and Drug Administration, European Medicines Agency)., Conclusion: We present an overview of existing methods to detect, quantify, or adjust for dissemination bias. It remains difficult to advise which method should be used as they are all limited and their validity has rarely been assessed. Therefore, a thorough literature search remains crucial in systematic reviews, and further steps to increase the availability of all research results need to be taken., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objective: To describe trends in the use of statistical and epidemiological methods in the medical literature over the past 2 decades., Study Design and Setting: We obtained all 1,028,786 articles from the PubMed Central Open-Access archive (retrieved May 9, 2015). We focused on 113,450 medical research articles. A Delphi panel identified 177 statistical/epidemiological methods pertinent to clinical researchers. We used a text-mining approach to determine if a specific statistical/epidemiological method was encountered in a given article. We report the proportion of articles using a specific method for the entire cross-sectional sample and also stratified into three blocks of time (1995-2005; 2006-2010; 2011-2015)., Results: Numeric descriptive statistics were commonplace (96.4% articles). Other frequently encountered methods groups included statistical inferential concepts (52.9% articles), epidemiological measures of association (53.5% articles) methods for diagnostic/classification accuracy (40.1% articles), hypothesis testing (28.8% articles), ANOVA (23.2% articles), and regression (22.6% articles). We observed relative percent increases in the use of: regression (103.0%), missing data methods (217.9%), survival analysis (147.6%), and correlated data analysis (192.2%)., Conclusions: This study identified commonly encountered and emergent methods used to investigate medical research problems. Clinical researchers must be aware of the methodological landscape in their field, as statistical/epidemiological methods underpin research claims., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Objectives: Within epidemiology, a stepped wedge trial design (i.e., a one-way crossover trial in which several arms start the intervention at different time points) is increasingly popular as an alternative to a classical cluster randomized controlled trial. Despite this increasing popularity, there is a huge variation in the methods used to analyze data from a stepped wedge trial design., Study Design and Setting: Four linear mixed models were used to analyze data from a stepped wedge trial design on two example data sets. The four methods were chosen because they have been (frequently) used in practice. Method 1 compares all the intervention measurements with the control measurements. Method 2 treats the intervention variable as a time-independent categorical variable comparing the different arms with each other. In method 3, the intervention variable is a time-dependent categorical variable comparing groups with different number of intervention measurements, whereas in method 4, the changes in the outcome variable between subsequent measurements are analyzed., Results: Regarding the results in the first example data set, methods 1 and 3 showed a strong positive intervention effect, which disappeared after adjusting for time. Method 2 showed an inverse intervention effect, whereas method 4 did not show a significant effect at all. In the second example data set, the results were the opposite. Both methods 2 and 4 showed significant intervention effects, whereas the other two methods did not. For method 4, the intervention effect attenuated after adjustment for time., Conclusion: Different methods to analyze data from a stepped wedge trial design reveal different aspects of a possible intervention effect. The choice of a method partly depends on the type of the intervention and the possible time-dependent effect of the intervention. Furthermore, it is advised to combine the results of the different methods to obtain an interpretable overall result., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: Propensity score (PS) and instrumental variable (IV) are analytical techniques used to adjust for confounding in observational research. More and more, they seem to be used simultaneously in studies evaluating health interventions. The present review aimed to analyze the agreement between PS and IV results in medical research published to date., Study Design and Setting: Review of all published observational studies that evaluated a clinical intervention using simultaneously PS and IV analyses, as identified in MEDLINE and Web of Science., Results: Thirty-seven studies, most of them published during the previous 5 years, reported 55 comparisons between results from PS and IV analyses. There was a slight/fair agreement between the methods [Cohen's kappa coefficient = 0.21 (95% confidence interval: 0.00, 0.41)]. In 23 cases (42%), results were nonsignificant for one method and significant for the other, and IV analysis results were nonsignificant in most situations (87%)., Conclusion: Discrepancies are frequent between PS and IV analyses and can be interpreted in various ways. This suggests that researchers should carefully consider their analytical choices, and readers should be cautious when interpreting results, until further studies clarify the respective roles of the two methods in observational comparative effectiveness research., (Copyright © 2015 Elsevier Inc. All rights reserved.)