Search

Objectives We assessed disagreements between nonrandomized controlled studies based on real-world data (NRCS-RWDs) and randomized controlled trials (RCTs). Study Design and Setting We systematically searched for studies that compared treatment effect estimates from NRCS-RWDs and RCTs on the same clinical question. We assessed the potential difference between NRCS-RWDs and RCTs related to internal and external validity. We calculated various meta-epidemiological measures to assess agreement. In case of disagreements, we tried to identify the probable causes of disagreements. Results We included 12 studies comparing 15 treatment effect estimates of NRCS-RWDs and RCTs. There were many potential causes of disagreement. Ninety-five percent confidence intervals overlapped for 12 of 15 treatment effect estimates. Our analysis on predicted vs. observed overlap showed that there were no more disagreements than expected by chance. We observed only two substantial differences between the 15 treatment effect estimates. In both cases, we identified risk of bias in the NRCS-RWDs as the most probable cause of disagreement. Conclusion Our findings suggest that there are clinical questions where the difference in risk of bias between a well-conducted NRCS-RWD and an RCT is negligible. In our analysis, threats to external validity appeared to have no or only a weak impact on the disagreements of treatment effect estimates.

Objectives The aim of the study was to identify the validity of effect estimates for serious rare adverse events in clinical study reports of antidepressants trials, across different meta-analysis methods. Study Design and Setting Four serious rare adverse events (all-cause mortality, suicidality, aggressive behavior, and akathisia) were meta-analyzed using different methods. The Yusuf-Peto odds ratio ignores studies with no events and was compared with the alternative approaches of generalized linear mixed models (GLMMs), conditional logistic regression, a Bayesian approach using Markov Chain Monte Carlo (MCMC), and a beta-binomial regression model. Results The estimates for the four outcomes did not change substantially across the different methods; the Yusuf-Peto method underestimated the treatment harm and overestimated its precision, especially when the estimated odds ratio deviated greatly from 1. For example, the odds ratio for suicidality for children and adolescents was 2.39 (95% confidence interval = 1.32–4.33), using the Yusuf-Peto method but increased to 2.64 (1.33–5.26) using conditional logistic regression, to 2.69 (1.19–6.09) using beta-binomial, to 2.73 (1.37–5.42) using the GLMM, and finally to 2.87 (1.42–5.98) using the MCMC approach. Conclusion The method used for meta-analysis of rare events data influences the estimates obtained, and the exclusion of double-zero event studies can give misleading results. To ensure reduction of bias and erroneous inferences, sensitivity analyses should be performed using different methods instead of the Yusuf-Peto approach, in particular the beta-binomial method, which was shown to be superior through a simulation study.

Objectives The propensity score (PS) method is increasingly used to assess treatment effects in nonrandomized trials. Although there are several methods to use the PS for analysis, matching treated and untreated patients by the PS is recommended by most researchers among other reasons because this allows assessing covariate balance before and after matching. Although the standardized difference is commonly applied to compute a measure of balance, it has two deficiencies: its distribution depends on the sample size and one cannot compare standardized differences for baseline covariates on different scales, that is, continuous, binary, ordinal, or nominal covariates. Study Design and Setting We introduce the z -difference to measure covariate balance in matched PS analyses and illustrate it by a recent matched PS analysis from cardiac surgery. Results The z -difference is simple to calculate, can be used with second moments for continuous covariates, and in most cases can also be computed from published data. Its full advantage emerges after displaying z -differences in a Q – Q plot, which allows balance comparisons with respect to (1) a randomized trial and (2) a perfectly matched PS analysis in the sense of Rubin and Thomas. Conclusion The z -difference can be used to measure covariate balance in matched PS analyses.

Objective Analyses comparing randomized to nonrandomized clinical trials suffer from the fact that the study populations are usually different. We aimed for a comparison of randomized clinical trials (RCTs) and propensity score (PS) analyses in similar populations. Study Design and Setting In a systematic review, we “meta-matched” RCTs and PS analyses that compared the off- and the on-pump technique in coronary artery bypass grafting. “Meta-confounders” were summarized in a “meta-propensity score” and were used for “meta-matching.” We compared treatment effects between RCTs and PS analyses for 10 previously defined binary clinical outcomes in this “meta-matched” population as differences in “meta-odds ratios.” Results For all clinical outcomes, the estimated differences in “meta-odds ratios” were below an absolute value of 0.15, all confidence intervals included the null. Conclusions In our example, treatment effects of off-pump versus on-pump surgery from RCTs and PS analyses were very similar in a “meta-matched” population of studies, indicating that only a small remaining bias is present in PS analyses.

Objective The estimation of the number needed to be exposed (NNE) with adjustment for covariates can be performed by inverting the corresponding adjusted risk difference. The latter can be estimated by several approaches based on binomial and Poisson regression with or without constraints. A novel proposal is given by logistic regression with average risk difference (LR-ARD) estimation. Finally, the use of ordinary linear regression and unadjusted estimation can be considered. Study Design and Setting LR-ARD is compared with alternative approaches regarding bias, precision, and coverage probability by means of an extensive simulation study. Results LR-ARD was found to be superior compared with the other approaches. In the case of balanced covariates and large sample sizes, unadjusted estimation and ordinary linear regression can also be used. In general, however, LR-ARD seems to be the most appropriate approach to estimate adjusted risk differences and NNEs. Conclusions To estimate risk differences and NNEs with adjustment for covariates, the LR-ARD approach should be used.