Search

Objectives: To assess the current practice of propensity score (PS) analysis in the medical literature, particularly the assessment and reporting of balance on confounders. Study Design and Setting: A PubMed search identified studies using PS methods from December 2011 through May 2012. For each article included in the review, information was extracted on important aspects of the PS such as the type of PS method used, variable selection for PS model, and assessment of balance. Results: Among 296 articles that were included in the review, variable selection for PS model was explicitly reported in 102 studies (34.4%). Covariate balance was checked and reported in 177 studies (59.8%). P-values were the most commonly used statistical tools to report balance (125 of 177, 70.6%). The standardized difference and graphical displays were reported in 45 (25.4%) and 11 (6.2%) articles, respectively. Matching on the PS was the most commonly used approach to control for confounding (68.9%), followed by PS adjustment (20.9%), PS stratification (13.9%), and inverse probability of treatment weighting (IPTW, 7.1%). Balance was more often checked in articles using PS matching and IPTW, 70.6% and 71.4%, respectively. Conclusion: The execution and reporting of covariate selection and assessment of balance is far from optimal. Recommendations on reporting of PS analysis are provided to allow better appraisal of the validity of PS-based studies. [ABSTRACT FROM AUTHOR]

Objective: To give a comprehensive comparison of the performance of commonly applied interaction tests. Methods: A literature review and simulation study was performed evaluating interaction tests on the odds ratio (OR) or the risk difference (RD) scales: Cochran Q (Q), Breslow-Day (BD), Tarone, unconditional score, likelihood ratio (LR), Wald, and relative excess risk due to interaction (RERI)-based tests. Results: Review results agreed with results from our simulation study, which showed that on the OR scale, in small sample sizes (eg, number of subjects ≤ 250) the type 1 error rates of the LR test was 0.10; the BD and Tarone tests showed results around 0.05. On the RD scale, the LR and RERI tests had error rates around 0.05. On both scales, tests did not differ regarding power. When exposure prevented the outcome RERI-based tests were relatively underpowered (eg, N = 100; RERI power 5 5% vs. Wald power = 18%). With increasing sample size, difference decreased. Conclusion: In small samples, interaction tests differed. On the OR scale, the Tarone and BD tests are recommended. On the RD scale, the LR and RERI-based tests performed best. However, RERI-based tests are underpowered compared with other tests, when exposure prevents the outcome, and sample size is limited. [ABSTRACT FROM AUTHOR]

Objective: To determine the comparability of subgroup-specific and interaction effects (differences between subgroups) between different study designs. Study Design and Setting: We compared effects of interventions based on observational studies, randomized clinical trials (RCTs), and individual patient data meta-analyses (IPDMAs) of RCTs (reference) on three clinical topics: (1) mammography screening and breast cancer mortality, (2) coronary artery bypass surgery (CABG) and all-cause mortality, and (3) statins and incidence of major coronary events. Main, subgroup-specific, and interaction effects were compared. Results: Main and subgroup-specific effects were comparable with respect to the direction of the effects. Differences in the magnitude of subgroup-specific effects in observational studies yielded different interactions compared with those in IPDMA. In the mammography example, the ratio of risk ratios (RRR) (i.e., interaction effect) among observational studies was 1.46 [95% confidence interval (CI): 1.09, 1.96] compared with an IPDMA effect of 1.10 (95% CI: 0.89, 1.37). For the CABG studies, the observational RRR was 1.03 (95% CI: 0.84, 1.26), whereas in the IPDMA, this was 1.40 (95% CI: 1.08, 1.1.81). Finally, in the statin example, the RRR was 1.35 (95% CI: 1.13, 1.61) and 0.90 (95% CI: 0.84, 0.97) for observational studies and IPDMA, respectively. Conclusion: Main and subgroup-specific effects based on observational data were similar to main and subgroup-specific effects in IPDMAs based on RCTs, yet interactions differed. [ABSTRACT FROM AUTHOR]

Objective: This study aimed to validate trial patient eligibility screening and baseline data collection using text-mining in electronic healthcare records (EHRs), comparing the results to those of an international trial., Study Design and Setting: In three medical centers with different EHR vendors, EHR-based text-mining was used to automatically screen patients for trial eligibility and extract baseline data on nineteen characteristics. First, the yield of screening with automated EHR text-mining search was compared with manual screening by research personnel. Second, the accuracy of extracted baseline data by EHR text mining was compared to manual data entry by research personnel., Results: Of the 92,466 patients visiting the out-patient cardiology departments, 568 (0.6%) were enrolled in the trial during its recruitment period using manual screening methods. Automated EHR data screening of all patients showed that the number of patients needed to screen could be reduced by 73,863 (79.9%). The remaining 18,603 (20.1%) contained 458 of the actual participants (82.4% of participants). In trial participants, automated EHR text-mining missed a median of 2.8% (Interquartile range [IQR] across all variables 0.4-8.5%) of all data points compared to manually collected data. The overall accuracy of automatically extracted data was 88.0% (IQR 84.7-92.8%)., Conclusion: Automatically extracting data from EHRs using text-mining can be used to identify trial participants and to collect baseline information., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Epidemiologic studies often suffer from incomplete data, measurement error (or misclassification), and confounding. Each of these can cause bias and imprecision in estimates of exposure-outcome relations. We describe and compare statistical approaches that aim to control all three sources of bias simultaneously., Study Design and Setting: We illustrate four statistical approaches that address all three sources of bias, namely, multiple imputation for missing data and measurement error, multiple imputation combined with regression calibration, full information maximum likelihood within a structural equation modeling framework, and a Bayesian model. In a simulation study, we assess the performance of the four approaches compared with more commonly used approaches that do not account for measurement error, missing values, or confounding., Results: The results demonstrate that the four approaches consistently outperform the alternative approaches on all performance metrics (bias, mean squared error, and confidence interval coverage). Even in simulated data of 100 subjects, these approaches perform well., Conclusion: There can be a large benefit of addressing measurement error, missing values, and confounding to improve the estimation of exposure-outcome relations, even when the available sample size is relatively small., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Objective: Article full texts are often inaccessible via the standard search engines of biomedical literature, such as PubMed and Embase, which are commonly used for systematic reviews. Excluding the full-text bodies from a literature search may result in a small or selective subset of articles being included in the review because of the limited information that is available in only title, abstract, and keywords. This article describes a comparison of search strategies based on a systematic literature review of all articles published in 5 top-ranked epidemiology journals between 2000 and 2017., Study Design and Setting: Based on a text-mining approach, we studied how nine different methodological topics were mentioned across text fields (title, abstract, keywords, and text body). The following methodological topics were studied: propensity score methods, inverse probability weighting, marginal structural modeling, multiple imputation, Kaplan-Meier estimation, number needed to treat, measurement error, randomized controlled trial, and latent class analysis., Results: In total, 31,641 Hypertext Markup Language (HTML) files were downloaded from the journals' websites. For all methodological topics and journals, at most 50% of articles with a mention of a topic in the text body also mentioned the topic in the title, abstract, or keywords. For several topics, a gradual decrease over calendar time was observed of reporting in the title, abstract, or keywords., Conclusion: Literature searches based on title, abstract, and keywords alone may not be sufficiently sensitive for studies of epidemiological research practice. This study also illustrates the potential value of full-text literature searches, provided there is accessibility of full-text bodies for literature searches., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols., Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy., Results: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in C-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively., Conclusion: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting., (Copyright © 2019 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: The Trials within Cohorts (TwiCs) design is an alternative for pragmatic randomized controlled trials (RCTs) and might overcome disadvantages such as difficult recruitment, dropout after randomization to control, and contamination. We investigated the applicability of the TwiCs design in an exercise oncology study regarding the recruitment process, representativeness of the study sample, contamination, participation, and dropout., Methods: The Utrecht cohort for Multiple BREast cancer intervention studies and Long-term evaLuAtion (UMBRELLA) Fit TwiCs evaluates an exercise intervention in inactive breast cancer patients. Eligible patients participating in the prospective UMBRELLA were identified and randomized. Patients randomized to the intervention (n = 130) were offered the intervention, whereas controls (n = 130) were not informed., Results: Fifty-two percent (n = 68) accepted the intervention. Because this rate was lower than expected, a larger sample size was required than initially estimated (n = 166). However, recruitment of 260 patients was still completed by one researcher within 30 months. Unselective eligibility screening and randomization before invitation improved representativeness. Disadvantage of the design might be inclusion of ineligible patients when cohort information is limited. Furthermore, the design faced higher noncompliance in the intervention group, but prevention of contamination., Conclusion: The TwiCs design improved logistics in recruitment and prevented contamination, but noncompliance due to refusal of the intervention was higher compared with conventional pragmatic exercise oncology RCTs, which may dilute the estimated intervention effect., (Copyright © 2019 The Authors. Published by Elsevier Inc. All rights reserved.)

Objective: To determine how often stepped-wedge cluster randomized controlled trials reach their planned sample size, and what reasons are reported for choosing a stepped-wedge trial design., Study Design and Setting: We conducted a PubMed literature search (period 2012 to 2017) and included articles describing the results of a stepped-wedge cluster randomized trial. We calculated the percentage of studies reaching their prespecified number of participants and clusters, and we summarized the reasons for choosing the stepped-wedge trial design as well as difficulties during enrollment., Results: Forty-six individual stepped-wedge studies from a total of 53 articles were included in our review. Of the 35 studies, for which recruitment rate could be calculated, 69% recruited their planned number of participants, with 80% having recruited the planned number of clusters. Ethical reasons were the most common motivation for choosing the stepped-wedge trial design. Most important difficulties during study conduct were dropout of clusters and delayed implementation of the intervention., Conclusion: About half of recently published stepped-wedge trials reached their planned sample size indicating that recruitment is also a major problem in these trials. Still, the stepped-wedge trial design can yield practical, ethical, and methodological advantages., (Copyright © 2018 Elsevier Inc. All rights reserved.)

Objective: To assess the impact on noninferiority decisions when using a single margin or single preserved fraction (PF) for all noninferiority trials within a pharmacological class., Study Design and Setting: A search in PubMed, EMBASE, and CENTRAL resulted in seven active-controlled statin trials (nine noninferiority comparisons) for treating hyperlipidemia. The impact of using a single margin was assessed by calculating whether this margin corresponds to different PFs among comparator statins which will demonstrate that the threshold of demonstrating noninferiority (in terms of the PF) varies among comparator statins. The use of a single PF was assessed by reanalyzing noninferiority in the included trials with new margins (based on the single PF) for each comparator statin., Results: The use of a single margin resulted in PFs that range between 81% and 89% for the different comparators (i.e., different thresholds). The use of a single PF resulted in four of nine (44%) different noninferiority conclusions compared with the original analyses., Conclusion: The threshold of demonstrating noninferiority with a single margin or single PF of the effect per pharmacological class may not be consistent with using a margin/PF for each comparator separately and may impact the conclusions of noninferiority., (Copyright © 2018 Elsevier Inc. All rights reserved.)

Objectives: In medical research, covariates (e.g., exposure and confounder variables) are often measured with error. While it is well accepted that this introduces bias and imprecision in exposure-outcome relations, it is unclear to what extent such issues are currently considered in research practice. The objective was to study common practices regarding covariate measurement error via a systematic review of general medicine and epidemiology literature., Study Design and Setting: Original research published in 2016 in 12 high impact journals was full-text searched for phrases relating to measurement error. Reporting of measurement error and methods to investigate or correct for it were quantified and characterized., Results: Two hundred and forty-seven (44%) of the 565 original research publications reported on the presence of measurement error. 83% of these 247 did so with respect to the exposure and/or confounder variables. Only 18 publications (7% of 247) used methods to investigate or correct for measurement error., Conclusions: Consequently, it is difficult for readers to judge the robustness of presented results to the existence of measurement error in the majority of publications in high impact journals. Our systematic review highlights the need for increased awareness about the possible impact of covariate measurement error. Additionally, guidance on the use of measurement error correction methods is necessary., (Copyright © 2018 Elsevier Inc. All rights reserved.)

Pragmatic trials aim to inform clinical decision making by measuring the effect of a treatment in clinical practice. The purpose of the PRECIS-2 tool is to support in designing a truly pragmatic trial. We comment on a study by Forbes et al., who assessed the applicability of the PRECIS-2 tool. The tool will prove particularly useful when implemented in the process of trial design. However, it is yet unclear how e.g., possible dependencies between PRECIS domains, or conducting a pragmatic trial within an existing data registry (e.g., electronic health records) affect the applicability of the tool., (Copyright © 2017 Elsevier Inc. All rights reserved.)

This paper addresses challenges of identifying, enrolling, and retaining participants in a trial conducted within a routine care setting. All patients who are potential candidates for the treatments in routine clinical practice should be considered eligible for a pragmatic trial. To ensure generalizability, the recruited sample should have a similar distribution of the treatment effect modifiers as the target population. In practice, this can be best achieved by including-within the selected sites-all patients without further selection. If relevant heterogeneity between subgroups is expected, increasing the relative proportion of the subgroup of patients in the heterogeneous trial could be considered (oversampling) or a separate trial in this subgroup can be planned. Selection will nevertheless occur. Low enrollment and loss to follow-up can introduce selection and can jeopardize validity as well as generalizability. Pragmatic trials are conducted in clinical practice rather than in a dedicated research setting, which could reduce recruitment rates. However, if a trial poses a minimal burden to the physician and the patient and routine clinical practice is maximally adhered to, the participation rate may be high and loss to follow-up will not be a specific problem for pragmatic trials., (Copyright © 2017 The Authors. Published by Elsevier Inc. All rights reserved.)

This is the introductory paper in a series of eight papers. In this series, we integrate the theoretical design options with the practice of conducting pragmatic trials. For most new market-approved treatments, the clinical evidence is insufficient to fully guide physicians and policy makers in choosing the optimal treatment for their patients. Pragmatic trials can fill this gap, by providing evidence on the relative effectiveness of a treatment strategy in routine clinical practice, already in an early phase of development, while maintaining the strength of randomized controlled trials. Selecting the setting, study population, mode of intervention, comparator, and outcome are crucial in designing pragmatic trials. In combination with monitoring and data collection that does not change routine care, this will enable appropriate generalization to the target patient group in clinical practice. To benefit from the full potential of pragmatic trials, there is a need for guidance and tools in designing these studies while ensuring operational feasibility. This paper introduces the concept of pragmatic trial design. The complex interplay between pragmatic design options, feasibility, stakeholder acceptability, validity, precision, and generalizability will be clarified. In this way, balanced design choices can be made in pragmatic trials with an optimal chance of success in practice., (Copyright © 2017 The Authors. Published by Elsevier Inc. All rights reserved.)

This second article in the series on pragmatic trials describes the challenges in selection of sites for pragmatic clinical trials and the impact on validity, precision, and generalizability of the results. The selection of sites is an important factor for the successful execution of a pragmatic trial and impacts the extent to which the results are applicable to future patients in clinical practice. The first step is to define usual care and understand the heterogeneity of sites, patient demographics, disease prevalence and country choice. Next, specific site characteristics are important to consider such as interest in the objectives of the trial, the level of research experience, availability of resources, and the expected number of eligible patients. It can be advisable to support the sites with implementing the trial-related activities and minimize the additional burden that the research imposes on routine clinical practice. Health care providers should be involved in an early phase of protocol development to generate engagement and ensure an appropriate selection of sites with patients who are representative of the future drug users., (Copyright © 2017 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Meta-analyses using individual patient data (IPD) rather than aggregated data are increasingly applied to analyze sources of heterogeneity between trials and have only recently been applied to unravel multicomponent, complex interventions. This study reflects on methodological challenges encountered in two IPD meta-analyses on self-management interventions in patients with heart failure or chronic obstructive pulmonary disease., Study Design and Setting: Critical reflection on prior IPD meta-analyses and discussion of literature., Results: Experience from two IPD meta-analyses illustrates methodological challenges. Despite close collaboration with principal investigators, assessing the effect of characteristics of complex interventions on the outcomes of trials is compromised by lack of sufficient details on intervention characteristics and limited data on fidelity and adherence. Furthermore, trials collected baseline variables in a highly diverse way, limiting the possibilities to study subgroups of patients in a consistent manner. Possible solutions are proposed based on lessons learnt from the methodological challenges., Conclusion: Future researchers of complex interventions should pay considerable attention to the causal mechanism underlying the intervention and conducting process evaluations. Future researchers on IPD meta-analyses of complex interventions should carefully consider their own causal assumptions and availability of required data in eligible trials before undertaking such resource-intensive IPD meta-analysis., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Objectives: To compare different methods to handle treatment when developing a prognostic model that aims to produce accurate probabilities of the outcome of individuals if left untreated., Study Design and Setting: Simulations were performed based on two normally distributed predictors, a binary outcome, and a binary treatment, mimicking a randomized trial or an observational study. Comparison was made between simply ignoring treatment (SIT), restricting the analytical data set to untreated individuals (AUT), inverse probability weighting (IPW), and explicit modeling of treatment (MT). Methods were compared in terms of predictive performance of the model and the proportion of incorrect treatment decisions., Results: Omitting a genuine predictor of the outcome from the prognostic model decreased model performance, in both an observational study and a randomized trial. In randomized trials, the proportion of incorrect treatment decisions was smaller when applying AUT or MT, compared to SIT and IPW. In observational studies, MT was superior to all other methods regarding the proportion of incorrect treatment decisions., Conclusion: If a prognostic model aims to produce correct probabilities of the outcome in the absence of treatment, ignoring treatments that affect that outcome can lead to suboptimal model performance and incorrect treatment decisions. Explicitly, modeling treatment is recommended., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: To illustrate how pragmatic trial design elements or inserting explanatory trial elements in pragmatic trials affect validity, generalizability, precision, and operational feasibility., Study Design and Setting: From illustrative examples identified through the IMI Get Real Consortium, we selected randomized drug trials with a pragmatic design feature. We searched all publications on these trials for information on how pragmatic trial design features affect validity, generalizability, precision, or feasibility., Results: We present examples from the Salford lung study, International Suicide Prevention Trial, Sequenced Treatment Alternatives to Relieve Depression, and Cluster Randomized Usual care vs. Caduet Investigation Assessing Long-term-risk trial. These examples show that incorporating pragmatic trial design elements in trials may affect generalizability, precision and validity and may lead to operational challenges different from traditional explanatory trials. Inserting explanatory trial elements into pragmatic trials may also affect validity, generalizability, and operational feasibility, especially when these trial elements are incorporated in one arm of the trial only. Design choices that positively affect one of these domains (e.g., generalizability) may negatively affect others (e.g., feasibility)., Conclusion: Consequences of incorporating pragmatic or explanatory trial design elements in pragmatic trials should be explicitly considered and balanced for all relevant domains, including validity, generalizability, precision, and operational feasibility. Tools are needed to make these consequences more transparent., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: Poor quality of reporting of confounding has been observed in observational studies prior the STrenghtening the Reporting of Observational studies in Epidemiology (STROBE) statement, a reporting guideline for observational studies. We assessed whether the reporting of confounding improved after the STROBE statement., Study Design and Setting: We searched MEDLINE for all articles about observational cohort and case-control studies on interventions with a hypothesized beneficial effect in five general medical and five epidemiologic journals published between January 2010 and December 2012. We abstracted data for the baseline period before the publication of the STROBE statement (January 2004-April 2007) from a prior study. Six relevant items related to confounding were scored for each article. A comparison of the median number of items reported in both periods was made., Results: In total, 174 articles published before and 220 articles published after the STROBE statement were included. The median number reported items was similar before and after the publication of the STROBE statement [median, 4; interquartile range [IQR], 3-5 vs. median, 4; IQR, 3.75-5]. However, the distribution of the number of reported items shifted somewhat to the right (P = 0.01)., Conclusion: Although the quality of reporting of confounding improved in certain aspects, the overall quality remains suboptimal., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: To assess the current practice of propensity score (PS) analysis in the medical literature, particularly the assessment and reporting of balance on confounders., Study Design and Setting: A PubMed search identified studies using PS methods from December 2011 through May 2012. For each article included in the review, information was extracted on important aspects of the PS such as the type of PS method used, variable selection for PS model, and assessment of balance., Results: Among 296 articles that were included in the review, variable selection for PS model was explicitly reported in 102 studies (34.4%). Covariate balance was checked and reported in 177 studies (59.8%). P-values were the most commonly used statistical tools to report balance (125 of 177, 70.6%). The standardized difference and graphical displays were reported in 45 (25.4%) and 11 (6.2%) articles, respectively. Matching on the PS was the most commonly used approach to control for confounding (68.9%), followed by PS adjustment (20.9%), PS stratification (13.9%), and inverse probability of treatment weighting (IPTW, 7.1%). Balance was more often checked in articles using PS matching and IPTW, 70.6% and 71.4%, respectively., Conclusion: The execution and reporting of covariate selection and assessment of balance is far from optimal. Recommendations on reporting of PS analysis are provided to allow better appraisal of the validity of PS-based studies., (Copyright © 2015 Elsevier Inc. All rights reserved.)