Search

Abstract Background Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients. Methods We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell’s c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I2-statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a “membership model” to obtain a gross estimate of transportability. Results Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70–0.78, range 0.52–0.84, I2 = 0.92), the mean intercept was -0.06 (95%CI -0.37–0.24, range -1.40–0.75, I2 = 0.97), and the mean slope was 0.96 (95%CI 0.78–1.13, range 0.53–1.31, I2 = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model. Conclusions We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls.

Abstract Background Nested case–control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data. Methods We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design. Results Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58–0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62–0.69) and 0.65 (0.61–0.69), respectively. The unweighted O/E ratio was 18.38 (17.67–19.06) in the NCC datasets, while it was 1.69 (1.42–1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53–1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment. Conclusions Nested case–control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure.

Objective Acute myeloid leukaemia (AML) prognosis is enhanced with intensive remission induction chemotherapy (ICT) in eligible patients. However, ICT eligibility perceptions may differ among healthcare professionals. This nationwide, population-based study aimed to explore regional variation in ICT application and its relation with overall survival (OS).Methods and analysis We compared nine Dutch regional networks using data from the Netherlands Cancer Registry. Regional variance was assessed for the entire population and age subgroups (ie, ≤60 years and >60 years) using multivariable mixed effects logistic and Cox proportional hazard regression analyses, expressed via median OR (MOR) and median HR (MHR).Results Including all adult AML patients from 2014 to 2018 (N=4060 patients; 58% males; median age, 70 years), 1761 (43%) received ICT. ICT application varied from 36% to 57% (MOR 1.36 (95% CI 1.11 to 1.58)) across regions, with minor variations for patients aged ≤60 years (MOR 1.16 (95% CI 1.00 to 1.40)) and more extensive differences for those aged >60 years (MOR 1.43 (95% CI 1.16 to 1.63)). Median OS spanned 4.9–8.4 months across regions (MHR 1.11 (95% CI 1.00 to 1.15)), with pronounced differences in older patients (MHR 1.12 (95% CI 1.08 to 1.20)) but negligible differences in the younger group (MHR 1.02 (95% CI 1.00 to 1.14)). Survival differences for the total population and the older patients decreased to respectively, MHR 1.09 (95% CI 1.00 to 1.13) and 1.10 (95% CI 1.04 to 1.18), after additional adjustment for the probability of receiving ICT within a region, indicating approximately 10% unexplained differences.Conclusion Regional disparities in ICT application and survival exist, especially in older AML patients. However, ICT application differences partially explain survival disparities, indicating the need for more standardised ICT eligibility criteria and a better understanding of underlying causes of outcome disparities.

Abstract Background In patients with aneurysmal subarachnoid hemorrhage suitable for endovascular coiling and neurosurgical clip-reconstruction, the aneurysm treatment decision-making process could be improved by considering heterogeneity of treatment effect and durability of treatment. We aimed to develop and validate a tool to predict individualized treatment benefit of endovascular coiling compared to neurosurgical clip-reconstruction. Methods We used randomized data (International Subarachnoid Aneurysm Trial, n = 2143) to develop models to predict 2-month functional outcome and to predict time-to-rebleed-or-retreatment. We modeled for heterogeneity of treatment effect by adding interaction terms of treatment with prespecified predictors and with baseline risk of the outcome. We predicted outcome with both treatments and calculated absolute treatment benefit. We described the patient characteristics of patients with ≥ 5% point difference in the predicted probability of favorable functional outcome (modified Rankin Score 0–2) and of no rebleed or retreatment within 10 years. Model performance was expressed with the c-statistic and calibration plots. We performed bootstrapping and leave-one-cluster-out cross-validation and pooled cluster-specific c-statistics with random effects meta-analysis. Results The pooled c-statistics were 0.72 (95% CI: 0.69–0.75) for the prediction of 2-month favorable functional outcome and 0.67 (95% CI: 0.63–0.71) for prediction of no rebleed or retreatment within 10 years. We found no significant interaction between predictors and treatment. The average predicted benefit in favorable functional outcome was 6% (95% CI: 3–10%) in favor of coiling, but 11% (95% CI: 9–13%) for no rebleed or retreatment in favor of clip-reconstruction. 134 patients (6%), young and in favorable clinical condition, had negligible functional outcome benefit of coiling but had a ≥ 5% point benefit of clip-reconstruction in terms of durability of treatment. Conclusions We show that young patients in favorable clinical condition and without extensive vasospasm have a negligible benefit in functional outcome of endovascular coiling – compared to neurosurgical clip-reconstruction – while at the same time having a substantially lower probability of retreatment or rebleeding from neurosurgical clip-reconstruction – compared to endovascular coiling. The SHARP prediction tool ( https://sharpmodels.shinyapps.io/sharpmodels/ ) could support and incentivize a multidisciplinary discussion about aneurysm treatment decision-making by providing individualized treatment benefit estimates.

Abstract Background Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for “personalizing” medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects. Methods We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike’s Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit. Results The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial. Conclusions An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions.

Abstract Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments.

Summary: Background: Cutaneous squamous cell carcinoma (cSCC) is a common skin cancer, affecting more than 2 million people worldwide yearly and metastasising in 2–5% of patients. However, current clinical staging systems do not provide estimates of absolute metastatic risk, hence missing the opportunity for more personalised treatment advice. We aimed to develop a clinico-pathological model that predicts the probability of metastasis in patients with cSCC. Methods: Nationwide cohorts from (1) all patients with a first primary cSCC in The Netherlands in 2007–2008 and (2) all patients with a cSCC in 2013–2015 in England were used to derive nested case–control cohorts. Pathology records of primary cSCCs that originated a loco-regional or distant metastasis were identified, and these cSCCs were matched to primary cSCCs of controls without metastasis (1:1 ratio). The model was developed on the Dutch cohort (n = 390) using a weighted Cox regression model with backward selection and validated on the English cohort (n = 696). Model performance was assessed using weighted versions of the C-index, calibration metrics, and decision curve analysis; and compared to the Brigham and Women's Hospital (BWH) and the American Joint Committee on Cancer (AJCC) staging systems. Members of the multidisciplinary Skin Cancer Outcomes (SCOUT) consortium were surveyed to interpret metastatic risk cutoffs in a clinical context. Findings: Eight out of eleven clinico-pathological variables were selected. The model showed good discriminative ability, with an optimism-corrected C-index of 0.80 (95% Confidence interval (CI) 0.75–0.85) in the development cohort and a C-index of 0.84 (95% CI 0.81–0.87) in the validation cohort. Model predictions were well-calibrated: the calibration slope was 0.96 (95% CI 0.76–1.16) in the validation cohort. Decision curve analysis showed improved net benefit compared to current staging systems, particularly for thresholds relevant for decisions on follow-up and adjuvant treatment. The model is available as an online web-based calculator (https://emc-dermatology.shinyapps.io/cscc-abs-met-risk/). Interpretation: This validated model assigns personalised metastatic risk predictions to patients with cSCC, using routinely reported histological and patient-specific risk factors. The model can empower clinicians and healthcare systems in identifying patients with high-risk cSCC and offering personalised care/treatment and follow-up. Use of the model for clinical decision-making in different patient populations must be further investigated. Funding: PPP Allowance made available by Health-Holland, Top Sector Life Sciences & Health, to stimulate public-private partnerships.

Abstract Background Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time. Methods We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit. Results Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data. Conclusions NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic.

Despite rapid and significant changes during the pandemic, prognostic models for COVID-19 patients do not currently account for data drifts. Here, the authors develop a framework for continuously monitoring and updating prognostic models and applied it to predict 28-day survival in COVID-19 patients.

Summary: Background: Loss of life expectancy (LOLE) may provide more intuitive information on the impact of cancer than relative survival over a fixed time horizon (e.g., 5-year relative survival). We aimed to assess the evolution of the LOLE using a nationwide, population-based cohort including patients diagnosed with one of 17 most frequent solid malignancies. Methods: From the Netherlands Cancer Registry, we selected adult patients diagnosed with one of the 17 most frequent solid malignancies in the Netherlands during 1989–2019, with survival follow-up until 2022. We used flexible parametric survival models to estimate the LOLE at diagnosis and the LOLE after surviving several years post-diagnosis (conditional LOLE; CLOLE) by cancer type, calendar year, age, sex, and disease stage. Findings: For all cancers combined, the LOLE consistently decreased from 1989 to 2019. This decrease was most pronounced for males with prostate cancer (e.g., from 6.9 [95% confidence interval [CI], 6.7–7.1] to 2.7 [95% CI, 2.5–3.0] for 65-year-olds) and females with breast cancer (e.g., from 6.6 [95% CI, 6.4–6.7] to 1.9 [95% CI, 1.8–2.0] for 65-year-olds). The LOLE among patients with cancers of the head and neck or the central nervous system remained constant over time. Overall, the CLOLE showed that the life years lost among patients with cancer decreased with each additional year survived post-diagnosis. For example, the LOLE at diagnosis for 65-year-old females diagnosed with breast cancer in 2019 was 1.9 [95% CI, 1.8–2.0] compared with 1.7 [95% CI, 1.6–1.8], 1.0 [95% CI, 0.9–1.1], and 0.5 [95% CI, 0.5–0.6] when surviving one, five, and ten years post-diagnosis, respectively. Estimates for other combinations of patient and tumour characteristics are available in a publicly available web-based application. Interpretation: Our findings suggested that the evolution of LOLE substantially varies across cancer type, age, and disease stage. LOLE estimates help patients better understand the impact of their specific cancer diagnosis on their life expectancy. Funding: None.

Abstract Background Prediction modeling studies often have methodological limitations, which may compromise model performance in new patients and settings. We aimed to examine the relation between methodological quality of model development studies and their performance at external validation. Methods We systematically searched for externally validated multivariable prediction models that predict functional outcome following moderate or severe traumatic brain injury. Risk of bias and applicability of development studies was assessed with the Prediction model Risk Of Bias Assessment Tool (PROBAST). Each model was rated for its presentation with sufficient detail to be used in practice. Model performance was described in terms of discrimination (AUC), and calibration. Delta AUC (dAUC) was calculated to quantify the percentage change in discrimination between development and validation for all models. Generalized estimation equations (GEE) were used to examine the relation between methodological quality and dAUC while controlling for clustering. Results We included 54 publications, presenting ten development studies of 18 prediction models, and 52 external validation studies, including 245 unique validations. Two development studies (four models) were found to have low risk of bias (RoB). The other eight publications (14 models) showed high or unclear RoB. The median dAUC was positive in low RoB models (dAUC 8%, [IQR − 4% to 21%]) and negative in high RoB models (dAUC − 18%, [IQR − 43% to 2%]). The GEE showed a larger average negative change in discrimination for high RoB models (− 32% (95% CI: − 48 to − 15) and unclear RoB models (− 13% (95% CI: − 16 to − 10)) compared to that seen in low RoB models. Conclusion Lower methodological quality at model development associates with poorer model performance at external validation. Our findings emphasize the importance of adherence to methodological principles and reporting guidelines in prediction modeling studies.

Background & Aims: When listing for liver transplantation, one can transplant as soon as possible or introduce a test-of-time to better select patients, as the tumor’s biological behavior is observed. Knowing the degree of harm caused by time itself is essential to advise patients and decide on the maximum duration of the test-of-time. Therefore, we investigated the causal effect of waiting time on post-transplant survival for patients with hepatocellular carcinoma. Methods: We analyzed the UNOS-OPTN dataset and exploited a natural experiment created by blood groups. Relations between variables and assumptions were described in a causal graph. Selection bias was addressed by inverse probability weighting. Confounding was avoided using instrumental variable analysis, with an additive hazards model in the second stage. The causal effect was evaluated by estimating the difference in 5-year overall survival if all patients waited 2 months instead of 12 months. Upper bounds of the test-of-time were evaluated for probable scenarios by means of simulation. Results: The F-statistic of the first stage was 86.3. The effect of waiting 12 months vs. 2 months corresponded with a drop in overall survival rate of 5.07% (95% CI 0.277–9.69) and 8.33% (95% CI 0.47–15.60) at 5- and 10-years post-transplant, respectively. The median survival dropped by 3.41 years from 16.21 years (95% CI 15.98–16.60) for those waiting 2 months to 12.80 years (95% CI 10.72–15.90) for those waiting 12 months. Conclusions: From a patient’s perspective, the choice between ablate-and-wait vs. immediate transplantation is in favor of immediate transplantation. From a policy perspective, the extra waiting time can be used to increase the utility of scarce donor livers. However, the duration of the test-of-time is bounded, and it likely should not exceed 8 months. Impact and implications: When listing patients with liver cancer for transplantation, it is unclear whether a test-of-time or immediate transplantation offer better outcomes at the population level. In this study, we found that increased liver transplant waiting times are detrimental in patients with liver cancer. Furthermore, our simulation showed that a pre-operative observational period can be useful to ensure good donor liver allocation, but that its duration should not exceed 8 months.

IntroductionIntegrated care is effective in reducing all-cause mortality in patients with atrial fibrillation (AF) in primary care, though time and resource intensive. The aim of the current study was to assess whether integrated care should be directed at all AF patients equally.MethodsThe ALL-IN trial (n = 1,240 patients, median age 77 years) was a cluster-randomized trial in which primary care practices were randomized to provide integrated care or usual care to AF patients aged 65 years and older. Integrated care comprised of (i) anticoagulation monitoring, (ii) quarterly checkups and (iii) easy-access consultation with cardiologists. For the current analysis, cox proportional hazard analysis with all clinical variables from the CHA2DS2-VASc score was used to predict all-cause mortality in the ALL-IN trial. Subsequently, the hazard ratio and absolute risk reduction were plotted as a function of this predicted mortality risk to explore treatment heterogeneity.ResultsUnder usual care, after a median of 2 years follow-up the absolute risk of all-cause mortality in the highest-risk quarter was 31.0%, compared to 4.6% in the lowest-risk quarter. On the relative scale, there was no evidence of treatment heterogeneity (p for interaction = 0.90). However, there was substantial treatment heterogeneity on the absolute scale: risk reduction in the lowest risk- quarter of risk 3.3% (95% CI -0.4% - 7.0) compared to 12.0% (95% CI 2.7% - 22.0) in the highest risk quarter.ConclusionWhile the relative degree of benefit from integrated AF care is similar in all patients, patients with a high all-cause mortality risk have a greater benefit on an absolute scale and should therefore be prioritized when implementing integrated care.

BackgroundConsidering the importance of cardiovascular disease (CVD) risk prediction for healthcare systems and the limited information available in the Middle East, we evaluated the SCORE and Globorisk models to predict CVD death in a country of this region.MethodsWe included 24 427 participants (11 187 men) aged 40-80 years from four population-based cohorts in Iran. Updating approaches were used to recalibrate the baseline survival and the overall effect of the predictors of the models. We assessed the models’ discrimination using C-index and then compared the observed with the predicted risk of death using calibration plots. The sensitivity and specificity of the models were estimated at the risk thresholds of 3%, 5%, 7%, and 10%. An agreement between models was assessed using the intra-class correlation coefficient (ICC). We applied decision analysis to provide perception into the consequences of using the models in general practice; for this reason, the clinical usefulness of the models was assessed using the net benefit (NB) and decision curve analysis. The NB is a sensitivity penalized by a weighted false positive (FP) rate in population level.ResultsAfter 154 522 person-years of follow-up, 437 cardiovascular deaths (280 men) occurred. The 10-year observed risks were 4.2% (95% CI: 3.7%-4.8%) in men and 2.1% (1.8-2%.5%) in women. The c-index for SCORE function was 0.784 (0.756-0.812) in men and 0.780 (0.744-0.815) in women. Corresponding values for Globorisk were 0.793 (0.766-0.820) and 0.793 (0.757-0.829). The deviation of the calibration slopes from one reflected a need for recalibration; after which, the predicted-to-observed ratio for both models was 1.02 in men and 0.95 in women. Models showed good agreement (ICC 0.93 in men, and 0.89 in women). Decision curve showed that using both models results in the same clinical usefulness at the risk threshold of 5%, in both men and women; however, at the risk threshold of 10%, Globorisk had better clinical usefulness in women (Difference: 8%, 95% CI: 4%-13%).ConclusionOriginal Globorisk and SCORE models overestimate the CVD risk in Iranian populations resulting in a high number of people who need intervention. Recalibration could adopt these models to precisely predict CVD mortality. Globorisk showed better performance clinically, only among high-risk women.

Abstract Background Suboptimal maternal health already from preconception onwards is strongly linked to an increased risk of birth complications. To enable identification of women at risk of birth complications, we aimed to develop a prediction model for birth complications using maternal preconception socio-demographic, lifestyle, medical history and early-pregnancy clinical characteristics in a general population. Methods In a population-based prospective cohort study among 8340 women, we obtained information on 33 maternal characteristics at study enrolment in early-pregnancy. These characteristics covered the preconception period and first half of pregnancy (

Abstract Background Assessing calibration—the agreement between estimated risk and observed proportions—is an important component of deriving and validating clinical prediction models. Methods for assessing the calibration of prognostic models for use with competing risk data have received little attention. Methods We propose a method for graphically assessing the calibration of competing risk regression models. Our proposed method can be used to assess the calibration of any model for estimating incidence in the presence of competing risk (e.g., a Fine-Gray subdistribution hazard model; a combination of cause-specific hazard functions; or a random survival forest). Our method is based on using the Fine-Gray subdistribution hazard model to regress the cumulative incidence function of the cause-specific outcome of interest on the predicted outcome risk of the model whose calibration we want to assess. We provide modifications of the integrated calibration index (ICI), of E50 and of E90, which are numerical calibration metrics, for use with competing risk data. We conducted a series of Monte Carlo simulations to evaluate the performance of these calibration measures when the underlying model has been correctly specified and when the model was mis-specified and when the incidence of the cause-specific outcome differed between the derivation and validation samples. We illustrated the usefulness of calibration curves and the numerical calibration metrics by comparing the calibration of a Fine-Gray subdistribution hazards regression model with that of random survival forests for predicting cardiovascular mortality in patients hospitalized with heart failure. Results The simulations indicated that the method for constructing graphical calibration curves and the associated calibration metrics performed as desired. We also demonstrated that the numerical calibration metrics can be used as optimization criteria when tuning machine learning methods for competing risk outcomes. Conclusions The calibration curves and numeric calibration metrics permit a comprehensive comparison of the calibration of different competing risk models.

Introduction Treatment decisions for aneurysmal subarachnoid haemorrhage patients should be supported by individualised predictions of the effects of aneurysm treatment. We present a study protocol and analysis plan for the development and external validation of models to predict benefit of neurosurgical versus endovascular aneurysm treatment on functional outcome and durability of treatment.Methods and analysis We will use data from the International Subarachnoid Aneurysm Trial for model development. The outcomes are functional outcome, measured with modified Rankin Scale at 12 months, and any retreatment or rebleed of the target aneurysm during follow-up. We will develop an ordinal logistic regression model and Cox regression model, considering age, World Federation of Neurological Surgeons grade, Fisher grade, vasospasm at presentation, aneurysm lumen size, aneurysm neck size, aneurysm location and time-to-aneurysm-treatment as predictors. We will test for interactions with treatment and with baseline risk and derive individualised predicted probabilities of treatment benefit. A benefit of ≥5% will be considered clinically relevant. Discriminative performance of the outcome predictions will be assessed with the c-statistic. Calibration will be assessed with calibration plots. Discriminative performance of the benefit predictions will be assessed with the c-for benefit. We will assess internal validity with bootstrapping and external validity with leave-one-out internal-external cross-validation.Ethics and dissemination The medical ethical research committee of the Erasmus MC University Medical Center Rotterdam approved the study protocol under the exemption category and waived the need for written informed consent (MEC-2020-0810). We will disseminate our results through an open-access peer-reviewed scientific publication and with a web-based clinical prediction tool.

Abstract Background Prehospital triage protocols typically try to select patients with Injury Severity Score (ISS) above 15 for direct transportation to a Level-1 trauma center. However, ISS does not necessarily discriminate between patients who benefit from immediate care at Level-1 trauma centers. The aim of this study was to assess which patients benefit from direct transportation to Level-1 trauma centers. Methods We used the American National Trauma Data Bank (NTDB), a retrospective observational cohort. All adult patients (ISS > 3) between 2015 and 2016 were included. Patients who were self-presenting or had isolated limb injury were excluded. We used logistic regression to assess the association of direct transportation to Level-1 trauma centers with in-hospital mortality adjusted for clinically relevant confounders. We used this model to define benefit as predicted probability of mortality associated with transportation to a non-Level-1 trauma center minus predicted probability associated with transportation to a Level-1 trauma center. We used a threshold of 1% as absolute benefit. Potential interaction terms with transportation to Level-1 trauma centers were included in a penalized logistic regression model to study which patients benefit. Results We included 388,845 trauma patients from 232 Level-1 centers and 429 Level-2/3 centers. A small beneficial effect was found for direct transportation to Level-1 trauma centers (adjusted Odds Ratio: 0.96, 95% Confidence Interval: 0.92–0.99) which disappeared when comparing Level-1 and 2 versus Level-3 trauma centers. In the risk approach, predicted benefit ranged between 0 and 1%. When allowing for interactions, 7% of the patients (n = 27,753) had more than 1% absolute benefit from direct transportation to Level-1 trauma centers. These patients had higher AIS Head and Thorax scores, lower GCS and lower SBP. A quarter of the patients with ISS > 15 were predicted to benefit from transportation to Level-1 centers (n = 26,522, 22%). Conclusions Benefit of transportation to a Level-1 trauma centers is quite heterogeneous across patients and the difference between Level-1 and Level-2 trauma centers is small. In particular, patients with head injury and signs of shock may benefit from care in a Level-1 trauma center. Future prehospital triage models should incorporate more complete risk profiles.

Abstract Background Recent evidence suggests that there is often substantial variation in the benefits and harms across a trial population. We aimed to identify regression modeling approaches that assess heterogeneity of treatment effect within a randomized clinical trial. Methods We performed a literature review using a broad search strategy, complemented by suggestions of a technical expert panel. Results The approaches are classified into 3 categories: 1) Risk-based methods (11 papers) use only prognostic factors to define patient subgroups, relying on the mathematical dependency of the absolute risk difference on baseline risk; 2) Treatment effect modeling methods (9 papers) use both prognostic factors and treatment effect modifiers to explore characteristics that interact with the effects of therapy on a relative scale. These methods couple data-driven subgroup identification with approaches to prevent overfitting, such as penalization or use of separate data sets for subgroup identification and effect estimation. 3) Optimal treatment regime methods (12 papers) focus primarily on treatment effect modifiers to classify the trial population into those who benefit from treatment and those who do not. Finally, we also identified papers which describe model evaluation methods (4 papers). Conclusions Three classes of approaches were identified to assess heterogeneity of treatment effect. Methodological research, including both simulations and empirical evaluations, is required to compare the available methods in different settings and to derive well-informed guidance for their application in RCT analysis.

Abstract Background Discriminative ability is an important aspect of prediction model performance, but challenging to assess in clustered (e.g., multicenter) data. Concordance (c)-indexes may be too extreme within small clusters. We aimed to define a new approach for the assessment of discriminative ability in clustered data. Methods We assessed discriminative ability of a prediction model for the binary outcome mortality after traumatic brain injury within centers of the CRASH trial. With multilevel logistic regression analysis, we estimated cluster-specific calibration slopes which we used to obtain the recently proposed calibrated model-based concordance (c-mbc) within each cluster. We compared the c-mbc with the naïve c-index in centers of the CRASH trial and in simulations of clusters with varying calibration slopes. Results The c-mbc was less extreme in distribution than the c-index in 19 European centers (internal validation; n = 1716) and 36 non-European centers (external validation; n = 3135) of the CRASH trial. In simulations, the c-mbc was biased but less variable than the naïve c-index, resulting in lower root mean squared errors. Conclusions The c-mbc, based on multilevel regression analysis of the calibration slope, is an attractive alternative to the c-index as a measure of discriminative ability in multicenter studies with patient clusters of limited sample size.

Abstract Background Stability in baseline risk and estimated predictor effects both geographically and temporally is a desirable property of clinical prediction models. However, this issue has received little attention in the methodological literature. Our objective was to examine methods for assessing temporal and geographic heterogeneity in baseline risk and predictor effects in prediction models. Methods We studied 14,857 patients hospitalized with heart failure at 90 hospitals in Ontario, Canada, in two time periods. We focussed on geographic and temporal variation in baseline risk (intercept) and predictor effects (regression coefficients) of the EFFECT-HF mortality model for predicting 1-year mortality in patients hospitalized for heart failure. We used random effects logistic regression models for the 14,857 patients. Results The baseline risk of mortality displayed moderate geographic variation, with the hospital-specific probability of 1-year mortality for a reference patient lying between 0.168 and 0.290 for 95% of hospitals. Furthermore, the odds of death were 11% lower in the second period than in the first period. However, we found minimal geographic or temporal variation in predictor effects. Among 11 tests of differences in time for predictor variables, only one had a modestly significant P value (0.03). Conclusions This study illustrates how temporal and geographic heterogeneity of prediction models can be assessed in settings with a large sample of patients from a large number of centers at different time periods.

OBJECTIVES: The study aimed to (i) determine the influence of poor health on competing exit routes from paid employment among older workers in Europe, (ii) assess whether these risks are different among welfare state regimes in Europe, and (iii) evaluate differences in estimates between two different competing risk approaches. METHODS: The study population consisted of 5273 respondents (6-years follow-up) from the Survey of Health, Ageing, and Retirement in Europe (SHARE). The effect of poor health on exit routes from paid employment was assessed with a cause-specific Cox model and a Fine & Gray (F&G) model. These two competing risk analyses were used to calculate absolute risks of labor force exit among welfare state regimes in Europe. RESULTS: In both models, poor health was a risk factor for disability benefit [hazard ratio (HR) 3.36; subdistribution hazard ratio (SHR) 3.22], and unemployment (HR 1.43, SHR 1.32). Both models produced similar absolute risks. In countries with a Bismarckian welfare state regime, low-educated older workers living alone and in poor health had an 11% risk of disability benefit, 7% of unemployment, 46% of early retirement, and 7% of becoming economically inactive. In countries with a Scandinavian welfare state regime, the risks were 10%, 7%, 29%, and 3%, respectively, and in Southern European welfare state regimes 4%, 5%, 35%, and 7%. CONCLUSIONS: Workers with poor health are more likely to leave the labor force than workers with good health. The absolute risks of early retirement and becoming economically inactive were lowest in countries with a Scandinavian welfare state regime. For disability benefit and unemployment, absolute risks were lowest in Southern European welfare state regimes. The direct estimation of absolute risks of leaving the labor force in the presence of competing exit routes is an appealing feature of the F&G model.

OBJECTIVES/SPECIFIC AIMS: Objective: Approximately 86 million people in the US have prediabetes, but only a fraction of them receive proven effective therapies to prevent diabetes. Further, the effectiveness of these therapies varies with individual risk of progression to diabetes. We estimated the value of targeting those individuals at highest diabetes risk for treatment, compared to treating all individuals meeting inclusion criteria for the Diabetes Prevention Program (DPP). METHODS/STUDY POPULATION: METHODS: Using a micro-simulation model, we estimated total lifetime costs and quality-adjusted life expectancy (QALE) for individuals receiving: (1) lifestyle intervention involving an intensive program focused on healthy diet and exercise, (2) metformin administration, or (3) no intervention. The model combines several components. First a Cox proportional hazards model predicted onset of diabetes from baseline characteristics for each pre-diabetic individual and yielded a probability distribution for each alternative. We derived this risk model from the Diabetes Prevention Program (DPP) clinical trial data and the follow-up study DPP-OS. The Michigan Diabetes Research Center Model for Diabetes then estimated costs and outcomes for individuals after diabetes diagnosis using standard of care diabetes treatment. Based on individual costs and QALE, we evaluated NMB of the two interventions at population and individual levels, stratified by risk quintiles for diabetes onset at 3 years. RESULTS/ANTICIPATED RESULTS: Results: Compared to usual care, lifestyle modification conferred positive benefits for all eligible individuals. Metformin’s NMB was negative for the lowest population risk quintile. By avoiding use among individuals who would not benefit, targeted administration of metformin conferred a benefit of $500-$800 per person, depending on duration of treatment effect. When treating only 20% of the population (e.g., due to capacity constraints), targeting conferred a NMB of $14,000-$18,000 per person for lifestyle modification and $16,000-$20,000 for metformin. DISCUSSION/SIGNIFICANCE OF IMPACT: Conclusions: Metformin confers value only among higher risk individuals, so targeting its use is worthwhile. While lifestyle modification confers value for all eligible individuals, prioritizing the intervention to high risk patients when capacity is constrained substantially increases societal benefits.