Search

Abstract Background Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs). Methods Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the Eavg-for-benefit, E50-for-benefit, and E90-for-benefit as the average, median, and 90th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately “perturbed models” were compared to those of the data-generating model, i.e., “optimal model”. To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest. Results As desired, performance metric values of “perturbed models” were consistently worse than those of the “optimal model” (Eavg-for-benefit ≥ 0.043 versus 0.002, E50-for-benefit ≥ 0.032 versus 0.001, E90-for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package “HTEPredictionMetrics”. Conclusion The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs.

Background and purpose: With currently available techniques, the prediction of pathologic complete response after neoadjuvant chemoradiotherapy is insufficient. The tumor-stroma ratio (TSR) has proven to be a predictor of survival for several types of cancer, including esophageal. The aim of this study was to investigate the value of TSR in predicting pathologic response after neoadjuvant chemoradiotherapy in esophageal cancer patients. Materials and methods: Patients with esophageal adenocarcinoma or squamous cell carcinoma who received neoadjuvant chemoradiotherapy followed by a resection were selected. Haematoxylin and eosin (H&E) stained sections of diagnostic biopsies were collected and TSR was independently assessed by two investigators. Patients were categorized in stroma-low (≤50% stroma) and stroma-high (>50% stroma) groups for further analyses. The tumor regression grade (TRG) was assessed on H&E stained sections of the resected primary tumor to determine pathologic response. Results: A total of 94 patients were included in this study, of which 76 patients were categorized as stroma-low and 18 as stroma-high. Forty-two (45%) patients had a major pathologic response (TRG 1–2), whereas 52 (55%) were considered non-responders. After adjustment for gender, tumor type, cT-status and differentiation grade, patients with a stroma-high tumor showed a higher chance of no response compared to patients with a stroma-low tumor (OR 3.57, 95%CI 1.03–12.31, P = 0.04). Conclusion: TSR showed to have the potential to aid in the prediction of pathologic response in esophageal cancer patients receiving neoadjuvant chemoradiotherapy. Larger validation studies are necessary before implementing this method in daily practice. Keywords: Esophageal cancer, Neoadjuvant chemoradiotherapy, Pathologic response, Prediction, Tumor-stroma ratio

Objectives: Average treatment effects from randomized trials do not reflect the heterogeneity of an individual's response to treatment. This study evaluates the appropriate proportions of patients for coronary artery bypass grafting, or percutaneous intervention based on the predicted/observed ratio of 10-year all-cause mortality in the SYNTAX population., Methods: The study included 1800 randomized patients and 1275 patients in the nested percutaneous (n = 198) or surgical (n = 1077) registries. The primary end point was 10-year all-cause mortality. The SYNTAX score II-2020 was validated internally in the randomized cohort and externally in the registry cohort. Proportions of patients with survival benefits from coronary artery bypass grafting or percutaneous intervention were determined using SYNTAX score II-2020., Results: Ten-year mortality was 23.8% for coronary artery bypass grafting, 28.6% for percutaneous intervention in the randomized cohort, 27.6% for coronary artery bypass grafting and 55.4% for percutaneous intervention in the registries. In the coronary artery bypass grafting registry, the SYNTAX score II-2020 predicted 10-year mortality with helpful calibration and discrimination (C-index: 0.70, intercept: 0.00, slope: 0.76). The proportion of patients deriving a predicted survival benefit from coronary artery bypass grafting over percutaneous intervention was 82.4% (2143/2602) and 17.7% (459/2602) for the entire SYNTAX trial population, translating into a 4.7 to 1 appropriate ratio of treatment allocation to coronary artery bypass grafting and percutaneous intervention., Conclusions: Choosing a revascularization modality should depend on an individual's long-term prognosis rather than average treatment effects. Additionally, patients should be informed about their predicted prognosis., Trial Registration: Registered on clinicaltrial.gov., Syntaxes: NCT03417050 (https://clinicaltrials.gov/ct2/show/NCT03417050)., Syntax: NCT00114972 (https://www.clinicaltrials.gov/ct2/show/NCT00114972)., (© The Author(s) 2024. Published by Oxford University Press on behalf of the European Association for Cardio-Thoracic Surgery.)

Background: Accurate bleeding risk stratification after percutaneous coronary intervention (PCI) is important for treatment individualization. However, there is still an unmet need for a more precise and standardized identification of high bleeding risk patients. We derived and validated a novel bleeding risk score by augmenting the PRECISE-DAPT score with the Academic Research Consortium for High Bleeding Risk (ARC-HBR) criteria., Methods: The derivation cohort comprised 29,188 patients undergoing PCI, of whom 1136 (3.9%) had a Bleeding Academic Research Consortium (BARC) 3 or 5 bleeding at 1 year, from four contemporary real-world registries and the XIENCE V USA trial. The PRECISE-DAPT score was refitted with a Fine-Gray model in the derivation cohort and extended with the ARC-HBR criteria. The primary outcome was BARC 3 or 5 bleeding within 1 year. Independent predictors of BARC 3 or 5 bleeding were selected at multivariable analysis (p<0.01). The discrimination of the score was internally assessed with apparent validation and cross-validation. The score was externally validated in 4578 patients from the MASTER DAPT trial and 5970 patients from the STOPDAPT-2 total cohort., Results: The PRECISE-HBR score (age, estimated glomerular filtration rate, hemoglobin, white-blood-cell count, previous bleeding, oral anticoagulation, and ARC-HBR criteria) showed an area under the curve (AUC) for 1-year BARC 3 or 5 bleeding of 0.73 (95% CI, 0.71-0.74) at apparent validation, 0.72 (95% CI, 0.70-0.73) at cross-validation, 0.74 (95% CI, 0.68-0.80) in the MASTER DAPT, and 0.73 (95% CI, 0.66-0.79) in the STOPDAPT-2, with superior discrimination than the PRECISE-DAPT (cross-validation: Δ AUC, 0.01; p=0.02; MASTER DAPT: Δ AUC, 0.05; p=0.004; STOPDAPT-2: Δ AUC, 0.02; p=0.20) and other risk scores. In the derivation cohort, a cut-off of 23 points identified 11,414 patients (39.1%) with a 1-year BARC 3 or 5 bleeding risk ≥4%. An alternative version of the score, including acute myocardial infarction on admission instead of white-blood-cell count, showed similar predictive ability., Conclusions: The PRECISE-HBR score is a contemporary, simple 7-item risk score to predict bleeding after PCI, offering a moderate improvement in discrimination over multiple existing scores. Further evaluation is required to assess its impact on clinical practice.

Clinical prediction models (CPMs) are tools that compute the risk of an outcome given a set of patient characteristics and are routinely used to inform patients, guide treatment decision-making, and resource allocation. Although much hope has been placed on CPMs to mitigate human biases, CPMs may potentially contribute to racial disparities in decision-making and resource allocation. While some policymakers, professional organizations, and scholars have called for eliminating race as a variable from CPMs, others raise concerns that excluding race may exacerbate healthcare disparities and this controversy remains unresolved. The Guidance for Unbiased predictive Information for healthcare Decision-making and Equity (GUIDE) provides expert guidelines for model developers and health system administrators on the transparent use of race in CPMs and mitigation of algorithmic bias across contexts developed through a 5-round, modified Delphi process from a diverse 14-person technical expert panel (TEP). Deliberations affirmed that race is a social construct and that the goals of prediction are distinct from those of causal inference, and emphasized: the importance of decisional context (e.g., shared decision-making versus healthcare rationing); the conflicting nature of different anti-discrimination principles (e.g., anticlassification versus antisubordination principles); and the importance of identifying and balancing trade-offs in achieving equity-related goals with race-aware versus race-unaware CPMs for conditions where racial identity is prognostically informative. The GUIDE, comprising 31 key items in the development and use of CPMs in healthcare, outlines foundational principles, distinguishes between bias and fairness, and offers guidance for examining subgroup invalidity and using race as a variable in CPMs. This GUIDE presents a living document that supports appraisal and reporting of bias in CPMs to support best practice in CPM development and use., (© 2024. The Author(s).)

First-line treatment for advanced-stage diffuse large B-cell lymphoma (DLBCL) typically involves 6x R-CHOP21 or 6x R-CHOP21 with two additional rituximab administrations (6x R-CHOP21 + 2 R). In contemporary practice, this treatment choice might be guided by interim PET scan results. This nationwide, population-based study investigates the comparative effectiveness of these treatment regimens in an era where interim PET-guided treatment decisions were not standard practice. Utilizing the Netherlands Cancer Registry, we identified 1577 adult patients diagnosed with advanced-stage DLBCL between 2014-2018 who completed either 6x R-CHOP21 (43%) or 6x R-CHOP21 + 2 R (57%). We used propensity scores to assess differences in event-free survival (EFS) and overall survival (OS). At five years, EFS (hazard ratio of 6x R-CHOP21 + 2 R versus 6x R-CHOP21 [HR] = 0.89; 95% confidence interval [CI], 0.72-1.09) and OS (HR = 0.93; 95% CI, 0.73-1.18) were not significantly different between both regimens. In exploratory risk-stratified analysis according to the International Prognostic Index (IPI), high-IPI patients (i.e., scores of 4-5) benefit most from 6x R-CHOP21 + 2 R (5-year absolute risk difference of EFS = 16.8%; 95% CI, -0.4%-34.1% and OS = 12.1%; 95% CI, -5.4-29.6%). Collectively, this analysis reveals no significant differences on average in EFS and OS between the two treatments. However, the potential benefits for high-risk patients treated with 6x R-CHOP21 + 2 R underscore the need for future research., (© 2024. The Author(s).)

Purpose: Financial toxicity, the subjective distress caused by objective financial burden, significantly impacts cancer survivors. Yet, enduring effects on survivors remain unclear. Therefore, we investigated the experienced objective financial burden and subjective financial distress in long-term cancer survivors., Methods: A cross-sectional nationwide online survey of adult cancer survivors ≥ 5y after diagnosis were analyzed. Objective financial burden was measured via extra expenses and income loss, while subjective financial distress covered psychological well-being, coping and support-seeking behavior, and financial concerns. Groups were compared (i.e., having cancer vs. former patients) by t-tests and chi-squared tests. Financial toxicity was visualized with Sankey plots and sunburst diagrams., Results: 4,675 respondents completed the survey, of whom 2,391 (51%) were ≥ 5y after their cancer diagnosis. Among them, 75% experienced income loss and/or extra expenses after diagnosis. One-third of the previously employed respondents relied on work disability benefits. Further, 'being unable to make ends meet' increased from 2% before diagnosis to 13% ≥ 5y after diagnosis (p < .001). Additionally, 58% reported negative psychological impacts of financial toxicity, and 47% worried about their financial future., Conclusions: Cancer survivors often face income loss and additional expenses, leading to ongoing financial difficulties that affect their psychological well-being. Despite this significant impact, there is a lack of guidance and support to help them manage these financial challenges. These findings highlight the need for healthcare professionals to recognize and address the financial challenges., Implications for Cancer Survivors: This study underscores the widespread financial challenges cancer survivors encounter, emphasizing the need for ongoing financial support and comprehensive assessments of their physical and psychological well-being., (© 2024. The Author(s).)

Objective: Isolated ambulatory phlebectomy is a potential treatment option for patients with an incompetent great saphenous vein (GSV) or anterior accessory saphenous vein and one or more incompetent tributaries. Being able to determine which patients will most likely benefit from isolated phlebectomy is important. This study aimed to identify predictors for avoidance of secondary axial ablation after isolated phlebectomy and to develop and externally validate a multivariable model for predicting this outcome., Methods: For model development, data from patients included in the SAPTAP trial were used. The investigated outcome was avoidance of ablation of the saphenous trunk one year after isolated ambulatory phlebectomy. Pre-defined candidate predictors were analysed with multivariable logistic regression. Predictors were selected using Akaike information criterion backward selection. Discriminative ability was assessed by the concordance index. Bootstrapping was used to correct regression coefficients, and the C index for overfitting. The model was externally validated using a population of 94 patients, with an incompetent GSV and one or more incompetent tributaries, who underwent isolated phlebectomy., Results: For model development, 225 patients were used, of whom 167 (74.2%) did not undergo additional ablation of the saphenous trunk one year after isolated phlebectomy. The final model consisted of three predictors for avoidance of axial ablation: tributary length (< 15 cm vs. > 30 cm: odds ratio [OR] 0.09, 95% confidence interval [CI] 0.02 - 0.40; 15 - 30 cm vs. > 30 cm: OR 0.18, 95% CI 0.09 - 0.38); saphenofemoral junction (SFJ) reflux (absent vs. present: OR 2.53, 95% CI 0.81 - 7.87); and diameter of the saphenous trunk (per millimetre change: OR 0.63, 95% CI 0.41 - 0.96). The discriminative ability of the model was moderate (0.72 at internal validation; 0.73 at external validation)., Conclusion: A model was developed for predicting avoidance of secondary ablation of the saphenous trunk one year after isolated ambulatory phlebectomy, which can be helpful in daily practice to determine the suitable treatment strategy in patients with an incompetent saphenous trunk and one or more incompetent tributaries. Patients having a longer tributary, smaller diameter saphenous trunk, and absence of terminal valve reflux in the SFJ are more likely to benefit from isolated phlebectomy., (Copyright © 2024 The Authors. Published by Elsevier B.V. All rights reserved.)

Background: Prediction models are often externally validated with data from a single study or cohort. However, the interpretation of performance estimates obtained with single-study external validation is not as straightforward as assumed. We aimed to illustrate this by conducting a large number of external validations of a prediction model for functional outcome in subarachnoid hemorrhage (SAH) patients., Methods: We used data from the Subarachnoid Hemorrhage International Trialists (SAHIT) data repository (n = 11,931, 14 studies) to refit the SAHIT model for predicting a dichotomous functional outcome (favorable versus unfavorable), with the (extended) Glasgow Outcome Scale or modified Rankin Scale score, at a minimum of three months after discharge. We performed leave-one-cluster-out cross-validation to mimic the process of multiple single-study external validations. Each study represented one cluster. In each of these validations, we assessed discrimination with Harrell's c-statistic and calibration with calibration plots, the intercepts, and the slopes. We used random effects meta-analysis to obtain the (reference) mean performance estimates and between-study heterogeneity (I 2 -statistic). The influence of case-mix variation on discriminative performance was assessed with the model-based c-statistic and we fitted a "membership model" to obtain a gross estimate of transportability., Results: Across 14 single-study external validations, model performance was highly variable. The mean c-statistic was 0.74 (95%CI 0.70-0.78, range 0.52-0.84, I 2  = 0.92), the mean intercept was -0.06 (95%CI -0.37-0.24, range -1.40-0.75, I 2  = 0.97), and the mean slope was 0.96 (95%CI 0.78-1.13, range 0.53-1.31, I 2  = 0.90). The decrease in discriminative performance was attributable to case-mix variation, between-study heterogeneity, or a combination of both. Incidentally, we observed poor generalizability or transportability of the model., Conclusions: We demonstrate two potential pitfalls in the interpretation of model performance with single-study external validation. With single-study external validation. (1) model performance is highly variable and depends on the choice of validation data and (2) no insight is provided into generalizability or transportability of the model that is needed to guide local implementation. As such, a single single-study external validation can easily be misinterpreted and lead to a false appreciation of the clinical prediction model. Cross-validation is better equipped to address these pitfalls., (© 2024. The Author(s).)

Background and Objectives: Whether patent foramen ovale (PFO) closure benefits older patients with PFO and cryptogenic stroke is unknown because randomized controlled trials (RCTs) have predominantly enrolled patients younger than 60 years of age. Our objective was to estimate anticipated effects of PFO closure in older patients to predict the numbers needed to plan an RCT., Methods: Effectiveness estimates are derived from major observational studies (Risk of Paradoxical Embolism [RoPE] Study and Oxford Vascular Study, together referred to as the "RoPE-Ox" database) and all 6 major RCTs (Systematic, Collaborative, PFO Closure Evaluation [SCOPE] Consortium). To estimate stroke recurrence risk, observed outcomes were calculated for patients older than 60 years in the age-inclusive observational databases (n = 549). To estimate the reduction in the rate of recurrent stroke associated with PFO closure vs medical therapy based on the RoPE score and the presence of high-risk PFO features, a Cox proportional hazards regression model was developed on the RCT data in the SCOPE database (n = 3,740). These estimates were used to calculate sample sizes required for a future RCT., Results: Five-year risk of stroke recurrence using Kaplan-Meier estimates was 13.7 (95% CI 10.5-17.9) overall, 14.9% (95% CI 10.2-21.6) in those with high-risk PFO features. Predicted relative reduction in the event rate with PFO closure was 12.9% overall, 48.8% in those with a high-risk PFO feature. Using these estimates, enrolling all older patients with cryptogenic stroke and PFO would require much larger samples than those used for prior PFO closure trials, but selectively enrolling patients with high-risk PFO features would require totals of 630 patients for 90% power and 471 patients for 80% power, with an average of 5 years of follow-up., Discussion: Based on our projections, anticipated effect sizes in older patients with high-risk features make a trial in these subjects feasible. With lengthening life expectancy in almost all regions of the world, the utility of PFO closure in older adults is increasingly important to explore.

Background: Nested case-control (NCC) designs are efficient for developing and validating prediction models that use expensive or difficult-to-obtain predictors, especially when the outcome is rare. Previous research has focused on how to develop prediction models in this sampling design, but little attention has been given to model validation in this context. We therefore aimed to systematically characterize the key elements for the correct evaluation of the performance of prediction models in NCC data., Methods: We proposed how to correctly evaluate prediction models in NCC data, by adjusting performance metrics with sampling weights to account for the NCC sampling. We included in this study the C-index, threshold-based metrics, Observed-to-expected events ratio (O/E ratio), calibration slope, and decision curve analysis. We illustrated the proposed metrics with a validation of the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA version 5) in data from the population-based Rotterdam study. We compared the metrics obtained in the full cohort with those obtained in NCC datasets sampled from the Rotterdam study, with and without a matched design., Results: Performance metrics without weight adjustment were biased: the unweighted C-index in NCC datasets was 0.61 (0.58-0.63) for the unmatched design, while the C-index in the full cohort and the weighted C-index in the NCC datasets were similar: 0.65 (0.62-0.69) and 0.65 (0.61-0.69), respectively. The unweighted O/E ratio was 18.38 (17.67-19.06) in the NCC datasets, while it was 1.69 (1.42-1.93) in the full cohort and its weighted version in the NCC datasets was 1.68 (1.53-1.84). Similarly, weighted adjustments of threshold-based metrics and net benefit for decision curves were unbiased estimates of the corresponding metrics in the full cohort, while the corresponding unweighted metrics were biased. In the matched design, the bias of the unweighted metrics was larger, but it could also be compensated by the weight adjustment., Conclusions: Nested case-control studies are an efficient solution for evaluating the performance of prediction models that use expensive or difficult-to-obtain biomarkers, especially when the outcome is rare, but the performance metrics need to be adjusted to the sampling procedure., (© 2024. The Author(s).)

Background: The introduction of adjuvant systemic treatment for patients with high-risk melanomas necessitates accurate staging of disease. However, inconsistencies in outcomes exist between disease stages as defined by the American Joint Committee on Cancer (8th edition). We aimed to develop a tool to predict patient-specific outcomes in people with melanoma rather than grouping patients according to disease stage., Methods: Patients older than 13 years with confirmed primary melanoma who underwent sentinel lymph node biopsy (SLNB) between Oct 29, 1997, and Nov 11, 2013, at four European melanoma centres (based in Berlin, Germany; Amsterdam and Rotterdam, the Netherlands; and Warsaw, Poland) were included in the development cohort. Potential predictors of recurrence-free and melanoma-specific survival assessed were sex, age, presence of ulceration, primary tumour location, histological subtype, Breslow thickness, sentinel node status, number of sentinel nodes removed, maximum diameter of the largest sentinel node metastasis, and Dewar classification. A prognostic model and nomogram were developed to predict 5-year recurrence-free survival on a continuous scale in patients with stage pT1b or higher melanomas. This model was also calibrated to predict melanoma-specific survival. Model performance was assessed by discrimination (area under the time-dependent receiver operating characteristics curve [AUC]) and calibration. External validation was done in a cohort of patients with primary melanomas who underwent SLNB between Jan 30, 1997, and Dec 12, 2013, at the Melanoma Institute Australia (Sydney, NSW, Australia)., Findings: The development cohort consisted of 4071 patients, of whom 2075 (51%) were female and 1996 (49%) were male. 889 (22%) had sentinel node-positive disease and 3182 (78%) had sentinel node-negative disease. The validation cohort comprised 4822 patients, of whom 1965 (41%) were female and 2857 (59%) were male. 891 (18%) had sentinel node-positive disease and 3931 (82%) had sentinel node-negative disease. Median follow-up was 4·8 years (IQR 2·3-7·8) in the development cohort and 5·0 years (2·2-8·9) in the validation cohort. In the development cohort, 5-year recurrence-free survival was 73·5% (95% CI 72·0-75·1) and 5-year melanoma-specific survival was 86·5% (85·3-87·8). In the validation cohort, the corresponding estimates were 66·1% (64·6-67·7) and 83·3% (82·0-84·6), respectively. The final model contained six prognostic factors: sentinel node status, Breslow thickness, presence of ulceration, age at SLNB, primary tumour location, and maximum diameter of the largest sentinel node metastasis. In the development cohort, for the model's prediction of recurrence-free survival, the AUC was 0·80 (95% CI 0·78-0·81); for prediction of melanoma-specific survival, the AUC was 0·81 (0·79-0·84). External validation showed good calibration for both outcomes, with AUCs of 0·73 (0·71-0·75) and 0·76 (0·74-0·78), respectively., Interpretation: Our prediction model and nomogram accurately predicted patient-specific risk probabilities for 5-year recurrence-free and melanoma-specific survival. These tools could have important implications for clinical decision making when considering adjuvant treatments in patients with high-risk melanomas., Funding: Erasmus Medical Centre Cancer Institute., Competing Interests: Declaration of interests RCS reports honoraria (paid to their institution) from Amgen. AAMvdV reports consultancy fees (paid to their institution) from Bristol Myers Squibb, Merck Sharp & Dohme, Sanofi, Pierre Fabre, Ipsen, Eisai, Merck, Pfizer, Novartis, and Roche. RPMS has received honoraria for advisory board participation from Merck Sharp & Dohme, Novartis, and Qbiotics, and speaker fees from Bristol Myers Squibb and Novartis. AHRV has received honoraria from Novartis. RAS has received fees for professional services from MetaOptima Technology, F Hoffmann-La Roche, Evaxion, Provectus, Qbiotics, Novartis, Merck Sharp & Dohme, NeraCare, Amgen, Bristol Myers Squibb, Myriad Genetics, and GlaxoSmithKline. GVL reports consultancy or advisory fees from Agenus, Amgen, Array Biopharma, AstraZeneca, Boehringer Ingelheim, Bristol Myers Squibb, Evaxion, Hexal, Highlight Therapeutics, Innovent Biologics, Novartis, OncoSec, PHMR, Pierre Fabre, Provectus, Qbiotics, and Regeneron. JFT has received honoraria for advisory board participation from Bristol Myers Squibb, Merck Sharp & Dohme, GlaxoSmithKline, and Provectus, and travel and conference support from GlaxoSmithKline, Provectus, and Novartis. PR reports research funding (paid to their institution) from Merck Sharp & Dohme and Pfizer, and has received honoraria for lectures and advisory board participation from Merck Sharp & Dohme, Bristol Myers Squibb, Novartis, Pierre Fabre, Sanofi, Merck, Philogen, and Blueprint Medicines. UK has received consulting fees from Merck Sharp & Dohme, travel support from Merck Serono and Pfizer, and grants for educational activities from Merck Serono, Bristol Myers Squibb, and Pierre-Fabre. ACJvA reports honoraria for consultancy and serving on advisory boards (all paid to their institution) from Amgen, Bristol Myers Squibb, Merck Sharp & Dohme, Merck, Pfizer, Neracare, Novartis, Pierre Fabre, Provectus, Sanofi, Sirius Medical, and 4SC, and research grants (paid to their institution) from Amgen, Merck, and Pfizer. DvK has participated on the data and safety monitoring boards of the FAST CABG and Multivessel Talent trials. DJG has served on data and safety monitoring boards for Amgen and Novartis (funds paid to their institution). All other authors declare no competing interests., (Copyright © 2024 Elsevier Ltd. All rights reserved.)

Background: In patients with aneurysmal subarachnoid hemorrhage suitable for endovascular coiling and neurosurgical clip-reconstruction, the aneurysm treatment decision-making process could be improved by considering heterogeneity of treatment effect and durability of treatment. We aimed to develop and validate a tool to predict individualized treatment benefit of endovascular coiling compared to neurosurgical clip-reconstruction., Methods: We used randomized data (International Subarachnoid Aneurysm Trial, n = 2143) to develop models to predict 2-month functional outcome and to predict time-to-rebleed-or-retreatment. We modeled for heterogeneity of treatment effect by adding interaction terms of treatment with prespecified predictors and with baseline risk of the outcome. We predicted outcome with both treatments and calculated absolute treatment benefit. We described the patient characteristics of patients with ≥ 5% point difference in the predicted probability of favorable functional outcome (modified Rankin Score 0-2) and of no rebleed or retreatment within 10 years. Model performance was expressed with the c-statistic and calibration plots. We performed bootstrapping and leave-one-cluster-out cross-validation and pooled cluster-specific c-statistics with random effects meta-analysis., Results: The pooled c-statistics were 0.72 (95% CI: 0.69-0.75) for the prediction of 2-month favorable functional outcome and 0.67 (95% CI: 0.63-0.71) for prediction of no rebleed or retreatment within 10 years. We found no significant interaction between predictors and treatment. The average predicted benefit in favorable functional outcome was 6% (95% CI: 3-10%) in favor of coiling, but 11% (95% CI: 9-13%) for no rebleed or retreatment in favor of clip-reconstruction. 134 patients (6%), young and in favorable clinical condition, had negligible functional outcome benefit of coiling but had a ≥ 5% point benefit of clip-reconstruction in terms of durability of treatment., Conclusions: We show that young patients in favorable clinical condition and without extensive vasospasm have a negligible benefit in functional outcome of endovascular coiling - compared to neurosurgical clip-reconstruction - while at the same time having a substantially lower probability of retreatment or rebleeding from neurosurgical clip-reconstruction - compared to endovascular coiling. The SHARP prediction tool ( https://sharpmodels.shinyapps.io/sharpmodels/ ) could support and incentivize a multidisciplinary discussion about aneurysm treatment decision-making by providing individualized treatment benefit estimates., (© 2024. The Author(s).)

Background: In patients with 3-vessel coronary artery disease (CAD) and/or left main CAD, individual risk prediction plays a key role in deciding between percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG)., Objectives: The aim of this study was to assess whether these individualized revascularization decisions can be improved by applying machine learning (ML) algorithms and integrating clinical, biological, and anatomical factors., Methods: In the SYNTAX (Synergy between PCI with Taxus and Cardiac Surgery) study, ML algorithms (Lasso regression, gradient boosting) were used to develop a prognostic index for 5-year death, which was combined, in the second stage, with assigned treatment (PCI or CABG) and prespecified effect-modifiers: disease type (3-vessel or left main CAD) and anatomical SYNTAX score. The model's discriminative ability to predict the risk of 5-year death and treatment benefit between PCI and CABG was cross-validated in the SYNTAX trial (n = 1,800) and externally validated in the CREDO-Kyoto (Coronary REvascularization Demonstrating Outcome Study in Kyoto) registry (n = 7,362), and then compared with the original SYNTAX score II 2020 (SSII-2020)., Results: The hybrid gradient boosting model performed best for predicting 5-year all-cause death with C-indexes of 0.78 (95% CI: 0.75-0.81) in cross-validation and 0.77 (95% CI: 0.76-0.79) in external validation. The ML models discriminated 5-year mortality better than the SSII-2020 in the external validation cohort and identified heterogeneity in the treatment benefit of CABG vs PCI., Conclusions: An ML-based approach for identifying individuals who benefit from CABG or PCI is feasible and effective. Implementation of this model in health care systems-trained to collect large numbers of parameters-may harmonize decision making globally. (Synergy Between PCI With TAXUS and Cardiac Surgery: SYNTAX Extended Survival [SYNTAXES]; NCT03417050; SYNTAX Study: TAXUS Drug-Eluting Stent Versus Coronary Artery Bypass Surgery for the Treatment of Narrowed Arteries; NCT00114972)., Competing Interests: Funding Support and Author Disclosures The SYNTAX trial, during 0 to 5 years follow-up, was funded by Boston Scientific Corporation. The sponsor had no role in the study design, data collection, data analyses, and interpretation of the study data, nor were they involved in the decision to publish the final paper. The principal investigators and authors had complete scientific freedom. Prof Serruys has received consultancy fees from SMT (Sahajanand Medical technological), Novartis, Xeltis, Merillife, and Philips. All other authors have reported that they have no relationships relevant to the contents of this paper to disclose., (Copyright © 2023. Published by Elsevier Inc.)

Background: Incisional hernia occurs approximately in 40% of high-risk patients after midline laparotomy. Prophylactic mesh placement has shown promising results, but long-term outcomes are needed. The present study aimed to assess the long-term incisional hernia rates of the previously conducted PRIMA trial with radiological follow-up., Methods: In the PRIMA trial, patients with increased risk of incisional hernia formation (AAA or BMI ≥27 kg/m 2 ) were randomised in a 1:2:2 ratio to primary suture, onlay mesh or sublay mesh closure in three different countries in eleven institutions. Incisional hernia during follow-up was diagnosed by any of: CT, ultrasound and physical examination, or during surgery. Assessors and patients were blinded until 2-year follow-up. Time-to-event analysis according to intention-to-treat principle was performed with the Kaplan-Meier method and Cox proportional hazard models. Trial registration: NCT00761475 (ClinicalTrials.gov)., Findings: Between 2009 and 2012, 480 patients were randomized: 107 primary suture, 188 onlay mesh and 185 sublay mesh. Five-year incisional hernia rates were 53.4% (95% CI: 40.4-64.8), 24.7% (95% CI: 12.7-38.8), 29.8% (95% CI: 17.9-42.6), respectively. Compared to primary suture, onlay mesh (HR: 0.390, 95% CI: 0.248-0.614, p < 0.001) and sublay mesh (HR: 0.485, 95% CI: 0.309-0.761, p = 0.002) were associated with a significantly lower risk of incisional hernia development., Interpretation: Prophylactic mesh placement remained effective in reducing incisional hernia occurrence after midline laparotomy in high-risk patients during long-term follow-up. Hernia rates in the primary suture group were higher than previously anticipated., Funding: B. Braun., Competing Interests: None of the authors declare conflict of interest., (© 2023 The Author(s).)

Introduction: Integrated care is effective in reducing all-cause mortality in patients with atrial fibrillation (AF) in primary care, though time and resource intensive. The aim of the current study was to assess whether integrated care should be directed at all AF patients equally., Methods: The ALL-IN trial (n = 1,240 patients, median age 77 years) was a cluster-randomized trial in which primary care practices were randomized to provide integrated care or usual care to AF patients aged 65 years and older. Integrated care comprised of (i) anticoagulation monitoring, (ii) quarterly checkups and (iii) easy-access consultation with cardiologists. For the current analysis, cox proportional hazard analysis with all clinical variables from the CHA2DS2-VASc score was used to predict all-cause mortality in the ALL-IN trial. Subsequently, the hazard ratio and absolute risk reduction were plotted as a function of this predicted mortality risk to explore treatment heterogeneity., Results: Under usual care, after a median of 2 years follow-up the absolute risk of all-cause mortality in the highest-risk quarter was 31.0%, compared to 4.6% in the lowest-risk quarter. On the relative scale, there was no evidence of treatment heterogeneity (p for interaction = 0.90). However, there was substantial treatment heterogeneity on the absolute scale: risk reduction in the lowest risk- quarter of risk 3.3% (95% CI -0.4% - 7.0) compared to 12.0% (95% CI 2.7% - 22.0) in the highest risk quarter., Conclusion: While the relative degree of benefit from integrated AF care is similar in all patients, patients with a high all-cause mortality risk have a greater benefit on an absolute scale and should therefore be prioritized when implementing integrated care., Competing Interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: G.J. Geersing, F.H. Rutten, and M.E.W. Hemels report unrestricted institutional grants for performing research in the field of atrial fibrillation from Boehringer-Ingelheim, Bayer Healthcare, BMS Pfizer and Daiichi Sankyo. I.C. van Gelder reports consultancy fees from Boston, BMS and Bayer to the institution, unrestricted research grants from the Netherlands Cardiovascular Research Initiative, unrestricted research grant from the European Union’s Horizon 2020 research and innovation programme under grant agreement: EHRA-PATHS (945260). M. Rienstra reports Consultancy fees from Bayer, Microport, InCarda Therapeutics to the institution, an unrestricted research grant from ZonMW and the Dutch Heart Foundation; DECISION project 848090001, unrestricted research grants from the Netherlands Cardiovascular Research Initiative: an initiative with support of the Dutch Heart Foundation; RACE V (CVON 2014–9), RED-CVD (CVON2017-11), an unrestricted research grant from Top Sector Life Sciences & Health to the Dutch Heart Foundation (PPP Allowance; CVON-AI (2018B017)), and an unrestricted research grant from the European Union’s Horizon 2020 research and innovation programme under grant agreement; EHRA-PATHS (945260). S. van Doorn reports an unrestricted institutional grant for performing research in the field of stroke diagnosis from Stoffels-Hornstra. This does not alter our adherence to PLOS ONE policies on sharing data and materials., (Copyright: © 2023 Trinks-Roerdink et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Background: Cutaneous squamous cell carcinoma (cSCC) is a common skin cancer, affecting more than 2 million people worldwide yearly and metastasising in 2-5% of patients. However, current clinical staging systems do not provide estimates of absolute metastatic risk, hence missing the opportunity for more personalised treatment advice. We aimed to develop a clinico-pathological model that predicts the probability of metastasis in patients with cSCC., Methods: Nationwide cohorts from (1) all patients with a first primary cSCC in The Netherlands in 2007-2008 and (2) all patients with a cSCC in 2013-2015 in England were used to derive nested case-control cohorts. Pathology records of primary cSCCs that originated a loco-regional or distant metastasis were identified, and these cSCCs were matched to primary cSCCs of controls without metastasis (1:1 ratio). The model was developed on the Dutch cohort (n = 390) using a weighted Cox regression model with backward selection and validated on the English cohort (n = 696). Model performance was assessed using weighted versions of the C-index, calibration metrics, and decision curve analysis; and compared to the Brigham and Women's Hospital (BWH) and the American Joint Committee on Cancer (AJCC) staging systems. Members of the multidisciplinary Skin Cancer Outcomes (SCOUT) consortium were surveyed to interpret metastatic risk cutoffs in a clinical context., Findings: Eight out of eleven clinico-pathological variables were selected. The model showed good discriminative ability, with an optimism-corrected C-index of 0.80 (95% Confidence interval (CI) 0.75-0.85) in the development cohort and a C-index of 0.84 (95% CI 0.81-0.87) in the validation cohort. Model predictions were well-calibrated: the calibration slope was 0.96 (95% CI 0.76-1.16) in the validation cohort. Decision curve analysis showed improved net benefit compared to current staging systems, particularly for thresholds relevant for decisions on follow-up and adjuvant treatment. The model is available as an online web-based calculator (https://emc-dermatology.shinyapps.io/cscc-abs-met-risk/)., Interpretation: This validated model assigns personalised metastatic risk predictions to patients with cSCC, using routinely reported histological and patient-specific risk factors. The model can empower clinicians and healthcare systems in identifying patients with high-risk cSCC and offering personalised care/treatment and follow-up. Use of the model for clinical decision-making in different patient populations must be further investigated., Funding: PPP Allowance made available by Health-Holland, Top Sector Life Sciences & Health, to stimulate public-private partnerships., Competing Interests: MW participated as speaker/advisory board member/consultant for Sanofi Genzyme, Sunpharma and LEO Pharma. DB is employed by SkylineDx. BRP was employed by SkylineDx before beginning the submitted work (December 2019–July 2021). All other authors declare no competing interests., (© 2023 The Author(s).)

After mild traumatic brain injury (mTBI), a substantial proportion of individuals do not fully recover on the Glasgow Outcome Scale Extended (GOSE) or experience persistent post-concussion symptoms (PPCS). We aimed to develop prognostic models for the GOSE and PPCS at 6 months after mTBI and to assess the prognostic value of different categories of predictors (clinical variables; questionnaires; computed tomography [CT]; blood biomarkers). From the Collaborative European NeuroTrauma Effectiveness Research in Traumatic Brain Injury (CENTER-TBI) study, we included participants aged 16 or older with Glasgow Coma Score (GCS) 13-15. We used ordinal logistic regression to model the relationship between predictors and the GOSE, and linear regression to model the relationship between predictors and the Rivermead Post-concussion Symptoms Questionnaire (RPQ) total score. First, we studied a pre-specified Core model. Next, we extended the Core model with other clinical and sociodemographic variables available at presentation (Clinical model). The Clinical model was then extended with variables assessed before discharge from hospital: early post-concussion symptoms, CT variables, biomarkers, or all three categories (extended models). In a subset of patients mostly discharged home from the emergency department, the Clinical model was extended with 2-3-week post-concussion and mental health symptoms. Predictors were selected based on Akaike's Information Criterion. Performance of ordinal models was expressed as a concordance index (C) and performance of linear models as proportion of variance explained (R2). Bootstrap validation was used to correct for optimism. We included 2376 mTBI patients with 6-month GOSE and 1605 patients with 6-month RPQ. The Core and Clinical models for GOSE showed moderate discrimination (C = 0.68 95% confidence interval 0.68 to 0.70 and C = 0.70[0.69 to 0.71], respectively) and injury severity was the strongest predictor. The extended models had better discriminative ability (C = 0.71[0.69 to 0.72] with early symptoms; 0.71[0.70 to 0.72] with CT variables or with blood biomarkers; 0.72[0.71 to 0.73] with all three categories). The performance of models for RPQ was modest (R2 = 4% Core; R2 = 9% Clinical), and extensions with early symptoms increased the R2 to 12%. The 2-3-week models had better performance for both outcomes in the subset of participants with these symptoms measured (C = 0.74 [0.71 to 0.78] vs. C = 0.63[0.61 to 0.67] for GOSE; R2 = 37% vs. 6% for RPQ). In conclusion, the models based on variables available before discharge have moderate performance for the prediction of GOSE and poor performance for the prediction of PPCS. Symptoms assessed at 2-3 weeks are required for better predictive ability of both outcomes. The performance of the proposed models should be examined in independent cohorts.

Objectives: Detection of residual oesophageal cancer after neoadjuvant chemoradiotherapy (nCRT) is important to guide treatment decisions regarding standard oesophagectomy or active surveillance. The aim was to validate previously developed 18 F-FDG PET-based radiomic models to detect residual local tumour and to repeat model development (i.e. 'model extension') in case of poor generalisability., Methods: This was a retrospective cohort study in patients collected from a prospective multicentre study in four Dutch institutes. Patients underwent nCRT followed by oesophagectomy between 2013 and 2019. Outcome was tumour regression grade (TRG) 1 (0% tumour) versus TRG 2-3-4 (≥1% tumour). Scans were acquired according to standardised protocols. Discrimination and calibration were assessed for the published models with optimism-corrected AUCs >0.77. For model extension, the development and external validation cohorts were combined., Results: Baseline characteristics of the 189 patients included [median age 66 years (interquartile range 60-71), 158/189 male (84%), 40/189 TRG 1 (21%) and 149/189 (79%) TRG 2-3-4] were comparable to the development cohort. The model including cT stage plus the feature 'sum entropy' had best discriminative performance in external validation (AUC 0.64, 95% confidence interval 0.55-0.73), with a calibration slope and intercept of 0.16 and 0.48 respectively. An extended bootstrapped LASSO model yielded an AUC of 0.65 for TRG 2-3-4 detection., Conclusion: The high predictive performance of the published radiomic models could not be replicated. The extended model had moderate discriminative ability. The investigated radiomic models appeared inaccurate to detect local residual oesophageal tumour and cannot be used as an adjunct tool for clinical decision-making in patients., (Copyright © 2023 The Author(s). Published by Wolters Kluwer Health, Inc.)

Background: Measuring the performance of models that predict individualized treatment effect is challenging because the outcomes of two alternative treatments are inherently unobservable in one patient. The C-for-benefit was proposed to measure discriminative ability. However, measures of calibration and overall performance are still lacking. We aimed to propose metrics of calibration and overall performance for models predicting treatment effect in randomized clinical trials (RCTs)., Methods: Similar to the previously proposed C-for-benefit, we defined observed pairwise treatment effect as the difference between outcomes in pairs of matched patients with different treatment assignment. We match each untreated patient with the nearest treated patient based on the Mahalanobis distance between patient characteristics. Then, we define the E avg -for-benefit, E 50 -for-benefit, and E 90 -for-benefit as the average, median, and 90 th quantile of the absolute distance between the predicted pairwise treatment effects and local-regression-smoothed observed pairwise treatment effects. Furthermore, we define the cross-entropy-for-benefit and Brier-for-benefit as the logarithmic and average squared distance between predicted and observed pairwise treatment effects. In a simulation study, the metric values of deliberately "perturbed models" were compared to those of the data-generating model, i.e., "optimal model". To illustrate these performance metrics, different modeling approaches for predicting treatment effect are applied to the data of the Diabetes Prevention Program: 1) a risk modelling approach with restricted cubic splines; 2) an effect modelling approach including penalized treatment interactions; and 3) the causal forest., Results: As desired, performance metric values of "perturbed models" were consistently worse than those of the "optimal model" (E avg -for-benefit ≥ 0.043 versus 0.002, E 50 -for-benefit ≥ 0.032 versus 0.001, E 90 -for-benefit ≥ 0.084 versus 0.004, cross-entropy-for-benefit ≥ 0.765 versus 0.750, Brier-for-benefit ≥ 0.220 versus 0.218). Calibration, discriminative ability, and overall performance of three different models were similar in the case study. The proposed metrics were implemented in a publicly available R-package "HTEPredictionMetrics"., Conclusion: The proposed metrics are useful to assess the calibration and overall performance of models predicting treatment effect in RCTs., (© 2023. The Author(s).)

Background: Loss of life expectancy (LOLE) may provide more intuitive information on the impact of cancer than relative survival over a fixed time horizon (e.g., 5-year relative survival). We aimed to assess the evolution of the LOLE using a nationwide, population-based cohort including patients diagnosed with one of 17 most frequent solid malignancies., Methods: From the Netherlands Cancer Registry, we selected adult patients diagnosed with one of the 17 most frequent solid malignancies in the Netherlands during 1989-2019, with survival follow-up until 2022. We used flexible parametric survival models to estimate the LOLE at diagnosis and the LOLE after surviving several years post-diagnosis (conditional LOLE; CLOLE) by cancer type, calendar year, age, sex, and disease stage., Findings: For all cancers combined, the LOLE consistently decreased from 1989 to 2019. This decrease was most pronounced for males with prostate cancer (e.g., from 6.9 [95% confidence interval [CI], 6.7-7.1] to 2.7 [95% CI, 2.5-3.0] for 65-year-olds) and females with breast cancer (e.g., from 6.6 [95% CI, 6.4-6.7] to 1.9 [95% CI, 1.8-2.0] for 65-year-olds). The LOLE among patients with cancers of the head and neck or the central nervous system remained constant over time. Overall, the CLOLE showed that the life years lost among patients with cancer decreased with each additional year survived post-diagnosis. For example, the LOLE at diagnosis for 65-year-old females diagnosed with breast cancer in 2019 was 1.9 [95% CI, 1.8-2.0] compared with 1.7 [95% CI, 1.6-1.8], 1.0 [95% CI, 0.9-1.1], and 0.5 [95% CI, 0.5-0.6] when surviving one, five, and ten years post-diagnosis, respectively. Estimates for other combinations of patient and tumour characteristics are available in a publicly available web-based application., Interpretation: Our findings suggested that the evolution of LOLE substantially varies across cancer type, age, and disease stage. LOLE estimates help patients better understand the impact of their specific cancer diagnosis on their life expectancy., Funding: None., Competing Interests: All authors declare no competing interests., (© 2023 The Authors.)

Introduction: Clinical prediction models (CPMs) for coronavirus disease 2019 (COVID-19) may support clinical decision making, treatment, and communication. However, attitudes about using CPMs for COVID-19 decision making are unknown., Methods: Online focus groups and interviews were conducted among health care providers, survivors of COVID-19, and surrogates (i.e., loved ones/surrogate decision makers) in the United States and the Netherlands. Semistructured questions explored experiences about clinical decision making in COVID-19 care and facilitators and barriers for implementing CPMs., Results: In the United States, we conducted 4 online focus groups with 1) providers and 2) surrogates and survivors of COVID-19 between January 2021 and July 2021. In the Netherlands, we conducted 3 focus groups and 4 individual interviews with 1) providers and 2) surrogates and survivors of COVID-19 between May 2021 and July 2021. Providers expressed concern about CPM validity and the belief that patients may interpret CPM predictions as absolute. They described CPMs as potentially useful for resource allocation, triaging, education, and research. Several surrogates and people who had COVID-19 were not given prognostic estimates but believed this information would have supported and influenced their decision making. A limited number of participants felt the data would not have applied to them and that they or their loved ones may not have survived, as poor prognosis may have suggested withdrawal of treatment., Conclusions: Many providers had reservations about using CPMs for people with COVID-19 due to concerns about CPM validity and patient-level interpretation of the outcome predictions. However, several people who survived COVID-19 and their surrogates indicated that they would have found this information useful for decision making. Therefore, information provision may be needed to improve provider-level comfort and patient and surrogate understanding of CPMs., Highlights: While clinical prediction models (CPMs) may provide an objective means of assessing COVID-19 prognosis, provider concerns about CPM validity and the interpretation of CPM predictions may limit their clinical use.Providers felt that CPMs may be most useful for resource allocation, triage, research, or educational purposes for COVID-19.Several survivors of COVID-19 and their surrogates felt that CPMs would have been informative and may have aided them in making COVID-19 treatment decisions, while others felt the data would not have applied to them.

Treatment effects are often anticipated to vary across groups of patients with different baseline risk. The Predictive Approaches to Treatment Effect Heterogeneity (PATH) statement focused on baseline risk as a robust predictor of treatment effect and provided guidance on risk-based assessment of treatment effect heterogeneity in a randomized controlled trial. The aim of this study is to extend this approach to the observational setting using a standardized scalable framework. The proposed framework consists of five steps: (1) definition of the research aim, i.e., the population, the treatment, the comparator and the outcome(s) of interest; (2) identification of relevant databases; (3) development of a prediction model for the outcome(s) of interest; (4) estimation of relative and absolute treatment effect within strata of predicted risk, after adjusting for observed confounding; (5) presentation of the results. We demonstrate our framework by evaluating heterogeneity of the effect of thiazide or thiazide-like diuretics versus angiotensin-converting enzyme inhibitors on three efficacy and nine safety outcomes across three observational databases. We provide a publicly available R software package for applying this framework to any database mapped to the Observational Medical Outcomes Partnership Common Data Model. In our demonstration, patients at low risk of acute myocardial infarction receive negligible absolute benefits for all three efficacy outcomes, though they are more pronounced in the highest risk group, especially for acute myocardial infarction. Our framework allows for the evaluation of differential treatment effects across risk strata, which offers the opportunity to consider the benefit-harm trade-off between alternative treatments., (© 2023. The Author(s).)

Background: Baseline outcome risk can be an important determinant of absolute treatment benefit and has been used in guidelines for "personalizing" medical decisions. We compared easily applicable risk-based methods for optimal prediction of individualized treatment effects., Methods: We simulated RCT data using diverse assumptions for the average treatment effect, a baseline prognostic index of risk, the shape of its interaction with treatment (none, linear, quadratic or non-monotonic), and the magnitude of treatment-related harms (none or constant independent of the prognostic index). We predicted absolute benefit using: models with a constant relative treatment effect; stratification in quarters of the prognostic index; models including a linear interaction of treatment with the prognostic index; models including an interaction of treatment with a restricted cubic spline transformation of the prognostic index; an adaptive approach using Akaike's Information Criterion. We evaluated predictive performance using root mean squared error and measures of discrimination and calibration for benefit., Results: The linear-interaction model displayed optimal or close-to-optimal performance across many simulation scenarios with moderate sample size (N = 4,250; ~ 785 events). The restricted cubic splines model was optimal for strong non-linear deviations from a constant treatment effect, particularly when sample size was larger (N = 17,000). The adaptive approach also required larger sample sizes. These findings were illustrated in the GUSTO-I trial., Conclusions: An interaction between baseline risk and treatment assignment should be considered to improve treatment effect predictions., (© 2023. The Author(s).)

Introduction: Treatment decisions for aneurysmal subarachnoid haemorrhage patients should be supported by individualised predictions of the effects of aneurysm treatment. We present a study protocol and analysis plan for the development and external validation of models to predict benefit of neurosurgical versus endovascular aneurysm treatment on functional outcome and durability of treatment., Methods and Analysis: We will use data from the International Subarachnoid Aneurysm Trial for model development. The outcomes are functional outcome, measured with modified Rankin Scale at 12 months, and any retreatment or rebleed of the target aneurysm during follow-up. We will develop an ordinal logistic regression model and Cox regression model, considering age, World Federation of Neurological Surgeons grade, Fisher grade, vasospasm at presentation, aneurysm lumen size, aneurysm neck size, aneurysm location and time-to-aneurysm-treatment as predictors. We will test for interactions with treatment and with baseline risk and derive individualised predicted probabilities of treatment benefit. A benefit of ≥5% will be considered clinically relevant. Discriminative performance of the outcome predictions will be assessed with the c-statistic. Calibration will be assessed with calibration plots. Discriminative performance of the benefit predictions will be assessed with the c-for benefit. We will assess internal validity with bootstrapping and external validity with leave-one-out internal-external cross-validation., Ethics and Dissemination: The medical ethical research committee of the Erasmus MC University Medical Center Rotterdam approved the study protocol under the exemption category and waived the need for written informed consent (MEC-2020-0810). We will disseminate our results through an open-access peer-reviewed scientific publication and with a web-based clinical prediction tool., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Supporting decisions for patients who present to the emergency department (ED) with COVID-19 requires accurate prognostication. We aimed to evaluate prognostic models for predicting outcomes in hospitalized patients with COVID-19, in different locations and across time., Methods: We included patients who presented to the ED with suspected COVID-19 and were admitted to 12 hospitals in the New York City (NYC) area and 4 large Dutch hospitals. We used second-wave patients who presented between September and December 2020 (2137 and 3252 in NYC and the Netherlands, respectively) to evaluate models that were developed on first-wave patients who presented between March and August 2020 (12,163 and 5831). We evaluated two prognostic models for in-hospital death: The Northwell COVID-19 Survival (NOCOS) model was developed on NYC data and the COVID Outcome Prediction in the Emergency Department (COPE) model was developed on Dutch data. These models were validated on subsequent second-wave data at the same site (temporal validation) and at the other site (geographic validation). We assessed model performance by the Area Under the receiver operating characteristic Curve (AUC), by the E-statistic, and by net benefit., Results: Twenty-eight-day mortality was considerably higher in the NYC first-wave data (21.0%), compared to the second-wave (10.1%) and the Dutch data (first wave 10.8%; second wave 10.0%). COPE discriminated well at temporal validation (AUC 0.82), with excellent calibration (E-statistic 0.8%). At geographic validation, discrimination was satisfactory (AUC 0.78), but with moderate over-prediction of mortality risk, particularly in higher-risk patients (E-statistic 2.9%). While discrimination was adequate when NOCOS was tested on second-wave NYC data (AUC 0.77), NOCOS systematically overestimated the mortality risk (E-statistic 5.1%). Discrimination in the Dutch data was good (AUC 0.81), but with over-prediction of risk, particularly in lower-risk patients (E-statistic 4.0%). Recalibration of COPE and NOCOS led to limited net benefit improvement in Dutch data, but to substantial net benefit improvement in NYC data., Conclusions: NOCOS performed moderately worse than COPE, probably reflecting unique aspects of the early pandemic in NYC. Frequent updating of prognostic models is likely to be required for transportability over time and space during a dynamic pandemic., (© 2022. The Author(s).)