Search

Background and Objectives: Having a sufficient sample size is crucial when developing a clinical prediction model. We reviewed details of sample size in studies developing prediction models for binary outcomes using machine learning (ML) methods within oncology and compared the sample size used to develop the models with the minimum required sample size needed when developing a regression-based model (N min )., Methods: We searched the Medline (via OVID) database for studies developing a prediction model using ML methods published in December 2022. We reviewed how sample size was justified. We calculated N min , which is the N min , and compared this with the sample size that was used to develop the models., Results: Only one of 36 included studies justified their sample size. We were able to calculate N min for 17 (47%) studies. 5/17 studies met N min , allowing to precisely estimate the overall risk and minimize overfitting. There was a median deficit of 302 participants with the event (n = 17; range: -21,331 to 2298) when developing the ML models. An additional three out of the 17 studies met the required sample size to precisely estimate the overall risk only., Conclusion: Studies developing a prediction model using ML in oncology seldom justified their sample size and sample sizes were often smaller than N min . As ML models almost certainly require a larger sample size than regression models, the deficit is likely larger. We recommend that researchers consider and report their sample size and at least meet the minimum sample size required when developing a regression-based model., Competing Interests: Declaration of competing interest There are no competing interests for any author., (Copyright © 2025 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: Multicategory prediction models (MPMs) can be used in health care when the primary outcome of interest has more than two categories. The application of MPMs is scarce, possibly due to added methodological complexities compared to binary outcome models. We provide a guide of how to develop, validate, and update clinical prediction models based on multinomial logistic regression., Study Design and Setting: We present guidance and recommendations based on recent methodological literature, illustrated by a previously developed and validated MPM for treatment outcomes in rheumatoid arthritis. Prediction models using multinomial logistic regression can be developed for nominal outcomes, but also for ordinal outcomes. This article is intended to supplement existing general guidance on prediction model research., Results: This guide is split into three parts: 1) outcome definition and variable selection, 2) model development, and 3) model evaluation (including performance assessment, internal and external validation, and model recalibration). We outline how to evaluate and interpret the predictive performance of MPMs. R code is provided., Conclusion: We recommend the application of MPMs in clinical settings where the prediction of a multicategory outcome is of interest. Future methodological research could focus on MPM-specific considerations for variable selection and sample size criteria for external validation., Competing Interests: Declaration of competing interest K.L.H. has received speaker honoraria from Abbvie and grant income from Bristol Myers Squibb and Pfizer. There are no competing interests for any other author., (Copyright © 2024 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objective: Class imbalance is sometimes considered a problem when developing clinical prediction models and assessing their performance. To address it, correction strategies involving manipulations of the training dataset, such as random undersampling or oversampling, are frequently used. The aim of this article is to illustrate the consequences of these class imbalance correction strategies on clinical prediction models' internal validity in terms of calibration and discrimination performances., Methods: We used both heuristic intuition and formal mathematical reasoning to characterize the relations between conditional probabilities of interest and probabilities targeted when using random undersampling or oversampling. We propose a plug-in estimator that represents a natural correction for predictions obtained from models that have been trained on artificially balanced datasets ("naïve" models). We conducted a Monte Carlo simulation with two different data generation processes and present a real-world example using data from the International Stroke Trial database to empirically demonstrate the consequences of applying random resampling techniques for class imbalance correction on calibration and discrimination (in terms of Area Under the ROC, AUC) for logistic regression and tree-based prediction models., Results: Across our simulations and in the real-world example, calibration of the naïve models was very poor. The models using the plug-in estimator generally outperformed the models relying on class imbalance correction in terms of calibration while achieving the same discrimination performance., Conclusion: Random resampling techniques for class imbalance correction do not generally improve discrimination performance (i.e., AUC), and their use is hard to justify when aiming at providing calibrated predictions. Improper use of such class imbalance correction techniques can lead to suboptimal data usage and less valid risk prediction models., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: MP reports having received partial funding from Novartis Pharma and being awarded a research grant from the Center for Stroke Research Berlin (private donations), both outside of the submitted work. Since July 2023, MW has been employed at Data Sciences & Quantitative Biology, Discovery Sciences, R&D, AstraZeneca, Cambridge, UK. JLR reports receiving a grant from Novartis Pharma for a research project on migraine outside of the submitted work. BVC, SK and MVS have no conflicts of interest to disclose., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: To systematically review the risk of bias and applicability of published prediction models for risk of central line-associated bloodstream infection (CLA-BSI) in hospitalized patients., Study Design and Setting: Systematic review of literature in PubMed, Embase, Web of Science Core Collection, and Scopus up to July 10, 2023. Two authors independently appraised risk models using CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) and assessed their risk of bias and applicability using Prediction model Risk Of Bias ASsessment Tool (PROBAST)., Results: Sixteen studies were included, describing 37 models. When studies presented multiple algorithms, we focused on the model that was selected as the best by the study authors. Eventually we appraised 19 models, among which 15 were regression models and four machine learning models. All models were at a high risk of bias, primarily due to inappropriate proxy outcomes, predictors that are unavailable at prediction time in clinical practice, inadequate sample size, negligence of missing data, lack of model validation, and absence of calibration assessment. 18 out of 19 models had a high concern for applicability, one model had unclear concern for applicability due to incomplete reporting., Conclusion: We did not identify a prediction model of potential clinical use. There is a pressing need to develop an applicable model for CLA-BSI., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Inc. All rights reserved.)

Objectives: In biomedical research, spin is the overinterpretation of findings, and it is a growing concern. To date, the presence of spin has not been evaluated in prognostic model research in oncology, including studies developing and validating models for individualized risk prediction., Study Design and Setting: We conducted a systematic review, searching MEDLINE and EMBASE for oncology-related studies that developed and validated a prognostic model using machine learning published between 1st January, 2019, and 5th September, 2019. We used existing spin frameworks and described areas of highly suggestive spin practices., Results: We included 62 publications (including 152 developed models; 37 validated models). Reporting was inconsistent between methods and the results in 27% of studies due to additional analysis and selective reporting. Thirty-two studies (out of 36 applicable studies) reported comparisons between developed models in their discussion and predominantly used discrimination measures to support their claims (78%). Thirty-five studies (56%) used an overly strong or leading word in their title, abstract, results, discussion, or conclusion., Conclusion: The potential for spin needs to be considered when reading, interpreting, and using studies that developed and validated prognostic models in oncology. Researchers should carefully report their prognostic model research using words that reflect their actual results and strength of evidence., (Copyright © 2023 The Author(s). Published by Elsevier Inc. All rights reserved.)

Objectives: To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process., Study Design and Setting: Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts., Results: Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance., Conclusions: The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic., (Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.)

Background: Two randomized controlled trials compared the neonatal and infant outcomes after fetoscopic endoluminal tracheal occlusion with expectant prenatal management in fetuses with severe and moderate isolated congenital diaphragmatic hernia, respectively. Fetoscopic endoluminal tracheal occlusion was carried out at 27 +0 to 29 +6 weeks' gestation (referred to as "early") for severe and at 30 +0 to 31 +6 weeks ("late") for moderate hypoplasia. The reported absolute increase in the survival to discharge was 13% (95% confidence interval, -1 to 28; P=.059) and 25% (95% confidence interval, 6-46; P=.0091) for moderate and severe hypoplasia., Objective: Data from the 2 trials were pooled to study the heterogeneity of the treatment effect by observed over expected lung-to-head ratio and explore the effect of gestational age at balloon insertion., Study Design: Individual participant data from the 2 trials were reanalyzed. Women were assessed between 2008 and 2020 at 14 experienced fetoscopic endoluminal tracheal occlusion centers and were randomized in a 1:1 ratio to either expectant management or fetoscopic endoluminal tracheal occlusion. All received standardized postnatal management. The combined data involved 287 patients (196 with moderate hypoplasia and 91 with severe hypoplasia). The primary endpoint was survival to discharge from the neonatal intensive care unit. The secondary endpoints were survival to 6 months of age, survival to 6 months without oxygen supplementation, and gestational age at live birth. Penalized regression was used with the following covariates: intervention (fetoscopic endoluminal tracheal occlusion vs expectant), early balloon insertion (yes vs no), observed over expected lung-to-head ratio, liver herniation (yes vs no), and trial (severe vs moderate). The interaction between intervention and the observed over expected lung-to-head ratio was evaluated to study treatment effect heterogeneity., Results: For survival to discharge, the adjusted odds ratio of fetoscopic endoluminal tracheal occlusion was 1.78 (95% confidence interval, 1.05-3.01; P=.031). The additional effect of early balloon insertion was highly uncertain (adjusted odds ratio, 1.53; 95% confidence interval, 0.60-3.91; P=.370). When combining these 2 effects, the adjusted odds ratio of fetoscopic endoluminal tracheal occlusion with early balloon insertion was 2.73 (95% confidence interval, 1.15-6.49). The results for survival to 6 months and survival to 6 months without oxygen dependence were comparable. The gestational age at delivery was on average 1.7 weeks earlier (95% confidence interval, 1.1-2.3) following fetoscopic endoluminal tracheal occlusion with late insertion and 3.2 weeks earlier (95% confidence interval, 2.3-4.1) following fetoscopic endoluminal tracheal occlusion with early insertion compared with expectant management. There was no evidence that the effect of fetoscopic endoluminal tracheal occlusion depended on the observed over expected lung-to-head ratio for any of the endpoints., Conclusion: This analysis suggests that fetoscopic endoluminal tracheal occlusion increases survival for both moderate and severe lung hypoplasia. The difference between the results for the Tracheal Occlusion To Accelerate Lung growth trials, when considered apart, may be because of the difference in the time point of balloon insertion. However, the effect of the time point of balloon insertion could not be robustly assessed because of a small sample size and the confounding effect of disease severity. Fetoscopic endoluminal tracheal occlusion with early balloon insertion in particular strongly increases the risk for preterm delivery., (Copyright © 2021 Elsevier Inc. All rights reserved.)

Covid-19 research made it painfully clear that the scandal of poor medical research, as denounced by Altman in 1994, persists today. The overall quality of medical research remains poor, despite longstanding criticisms. The problems are well known, but the research community fails to properly address them. We suggest that most problems stem from an underlying paradox: although methodology is undeniably the backbone of high-quality and responsible research, science consistently undervalues methodology. The focus remains more on the destination (research claims and metrics) than on the journey. Notwithstanding, research should serve society more than the reputation of those involved. While we notice that many initiatives are being established to improve components of the research cycle, these initiatives are too disjointed. The overall system is monolithic and slow to adapt. We assert that top-down action is needed from journals, universities, funders and governments to break the cycle and put methodology first. These actions should involve the widespread adoption of registered reports, balanced research funding between innovative, incremental and methodological research projects, full recognition and demystification of peer review, improved methodological review of reports, adherence to reporting guidelines, and investment in methodological education and research. Currently, the scientific enterprise is doing a major disservice to patients and society., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Objective: Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology., Study Design and Setting: We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical prediction model using machine learning methods (as defined by primary study authors) in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement to assess the reporting quality of included publications. We described overall reporting adherence of included publications and by each section of TRIPOD., Results: Sixty-two publications met the inclusion criteria. 48 were development studies and 14 were development with validation studies. 152 models were developed across all publications. Median adherence to TRIPOD reporting items was 41% [range: 10%-67%] and at least 50% adherence was found in 19% (n=12/62) of publications. Adherence was lower in development only studies (median: 38% [range: 10%-67%]); and higher in development with validation studies (median: 49% [range: 33%-59%])., Conclusion: Reporting of clinical prediction models using machine learning in oncology is poor and needs urgent improvement, so readers and stakeholders can appraise the study methods, understand study findings, and reduce research waste., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: Receiver operating characteristic (ROC) curves show how well a risk prediction model discriminates between patients with and without a condition. We aim to investigate how ROC curves are presented in the literature and discuss and illustrate their potential limitations., Study Design and Setting: We conducted a pragmatic literature review of contemporary publications that externally validated clinical prediction models. We illustrated limitations of ROC curves using a testicular cancer case study and simulated data., Results: Of 86 identified prediction modeling studies, 52 (60%) presented ROC curves without thresholds and one (1%) presented an ROC curve with only a few thresholds. We illustrate that ROC curves in their standard form withhold threshold information have an unstable shape even for the same area under the curve (AUC) and are problematic for comparing model performance conditional on threshold. We compare ROC curves with classification plots, which show sensitivity and specificity conditional on risk thresholds., Conclusion: ROC curves do not offer more information than the AUC to indicate discriminative ability. To assess the model's performance for decision-making, results should be provided conditional on risk thresholds. Therefore, if discriminatory ability must be visualized, classification plots are attractive., (Copyright © 2020. Published by Elsevier Inc.)

Objective: We aimed to explore the added value of common machine learning (ML) algorithms for prediction of outcome for moderate and severe traumatic brain injury., Study Design and Setting: We performed logistic regression (LR), lasso regression, and ridge regression with key baseline predictors in the IMPACT-II database (15 studies, n = 11,022). ML algorithms included support vector machines, random forests, gradient boosting machines, and artificial neural networks and were trained using the same predictors. To assess generalizability of predictions, we performed internal, internal-external, and external validation on the recent CENTER-TBI study (patients with Glasgow Coma Scale <13, n = 1,554). Both calibration (calibration slope/intercept) and discrimination (area under the curve) was quantified., Results: In the IMPACT-II database, 3,332/11,022 (30%) died and 5,233(48%) had unfavorable outcome (Glasgow Outcome Scale less than 4). In the CENTER-TBI study, 348/1,554(29%) died and 651(54%) had unfavorable outcome. Discrimination and calibration varied widely between the studies and less so between the studied algorithms. The mean area under the curve was 0.82 for mortality and 0.77 for unfavorable outcomes in the CENTER-TBI study., Conclusion: ML algorithms may not outperform traditional regression approaches in a low-dimensional setting for outcome prediction after moderate or severe traumatic brain injury. Similar to regression-based prediction models, ML algorithms should be rigorously validated to ensure applicability to new populations., (Copyright © 2020 The Authors. Published by Elsevier Inc. All rights reserved.)

Background: Early pregnancy losses are common, but their psychologic sequelae are often overlooked. Previous studies have established links between miscarriage and early symptoms of anxiety and depression. However, the incidence of posttraumatic stress symptoms and the psychologic response specifically to ectopic pregnancies have not been investigated., Objective: The purpose of this study was to investigate levels of posttraumatic stress, depression, and anxiety in women in the 9 months after early pregnancy loss, with a focus on miscarriage and ectopic pregnancy. Morbidity at 1 month was compared with a control group in healthy pregnancy., Study Design: This was a prospective cohort study. Consecutive women were recruited from the early pregnancy and antenatal clinics at 3 London hospitals and received emailed surveys that contained standardized psychologic assessments that included the Hospital Anxiety and Depression Scale and Posttraumatic stress Diagnostic Scale, at 1, 3, and 9 months after loss. Control subjects were assessed after a dating scan. We assessed the proportion of participants who met the screening criteria for posttraumatic stress and moderate/severe anxiety or depression. We used logistic regression to calculate adjusted odds ratios., Results: Seven hundred thirty-seven of 1098 women (67%) with early pregnancy loss (including 537 miscarriages and 116 ectopic pregnancies) and 171 of 187 control subjects (91%) agreed to participate. Four hundred ninety-two of the women with losses (67%) completed the Hospital Anxiety and Depression Scale after 1 month; 426 women (58%) completed it after 3 months, and 338 women (46%) completed it after 9 months. Eighty-seven control subjects (51%) participated. Criteria for posttraumatic stress were met in 29% of women with early pregnancy loss after 1 month and in 18% after 9 months (odds ratio per month, 0.80; 95% confidence interval, 0.72-0.89). Moderate/severe anxiety was reported in 24% after 1 month and in 17% after 9 months (odds ratio per month, 0.69; 95% confidence interval, 0.50-0.94). Moderate/severe depression was reported in 11% of the women after 1 month and 6% of the women after 9 months (odds ratio per month, 0.87; 95% confidence interval, 0.53-1.44). After miscarriage, proportions after 9 months were 16% for posttraumatic stress, 17% for anxiety, and 5% for depression. Corresponding figures after ectopic pregnancy were 21%, 23%, and 11%, respectively. In contrast, among control women with viable pregnancies, 13% reported moderate-to-severe anxiety (odds ratio loss at 1 month vs controls: 2.14; 95% confidence interval, 1.14-4.36), and 2% reported moderate-to-severe depression (odds ratio loss at 1 month vs control subjects: 3.88; 95% confidence interval, 1.27-19.2)., Conclusion: Women experience high levels of posttraumatic stress, anxiety, and depression after early pregnancy loss. Distress declines over time but remains at clinically important levels at 9 months., (Copyright © 2019. Published by Elsevier Inc.)

Objectives: The aim of this study was to quantify the impact of predictor measurement heterogeneity on prediction model performance. Predictor measurement heterogeneity refers to variation in the measurement of predictor(s) between the derivation of a prediction model and its validation or application. It arises, for instance, when predictors are measured using different measurement instruments or protocols., Study Design and Setting: We examined the effects of various scenarios of predictor measurement heterogeneity in real-world clinical examples using previously developed prediction models for diagnosis of ovarian cancer, mutation carriers for Lynch syndrome, and intrauterine pregnancy., Results: Changing the measurement procedure of a predictor influenced the performance at validation of the prediction models in nine clinical examples. Notably, it induced model miscalibration. The calibration intercept at validation ranged from -0.70 to 1.43 (0 for good calibration), whereas the calibration slope ranged from 0.50 to 1.67 (1 for good calibration). The difference in C-statistic and scaled Brier score between derivation and validation ranged from -0.08 to +0.08 and from -0.40 to +0.16, respectively., Conclusion: This study illustrates that predictor measurement heterogeneity can influence the performance of a prediction model substantially, underlining that predictor measurements used in research settings should resemble clinical practice. Specification of measurement heterogeneity can help researchers explaining discrepancies in predictive performance between derivation and validation setting., (Copyright © 2019 The Authors. Published by Elsevier Inc. All rights reserved.)

Objectives: The objective of this study was to compare performance of logistic regression (LR) with machine learning (ML) for clinical prediction modeling in the literature., Study Design and Setting: We conducted a Medline literature search (1/2016 to 8/2017) and extracted comparisons between LR and ML models for binary outcomes., Results: We included 71 of 927 studies. The median sample size was 1,250 (range 72-3,994,872), with 19 predictors considered (range 5-563) and eight events per predictor (range 0.3-6,697). The most common ML methods were classification trees, random forests, artificial neural networks, and support vector machines. In 48 (68%) studies, we observed potential bias in the validation procedures. Sixty-four (90%) studies used the area under the receiver operating characteristic curve (AUC) to assess discrimination. Calibration was not addressed in 56 (79%) studies. We identified 282 comparisons between an LR and ML model (AUC range, 0.52-0.99). For 145 comparisons at low risk of bias, the difference in logit(AUC) between LR and ML was 0.00 (95% confidence interval, -0.18 to 0.18). For 137 comparisons at high risk of bias, logit(AUC) was 0.34 (0.20-0.47) higher for ML., Conclusion: We found no evidence of superior performance of ML over LR. Improvements in methodology and reporting are needed for studies that compare modeling algorithms., (Copyright © 2019 Elsevier Inc. All rights reserved.)

Purpose: Controversy remains around acceptable surgical delay of acute hip fractures with current guidelines ranging from 24 to 48 h. Increasing healthcare costs force us to consider the economic burden as well. We aimed to evaluate the adjusted effect of surgical delay for hip fracture surgery on early mortality, healthcare costs and readmission rate. We hypothesized that shorter delays resulted in lower early mortality and costs., Methods: In this retrospective cohort study 2573 consecutive patients aged ≥50 years were included, who underwent surgery for acute hip fractures between 2009 and 2017. Main endpoints were thirty- and ninety-day mortality, total cost, and readmission rate. Multivariable regression included sex, age and ASA score as covariates., Results: Thirty-day mortality was 5% (n = 133), ninety-day mortality 12% (n = 304). Average total cost was €11960, dominated by hospitalization (59%) and honoraria (23%). Per 24 h delay, the adjusted odds ratio was 1.07 (95% CI 0.98-1.18) for thirty-day mortality, 1.12 (95% CI 1.04-1.19) for ninety-day mortality, and 0.99 (95% CI = 0.88-1.12) for readmission. Per 24 h delay, costs increased with 7% (95% CI 6-8%). For mortality, delay was a weaker predictor than sex, age, and ASA score. For costs, delay was the strongest predictor. We did not find clear cut-points for surgical delay after which mortality or costs increased abruptly., Conclusions: Despite only modest associations with mortality, we observed a steady increase in healthcare costs when delaying surgery. Hence, a more pragmatic approach with surgery as soon as medically and organizationally possible seems justifiable over rigorous implementation of the current guidelines., (Copyright © 2018 Elsevier Ltd. All rights reserved.)

Objective: To evaluate limitations of common statistical modeling approaches in deriving clinical prediction models and explore alternative strategies., Study Design and Setting: A previously published model predicted the likelihood of having a mutation in germline DNA mismatch repair genes at the time of diagnosis of colorectal cancer. This model was based on a cohort where 38 mutations were found among 870 participants, with validation in an independent cohort with 35 mutations. The modeling strategy included stepwise selection of predictors from a pool of over 37 candidate predictors and dichotomization of continuous predictors. We simulated this strategy in small subsets of a large contemporary cohort (2,051 mutations among 19,866 participants) and made comparisons to other modeling approaches. All models were evaluated according to bias and discriminative ability (concordance index, c) in independent data., Results: We found over 50% bias for five of six originally selected predictors, unstable model specification, and poor performance at validation (median c = 0.74). A small validation sample hampered stable assessment of performance. Model prespecification based on external knowledge and using continuous predictors led to better performance (c = 0.836 and c = 0.852 with 38 and 2,051 events respectively)., Conclusion: Prediction models perform poorly if based on small numbers of events and developed with common but suboptimal statistical approaches. Alternative modeling strategies to best exploit available predictive information need wider implementation, with collaborative research to increase sample sizes., (Copyright © 2017 Elsevier Inc. All rights reserved.)

Objective: Calibrated risk models are vital for valid decision support. We define four levels of calibration and describe implications for model development and external validation of predictions., Study Design and Setting: We present results based on simulated data sets., Results: A common definition of calibration is "having an event rate of R% among patients with a predicted risk of R%," which we refer to as "moderate calibration." Weaker forms of calibration only require the average predicted risk (mean calibration) or the average prediction effects (weak calibration) to be correct. "Strong calibration" requires that the event rate equals the predicted risk for every covariate pattern. This implies that the model is fully correct for the validation setting. We argue that this is unrealistic: the model type may be incorrect, the linear predictor is only asymptotically unbiased, and all nonlinear and interaction effects should be correctly modeled. In addition, we prove that moderate calibration guarantees nonharmful decision making. Finally, results indicate that a flexible assessment of calibration in small validation data sets is problematic., Conclusion: Strong calibration is desirable for individualized decision support but unrealistic and counter productive by stimulating the development of overly complex models. Model development and external validation should focus on moderate calibration., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Background: Accurate methods to preoperatively characterize adnexal tumors are pivotal for optimal patient management. A recent metaanalysis concluded that the International Ovarian Tumor Analysis algorithms such as the Simple Rules are the best approaches to preoperatively classify adnexal masses as benign or malignant., Objective: We sought to develop and validate a model to predict the risk of malignancy in adnexal masses using the ultrasound features in the Simple Rules., Study Design: This was an international cross-sectional cohort study involving 22 oncology centers, referral centers for ultrasonography, and general hospitals. We included consecutive patients with an adnexal tumor who underwent a standardized transvaginal ultrasound examination and were selected for surgery. Data on 5020 patients were recorded in 3 phases from 2002 through 2012. The 5 Simple Rules features indicative of a benign tumor (B-features) and the 5 features indicative of malignancy (M-features) are based on the presence of ascites, tumor morphology, and degree of vascularity at ultrasonography. Gold standard was the histopathologic diagnosis of the adnexal mass (pathologist blinded to ultrasound findings). Logistic regression analysis was used to estimate the risk of malignancy based on the 10 ultrasound features and type of center. The diagnostic performance was evaluated by area under the receiver operating characteristic curve, sensitivity, specificity, positive likelihood ratio (LR+), negative likelihood ratio (LR-), positive predictive value (PPV), negative predictive value (NPV), and calibration curves., Results: Data on 4848 patients were analyzed. The malignancy rate was 43% (1402/3263) in oncology centers and 17% (263/1585) in other centers. The area under the receiver operating characteristic curve on validation data was very similar in oncology centers (0.917; 95% confidence interval, 0.901-0.931) and other centers (0.916; 95% confidence interval, 0.873-0.945). Risk estimates showed good calibration. In all, 23% of patients in the validation data set had a very low estimated risk (<1%) and 48% had a high estimated risk (≥30%). For the 1% risk cutoff, sensitivity was 99.7%, specificity 33.7%, LR+ 1.5, LR- 0.010, PPV 44.8%, and NPV 98.9%. For the 30% risk cutoff, sensitivity was 89.0%, specificity 84.7%, LR+ 5.8, LR- 0.13, PPV 75.4%, and NPV 93.9%., Conclusion: Quantification of the risk of malignancy based on the Simple Rules has good diagnostic performance both in oncology centers and other centers. A simple classification based on these risk estimates may form the basis of a clinical management system. Patients with a high risk may benefit from surgery by a gynecological oncologist, while patients with a lower risk may be managed locally., (Copyright © 2016 Elsevier Inc. All rights reserved.)

Objectives: This study aims to investigate the influence of the amount of clustering [intraclass correlation (ICC) = 0%, 5%, or 20%], the number of events per variable (EPV) or candidate predictor (EPV = 5, 10, 20, or 50), and backward variable selection on the performance of prediction models., Study Design and Setting: Researchers frequently combine data from several centers to develop clinical prediction models. In our simulation study, we developed models from clustered training data using multilevel logistic regression and validated them in external data., Results: The amount of clustering was not meaningfully associated with the models' predictive performance. The median calibration slope of models built in samples with EPV = 5 and strong clustering (ICC = 20%) was 0.71. With EPV = 5 and ICC = 0%, it was 0.72. A higher EPV related to an increased performance: the calibration slope was 0.85 at EPV = 10 and ICC = 20% and 0.96 at EPV = 50 and ICC = 20%. Variable selection sometimes led to a substantial relative bias in the estimated predictor effects (up to 118% at EPV = 5), but this had little influence on the model's performance in our simulations., Conclusion: We recommend at least 10 EPV to fit prediction models in clustered data using logistic regression. Up to 50 EPV may be needed when variable selection is performed., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Clinical risk calculators are now widely available but have generally been implemented in a static and one-size-fits-all fashion. The objective of this study was to challenge these notions and show via a case study concerning risk-based screening for prostate cancer how calculators can be dynamically and locally tailored to improve on-site patient accuracy. Yearly data from five international prostate biopsy cohorts (3 in the US, 1 in Austria, 1 in England) were used to compare 6 methods for annual risk prediction: static use of the online US-developed Prostate Cancer Prevention Trial Risk Calculator (PCPTRC); recalibration of the PCPTRC; revision of the PCPTRC; building a new model each year using logistic regression, Bayesian prior-to-posterior updating, or random forests. All methods performed similarly with respect to discrimination, except for random forests, which were worse. All methods except for random forests greatly improved calibration over the static PCPTRC in all cohorts except for Austria, where the PCPTRC had the best calibration followed closely by recalibration. The case study shows that a simple annual recalibration of a general online risk tool for prostate cancer can improve its accuracy with respect to the local patient practice at hand., (Copyright © 2015 Elsevier Inc. All rights reserved.)

When validating risk models (or probabilistic classifiers), calibration is often overlooked. Calibration refers to the reliability of the predicted risks, i.e. whether the predicted risks correspond to observed probabilities. In medical applications this is important because treatment decisions often rely on the estimated risk of disease. The aim of this paper is to present generic tools to assess the calibration of multiclass risk models. We describe a calibration framework based on a vector spline multinomial logistic regression model. This framework can be used to generate calibration plots and calculate the estimated calibration index (ECI) to quantify lack of calibration. We illustrate these tools in relation to risk models used to characterize ovarian tumors. The outcome of the study is the surgical stage of the tumor when relevant and the final histological outcome, which is divided into five classes: benign, borderline malignant, stage I, stage II-IV, and secondary metastatic cancer. The 5909 patients included in the study are randomly split into equally large training and test sets. We developed and tested models using the following algorithms: logistic regression, support vector machines, k nearest neighbors, random forest, naive Bayes and nearest shrunken centroids. Multiclass calibration plots are interesting as an approach to visualizing the reliability of predicted risks. The ECI is a convenient tool for comparing models, but is less informative and interpretable than calibration plots. In our case study, logistic regression and random forest showed the highest degree of calibration, and the naive Bayes the lowest., (Copyright © 2015 Elsevier Inc. All rights reserved.)

Purpose: To investigate the clinical behavior of triple negative breast cancer (TNC), including age distribution, occurrence of LN (lymph node) invasion and prognosis in different histological subtypes., Methods: For this cohort study we used data on 476 patients with newly diagnosed TNC at the University Hospitals Leuven (Belgium) between 1999 and 2009. Of these, 395 received upfront surgery, 68 neoadjuvant chemotherapy and 21 had metastases at diagnosis., Results: Apocrine and invasive lobular TNC occur more often in older patients compared to IDC-NOS. Of the primarily operated patients with TNC, 35.1% has pathological LN involvement. There were no significant differences in nodal invasion between different histological subtypes, but most subtypes contained few patients. In contrast to previous reports, 6/14 of apocrine TNC had LN involvement. Disease free survival (DFS) was different in different histological subtypes, but group sizes were insufficient to be able to draw firm conclusions. Within the histologically 'homogeneous' IDC-NOS group with primary surgery and outcome data (n = 300), DFS with 3.5 year median follow-up decreased with increasing age, but chemotherapy and radiotherapy were much less frequently given with increasing age. In multivariable analysis, lower age, presence of LN involvement, lack of administration of chemotherapy and radiotherapy were significant predictors of relapse., Conclusion: TNC is not a uniform disease. Different histological subtypes have different age distribution and behavior. The prognosis of the most common histological subgroup, IDC-NOS, is better in older patients, but this is counterbalanced by significantly decreased use of chemotherapy and radiotherapy., (Copyright © 2013 Elsevier Ltd. All rights reserved.)

Objectives: Prediction models may perform poorly in a new setting. We aimed to determine which model updating methods should be applied for models predicting polytomous outcomes, which often suffer from one or more categories with low prevalence., Study Design and Setting: We used case studies on testicular and ovarian tumors. The original regression models were based on 502 and 2,037 patients and validated on 273 and 1,107 patients, respectively. The polytomous models combined dichotomous models for category A vs. B + C and B vs. C (sequential dichotomous modeling). Simple recalibration, revision, and redevelopment methods were considered. To assess discrimination (using dichotomous and polytomous c-statistics) and calibration (by comparing observed and expected prevalences) of these methods, the validation data were divided into updating and test parts. Five hundred such divisions were randomly generated, and the average test set results reported., Results: None of the updating methods could improve discrimination of the original models, but recalibration, revision, and redevelopment strongly improved calibration. Redevelopment was unstable with respect to overfitting and performance., Conclusion: Simple dichotomous updating methods behaved well when applied to polytomous models. Our results suggest that recalibration is preferred, but larger validation sets may make revision or redevelopment a sensible alternative., (Copyright © 2013 Elsevier Inc. All rights reserved.)

Background: Aromatase inhibitors (AIs) frequently lead to the AI-induced musculoskeletal syndrome (AIMSS). Looking into its pathophysiology, 6 months of AI therapy thickens the tendon sheath with intra-articular fluid (IAF) retention and loss of grip strength. We here report 24-month follow-up data., Patients and Methods: A prospective cohort study of 33 postmenopausal breast cancer patients received adjuvant endocrine therapy; 27 received an AI and 6 received tamoxifen. At baseline, 6 and 24 months patients had a rheumatologic examination, including a grip strength test, and magnetic resonance imaging of both hands and wrists. The primary end point was tenosynovial changes; secondary end points were changes in morning stiffness, grip strength and IAF., Results: Twenty-three AI and 5 tamoxifen patients completed all investigations. Between month 6 and 24, IAF further increased in AI users (P = 0.04) but not in tamoxifen users, and grip strength further decreased in both groups. The worsened tenosynovial changes were strongly correlated with a decrease in grip strength. At 24 months, morning stiffness continued to be present in over a third of AI users., Conclusion: AIMSS represents a substantial problem in breast cancer patients. It is associated with tenosynovial changes, IAF retention, joint stiffness and loss of grip strength that do not improve with prolonged use.

Background: We studied the stellate ganglion block (SGB) recently suggested for the treatment of severe vasomotor symptoms and sleep disturbances in breast cancer survivors. Following an initial pilot study, which focused on the acceptability and safety of SGB for this important problem, we evaluated its short- and long-term efficacy., Materials and Methods: Postmenopausal breast cancer survivors with severe vasomotor symptoms resistant to standard nonhormonal pharmacological intervention were eligible. Diaries were used to measure daily hot flash scores (frequency and intensity) and sleep quality (Pittsburgh Sleep Quality Index) during scheduled visits at baseline, 1, 4, 12 and 24 weeks following the SGB. Efficacy data were analyzed using longitudinal regression models., Results: Thirty-four patients participated and none refused the SGB procedure. Most patients received more than one SGB. The pilot study found SGB to be safe. In the main study, hot flash scores were reduced from baseline by 64% [95% confidence interval (CI) -74% to -49%] and 47% (95% CI -62% to -27%) at weeks 1 and 24, respectively. The odds ratio of better sleep quality relative to baseline was 3.4 at week 1 (95% CI 1.6-7.2) and 4.3 at week 24 (95% CI 1.9-9.8)., Conclusion: In the short term, SGB appears to be an effective treatment with acceptable morbidity for some breast cancer survivors with therapy-resistant vasomotor symptoms and/or sleep disturbances. Although sleep quality was maintained out to 24 weeks the efficacy of SGB for hot flashes was reduced over time. A randomized controlled trial is needed to confirm these findings.

Background: Our preliminary results showed that tenosynovial changes and decrease in grip strength are associated with the aromatase inhibitor-induced musculoskeletal syndrome (AIMSS). Here, we report the final results and assess the relationship between grip strength and body mass index (BMI)., Patients and Methods: We conducted a prospective study including postmenopausal early breast cancer patients receiving either an aromatase inhibitor (AI) or tamoxifen. Primary end point was change from baseline in tenosynovial abnormalities. Secondary end points were changes from baseline in morning stiffness, intra-articular fluid and grip strength and its association with BMI., Results: After 6 months of therapy, 74% [95% confidence interval (CI) 51% to 89%] of AI-treated patients had worsened tenosynovial abnormalities, 56% (95% CI 34% to 75%) had increased intra-articular fluid, and 22% (95% CI 9% to 45%) had increased morning stiffness. Grip strength decreased 8% for the left hand (95% CI 2% to 21%) and 11% for the right (95% CI 4% to 17%). Regression analysis suggested that grip strength decreased more for subjects with high or with low BMI., Conclusions: AIMSS is characterized by tenosynovial changes, intra-articular fluid and morning stiffness. We hypothesize that the quadratic association between BMI and loss of grip strength reflects AI-induced changes on the endocrine control of the growth hormone insulin-like growth factor-I pathway.

Study Objective: To identify factors associated with the need to perform uterine morcellation during total laparoscopic hysterectomy (TLH). A secondary aim was to establish new cut-offs based on uterine weight for the probability of morcellation., Design: Prospective observational study (Canadian Task Force Classification II-2)., Setting: Tertiary referral laparoscopic unit., Patients: All women scheduled to undergo TLH in the study period were included., Interventions: Age, parity, operating time, estimated blood loss, and final uterine weight at histology were recorded. Logistic regression analysis was performed to determine the factors associated with the need to perform uterine morcellation at the time of TLH. Multiple imputation (MI) was used to impute missing values., Measurements and Main Results: A total of 112 consecutive women underwent TLH and were included in the final analysis. In all, 56 (50%) of 112 women underwent TLH without morcellation (i.e., it was possible to deliver the uterine specimen vaginally) and 56 (50%) of 112 women underwent TLH with morcellation (i.e., it was not possible to deliver the uterine specimen vaginally and, therefore, morcellation was performed). Median age in each group was 45 and 46 years, respectively. Sixteen (70%) of 23 nulliparous women underwent morcellation compared with 40 (45%) of 89 parous women. Multivariable logistic regression analysis revealed that nulliparity (OR = 6.45, 95% CI = 1.74-23.9) and uterine weight (OR/100-g increase = 4.97, 95% CI = 2.13-11.6) increased the odds of morcellation. All 20 women with a uterine weight of at least 350 g required morcellation. Based on the MI analysis results, uterine weight was at least 350 g in 1 of 5 patients, with 99.5% of the women having uterine weight of 350 g or more that required morcellation., Conclusion: Nulliparity and increasing uterine weight are associated with the need to perform uterine morcellation in TLH. Studies are needed to find a reliable method for estimating uterine weight preoperatively.

Unlabelled: OBJECTIVE STUDY: To estimate the value of preoperative bimanual examination of the pelvis in women undergoing total laparoscopic hysterectomy (TLH)., Design: Prospective observational cohort study., Setting: Private hospitals and centre., Patients: One hundred fourteen consecutive women undergoing TLH., Intervention: All women who were scheduled to undergo TLH from May 2005 through June 2006 had a bimanual examination of the pelvis performed before surgery and the size of the uterus clinically estimated and recorded as gestational equivalents. The operating time, the estimated blood loss (EBL) during TLH, and the final weight of the uterus at histologic study were recorded. Spearman correlation coefficient analysis was used to determine whether there was a correlation between the estimated uterine size before surgery and actual uterine weight, operating time, and EBL., Measurements and Main Results: Of the one hundred fourteen consecutive women eligible for the study, 75 had complete data and therefore were included in the final analysis. The median age was 46 years (range 34-71 years); 22.7% (17/75) had a clinically estimated normal uterus, 10.7% (8/75) had an 8-10/40 uterus, 12% (9/75) had a 10-12/40 uterus, 14.6% (11/75) had a 12-14/40 uterus, 20.0% (15/75) had a 14-16/40 uterus, 9.3% (7/75) had a 16-18/40 uterus, and 10.7% (8/75) had an 18-20/40 uterus. The median operating time was 110 minutes (range 59-240 minutes); the median EBL was 80 mL (range 20-1000 mL); and the median weight of the uterus was 181 g (range 52-1080 g). Histologic diagnoses included leiomyomata in 64.0% (48/75), adenomyosis in 44.0% (33/75), endometriosis in 22.7% (17/75), endocervical polyp in 4.0% (3/75), and normal uterus in 8.0% (6/75). The Spearman correlations between clinical size of the uterus and the weight of the uterus, the EBL, and the operating time were 0.81, 0.33, and 0.29, respectively; that is, the 2 variables tended to increase together. These correlations were all significant (p <.0001, .0044, and .0114, respectively)., Conclusions: This study showed significant correlation between clinical estimate of uterine size and histologic weight of the uterus, operating time, and EBL in women undergoing laparoscopic hysterectomy. These findings are of great value in preoperative counseling in relation to the risk of bleeding and the potential need for blood transfusion, and in operating room planning.