Search

Objective: Evaluate the completeness of reporting of prognostic prediction models developed using machine learning methods in the field of oncology., Study Design and Setting: We conducted a systematic review, searching the MEDLINE and Embase databases between 01/01/2019 and 05/09/2019, for non-imaging studies developing a prognostic clinical prediction model using machine learning methods (as defined by primary study authors) in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement to assess the reporting quality of included publications. We described overall reporting adherence of included publications and by each section of TRIPOD., Results: Sixty-two publications met the inclusion criteria. 48 were development studies and 14 were development with validation studies. 152 models were developed across all publications. Median adherence to TRIPOD reporting items was 41% [range: 10%-67%] and at least 50% adherence was found in 19% (n=12/62) of publications. Adherence was lower in development only studies (median: 38% [range: 10%-67%]); and higher in development with validation studies (median: 49% [range: 33%-59%])., Conclusion: Reporting of clinical prediction models using machine learning in oncology is poor and needs urgent improvement, so readers and stakeholders can appraise the study methods, understand study findings, and reduce research waste., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Competing Interests: Competing interests: None.

Continuous predictors are routinely encountered when developing a prognostic model. Investigators, who are often non-statisticians, must decide how to handle continuous predictors in their models. Categorising continuous measurements into two or more categories has been widely discredited, yet is still frequently done because of its simplicity, investigator ignorance of the potential impact and of suitable alternatives, or to facilitate model uptake. We examine three broad approaches for handling continuous predictors on the performance of a prognostic model, including various methods of categorising predictors, modelling a linear relationship between the predictor and outcome and modelling a nonlinear relationship using fractional polynomials or restricted cubic splines. We compare the performance (measured by the c-index, calibration and net benefit) of prognostic models built using each approach, evaluating them using separate data from that used to build them. We show that categorising continuous predictors produces models with poor predictive performance and poor clinical usefulness. Categorising continuous predictors is unnecessary, biologically implausible and inefficient and should not be used in prognostic model development. © 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd., (© 2016 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.)

Objectives: To compare different methods to handle treatment when developing a prognostic model that aims to produce accurate probabilities of the outcome of individuals if left untreated., Study Design and Setting: Simulations were performed based on two normally distributed predictors, a binary outcome, and a binary treatment, mimicking a randomized trial or an observational study. Comparison was made between simply ignoring treatment (SIT), restricting the analytical data set to untreated individuals (AUT), inverse probability weighting (IPW), and explicit modeling of treatment (MT). Methods were compared in terms of predictive performance of the model and the proportion of incorrect treatment decisions., Results: Omitting a genuine predictor of the outcome from the prognostic model decreased model performance, in both an observational study and a randomized trial. In randomized trials, the proportion of incorrect treatment decisions was smaller when applying AUT or MT, compared to SIT and IPW. In observational studies, MT was superior to all other methods regarding the proportion of incorrect treatment decisions., Conclusion: If a prognostic model aims to produce correct probabilities of the outcome in the absence of treatment, ignoring treatments that affect that outcome can lead to suboptimal model performance and incorrect treatment decisions. Explicitly, modeling treatment is recommended., (Copyright © 2016 Elsevier Inc. All rights reserved.)

After developing a prognostic model, it is essential to evaluate the performance of the model in samples independent from those used to develop the model, which is often referred to as external validation. However, despite its importance, very little is known about the sample size requirements for conducting an external validation. Using a large real data set and resampling methods, we investigate the impact of sample size on the performance of six published prognostic models. Focussing on unbiased and precise estimation of performance measures (e.g. the c-index, D statistic and calibration), we provide guidance on sample size for investigators designing an external validation study. Our study suggests that externally validating a prognostic model requires a minimum of 100 events and ideally 200 (or more) events., (© 2015 The Authors. Statistics in Medicine Published by John Wiley & Sons Ltd.)

Context: Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed., Objective: The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes., Evidence Acquisition: This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors., Evidence Synthesis: The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document., Conclusions: To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org)., Patient Summary: The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes., (Copyright © 2014 The Authors. Published by Elsevier B.V. All rights reserved.)

Background: Prediction models are developed to aid health care providers in estimating the probability or risk that a specific disease or condition is present (diagnostic models) or that a specific event will occur in the future (prognostic models), to inform their decision making. However, the overwhelming evidence shows that the quality of reporting of prediction model studies is poor. Only with full and clear reporting of information on all aspects of a prediction model can risk of bias and potential usefulness of prediction models be adequately assessed., Methods: The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) Initiative developed a set of recommendations for the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. This article describes how the TRIPOD Statement was developed. An extensive list of items based on a review of the literature was created, which was reduced after a Web-based survey and revised during a 3-day meeting in June 2011 with methodologists, health care professionals, and journal editors. The list was refined during several meetings of the steering group and in e-mail discussions with the wider group of TRIPOD contributors., Results: The resulting TRIPOD Statement is a checklist of 22 items, deemed essential for transparent reporting of a prediction model study. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. The TRIPOD Statement is best used in conjunction with the TRIPOD explanation and elaboration document., Conclusions: To aid the editorial process and readers of prediction model studies, it is recommended that authors include a completed checklist in their submission (also available at www.tripod-statement.org)., (© 2015 The Authors.)

The TRIPOD (Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) Statement includes a 22-item checklist, which aims to improve the reporting of studies developing, validating, or updating a prediction model, whether for diagnostic or prognostic purposes. The TRIPOD Statement aims to improve the transparency of the reporting of a prediction model study regardless of the study methods used. This explanation and elaboration document describes the rationale; clarifies the meaning of each item; and discusses why transparent reporting is important, with a view to assessing risk of bias and clinical usefulness of the prediction model. Each checklist item of the TRIPOD Statement is explained in detail and accompanied by published examples of good reporting. The document also provides a valuable reference of issues to consider when designing, conducting, and analyzing prediction model studies. To aid the editorial process and help peer reviewers and, ultimately, readers and systematic reviewers of prediction model studies, it is recommended that authors include a completed checklist in their submission. The TRIPOD checklist can also be downloaded from www.tripod-statement.org.

Objectives: Assessment of patient-reported outcomes (PROs) in oncology is of critical importance because it provides unique information that may also predict clinical outcomes.Methods: We conducted a systematic review of prognostic factor studies to examine the prognostic value of PROs for survival in cancer. A systematic literature search was performed in PubMed for studies published between 2013 and 2018. We considered any study, regardless of the research design, that included at least 1 PRO domain in the final multivariable prognostic model. The protocol (EPIPHANY) was published and registered in the International Prospective Register of Systematic Reviews (CRD42018099160).Results: Eligibility criteria selected 138 studies including 158 127 patients, of which 43 studies were randomized, controlled trials. Overall, 120 (87%) studies reported at least 1 PRO to be statistically significantly prognostic for overall survival. Lung (n = 41, 29.7%) and genitourinary (n = 27, 19.6%) cancers were most commonly investigated. The prognostic value of PROs was investigated in secondary data analyses in 101 (73.2%) studies. The EORTC QLQ-C30 questionnaire was the most frequently used measure, and its physical functioning scale (range 0-100) the most frequent independent prognostic PRO, with a pooled hazard ratio estimate of 0.88 per 10-point increase (95% CI 0.84-0.92).Conclusions: There is convincing evidence that PROs provide independent prognostic information for overall survival across cancer populations and disease stages. Further research is needed to translate current evidence-based data into prognostic tools to aid in clinical decision making. [ABSTRACT FROM AUTHOR]

It is widely recommended that any developed-diagnostic or prognostic-prediction model is externally validated in terms of its predictive performance measured by calibration and discrimination. When multiple validations have been performed, a systematic review followed by a formal meta-analysis helps to summarize overall performance across multiple settings, and reveals under which circumstances the model performs suboptimal (alternative poorer) and may need adjustment. We discuss how to undertake meta-analysis of the performance of prediction models with either a binary or a time-to-event outcome. We address how to deal with incomplete availability of study-specific results (performance estimates and their precision), and how to produce summary estimates of the c-statistic, the observed:expected ratio and the calibration slope. Furthermore, we discuss the implementation of frequentist and Bayesian meta-analysis methods, and propose novel empirically-based prior distributions to improve estimation of between-study heterogeneity in small samples. Finally, we illustrate all methods using two examples: meta-analysis of the predictive performance of EuroSCORE II and of the Framingham Risk Score. All examples and meta-analysis models have been implemented in our newly developed R package "metamisc". [ABSTRACT FROM AUTHOR]

Prediction models in health care use predictors to estimate for an individual the probability that a condition or disease is already present (diagnostic model) or will occur in the future (prognostic model). Publications on prediction models have become more common in recent years, and competing prediction models frequently exist for the same outcome or target population. Health care providers, guideline developers, and policymakers are often unsure which model to use or recommend, and in which persons or settings. Hence, systematic reviews of these studies are increasingly demanded, required, and performed. A key part of a systematic review of prediction models is examination of risk of bias and applicability to the intended population and setting. To help reviewers with this process, the authors developed PROBAST (Prediction model Risk Of Bias ASsessment Tool) for studies developing, validating, or updating (for example, extending) prediction models, both diagnostic and prognostic. PROBAST was developed through a consensus process involving a group of experts in the field. It includes 20 signaling questions across 4 domains (participants, predictors, outcome, and analysis). This explanation and elaboration document describes the rationale for including each domain and signaling question and guides researchers, reviewers, readers, and guideline developers in how to use them to assess risk of bias and applicability concerns. All concepts are illustrated with published examples across different topics. The latest version of the PROBAST checklist, accompanying documents, and filled-in examples can be downloaded from www.probast.org. [ABSTRACT FROM AUTHOR]

: Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary [ABSTRACT FROM AUTHOR]

Background Despite its rarity, the prognosis of pancreatic cancer is very poor and it is a major cause of cancer mortality; being ranked fourth in the world, it has one of the worst survival rates of any cancer. Aim To evaluate the performance of QCancer® (Pancreas) for predicting the absolute risk of pancreatic cancer in an independent UK cohort of patients, from general practice records. Design and setting Prospective cohort study to evaluate the performance QCancer (Pancreas) prediction models in 364 practices from the UK, contributing to The Health Improvement Network (THIN) database. Method Records were extracted from the THIN database for 2.15 million patients registered with a general practice surgery between 1 January 2000 and 30 June 2008, aged 30-84 years (3.74 million person-years), with 618 pancreatic cancer cases. Pancreatic cancer was defined as incident diagnosis of pancreatic cancer during the 2 years after study entry. Results The results from this independent and external validation of QCancer (Pancreas) demonstrated good performance data on a large cohort of general practice patients. QCancer (Pancreas) had very good discrimination properties, with areas under the receiver operating characteristic curve of 0.89 and 0.92 for females and males respectively. QCancer (Pancreas) explained 60% and 67% of the variation in females and males respectively. QCancer (Pancreas) over-predicted risk in both females and males, notably in older patients. Conclusion QCancer (Pancreas) is potentially useful for identifying undetected cases of pancreatic cancer in primary care in the UK. [ABSTRACT FROM AUTHOR]

Collecting data for an individual participant data meta‐analysis (IPDMA) project can be time consuming and resource intensive and could still have insufficient power to answer the question of interest. Therefore, researchers should consider the power of their planned IPDMA before collecting IPD. Here we propose a method to estimate the power of a planned IPDMA project aiming to synthesise multiple cohort studies to investigate the (unadjusted or adjusted) effects of potential prognostic factors for a binary outcome. We consider both binary and continuous factors and provide a three‐step approach to estimating the power in advance of collecting IPD, under an assumption of the true prognostic effect of each factor of interest. The first step uses routinely available (published) aggregate data for each study to approximate Fisher's information matrix and thereby estimate the anticipated variance of the unadjusted prognostic factor effect in each study. These variances are then used in step 2 to estimate the anticipated variance of the summary prognostic effect from the IPDMA. Finally, step 3 uses this variance to estimate the corresponding IPDMA power, based on a two‐sided Wald test and the assumed true effect. Extensions are provided to adjust the power calculation for the presence of additional covariates correlated with the prognostic factor of interest (by using a variance inflation factor) and to allow for between‐study heterogeneity in prognostic effects. An example is provided for illustration, and Stata code is supplied to enable researchers to implement the method. [ABSTRACT FROM AUTHOR]

Background: The prevalence of malignant pleural effusion is increasing worldwide, but prognostic biomarkers to plan treatment and to understand the underlying mechanisms of disease progression remain unidentified. The PROMISE study was designed with the objectives to discover, validate, and prospectively assess biomarkers of survival and pleurodesis response in malignant pleural effusion and build a score that predicts survival.Methods: In this multicohort study, we used five separate and independent datasets from randomised controlled trials to investigate potential biomarkers of survival and pleurodesis. Mass spectrometry-based discovery was used to investigate pleural fluid samples for differential protein expression in patients from the discovery group with different survival and pleurodesis outcomes. Clinical, radiological, and biological variables were entered into least absolute shrinkage and selection operator regression to build a model that predicts 3-month mortality. We evaluated the model using internal and external validation.Findings: 17 biomarker candidates of survival and seven of pleurodesis were identified in the discovery dataset. Three independent datasets (n=502) were used for biomarker validation. All pleurodesis biomarkers failed, and gelsolin, macrophage migration inhibitory factor, versican, and tissue inhibitor of metalloproteinases 1 (TIMP1) emerged as accurate predictors of survival. Eight variables (haemoglobin, C-reactive protein, white blood cell count, Eastern Cooperative Oncology Group performance status, cancer type, pleural fluid TIMP1 concentrations, and previous chemotherapy or radiotherapy) were validated and used to develop a survival score. Internal validation with bootstrap resampling and external validation with 162 patients from two independent datasets showed good discrimination (C statistic values of 0·78 [95% CI 0·72-0·83] for internal validation and 0·89 [0·84-0·93] for external validation of the clinical PROMISE score).Interpretation: To our knowledge, the PROMISE score is the first prospectively validated prognostic model for malignant pleural effusion that combines biological and clinical parameters to accurately estimate 3-month mortality. It is a robust, clinically relevant prognostic score that can be applied immediately, provide important information on patient prognosis, and guide the selection of appropriate management strategies.Funding: European Respiratory Society, Medical Research Funding-University of Oxford, Slater & Gordon Research Fund, and Oxfordshire Health Services Research Committee Research Grants. [ABSTRACT FROM AUTHOR]

Background: We compared two strategies for treating patients infected with multidrug-resistant human immunodeficiency virus (HIV). Methods: Patients with multidrug-resistant HIV and HIV RNA levels of more than 5000 copies per milliliter were randomly assigned to a four-month structured interruption of treatment followed by a change in antiretroviral regimen (treatment-interruption group) or to an immediate change in regimen (control group). Genotypic and phenotypic resistance testing was performed. Disease progression, death, and changes in genotypic resistance, CD4 cell counts, HIV RNA levels, and quality of life were assessed. Results: After a median follow-up of 11.6 months, disease progression or death occurred in 22 of the 138 patients in the treatment-interruption group and in 12 of the 132 patients in the control group (P=0.01), with a hazard ratio of 2.57 (95 percent confidence interval, 1.2 to 5.5) for the treatment-interruption group. There were eight deaths in each group. In the treatment-interruption group, the mutant HIV populations completely or partially reverted to wild type by four months in 64.0 percent of patients. As compared with the control group, the treatment-interruption group had a mean CD4 cell count that was 85 cells per cubic millimeter lower from months 0 through 4 (P<0.001), 47 cells per cubic millimeter lower from months 5 through 8 (P<0.001), and 31 cells per cubic millimeter lower after eight months (P=0.11). The mean HIV RNA levels were 1.2 log copies per milliliter higher (on a base-10 scale) in the treatment-interruption group during months 0 through 4 (P<0.001), but they were not significantly different from those in the control group after month 4. The overall quality of life was similar in the two groups. Conclusions: In patients infected with multidrug-resistant HIV, structured interruption of treatment was associated with greater progression of disease and did not confer immunologic or virologic benefits or improve the overall quality of life. N Engl J Med 2003;349:837-46. [ABSTRACT FROM AUTHOR]