Search

Purpose: Machine learning models developed from electronic health records data have been increasingly used to predict risk of mortality for general oncology patients. But these models may have suboptimal performance because of patient heterogeneity. The objective of this work is to develop a new modeling approach to predicting short-term mortality that accounts for heterogeneity across multiple subgroups in the presence of a large number of electronic health record predictors., Methods: We proposed a two-stage approach to addressing heterogeneity among oncology patients of different cancer types for predicting their risk of mortality. Structured data were extracted from the University of Pennsylvania Health System for 20,723 patients of 11 cancer types, where 1,340 (6.5%) patients were deceased. We first modeled the overall risk for all patients without differentiating cancer types, as is done in the current practice. We then developed cancer type-specific models using the overall risk score as a predictor along with preselected type-specific predictors. The overall and type-specific models were compared with respect to discrimination using the area under the precision-recall curve (AUPRC) and calibration using the calibration slope. We also proposed metrics that characterize the degree of risk heterogeneity by comparing risk predictors in the overall and type-specific models., Results: The two-stage modeling resulted in improved calibration and discrimination across all 11 cancer types. The improvement in AUPRC was significant for hematologic malignancies including leukemia, lymphoma, and myeloma. For instance, the AUPRC increased from 0.358 to 0.519 (∆ = 0.161; 95% CI, 0.102 to 0.224) and from 0.299 to 0.354 (∆ = 0.055; 95% CI, 0.009 to 0.107) for leukemia and lymphoma, respectively. For all 11 cancer types, the two-stage approach generated well-calibrated risks. A high degree of heterogeneity between type-specific and overall risk predictors was observed for most cancer types., Conclusion: Our two-stage modeling approach that accounts for cancer type-specific risk heterogeneity has improved calibration and discrimination than a model agnostic to cancer types., Competing Interests: Ravi B. ParikhStock and Other Ownership Interests: Merck, Google, GNS HealthcareConsulting or Advisory Role: GNS Healthcare, Cancer Study Group, Onc.AITravel, Accommodations, Expenses: Conquer Cancer Foundation, Flatiron HealthNo other potential conflicts of interest were reported.

Background: Identifying individuals at risk for future hospitalization or death has been a major priority of population health management strategies. High-risk individuals are a heterogeneous group, and existing studies describing heterogeneity in high-risk individuals have been limited by data focused on clinical comorbidities and not socioeconomic or behavioral factors. We used machine learning clustering methods and linked comorbidity-based, sociodemographic, and psychobehavioral data to identify subgroups of high-risk Veterans and study long-term outcomes, hypothesizing that factors other than comorbidities would characterize several subgroups., Methods and Findings: In this cross-sectional study, we used data from the VA Corporate Data Warehouse, a national repository of VA administrative claims and electronic health data. To identify high-risk Veterans, we used the Care Assessment Needs (CAN) score, a routinely-used VA model that predicts a patient's percentile risk of hospitalization or death at one year. Our study population consisted of 110,000 Veterans who were randomly sampled from 1,920,436 Veterans with a CAN score≥75th percentile in 2014. We categorized patient-level data into 119 independent variables based on demographics, comorbidities, pharmacy, vital signs, laboratories, and prior utilization. We used a previously validated density-based clustering algorithm to identify 30 subgroups of high-risk Veterans ranging in size from 50 to 2,446 patients. Mean CAN score ranged from 72.4 to 90.3 among subgroups. Two-year mortality ranged from 0.9% to 45.6% and was highest in the home-based care and metastatic cancer subgroups. Mean inpatient days ranged from 1.4 to 30.5 and were highest in the post-surgery and blood loss anemia subgroups. Mean emergency room visits ranged from 1.0 to 4.3 and were highest in the chronic sedative use and polysubstance use with amphetamine predominance subgroups. Five subgroups were distinguished by psychobehavioral factors and four subgroups were distinguished by sociodemographic factors., Conclusions: High-risk Veterans are a heterogeneous population consisting of multiple distinct subgroups-many of which are not defined by clinical comorbidities-with distinct utilization and outcome patterns. To our knowledge, this represents the largest application of ML clustering methods to subgroup a high-risk population. Further study is needed to determine whether distinct subgroups may benefit from individualized interventions., Competing Interests: The authors have declared that no competing interests exist.

Importance: Machine learning (ML) algorithms can identify patients with cancer at risk of short-term mortality to inform treatment and advance care planning. However, no ML mortality risk prediction algorithm has been prospectively validated in oncology or compared with routinely used prognostic indices., Objective: To validate an electronic health record-embedded ML algorithm that generated real-time predictions of 180-day mortality risk in a general oncology cohort., Design, Setting, and Participants: This prognostic study comprised a prospective cohort of patients with outpatient oncology encounters between March 1, 2019, and April 30, 2019. An ML algorithm, trained on retrospective data from a subset of practices, predicted 180-day mortality risk between 4 and 8 days before a patient's encounter. Patient encounters took place in 18 medical or gynecologic oncology practices, including 1 tertiary practice and 17 general oncology practices, within a large US academic health care system. Patients aged 18 years or older with outpatient oncology or hematology and oncology encounters were included in the analysis. Patients were excluded if their appointment was scheduled after weekly predictions were generated and if they were only evaluated in benign hematology, palliative care, or rehabilitation practices., Exposures: Gradient-boosting ML binary classifier., Main Outcomes and Measures: The primary outcome was the patients' 180-day mortality from the index encounter. The primary performance metric was the area under the receiver operating characteristic curve (AUC)., Results: Among 24 582 patients, 1022 (4.2%) died within 180 days of their index encounter. Their median (interquartile range) age was 64.6 (53.6-73.2) years, 15 319 (62.3%) were women, 18 015 (76.0%) were White, and 10 658 (43.4%) were seen in the tertiary practice. The AUC was 0.89 (95% CI, 0.88-0.90) for the full cohort. The AUC varied across disease-specific groups within the tertiary practice (AUC ranging from 0.74 to 0.96) but was similar between the tertiary and general oncology practices. At a prespecified 40% mortality risk threshold used to differentiate high- vs low-risk patients, observed 180-day mortality was 45.2% (95% CI, 41.3%-49.1%) in the high-risk group vs 3.1% (95% CI, 2.9%-3.3%) in the low-risk group. Integrating the algorithm into the Eastern Cooperative Oncology Group and Elixhauser comorbidity index-based classifiers resulted in favorable reclassification (net reclassification index, 0.09 [95% CI, 0.04-0.14] and 0.23 [95% CI, 0.20-0.27], respectively)., Conclusions and Relevance: In this prognostic study, an ML algorithm was feasibly integrated into the electronic health record to generate real-time, accurate predictions of short-term mortality for patients with cancer and outperformed routinely used prognostic indices. This algorithm may be used to inform behavioral interventions and prompt earlier conversations about goals of care and end-of-life preferences among patients with cancer.

Introduction: Patients with cancer often receive care that is not aligned with their personal values and goals. Serious illness conversations (SICs) between clinicians and patients can help increase a patient's understanding of their prognosis, goals and values., Methods and Analysis: In this study, we describe the design of a stepped-wedge cluster randomized trial to evaluate the impact of an intervention that employs machine learning-based prognostic algorithms and behavioral nudges to prompt oncologists to have SICs with patients at high risk of short-term mortality. Data are collected on documented SICs, documented advance care planning discussions, and end-of-life care utilization (emergency room and inpatient admissions, chemotherapy and hospice utilization) for patients of all enrolled clinicians., Conclusion: This trial represents a novel application of machine-generated mortality predictions combined with behavioral nudges in the routine care of outpatients with cancer. Findings from the trial may inform strategies to encourage early serious illness conversations and the application of mortality risk predictions in clinical settings., Trial Registration: Clinicaltrials.gov Identifier: NCT03984773., Competing Interests: Declaration of Competing Interest The authors have no conflict of interest disclosures relevant to this project. Outside of the scope of this work, Dr. Parikh serves as a consultant for GNS Healthcare. Dr. Patel is supported by career development awards from the Department of Veterans Affairs HSR&D, and is a founder of Catalyst Health, a technology and behavior change consulting firm., (Copyright © 2020 Elsevier Inc. All rights reserved.)

Importance: Machine learning algorithms could identify patients with cancer who are at risk of short-term mortality. However, it is unclear how different machine learning algorithms compare and whether they could prompt clinicians to have timely conversations about treatment and end-of-life preferences., Objectives: To develop, validate, and compare machine learning algorithms that use structured electronic health record data before a clinic visit to predict mortality among patients with cancer., Design, Setting, and Participants: Cohort study of 26 525 adult patients who had outpatient oncology or hematology/oncology encounters at a large academic cancer center and 10 affiliated community practices between February 1, 2016, and July 1, 2016. Patients were not required to receive cancer-directed treatment. Patients were observed for up to 500 days after the encounter. Data analysis took place between October 1, 2018, and September 1, 2019., Exposures: Logistic regression, gradient boosting, and random forest algorithms., Main Outcomes and Measures: Primary outcome was 180-day mortality from the index encounter; secondary outcome was 500-day mortality from the index encounter., Results: Among 26 525 patients in the analysis, 1065 (4.0%) died within 180 days of the index encounter. Among those who died, the mean age was 67.3 (95% CI, 66.5-68.0) years, and 500 (47.0%) were women. Among those who were alive at 180 days, the mean age was 61.3 (95% CI, 61.1-61.5) years, and 15 922 (62.5%) were women. The population was randomly partitioned into training (18 567 [70.0%]) and validation (7958 [30.0%]) cohorts at the patient level, and a randomly selected encounter was included in either the training or validation set. At a prespecified alert rate of 0.02, positive predictive values were higher for the random forest (51.3%) and gradient boosting (49.4%) algorithms compared with the logistic regression algorithm (44.7%). There was no significant difference in discrimination among the random forest (area under the receiver operating characteristic curve [AUC], 0.88; 95% CI, 0.86-0.89), gradient boosting (AUC, 0.87; 95% CI, 0.85-0.89), and logistic regression (AUC, 0.86; 95% CI, 0.84-0.88) models (P for comparison = .02). In the random forest model, observed 180-day mortality was 51.3% (95% CI, 43.6%-58.8%) in the high-risk group vs 3.4% (95% CI, 3.0%-3.8%) in the low-risk group; at 500 days, observed mortality was 64.4% (95% CI, 56.7%-71.4%) in the high-risk group and 7.6% (7.0%-8.2%) in the low-risk group. In a survey of 15 oncology clinicians with a 52.1% response rate, 100 of 171 patients (58.8%) who had been flagged as having high risk by the gradient boosting algorithm were deemed appropriate for a conversation about treatment and end-of-life preferences in the upcoming week., Conclusions and Relevance: In this cohort study, machine learning algorithms based on structured electronic health record data accurately identified patients with cancer at risk of short-term mortality. When the gradient boosting algorithm was applied in real time, clinicians believed that most patients who had been identified as having high risk were appropriate for a timely conversation about treatment and end-of-life preferences.

Importance: Patients with cancer who die soon after starting chemotherapy incur costs of treatment without the benefits. Accurately predicting mortality risk before administering chemotherapy is important, but few patient data-driven tools exist., Objective: To create and validate a machine learning model that predicts mortality in a general oncology cohort starting new chemotherapy, using only data available before the first day of treatment., Design, Setting, and Participants: This retrospective cohort study of patients at a large academic cancer center from January 1, 2004, through December 31, 2014, determined date of death by linkage to Social Security data. The model was derived using data from 2004 through 2011, and performance was measured on nonoverlapping data from 2012 through 2014. The analysis was conducted from June 1 through August 1, 2017. Participants included 26 946 patients starting 51 774 new chemotherapy regimens., Main Outcomes and Measures: Thirty-day mortality from the first day of a new chemotherapy regimen. Secondary outcomes included model discrimination by predicted mortality risk decile among patients receiving palliative chemotherapy, and 180-day mortality from the first day of a new chemotherapy regimen., Results: Among the 26 946 patients included in the analysis, mean age was 58.7 years (95% CI, 58.5-58.9 years); 61.1% were female (95% CI, 60.4%-61.9%); and 86.9% were white (95% CI, 86.4%-87.4%). Thirty-day mortality from chemotherapy start was 2.1% (95% CI, 1.9%-2.4%). Among the 9114 patients in the validation set, the most common primary cancers were breast (21.1%; 95% CI, 20.2%-21.9%), colorectal (19.3%; 95% CI, 18.5%-20.2%), and lung (18.0%; 95% CI, 17.2%-18.8%). Model predictions were accurate for all patients (area under the curve [AUC], 0.940; 95% CI, 0.930-0.951). Predictions for patients starting palliative chemotherapy (46.6% of regimens; 95% CI, 45.8%-47.3%), for whom prognosis is particularly important, remained highly accurate (AUC, 0.924; 95% CI, 0.910-0.939). To illustrate model discrimination, patients were ranked initiating palliative chemotherapy by model-predicted mortality risk, and observed mortality was calculated by risk decile. Thirty-day mortality in the highest-risk decile was 22.6% (95% CI, 19.6%-25.6%); in the lowest-risk decile, no patients died. Predictions remained accurate across all primary cancers, stages, and chemotherapies, even for clinical trial regimens that first appeared in years after the model was trained (AUC, 0.942; 95% CI, 0.882-1.000). The same model also performed well for prediction of 180-day mortality (AUC for all patients, 0.870 [95% CI, 0.862-0.877]; highest- vs lowest-risk decile mortality, 74.8% [95% CI, 72.7%-77.0%] vs 0.2% [95% CI, 0.01%-0.4%]). Predictions were more accurate than estimates from randomized clinical trials of individual chemotherapies or the Surveillance, Epidemiology, and End Results data set., Conclusions and Relevance: A machine learning algorithm using electronic health record data accurately predicted short-term mortality among patients starting chemotherapy. Further research is necessary to determine the generalizability and feasibility of applying this algorithm in clinical settings.

Modern artificial intelligence (AI) tools built on high‐dimensional patient data are reshaping oncology care, helping to improve goal‐concordant care, decrease cancer mortality rates, and increase workflow efficiency and scope of care. However, data‐related concerns and human biases that seep into algorithms during development and post‐deployment phases affect performance in real‐world settings, limiting the utility and safety of AI technology in oncology clinics. To this end, the authors review the current potential and limitations of predictive AI for cancer diagnosis and prognostication as well as of generative AI, specifically modern chatbots, which interfaces with patients and clinicians. They conclude the review with a discussion on ongoing challenges and regulatory opportunities in the field. The authors cover emerging use cases of predictive and generative artificial intelligence in three broad use cases: diagnosis, prognostication, and patient communication. They discuss limitations including lack of explainability, bias, and performance drift and strategies to overcome them. [ABSTRACT FROM AUTHOR]

Sudden changes in health care utilization during the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic may have impacted the performance of clinical predictive models that were trained prior to the pandemic. In this study, we evaluated the performance over time of a machine learning, electronic health record-based mortality prediction algorithm currently used in clinical practice to identify patients with cancer who may benefit from early advance care planning conversations. We show that during the pandemic period, algorithm identification of high-risk patients had a substantial and sustained decline. Decreases in laboratory utilization during the peak of the pandemic may have contributed to drift. Calibration and overall discrimination did not markedly decline during the pandemic. This argues for careful attention to the performance and retraining of predictive algorithms that use inputs from the pandemic period. [ABSTRACT FROM AUTHOR]

PURPOSE: Machine learning (ML) algorithms that incorporate routinely collected patient-reported outcomes (PROs) alongside electronic health record (EHR) variables may improve prediction of short-term mortality and facilitate earlier supportive and palliative care for patients with cancer. METHODS: We trained and validated two-phase ML algorithms that incorporated standard PRO assessments alongside approximately 200 routinely collected EHR variables, among patients with medical oncology encounters at a tertiary academic oncology and a community oncology practice. RESULTS: Among 12,350 patients, 5,870 (47.5%) completed PRO assessments. Compared with EHR- and PRO-only algorithms, the EHR + PRO model improved predictive performance in both tertiary oncology (EHR + PRO v EHR v PRO: area under the curve [AUC] 0.86 [0.85-0.87] v 0.82 [0.81-0.83] v 0.74 [0.74-0.74]) and community oncology (area under the curve 0.89 [0.88-0.90] v 0.86 [0.85-0.88] v 0.77 [0.76-0.79]) practices. CONCLUSION: Routinely collected PROs contain added prognostic information not captured by an EHR-based ML mortality risk algorithm. Augmenting an EHR-based algorithm with PROs resulted in a more accurate and clinically relevant model, which can facilitate earlier and targeted supportive care for patients with cancer. How do patient-reported outcomes improve machine learning algorithms to predict mortality in cancer? In this predictive modeling study, @ravi_b_parikh @PennMedicine and colleagues find that ML algorithms based on PRO and EHR data significantly outperform other algorithms. [ABSTRACT FROM AUTHOR]

Background: Early serious illness conversations (SICs) about goals of care and prognosis improve mood, quality of life, and end-of-life care quality. Algorithm-based behavioral nudges to oncologists increase the frequency and timeliness of such conversations. However, clinicians' perspectives on such nudges are unknown. Design: Qualitative study consisting of semistructured interviews among medical oncology clinicians who participated in a stepped-wedge cluster randomized trial of Conversation Connect, an algorithm-based intervention consisting of behavioral nudges to promote early SICs in the outpatient oncology setting. Results: Of 79 eligible oncology clinicians, 56 (71%) were approached to participate in interviews and 25 (45%) accepted. Key facilitators to algorithm-based nudges included prompting documentation of conversations, peer comparisons, performance reports, and validating norms around early conversations. Barriers included cancer-specific heterogeneity in algorithm performance and the frequency and tone of text messages. Areas of improvement included utilizing different information channels, identifying patients earlier in the disease trajectory, and incorporating patient-targeted messaging that emphasizes the value of early conversations. Conclusions: Oncology clinicians identified key facilitators and barriers to Conversation Connect. These insights inform future algorithm-based supportive care interventions in oncology. Controlled trial (NCT03984773). [ABSTRACT FROM AUTHOR]

Policy PointsWith increasing integration of artificial intelligence and machine learning in medicine, there are concerns that algorithm inaccuracy could lead to patient injury and medical liability.While prior work has focused on medical malpractice, the artificial intelligence ecosystem consists of multiple stakeholders beyond clinicians. Current liability frameworks are inadequate to encourage both safe clinical implementation and disruptive innovation of artificial intelligence.Several policy options could ensure a more balanced liability system, including altering the standard of care, insurance, indemnification, special/no‐fault adjudication systems, and regulation. Such liability frameworks could facilitate safe and expedient implementation of artificial intelligence and machine learning in clinical care. [ABSTRACT FROM AUTHOR]

Machine learning algorithms may address prognostic inaccuracy among clinicians by identifying patients at risk of short-term mortality and facilitating earlier discussions about hospice enrollment, discontinuation of therapy, or other management decisions. In the present study, we used prospective predictions from a real-time machine learning prognostic algorithm to identify two trajectories of all-cause mortality risk for decedents with cancer. We show that patients with an unpredictable trajectory, where mortality risk rises only close to death, are significantly less likely to receive guideline-based end-of-life care and may not benefit from the integration of prognostic algorithms in practice. [ABSTRACT FROM AUTHOR]