Search

ObjectiveWe aimed to discover computationally-derived phenotypes of opioid-related patient presentations to the emergency department (ED) via clinical notes and structured electronic health record (EHR) data.MethodsThis was a retrospective study of ED visits from 2013-2020 across ten sites within a regional healthcare network. We derived phenotypes from visits for patients 18 years of age with at least one prior or current documentation of an opioid-related diagnosis. Natural language processing was used to extract clinical entities from notes, which were combined with structured data within the EHR to create a set of features. We performed Latent Dirichlet allocation to identify topics within these features. Groups of patient presentations with similar attributes were identified by cluster analysis.ResultsIn total 82,577 ED visits met inclusion criteria. The 30 topics discovered ranged from those related to substance use disorder, chronic conditions, mental health, and medical management. Clustering on these topics identified nine unique cohorts with one-year survivals ranging from 84.2-96.8%, rates of one-year ED returns from 9-34%, rates of one-year opioid event 10-17%, rates of medications for opioid use disorder from 17-43%, and a median Carlson comorbidity index of 2-8. Two cohorts of phenotypes were identified related to chronic substance use disorder, or acute overdose.ConclusionsOur results indicate distinct phenotypic clusters with varying patient-oriented outcomes which provide future targets better allocation of resources and therapeutics. This highlights the heterogeneity of the overall population, and the need to develop targeted interventions for each population.

The objective of the study is to identify healthcare events leading to a diagnosis of dementia from a large real-world dataset. This study uses a data-driven approach to identify temporally ordered pairs and trajectories of healthcare codes in the electronic health record (EHR). This allows for discovery of novel temporal risk factors leading to an outcome of interest that may not otherwise be obvious. We identified several known (Down syndrome RR: 116.1, thiamine deficiency RR: 76.1, and Parkinson’s disease RR: 41.1) and unknown (Brief psychotic disorder RR: 68.6, Toxic effect of metals RR: 40.4, and Schizoaffective disorders RR: 40.0) factors for a specific dementia diagnosis. The associations with the greatest risk for any dementia diagnosis were found to be primarily related to mental health (Brief psychotic disorder RR: 266.5, Dissociative and conversion disorders RR: 169.8), or neurologic conditions or procedures (Dystonia RR: 121.9, Lumbar Puncture RR: 119.0). Trajectory and clustering analysis identified factors related to cerebrovascular disorders, as well as diagnoses which increase the risk of toxic imbalances. The results of this study have the ability to provide valuable insights into potential patient progression towards dementia and improve recognition of patients at risk for developing dementia.

BackgroundChatGPT is a 175 billion parameter natural language processing model which can generate conversation style responses to user input.ObjectiveTo evaluate the performance of ChatGPT on questions within the scope of United States Medical Licensing Examination (USMLE) Step 1 and Step 2 exams, as well as analyze responses for user interpretability.MethodsWe used two novel sets of multiple choice questions to evaluate ChatGPT’s performance, each with questions pertaining to Step 1 and Step 2. The first was derived from AMBOSS, a commonly used question bank for medical students, which also provides statistics on question difficulty and the performance on an exam relative to the userbase. The second, was the National Board of Medical Examiners (NBME) Free 120-question exams. After prompting ChatGPT with each question, ChatGPT’s selected answer was recorded, and the text output evaluated across three qualitative metrics: logical justification of the answer selected, presence of information internal to the question, and presence of information external to the question.ResultsOn the four datasets, AMBOSS-Step1, AMBOSS-Step2, NBME-Free-Step1, and NBMEFree-Step2, ChatGPT achieved accuracies of 44%, 42%, 64.4%, and 57.8%. The model demonstrated a significant decrease in performance as question difficulty increased (P=.012) within the AMBOSSStep1 dataset. We found logical justification for ChatGPT’s answer selection was present in 100% of outputs. Internal information to the question was present in>90% of all questions. The presence of information external to the question was respectively 54.5% and 27% lower for incorrect relative to correct answers on the NBME-Free-Step1 and NBME-Free-Step2 datasets (PConclusionChatGPT marks a significant improvement in natural language processing models on the tasks of medical question answering. By performing at greater than 60% threshold on the NBME-FreeStep-1 dataset we show that the model is comparable to a third year medical student. Additionally, due to the dialogic nature of the response to questions, we demonstrate ChatGPT’s ability to provide reasoning and informational context across the majority of answers. These facts taken together make a compelling case for the potential applications of ChatGPT as a medical education tool.

Purpose: Although deep learning (DL) models have shown increasing ability to accurately classify diagnostic images in oncology, significantly large amounts of well-curated data are often needed to match human level performance. Given the relative paucity of imaging datasets for less prevalent cancer types, there is an increasing need for methods which can improve the performance of deep learning models trained using limited diagnostic images. Deep metric learning (DML) is a potential method which can improve accuracy in deep learning models trained on limited datasets. Leveraging a triplet-loss function, DML exponentially increases training data compared to a traditional DL model. In this study, we investigated the utility of DML to improve the accuracy of DL models trained to classify cancerous lesions found on screening mammograms. Methods: Using a dataset of 2620 lesions found on routine screening mammogram, we trained both a traditional DL and DML models to classify suspicious lesions as cancerous or benign. The VGG16 architecture was used as the basis for the DL and DML models. Model performance was compared by calculating model accuracy, sensitivity, and specificity on a blinded test set of 378 lesions. In addition to individual model performance, we also measured agreement accuracy when both the DL and DML models were combined. Sub-analyses were conducted to identify phenotypes which were best suited for each model type. Both models underwent hyperparameters optimization to identify ideal batch size, learning rate, and regularization to prevent overfitting. Results: We found that the combination of the traditional DL model with DML model resulted in the highest overall accuracy (78.7%) representing a 7.1% improvement compared to the traditional DL model (p Conclusion: Our study suggests that addition of DML models with traditional DL models can improve diagnostic image classification performance in cancer. Our results suggest DML models may provide increased specificity and help with classification of unique populations often misclassified by traditional DL models. Further studied investigating the utility of DML on other cancer imaging tasks are necessary to successfully build more robust DL models in cancer imaging. Citation Format: Justin Du, Sachin Umrao, Enoch Chang, Marina Joel, Aidan Gilson, Guneet Janda, Rachel Choi, Yongfeng Hui, Sanjay Aneja. The utility of deep metric learning for breast cancer identification on mammographic images [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 184.

Although deep learning analysis of diagnostic imaging has shown increasing effectiveness in modeling non-small cell lung cancer (NSCLC) outcomes, a minority of proposed deep learning algorithms have been externally validated. Given a majority of these models are built on single institutional datasets, their generalizability across the entire population remains understudied. Moreover, the effect of biases that exist among institutional training dataset on overall generalizability of deep learning prognostic models is unclear. We attempted to identify demographic and clinical characteristics which if over-represented within training data could affect the generalizability of deep learning models aimed at predicting survival in patients with non-small cell lung cancer (NSCLC). Using a dataset of pre-treatment CT images of 422 patients diagnosed with non-small cell lung cancer (NSCLC), we examined deep learning model performance across demographic and tumor specific factors. Demographic factors of interest included age and gender. Clinical factors of interest included tumor histology, overall stage, T-Stage, and N-Stage. The effect of bias among training data was examined by varying the representation of demographic and clinical populations within the training and validation datasets. Model generalizability was measured by comparing AUC values among validation datasets (biased versus unbiased). AUC was estimated using 1,000 bootstrapped samples of 400 patients from validation cohorts. We found training datasets with biased representation of NSCLC histologist to be associated with greatest decrease in generalizability. Specifically, we found over-representation of adenocarcinoma within training datasets to be associated with an AUC reduction of 0.320 (0.296 - 0.344 CI, p Citation Format: Aidan Gilson, Justin Du, Guneet Janda, Sachin Umrao, Marina Joel, Rachel Choi, Roy Herbst, Harlan Krumholz, Sanjay Aneja. The impact of phenotypic bias in the generalizability of deep learning models in non-small cell lung cancer [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-074.

Purpose: Deep learning (DL) models have shown the ability to automate the classification of diagnostic images used for cancer detection. Unfortunately, recent evidence has suggested DL models are also vulnerable to adversarial image attacks by manipulating image pixels to force models to make incorrect predictions with high confidence. The existence of adversarial images, which are imperceptible from unmodified images to the human eye, poses a roadblock to the safe implementation of DL models in clinical settings. The extent to which diagnostic imaging is vulnerable to adversarial image attacks remains underexplored. We investigated the effectiveness of adversarial imaging attacks on DL models for three common imaging tasks within oncology. Additionally, we explored whether adversarial image attack vulnerability could be used as a metric to improve deep learning model performance. Methods: We employed adversarial imaging attacks on DL models for three common imaging tasks within oncology: 1) Classifying malignant lung nodules on CT imaging, 2) Classifying brain metastases on MRI imaging, 3) Classifying malignant breast lesions on mammograms. To assess relative vulnerability to adversarial image attacks, we also employed two DL models on non-medical images: 1) CIFAR10, 2) MNIST. We considered three first-order adversarial attacks: Fast Gradient Sign Method, Projected Gradient Descent, and Basic Iterative Method. Vulnerability to adversarial image attacks was assessed by comparing model accuracy at fixed levels of image perturbations. Model performance was also measured after removing images which were most susceptible to adversarial imaging attacks. Results: We observed that all three diagnostic imaging types were susceptible to adversarial imaging attacks. Overall diagnostic images were more vulnerable to adversarial attacks compared to non-medical images. Mammograms [29.6% accuracy] appeared to be the most vulnerable to adversarial imaging attacks followed by lung CTs [30.6% accuracy] and brain MRIs [30.8% accuracy]. Finally, we determined that removing images most vulnerable to adversarial manipulation leads to improved deep learning model performance [Mammogram: 73.6 % accuracy, CT: 83.0% accuracy, MRI 84.2% accuracy]. Conclusion: Our study demonstrates that diagnostic imaging modalities in cancer are likely more vulnerable to adversarial attacks than non-medical images. Susceptibility to adversarial imaging attacks varies across different diagnostic imaging modalities. Adversarial susceptibility for an individual image can be used as a valuable metric to improve DL model performance on diagnostic images. Citation Format: Marina Joel, Sachin Umrao, Enoch Chang, Rachel Choi, Daniel Yang, Aidan Gilson, Roy Herbst, Harlan Krumholz, Sanjay Aneja. Exploring adversarial image attacks on deep learning models in oncology [abstract]. In: Proceedings of the AACR Virtual Special Conference on Artificial Intelligence, Diagnosis, and Imaging; 2021 Jan 13-14. Philadelphia (PA): AACR; Clin Cancer Res 2021;27(5_Suppl):Abstract nr PO-078.