Search

Summary Objective To develop a machine learning-based prediction model for incident radiographic osteoarthritis (OA) of the knee over 8 years using MRI-based cartilage biochemical composition and knee joint structure, demographics, and clinical predictors including muscle strength and symptoms. Design Individuals (n=1044) with baseline Kellgren Lawrence (KL) grade 0-1 in the right knee from the Osteoarthritis Initiative database were analyzed. 3T MRI at baseline was used to quantify knee cartilage T2, and Whole-Organ Magnetic Resonance Imaging Scores (WORMS) were obtained for cartilage, meniscus, and bone marrow. The outcome was set as true if a subject developed KL grade 2-4 OA in the right knee over 8 years (n=183) and false if the subject remained at KL 0-1 over 8 years (n=861). We developed and compared three models: Model 1: 112 predictors based on OA risk factors; Model 2: top ten predictors based on feature importance score from Model 1 and clinical relevance; Model 3: Model 2 without the imaging predictors. We compared the models using the area under the R OC curve derived from hold-out data. Results The 10-predictor model (Model 2, that includes cartilage and meniscus WORMS scores and cartilage T2) had a slightly lower AUC (0.772) compared to the model with 112 predictors (Model 1: AUC=0.792, p=0.739); and had a significantly higher AUC compared to the model without MR imaging predictors (Model 3, AUC=0.669, p=0.011). Conclusions A 10-predictor model including MRI parameters coupled with demographics, symptoms, muscle, and physical activity scores provides good prediction of incident radiographic OA over 8 years.

BackgroundIdentifying visually and semantically similar radiological images in a database can facilitate the creation of decision support tools, teaching files, and research cohorts. Existing content-based image retrieval tools are often limited to searching by pixel-wise difference or vector distance of model predictions. Vision transformers (ViT) use attention to simultaneously take into account radiological diagnosis and visual appearance.PurposeWe aim to develop a ViT-based image retrieval framework and evaluate the algorithm on NIH Chest Radiographs (CXR) and NLST Chest CTs.Materials and MethodsThe model was trained on 112,120 CXR and 111,955 CT images. For CXR, a ViT binary classifier was trained on 4 ground truth labels (Cardiomegaly, Opacity, Emphysema, No Finding) and ensembled to produce multilabel classifications for each CXR. For CT, a regression model was trained to minimize L1 loss on the continuous ground truth labels of patient weight. The ViT image embedding layer was treated as a global image descriptor, using the L2 distance between descriptors as a similarity measure. To qualitatively evaluate the model, five radiologists performed a reader performance study with random query images (25 CT, 25 CXR). For each image, they chose the 5 most similar images from a set of 10 images (the 5 closest and 5 furthest images from the query in model space). Inter-radiologist and radiologist-model agreement statistics were calculated.ResultsThe CXR model achieved nDCG@5 of 0.73 (pConclusionOur ViT architecture retrieved visually and semantically similar radiological images.Summary StatementThis study evaluates the efficacy of using ViT based image embeddings for CBIR tasks for CXR and CT images, finding that it performs well on visual and semantic recognition tasks.Key ResultsThe CXR model achieved nDCG@5 of 0.73 (pThe CT model achieved nDCG of 16.85 (pInter-radiologist Fleiss Kappa of 0.51 and radiologist consensus to model Cohen’s Kappa of 0.65 were observed.

Federated learning (FL) is a method used for training artificial intelligence models with data from multiple sources while maintaining data anonymity, thus removing many barriers to data sharing. Here we used data from 20 institutes across the globe to train a FL model, called EXAM (electronic medical record (EMR) chest X-ray AI model), that predicts the future oxygen requirements of symptomatic patients with COVID-19 using inputs of vital signs, laboratory data and chest X-rays. EXAM achieved an average area under the curve (AUC) >0.92 for predicting outcomes at 24 and 72 h from the time of initial presentation to the emergency room, and it provided 16% improvement in average AUC measured across all participating sites and an average increase in generalizability of 38% when compared with models trained at a single site using that site’s data. For prediction of mechanical ventilation treatment or death at 24 h at the largest independent test site, EXAM achieved a sensitivity of 0.950 and specificity of 0.882. In this study, FL facilitated rapid data science collaboration without data exchange and generated a model that generalized across heterogeneous, unharmonized datasets for prediction of clinical outcomes in patients with COVID-19, setting the stage for the broader use of FL in healthcare. Federated learning, a method for training artificial intelligence algorithms that protects data privacy, was used to predict future oxygen requirements of symptomatic patients with COVID-19 using data from 20 different institutes across the globe.

BackgroundEmphysema is a common pulmonary pathology known to be associated with increased risk of lung cancer and lung biopsy complications. Prevailing quantitation method of calculating voxel-wise percentage of low attenuation area (LAA) of lung tissue from CT scans is prone to noise and error due overcounting of single voxel LAA and incomplete segmentation of airways.PurposeWe aim to develop an accurate algorithm to quantitatively measure emphysema and classify its severity..Methods and MaterialsTwo chest CT datasets were obtained from two tertiary hospitals as training and external validation datasets. Exclusion criteria included any patients whose emphysema extent was not specified by the accompanying report. The training dataset included 722 patients, and the validation dataset included 1006 patients. Following lung segmentation and airways removal, we applied convolution of the segmented lung with averaging kernels of different sizes in 2D and 3D. Cutoffs between “none,” “mild to moderate,” and “severe” emphysema were determined via weighted logistic regression on the training dataset, and the categorical emphysema extent was obtained for each patient. The main measure for evaluating model performance was area under the curve (AUC) of the receiver operating characteristic (ROC) on the training dataset and accuracy of classification on both the training and the validation dataset. The 1×1×1 kernel, which is equivalent to the traditional LAA score, was used for comparison to other kernels for performance evaluation.ResultsThe best model used a 3D 3×3×3 kernel for average filtering with airways post processing and achieved a mean AUC of 0.782 and 0.985 for “none”-versus-rest and “severe”-versus-rest classifications respectively. It achieved a 0.676 and 0.757 multiclass classification accuracy on the training and validation dataset respectively.Conclusions and RelevanceWe present an automated pipeline that can achieve accurate emphysema quantification and severity classification. We showed that convolving the segmented lung with a 3D 3×3×3 kernel and post-processing to remove airways can reliably quantify emphysema.

Background A systematic approach to MRI protocol assignment is essential for the efficient delivery of safe patient care. Advances in natural language processing (NLP) allow for the development of accurate automated protocol assignment. We aim to develop, evaluate, and deploy an NLP model that automates protocol assignment, given the clinician indication text. Methods We collected 7139 spine MRI protocols (routine or contrast) and 990 head MRI protocols (routine brain, contrast brain, or other) from a single institution. Protocols were split into training (n = 4997 for spine MRI; n = 839 for head MRI), validation (n = 1071 for spine MRI, fivefold cross-validation used for head MRI), and test (n = 1071 for spine MRI; n = 151 for head MRI) sets. fastText and XGBoost were used to develop 2 NLP models to classify spine and head MRI protocols, respectively. A Flask-based web app was developed to be deployed via Heroku. Results The spine MRI model had an accuracy of 83.38% and a receiver operator characteristic area under the curve (ROC-AUC) of 0.8873. The head MRI model had an accuracy of 85.43% with a routine brain protocol ROC-AUC of 0.9463 and contrast brain protocol ROC-AUC of 0.9284. Cancer, infectious, and inflammatory related keywords were associated with contrast administration. Structural anatomic abnormalities and stroke/altered mental status were indicative of routine spine and brain MRI, respectively. Error analysis revealed increasing the sample size may improve performance for head MRI protocols. A web version of the model is provided for demonstration and deployment. Conclusion We developed and web-deployed two NLP models that accurately predict spine and head MRI protocol assignment, which could improve radiology workflow efficiency.

PURPOSE: To develop radiology domain–specific bidirectional encoder representations from transformers (BERT) models that can identify speech recognition (SR) errors and suggest corrections in radiology reports. MATERIALS AND METHODS: A pretrained BERT model, Clinical BioBERT, was further pretrained on a corpus of 114 008 radiology reports between April 2016 and August 2019 that were retrospectively collected from two hospitals. Next, the model was fine-tuned on a training dataset of generated insertion, deletion, and substitution errors, creating Radiology BERT. This model was retrospectively evaluated on an independent dataset of radiology reports with generated errors (n = 18 885) and on unaltered report sentences (n = 2000) and prospectively evaluated on true clinical SR errors (n = 92). Correction Radiology BERT was separately trained to suggest corrections for detected deletion and substitution errors. Area under the receiver operating characteristic curve (AUC) and bootstrapped 95% CIs were calculated for each evaluation dataset. RESULTS: Radiology-specific BERT had AUC values of >.99 (95% CI: >0.99, >0.99), 0.94 (95% CI: 0.93, 0.94), 0.98 (95% CI: 0.98, 0.98), and 0.97 (95% CI: 0.97, 0.97) for detecting insertion, deletion, substitution, and all errors, respectively, on the independently generated test set. Testing on unaltered report impressions revealed a sensitivity of 82% (28 of 34; 95% CI: 70%, 93%) and specificity of 88% (1521 of 1728; 95% CI: 87%, 90%). Testing on prospective SR errors showed an accuracy of 75% (69 of 92; 95% CI: 65%, 83%). Finally, the correct word was the top suggestion for 45.6% (475 of 1041; 95% CI: 42.5%, 49.3%) of errors. CONCLUSION: Radiology-specific BERT models fine-tuned on generated errors were able to identify SR errors in radiology reports and suggest corrections. Keywords: Computer Applications, Technology Assessment Supplemental material is available for this article. © RSNA, 2022 See also the commentary by Abajian and Cheung in this issue.

OBJECTIVE. The purpose of our study was to determine the CT features of benign intrapulmonary lymph nodes in pediatric patients with known extrapulmonary solid malignancy. MATERIALS AND METHODS. A retrospective review of surgical pathology archives was performed to identify consecutive chest CT studies of pediatric patients (≤ 18 years) with extrapulmonary solid malignancy and histologically confirmed benign intrapulmonary lymph nodes between January 1, 2004, and March 15, 2020. CT features of intrapulmonary lymph nodes-including size, shape, margin, type, associated calcification or fat, and location-were independently evaluated by two pediatric radiologist reviewers. The CT features of benign intrapulmonary lymph nodes in pediatric patients were analyzed using summary statistics. Interobserver agreement was measured with the kappa coefficient. RESULTS. There were 36 pathology-confirmed benign intrapulmonary lymph nodes in 27 pediatric patients (18 boys and nine girls; mean age, 12 years; age range, 1-18.2 years). Twenty-three (63.9%) of the benign intrapulmonary lymph nodes were biopsied from the right lung and 13 (36.1%) from the left lung (p = .03). The mean size, determined from CT studies, of benign intrapulmonary lymph nodes was 3.6 mm (SD, 1.4 mm; range, 1.3-7.8 mm). Triangular shape (25/36, 69.4%) was the most common shape of the benign intrapulmonary lymph nodes. Less commonly seen shapes of benign intrapulmonary lymph nodes were oval (6/36, 16.7%), round (3/36, 8.3%), and trapezoidal (2/36, 5.6%). All benign intrapulmonary lymph nodes were smoothly marginated and solid without associated calcification or fat. Of the 36 benign intrapulmonary lymph nodes, 15 (41.7%) were pleura-based; 11 (30.6%), perifissural; and 10 (27.8%), parenchymal. The kappa value for interobserver agreement between the two reviewers was 0.917 (95% CI, 0.825-1.000; standard error, 0.047), which corresponds to near-perfect agreement. CONCLUSION. In pediatric patients with known extrapulmonary solid malignancy, benign intrapulmonary lymph nodes are subcentimeter (mean size, 3.6 mm), smoothly marginated, and solid without containing calcification or fat on CT. In particular, triangular shape was the most commonly encountered shape of a benign intrapulmonary lymph node.

Our objective was to develop deep learning models with chest radiograph data to predict healthcare costs and classify top-50% spenders. 21,872 frontal chest radiographs were retrospectively collected from 19,524 patients with at least 1-year spending data. Among the patients, 11,003 patients had 3 years of cost data, and 1678 patients had 5 years of cost data. Model performances were measured with area under the receiver operating characteristic curve (ROC-AUC) for classification of top-50% spenders and Spearman ρ for prediction of healthcare cost. The best model predicting 1-year (N=21,872) expenditure achieved ROC-AUC of 0.806 [95% CI, 0.793-0.819] for top-50% spender classification and ρ of 0.561 [0.536-0.586] for regression. Similarly, for predicting 3-year (N=12,395) expenditure, ROC-AUC of 0.771 [0.750-0.794] and ρ of 0.524 [0.489-0.559]; for predicting 5-year (N=1,779) expenditure ROC-AUC of 0.729 [0.667-0.729] and ρ of 0.424 [0.324-0.529]. Our deep learning model demonstrated the feasibility of predicting health care expenditure as well as classifying top 50% healthcare spenders at 1, 3, and 5 year(s), implying the feasibility of combining deep learning with information-rich imaging data to uncover hidden associations that may allude physicians. Such a model can be a starting point of making an accurate budget in reimbursement models in healthcare industries.

Background Radiomics-based biomarkers enable the prognostication of resected non-small cell lung cancer (NSCLC), but their effectiveness in clinical stage and pathologic stage IA pure-solid tumors requires further determination. Purpose To construct an efficient radiomics signature for survival risk stratification personalized for patients with clinical stage and pathologic stage IA pure-solid NSCLC. Materials and Methods In this retrospective study, six radiomics signatures were constructed for patients with stage IA pure-solid NSCLC who underwent resection between January 2011 and December 2013 at authors' institution and were tested in the radiogenomics data set. The radiomics features were extracted from the intratumoral two-dimensional region, three-dimensional volume, and peritumoral area using PyRadiomics. The discriminative abilities of the signatures were quantified using the area under the time-dependent receiver operating characteristic curve (AUC), and the optimal signature was selected for patient stratification. Results The study included 592 patients with stage IA pure-solid NSCLC (median age, 61 years; interquartile range, 55-66 years; 269 women) for radiomics analysis: 381 patients for training, 163 for internal validation, and 48 for external validation. The radiomics signature combining three-region features yielded the highest 3- and 5-year AUCs of 0.77 and 0.78, respectively, in the internal validation set and 0.76 and 0.75, respectively, in the external validation set. Multivariable analysis suggested that the radiomics signature remained an independent prognostic factor (hazard ratio, 6.2; 95% CI: 3.5, 11.0

The advancements of artificial intelligence (AI) for osteoarthritis (OA) applications have been rapid in recent years, particularly innovations of deep learning for image classification, lesion detection, cartilage segmentation, and prediction modeling of future knee OA development. This review article focuses on AI applications in OA research, first describing machine learning (ML) techniques and workflow, followed by how these algorithms are used for OA classification tasks through imaging and non-imaging-based ML models. Deep learning applications for OA research, including analysis of both radiographs for automatic detection of OA severity, and MR images for detection of cartilage/meniscus lesions and cartilage segmentation for automatic T2 quantification will be described. In addition, information on ML models that identify individuals at high risk of OA development will be provided. The future vision of machine learning applications in imaging of OA and cartilage hinges on implementation of AI for optimizing imaging protocols, quantitative assessment of cartilage, and automated analysis of disease burden yielding a faster and more efficient workflow for a radiologist with a higher level of reproducibility and precision. It may also provide risk assessment tools for individual patients, which is an integral part of precision medicine.

Benign papillary and sclerosing lesions of the breast (intraductal papillomas, complex sclerosing lesions, radial scars) are considered high-risk lesions due to the potential for upgrade to carcinoma on subsequent surgical excision. Optimal clinical management of such lesions remains unclear due to variable reported upgrade rates. Apocrine metaplasia is a common finding in breast tissue and its role in MRI enhancing lesions is increasingly being recognized. The purpose of this study was to investigate the MRI features of papillary and sclerosing lesions of the breast, evaluate the clinical management and upgrade rate of such lesions, and examine the contribution of apocrine metaplasia to the imaging findings. A 13-year retrospective review of MRI-guided biopsies identified 70 MRI-detected and -biopsied papillary and sclerosing lesions. Sixteen lesions without atypia underwent surgical excision; only one case (6%) was upgraded to pleomorphic lobular carcinoma in situ. The majority (64%) of biopsies contained apocrine metaplasia either within or adjacent to the targeted lesion. We found that half of MRI-detected lesions had T2 hyperintense foci (2-5 mm) or masses (>5 mm) adjacent to the lesion. Histologic correlation showed apocrine cysts were likely responsible for this imaging finding in 56% of these cases.

BackgroundReidentification of prior nodules for temporal comparison is an important but time-consuming step in lung cancer screening. We develop and evaluate an automated nodule detector that utilizes the axial-slice number of nodules found in radiology reports to generate high precision nodule predictions.Methods888 CTs from Lung Nodule Analysis were used to train a 2-dimensional (2D) object detection neural network. A pipeline of 2D object detection, 3D unsupervised clustering, false positive reduction, and axial-slice numbers were used to generate nodule candidates. 47 CTs from the National Lung Cancer Screening Trial (NLST) were used for model evaluation.ResultsOur nodule detector achieved a precision of 0.962 at a recall of 0.573 on the NLST test set for any nodule. When adjusting for unintended nodule predictions, we achieved a precision of 0.931 at a recall 0.561, which corresponds to 0.06 false positives per CT. Error analysis revealed better detection of nodules with soft tissue attenuation compared to ground glass and undeterminable attenuation. Nodule margins, size, location, and patient demographics did not differ between correct and incorrect predictions.ConclusionsUtilization of axial-slice numbers from radiology reports allowed for development of a lung nodule detector with a low false positive rate compared to prior feature-engineering and machine learning approaches. This high precision nodule detector can reduce time spent on reidentification of prior nodules during lung cancer screening and can rapidly develop new institutional datasets to explore novel applications of computer vision in lung cancer imaging.

Rationale and Objectives Radiology turnaround time is an important quality measure that can impact hospital workflow and patient outcomes. We aimed to develop a machine learning model to predict delayed turnaround time during non-business hours and identify factors that contribute to this delay. Materials and Methods This retrospective study consisted of 15,117 CT cases from May 2018 to May 2019 during non-business hours at two hospital campuses after applying exclusion criteria. Of these 15,177 cases, 7,532 were inpatient cases and 7,585 were emergency cases. Order time, scan time, first communication by radiologist, free-text indications, and other clinical metadata were extracted. A combined XGBoost classifier and Random Forest natural language processing model was trained with 85% of the data and tested with 15% of the data. The model predicted two measures of delay: when the exam was ordered to first communication (total time) and when the scan was completed to first communication (interpretation time). The model was analyzed with the area under the curve (AUC) of receiver operating characteristic (ROC) and feature importance. Source code: https://bit.ly/2UrLiVJ Results The algorithm reached an AUC of 0.85, with a 95% confidence interval [0.83, 0.87], when predicting delays greater than 245 minutes for “total time” and 0.71, with a 95% confidence interval [0.68, 0.73], when predicting delays greater than 57 minutes for “interpretation time”. At our institution, CT scan description (e.g. "CTA chest pulmonary embolism protocol"), time of day, and year in training were more predictive features compared to body part, inpatient status, and hospital campus for both interpretation and total time delay. Conclusion This algorithm can be applied clinically when a physician is ordering the scan to reasonably predict delayed turnaround time. Such a model can be leveraged to identify factors associated with delays and emphasize areas for improvement to patient outcomes.

ObjectivesInvestigate long-term effects of repeated transarterial chemoembolization (TACE) on portal venous pressure (PVP) using non-invasive surrogate markers of portal hypertension.MethodsRetrospective, Institutional Review Board-approved study. 99 patients [hepatocellular carcinoma (HCC) group (n=57); liver metastasis group (n=42)] who underwent 279TACEs and had longitudinal pre-/post-therapy contrast-enhanced-MRI (n=388) and complete blood work were included. Outcomes of interest were platelet count (PC), spleen volume, ascites and portosystemic collaterals. Variables included TACE type/number, tumor type, microcatheter location, Child-Pugh, baseline tumor burden (tumor number/total/largest size), vessel invasion, alpha-fetoprotein, Eastern Cooperative Oncology Group (ECOG) performance status, and Model for End-Stage Liver Disease (MELD) score. Generalized Estimating Equations assessed the associations between TACE and outcomes. Power analysis determined the sample size was sufficient.ResultsNo significant change in PC over time was observed in either groups, regardless of liver function (P>0.05). Baseline spleen volume was 226 cm3 for metastatic group, and was larger by 204 cm3 for HCC group (P3 (95%CI: 8-32; P=0.001) for both groups after 1stTACE and by 16cm3/TACE (P=0.099) over the full follow-up (up to 9TACEs). Spleen volume also tended to increase by 23cm3 (95%CI: -1–48; P=0.064) with higher tumor burden. Odds of developing moderate/severe ascites for metastatic patients was decreased by 0.5 (95%CI: 0.3–0.9; P=0.014), regardless of the Child-Pugh, and increased by 1.5 (95%CI: 1.2–1.9; PP=0.008). PVP-related complications such as variceal bleeding post-TACE were low (0.4%).ConclusionRepeated TACEs did seem to have an impact on PVP. However, the increase in PVP had marginal effects with low portal hypertension-related complications.

Federated learning (FL) is a method used for training artificial intelligence models with data from multiple sources while maintaining data anonymity, thus removing many barriers to data sharing. Here we used data from 20 institutes across the globe to train a FL model, called EXAM (electronic medical record (EMR) chest X-ray AI model), that predicts the future oxygen requirements of symptomatic patients with COVID-19 using inputs of vital signs, laboratory data and chest X-rays. EXAM achieved an average area under the curve (AUC)0.92 for predicting outcomes at 24 and 72 h from the time of initial presentation to the emergency room, and it provided 16% improvement in average AUC measured across all participating sites and an average increase in generalizability of 38% when compared with models trained at a single site using that site's data. For prediction of mechanical ventilation treatment or death at 24 h at the largest independent test site, EXAM achieved a sensitivity of 0.950 and specificity of 0.882. In this study, FL facilitated rapid data science collaboration without data exchange and generated a model that generalized across heterogeneous, unharmonized datasets for prediction of clinical outcomes in patients with COVID-19, setting the stage for the broader use of FL in healthcare.

Although machine learning (ML) has made significant improvements in radiology, few algorithms have been integrated into clinical radiology workflow. Complex radiology IT environments and Picture Archiving and Communication System (PACS) pose unique challenges in creating a practical ML schema. However, clinical integration and testing are critical to ensuring the safety and accuracy of ML algorithms. This study aims to propose, develop, and demonstrate a simple, efficient, and understandable hardware and software system for integrating ML models into the standard radiology workflow and PACS that can serve as a framework for testing ML algorithms. A Digital Imaging and Communications in Medicine/Graphics Processing Unit (DICOM/GPU) server and software pipeline was established at a metropolitan county hospital intranet to demonstrate clinical integration of ML algorithms in radiology. A clinical ML integration schema, agnostic to the hospital IT system and specific ML models/frameworks, was implemented and tested with a breast density classification algorithm and prospectively evaluated for time delays using 100 digital 2D mammograms. An open-source clinical ML integration schema was successfully implemented and demonstrated. This schema allows for simple uploading of custom ML models. With the proposed setup, the ML pipeline took an average of 26.52 s per second to process a batch of 100 studies. The most significant processing time delays were noted in model load and study stability times. The code is made available at “ http://bit.ly/2Z121hX ”. We demonstrated the feasibility to deploy and utilize ML models in radiology without disrupting existing radiology workflow.

Background A multitask deep learning model might be useful in large epidemiologic studies wherein detailed structural assessment of osteoarthritis still relies on expert radiologists' readings. The potential of such a model in clinical routine should be investigated. Purpose To develop a multitask deep learning model for grading radiographic hip osteoarthritis features on radiographs and compare its performance to that of attending-level radiologists. Materials and Methods This retrospective study analyzed hip joints seen on weight-bearing anterior-posterior pelvic radiographs from participants in the Osteoarthritis Initiative (OAI). Participants were recruited from February 2004 to May 2006 for baseline measurements, and follow-up was performed 48 months later. Femoral osteophytes (FOs), acetabular osteophytes (AOs), and joint-space narrowing (JSN) were graded as absent, mild, moderate, or severe according to the Osteoarthritis Research Society International atlas. Subchondral sclerosis and subchondral cysts were graded as present or absent. The participants were split at 80% (n = 3494), 10% (n = 437), and 10% (n = 437) by using split-sample validation into training, validation, and testing sets, respectively. The multitask neural network was based on DenseNet-161, a shared convolutional features extractor trained with multitask loss function. Model performance was evaluated in the internal test set from the OAI and in an external test set by using temporal and geographic validation consisting of routine clinical radiographs. Results A total of 4368 participants (mean age, 61.0 years ± 9.2 [standard deviation]; 2538 women) were evaluated (15 364 hip joints on 7738 weight-bearing anterior-posterior pelvic radiographs). The accuracy of the model for assessing these five features was 86.7% (1333 of 1538) for FOs, 69.9% (1075 of 1538) for AOs, 81.7% (1257 of 1538) for JSN, 95.8% (1473 of 1538) for subchondral sclerosis, and 97.6% (1501 of 1538) for subchondral cysts in the internal test set, and 82.7% (86 of 104) for FOS, 65.4% (68 of 104) for AOs, 80.8% (84 of 104) for JSN, 88.5% (92 of 104) for subchondral sclerosis, and 91.3% (95 of 104) for subchondral cysts in the external test set. Conclusion A multitask deep learning model is a feasible approach to reliably assess radiographic features of hip osteoarthritis. © RSNA, 2020 Online supplemental material is available for this article.

Purpose To compare image quality and radiation exposure between a new angiographic imaging system and the preceding generation system during uterine artery embolization (UAE). Materials and Methods In this retrospective, IRB-approved two-arm study, 54 patients with symptomatic uterine fibroids were treated with UAE on two different angiographic imaging systems. The new system includes optimized acquisition parameters and real-time image processing algorithms. Air kerma (AK), dose area product (DAP) and acquisition time for digital fluoroscopy (DF) and digital subtraction angiography (DSA) were recorded. Body mass index was noted as well. DF image quality was assessed objectively by image noise measurements. DSA image quality was rated by two blinded, independent readers on a four-rank scale. Statistical differences were assessed with unpaired t tests and Wilcoxon rank-sum tests. Results There was no significant difference between the patients treated on the new (n = 36) and the old system (n = 18) regarding age (p = 0.10), BMI (p = 0.18), DF time (p = 0.35) and DSA time (p = 0.17). The new system significantly reduced the cumulative AK and DAP by 64 and 72%, respectively (median 0.58 Gy and 145.9 Gy*cm2 vs. 1.62 Gy and 526.8 Gy*cm2, p

Objectives To compare the diagnostic accuracy of hand-held point of care (POC) versus conventional sonography in a general diagnostic setting with the intention to inform medical providers or clinicians on the rational use of POC ultrasound in resource limited settings. Methods Over 3 months in 2010, 47 patients were prospectively enrolled at a single academic center to obtain 54 clinical conventional ultrasound examinations and 54 study-only POC ultrasound examinations. Indications were 48% abdominal, 26% retroperitoneal, and 24% obstetrical. Nine blinded readers (sonographers, residents, and attending radiologists) sequentially assigned diagnoses to POC and then conventional studies, yielding 476 interpreted study pairs. Diagnostic accuracy was obtained by comparing POC and conventional diagnoses to a reference diagnosis established by the unblinded, senior author. Analysis was stratified by study type, body mass index (BMI), diagnostic confidence, and image quality. Results The mean diagnostic accuracy of conventional sonography was 84% compared with 74% for POC (P < .001). This difference was constant regardless of reader, exam type, or BMI. The sensitivity and specificity to detect abnormalities with conventional was 85 and 83%, compared with 75 and 68% for POC. The POC sonography demonstrated greater variability in image quality and diagnostic confidence, and this accounted for lower diagnostic accuracy. When image quality and diagnostic confidence were similar between POC and conventional examinations, there was no difference in accuracy. Conclusions Point of care was nearly as accurate as conventional sonography for basic, focused examinations. Observed differences in accuracy were attributed to greater variation in POC image quality.

Allograft rejection is a major concern in kidney transplantation. Inflammatory processes in patients with kidney allografts involve various patterns of immune cell recruitment and distributions. Lymphoid aggregates (LAs) are commonly observed in patients with kidney allografts and their presence and localization may correlate with severity of acute rejection. Alongside with other markers of inflammation, LAs assessment is currently performed by pathologists manually in a qualitative way, which is both time consuming and far from precise. Here we present the first automated method of identifying LAs and measuring their densities in whole slide images of transplant kidney biopsies. We trained a deep convolutional neural network based on U-Net on 44 core needle kidney biopsy slides, monitoring loss on a validation set (n=7 slides). The model was subsequently tested on a hold-out set (n=10 slides). We found that the coarse pattern of LAs localization agrees between the annotations and predictions, which is reflected by high correlation between the annotated and predicted fraction of LAs area per slide (Pearson R of 0.9756). Furthermore, the network achieves an auROC of 97.78 ± 0.93% and an IoU score of 69.72 ± 6.24 % per LA-containing slide in the test set. Our study demonstrates that a deep convolutional neural network can accurately identify lymphoid aggregates in digitized histological slides of kidney. This study presents a first automatic DL-based approach for quantifying inflammation marks in allograft kidney, which can greatly improve precision and speed of assessment of allograft kidney biopsies when implemented as a part of computer-aided diagnosis system.

Purpose: Hip fractures are a common cause of morbidity and mortality. Automatic identification and classification of hip fractures using deep learning may improve outcomes by reducing diagnostic errors and decreasing time to operation. Methods: Hip and pelvic radiographs from 1118 studies were reviewed and 3034 hips were labeled via bounding boxes and classified as normal, displaced femoral neck fracture, nondisplaced femoral neck fracture, intertrochanteric fracture, previous ORIF, or previous arthroplasty. A deep learning-based object detection model was trained to automate the placement of the bounding boxes. A Densely Connected Convolutional Neural Network (DenseNet) was trained on a subset of the bounding box images, and its performance evaluated on a held out test set and by comparison on a 100-image subset to two groups of human observers: fellowship-trained radiologists and orthopaedists, and senior residents in emergency medicine, radiology, and orthopaedics. Results: The binary accuracy for fracture of our model was 93.8% (95% CI, 91.3-95.8%), with sensitivity of 92.7% (95% CI, 88.7-95.6%), and specificity 95.0% (95% CI, 91.5-97.3%). Multiclass classification accuracy was 90.4% (95% CI, 87.4-92.9%). When compared to human observers, our model achieved at least expert-level classification under all conditions. Additionally, when the model was used as an aid, human performance improved, with aided resident performance approximating unaided fellowship-trained expert performance. Conclusions: Our deep learning model identified and classified hip fractures with at least expert-level accuracy, and when used as an aid improved human performance, with aided resident performance approximating that of unaided fellowship-trained attendings., Comment: Presented at Orthopaedic Research Society, Austin, TX, Feb 2, 2019, currently in submission for publication

PURPOSE: To demonstrate the feasibility of CT-less attenuation and scatter correction (ASC) in the image space using deep learning for whole-body PET, with a focus on the potential benefits and pitfalls. MATERIALS AND METHODS: In this retrospective study, 110 whole-body fluorodeoxyglucose (FDG) PET/CT studies acquired in 107 patients (mean age ± standard deviation, 58 years ± 18; age range, 11–92 years; 72 females) from February 2016 through January 2018 were randomly collected. A total of 37.3% (41 of 110) of the studies showed metastases, with diverse FDG PET findings throughout the whole body. A U-Net–based network was developed for directly transforming noncorrected PET (PET(NC)) into attenuation- and scatter-corrected PET (PET(ASC)). Deep learning–corrected PET (PET(DL)) images were quantitatively evaluated by using the standardized uptake value (SUV) of the normalized root mean square error, the peak signal-to-noise ratio, and the structural similarity index, in addition to a joint histogram for statistical analysis. Qualitative reviews by radiologists revealed the potential benefits and pitfalls of this correction method. RESULTS: The normalized root mean square error (0.21 ± 0.05 [mean SUV ± standard deviation]), mean peak signal-to-noise ratio (36.3 ± 3.0), mean structural similarity index (0.98 ± 0.01), and voxelwise correlation (97.62%) of PET(DL) demonstrated quantitatively high similarity with PET(ASC). Radiologist reviews revealed the overall quality of PET(DL). The potential benefits of PET(DL) include a radiation dose reduction on follow-up scans and artifact removal in the regions with attenuation correction– and scatter correction–based artifacts. The pitfalls involve potential false-negative results due to blurring or missing lesions or false-positive results due to pseudo–low-uptake patterns. CONCLUSION: Deep learning–based direct ASC at whole-body PET is feasible and potentially can be used to overcome the current limitations of CT-based approaches, benefiting patients who are sensitive to radiation from CT. Supplemental material is available for this article. © RSNA, 2020

BACKGROUND: There has been increasing interest in machine learning based natural language processing (NLP) methods in radiology; however, models have often used word embeddings trained on general web corpora due to lack of a radiology-specific corpus. PURPOSE: We examined the potential of Radiopaedia to serve as a general radiology corpus to produce radiology specific word embeddings that could be used to enhance performance on a NLP task on radiological text. MATERIALS AND METHODS: Embeddings of dimension 50, 100, 200, and 300 were trained on articles collected from Radiopaedia using a GloVe algorithm and evaluated on analogy completion. A shallow neural network using input from either our trained embeddings or pre-trained Wikipedia2014+Gigaword5 (WG) embeddings was used to label the Radiopaedia articles. Labeling performance was evaluated based on exact match accuracy and Hamming loss. The McNemar’s test with continuity and the Benjamini-Hochberg correction and a 5×2 cross validation paired two-tailed t-test were used to assess statistical significance. RESULTS: For accuracy in the analogy task, 50-dimensional (50-D) Radiopaedia embeddings outperformed WG embeddings on tumor origin analogies (p < 0.05) and organ adjectives (p < 0.01) whereas WG embeddings tended to outperform on inflammation location and bone vs. muscle analogies (p < 0.01). The two embeddings had comparable performance on other subcategories. In the labeling task, the Radiopaedia-based model outperformed the WG based model at 50, 100, 200, and 300-D for exact match accuracy (p < 0.001, p < 0.001, p< 0.01, and p < 0.05, respectively) and Hamming loss (p < 0.001, p < 0.001, p < 0.01, and p < 0.05, respectively). CONCLUSION: We have developed a set of word embeddings from Radiopaedia and shown that they can preserve relevant medical semantics and augment performance on a radiology NLP task. Our results suggest that the cultivation of a radiology-specific corpus can benefit radiology NLP models in the future.

PURPOSE: To develop radiomics-based CT scores for assessing lung disease severity and exacerbation risk in adult patients with cystic fibrosis (CF). MATERIALS AND METHODS: This two-center retrospective observational study was approved by an institutional ethics committee, and the need for patient consent was waived. A total of 215 outpatients with CF referred for unenhanced follow-up chest CT were evaluated in two different centers between January 2013 and December 2016. After lung segmentation, chest CT scans from center 1 (training cohort, 162 patients [median age, 29 years; interquartile range {IQR}, 24–36 years; 84 men]) were used to build CT scores from 38 extracted CT features, using five different machine learning techniques trained to predict a clinical prognostic score, the Nkam score. The correlations between the developed CT scores, two different clinical prognostic scores (Liou and CF-ABLE), forced expiratory volume in 1 second (FEV(1)), and risk of respiratory exacerbations were evaluated in the test cohort (center 2, 53 patients [median age, 27 years; IQR, 22–35 years; 34 men]) using the Spearman rank coefficient. RESULTS: In the test cohort, all radiomics-based CT scores showed moderate to strong correlation with the Nkam score (R = 0.57 to 0.63, P < .001) and Liou scores (R = −0.55 to −0.65, P < .001), whereas the correlation with CF-ABLE score was weaker (R = 0.28 to 0.38, P = .005 to .048). The developed CT scores showed strong correlation with predicted FEV(1) (R = −0.62 to −0.66, P < .001) and weak to moderate correlation with the number of pulmonary exacerbations to occur in the 12 months after the CT examination (R = 0.38 to 0.55, P < .001 to P = .006). CONCLUSION: Radiomics can be used to build automated CT scores that correlate to clinical severity and exacerbation risk in adult patients with CF. Supplemental material is available for this article. See also the commentary by Elicker and Sohn in this issue. © RSNA, 2020

PurposeTo compare the visibility of liver metastases on dual-phase cone-beam CT (DP-CBCT) and digital subtraction angiography (DSA), with reference to preinterventional contrast-enhanced magnetic resonance imaging (CE-MRI) of the liver.MethodsThis IRB-approved, retrospective study included 28 patients with neuroendocrine (NELM), colorectal (CRCLM), or sarcoma (SLM) liver metastases who underwent DP-CBCT during intra-arterial therapy (IAT) between 01/2010 and 10/2014. DP-CBCT was acquired after a single contrast agent injection in the tumor-feeding arteries at early and delayed arterial phases (EAP and DAP). The visibility of each lesion was graded by two radiologists in consensus on a three-rank scale (complete, partial, none) on DP-CBCT and DSA images using CE-MRI as reference.Results47 NELM, 43 CRCLM, and 16 SLM were included. On DSA 85.1, 44.1, and 37.5% of NELM, CRCLM, and SLM, were at least partially depicted, respectively. EAP-CBCT yielded significantly higher sensitivities of 88.3 and 87.5% for CRCLM and SLM, respectively (p&nbsp

Purpose To assess the visibility of radiopaque microspheres during transarterial embolization (TAE) in the VX2 rabbit liver tumor model by using multimodality imaging, including single-snapshot radiography, cone-beam computed tomography (CT), multidetector CT, and micro-CT. Materials and Methods The study was approved by the institutional animal care and use committee. Fifteen VX2-tumor-bearing rabbits were assigned to three groups depending on the type of embolic agent injected: 70-150-μm radiopaque microspheres in saline (radiopaque microsphere group), 70-150-μm radiopaque microspheres in contrast material (radiopaque microsphere plus contrast material group), and 70-150-μm radiolucent microspheres in contrast material (nonradiopaque microsphere plus contrast material group). Rabbits were imaged with single-snapshot radiography, cone-beam CT, and multidetector CT. Three to 5 weeks after sacrifice, excised livers were imaged with micro-CT and histologic analysis was performed. The visibility of the embolic agent was assessed with all modalities before and after embolization by using a qualitative three-point scale score reading study and a quantitative assessment of the signal-to-noise ratio (SNR) change in various regions of interest, including the tumor and its feeding arteries. The Kruskal-Wallis test was used to compare the rabbit characteristics across groups, and the Wilcoxon signed rank test was used to compare SNR measurements before and after embolization. Results Radiopaque microspheres were qualitatively visualized within tumor feeding arteries and targeted tissue with all imaging modalities (P.05), and their presence was confirmed with histologic examination. SNRs of radiopaque microsphere deposition increased after TAE on multidetector CT, cone-beam CT, and micro-CT images (P.05). Similar results were obtained when contrast material was added to radiopaque microspheres, except for additional image attenuation due to tumor enhancement. For the group with nonradiopaque microspheres and contrast material, retained tumoral contrast remained qualitatively visible with all modalities except for micro-CT, which demonstrated soluble contrast material washout over time. Conclusion Radiopaque microspheres were visible with all imaging modalities and helped increase conspicuity of the tumor as well as its feeding arteries after TAE in a rabbit VX2 liver tumor model. (©) RSNA, 2015.

PurposeTo compare currently available non-three-dimensional methods (Response Evaluation Criteria in Solid Tumors [RECIST], European Association for Study of the Liver [EASL], modified RECIST [mRECIST[) with three-dimensional (3D) quantitative methods of the index tumor as early response markers in predicting patient survival after initial transcatheter arterial chemoembolization (TACE).Materials and methodsThis was a retrospective single-institution HIPAA-compliant and institutional review board-approved study. From November 2001 to November 2008, 491 consecutive patients underwent intraarterial therapy for liver cancer with either conventional TACE or TACE with drug-eluting beads. A diagnosis of hepatocellular carcinoma (HCC) was made in 290 of these patients. The response of the index tumor on pre- and post-TACE magnetic resonance images was assessed retrospectively in 78 treatment-naïve patients with HCC (63 male; mean age, 63 years ± 11 [standard deviation]). Each response assessment method (RECIST, mRECIST, EASL, and 3D methods of volumetric RECIST [vRECIST] and quantitative EASL [qEASL]) was used to classify patients as responders or nonresponders by following standard guidelines for the uni- and bidimensional measurements and by using the formula for a sphere for the 3D measurements. The Kaplan-Meier method with the log-rank test was performed for each method to evaluate its ability to help predict survival of responders and nonresponders. Uni- and multivariate Cox proportional hazard ratio models were used to identify covariates that had significant association with survival.ResultsThe uni- and bidimensional measurements of RECIST (hazard ratio, 0.6; 95% confidence interval [CI]: 0.3, 1.0; P = .09), mRECIST (hazard ratio, 0.6; 95% CI: 0.6, 1.0; P = .05), and EASL (hazard ratio, 1.1; 95% CI: 0.6, 2.2; P = .75) did not show a significant difference in survival between responders and nonresponders, whereas vRECIST (hazard ratio, 0.6; 95% CI: 0.3, 1.0; P = .04), qEASL (Vol) (hazard ratio, 0.5; 95% CI: 0.3, 0.9; P = .02), and qEASL (%) (hazard ratio, 0.3; 95% CI: 0.15, 0.60; P < .001) did show a significant difference between these groups.ConclusionThe 3D-based imaging biomarkers qEASL and vRECIST were tumor response criteria that could be used to predict patient survival early after initial TACE and enabled clear identification of nonresponders.

PurposeTo compare liver coverage and tumor detectability by using preprocedural magnetic resonance (MR) images as a reference, as well as radiation exposure of cone-beam computed tomography (CT) with different rotational trajectories.Materials and methodsFifteen patients (nine men and six women; mean age ± standard deviation, 65 years ± 5) with primary or secondary liver cancer were retrospectively included in this institutional review board-approved study. A modified cone-beam CT protocol was used in which the C-arm rotates from +55° to -185° (open arc cone-beam CT) instead of -120° to +120° (closed arc cone-beam CT). Each patient underwent two sessions of transarterial chemoembolization between February 2013 and March 2014 with closed arc and open arc cone-beam CT (during the first and second transarterial chemoembolization sessions, respectively, as part of the institutional transarterial chemoembolization protocol). For each cone-beam CT examination, liver volume and tumor detectability were assessed by using MR images as the reference. Radiation exposure was compared by means of a phantom study. For statistical analysis, paired t tests and a Wilcoxon signed rank test were performed.ResultsMean liver volume imaged was 1695 cm(3) ± 542 and 1857 cm(3) ± 571 at closed arc and open arc cone-beam CT, respectively. The coverage of open arc cone-beam CT was significantly higher compared with closed arc cone-beam CT (97% vs 86% of the MR imaging liver volume, P = .002). In eight patients (53%), tumors were partially or completely outside the closed arc cone-beam CT field of view. All tumors were within the open arc cone-beam CT field of view. The open arc cone-beam CT radiation exposure by means of weighted CT index was slightly lower compared with that of closed arc cone-beam CT (-5.1%).ConclusionOpen arc cone-beam CT allowed for a significantly improved intraprocedural depiction of peripheral hepatic tumors while achieving a slight radiation exposure reduction.

Importance: Mammography screening currently relies on subjective human interpretation. Artificial intelligence (AI) advances could be used to increase mammography screening accuracy by reducing missed cancers and false positives. Objective: To evaluate whether AI can overcome human mammography interpretation limitations with a rigorous, unbiased evaluation of machine learning algorithms. Design, Setting, and Participants: In this diagnostic accuracy study conducted between September 2016 and November 2017, an international, crowdsourced challenge was hosted to foster AI algorithm development focused on interpreting screening mammography. More than 1100 participants comprising 126 teams from 44 countries participated. Analysis began November 18, 2016. Main Outcomes and Measurements: Algorithms used images alone (challenge 1) or combined images, previous examinations (if available), and clinical and demographic risk factor data (challenge 2) and output a score that translated to cancer yes/no within 12 months. Algorithm accuracy for breast cancer detection was evaluated using area under the curve and algorithm specificity compared with radiologists' specificity with radiologists' sensitivity set at 85.9% (United States) and 83.9% (Sweden). An ensemble method aggregating top-performing AI algorithms and radiologists' recall assessment was developed and evaluated. Results: Overall, 144 231 screening mammograms from 85 580 US women (952 cancer positive ≤12 months from screening) were used for algorithm training and validation. A second independent validation cohort included 166 578 examinations from 68 008 Swedish women (780 cancer positive). The top-performing algorithm achieved an area under the curve of 0.858 (United States) and 0.903 (Sweden) and 66.2% (United States) and 81.2% (Sweden) specificity at the radiologists' sensitivity, lower than community-practice radiologists' specificity of 90.5% (United States) and 98.5% (Sweden). Combining top-performing algorithms and US radiologist assessments resulted in a higher area under the curve of 0.942 and achieved a significantly improved specificity (92.0%) at the same sensitivity. Conclusions and Relevance: While no single AI algorithm outperformed radiologists, an ensemble of AI algorithms combined with radiologist assessment in a single-reader screening environment improved overall accuracy. This study underscores the potential of using machine learning methods for enhancing mammography screening interpretation. CA extern

PURPOSE: To develop and validate a deep learning algorithm that predicts the final diagnosis of Alzheimer disease (AD), mild cognitive impairment, or neither at fluorine 18 ((18)F) fluorodeoxyglucose (FDG) PET of the brain and compare its performance to that of radiologic readers. MATERIALS AND METHODS: Prospective (18)F-FDG PET brain images from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (2109 imaging studies from 2005 to 2017, 1002 patients) and retrospective independent test set (40 imaging studies from 2006 to 2016, 40 patients) were collected. Final clinical diagnosis at follow-up was recorded. Convolutional neural network of InceptionV3 architecture was trained on 90% of ADNI data set and tested on the remaining 10%, as well as the independent test set, with performance compared to radiologic readers. Model was analyzed with sensitivity, specificity, receiver operating characteristic (ROC), saliency map, and t-distributed stochastic neighbor embedding. RESULTS: The algorithm achieved area under the ROC curve of 0.98 (95% confidence interval: 0.94, 1.00) when evaluated on predicting the final clinical diagnosis of AD in the independent test set (82% specificity at 100% sensitivity), an average of 75.8 months prior to the final diagnosis, which in ROC space outperformed reader performance (57% [four of seven] sensitivity, 91% [30 of 33] specificity; P < .05). Saliency map demonstrated attention to known areas of interest but with focus on the entire brain. CONCLUSION: By using fluorine 18 fluorodeoxyglucose PET of the brain, a deep learning algorithm developed for early prediction of Alzheimer disease achieved 82% specificity at 100% sensitivity, an average of 75.8 months prior to the final diagnosis. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by Larvie in this issue.

Purpose To develop and validate a deep learning-based automatic detection algorithm (DLAD) for malignant pulmonary nodules on chest radiographs and to compare its performance with physicians including thoracic radiologists. Materials and Methods For this retrospective study, DLAD was developed by using 43 292 chest radiographs (normal radiograph-to-nodule radiograph ratio, 34 067:9225) in 34 676 patients (healthy-to-nodule ratio, 30 784:3892; 19 230 men [mean age, 52.8 years; age range, 18-99 years]; 15 446 women [mean age, 52.3 years; age range, 18-98 years]) obtained between 2010 and 2015, which were labeled and partially annotated by 13 board-certified radiologists, in a convolutional neural network. Radiograph classification and nodule detection performances of DLAD were validated by using one internal and four external data sets from three South Korean hospitals and one U.S. hospital. For internal and external validation, radiograph classification and nodule detection performances of DLAD were evaluated by using the area under the receiver operating characteristic curve (AUROC) and jackknife alternative free-response receiver-operating characteristic (JAFROC) figure of merit (FOM), respectively. An observer performance test involving 18 physicians, including nine board-certified radiologists, was conducted by using one of the four external validation data sets. Performances of DLAD, physicians, and physicians assisted with DLAD were evaluated and compared. Results According to one internal and four external validation data sets, radiograph classification and nodule detection performances of DLAD were a range of 0.92-0.99 (AUROC) and 0.831-0.924 (JAFROC FOM), respectively. DLAD showed a higher AUROC and JAFROC FOM at the observer performance test than 17 of 18 and 15 of 18 physicians, respectively (P < .05), and all physicians showed improved nodule detection performances with DLAD (mean JAFROC FOM improvement, 0.043; range, 0.006-0.190; P < .05). Conclusion This deep learning-based automatic detection algorithm outperformed physicians in radiograph classification and nodule detection performance for malignant pulmonary nodules on chest radiographs, and it enhanced physicians' performances when used as a second reader. © RSNA, 2018 Online supplemental material is available for this article.

Magnetic resonance imaging (MRI) protocoling can be time- and resource-intensive, and protocols can often be suboptimal dependent upon the expertise or preferences of the protocoling radiologist. Providing a best-practice recommendation for an MRI protocol has the potential to improve efficiency and decrease the likelihood of a suboptimal or erroneous study. The goal of this study was to develop and validate a machine learning-based natural language classifier that can automatically assign the use of intravenous contrast for musculoskeletal MRI protocols based upon the free-text clinical indication of the study, thereby improving efficiency of the protocoling radiologist and potentially decreasing errors. We utilized a deep learning-based natural language classification system from IBM Watson, a question-answering supercomputer that gained fame after challenging the best human players on Jeopardy! in 2011. We compared this solution to a series of traditional machine learning-based natural language processing techniques that utilize a term-document frequency matrix. Each classifier was trained with 1240 MRI protocols plus their respective clinical indications and validated with a test set of 280. Ground truth of contrast assignment was obtained from the clinical record. For evaluation of inter-reader agreement, a blinded second reader radiologist analyzed all cases and determined contrast assignment based on only the free-text clinical indication. In the test set, Watson demonstrated overall accuracy of 83.2% when compared to the original protocol. This was similar to the overall accuracy of 80.2% achieved by an ensemble of eight traditional machine learning algorithms based on a term-document matrix. When compared to the second reader’s contrast assignment, Watson achieved 88.6% agreement. When evaluating only the subset of cases where the original protocol and second reader were concordant (n = 251), agreement climbed further to 90.0%. The classifier was relatively robust to spelling and grammatical errors, which were frequent. Implementation of this automated MR contrast determination system as a clinical decision support tool may save considerable time and effort of the radiologist while potentially decreasing error rates, and require no change in order entry or workflow. ELECTRONIC SUPPLEMENTARY MATERIAL: The online version of this article (10.1007/s10278-017-0021-3) contains supplementary material, which is available to authorized users.