Search

To address the rapid evolution of artificial intelligence in medical imaging, the authors present the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) 2024 Update.

The Radiological Society of North of America (RSNA) and the Medical Image Computing and Computer Assisted Intervention (MICCAI) Society have led a series of joint panels and seminars focused on the present impact and future directions of artificial intelligence (AI) in radiology. These conversations have collected viewpoints from multidisciplinary experts in radiology, medical imaging, and machine learning on the current clinical penetration of AI technology in radiology and how it is impacted by trust, reproducibility, explainability, and accountability. The collective points-both practical and philosophical-define the cultural changes for radiologists and AI scientists working together and describe the challenges ahead for AI technologies to meet broad approval. This article presents the perspectives of experts from MICCAI and RSNA on the clinical, cultural, computational, and regulatory considerations-coupled with recommended reading materials-essential to adopt AI technology successfully in radiology and, more generally, in clinical practice. The report emphasizes the importance of collaboration to improve clinical deployment, highlights the need to integrate clinical and medical imaging data, and introduces strategies to ensure smooth and incentivized integration. Keywords: Adults and Pediatrics, Computer Applications-General (Informatics), Diagnosis, Prognosis © RSNA, 2024.

Supplemental material is available for this article.

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Supplemental material is available for this article.

Background There is increasing interest in noncontrast breast MRI alternatives for tumor visualization to increase the accessibility of breast MRI. Purpose To evaluate the feasibility and accuracy of generating simulated contrast-enhanced T1-weighted breast MRI scans from precontrast MRI sequences in biopsy-proven invasive breast cancer with use of deep learning. Materials and Methods Women with invasive breast cancer and a contrast-enhanced breast MRI examination that was performed for initial evaluation of the extent of disease between January 2015 and December 2019 at a single academic institution were retrospectively identified. A three-dimensional, fully convolutional deep neural network simulated contrast-enhanced T1-weighted breast MRI scans from five precontrast sequences (T1-weighted non-fat-suppressed [FS], T1-weighted FS, T2-weighted FS, apparent diffusion coefficient, and diffusion-weighted imaging). For qualitative assessment, four breast radiologists (with 3-15 years of experience) blinded to whether the method of contrast was real or simulated assessed image quality (excellent, acceptable, good, poor, or unacceptable), presence of tumor enhancement, and maximum index mass size by using 22 pairs of real and simulated contrast-enhanced MRI scans. Quantitative comparison was performed using whole-breast similarity and error metrics and Dice coefficient analysis of enhancing tumor overlap. Results Ninety-six MRI examinations in 96 women (mean age, 52 years ± 12 [SD]) were evaluated. The readers assessed all simulated MRI scans as having the appearance of a real MRI scan with tumor enhancement. Index mass sizes on real and simulated MRI scans demonstrated good to excellent agreement (intraclass correlation coefficient, 0.73-0.86; P < .001) without significant differences (mean differences, -0.8 to 0.8 mm; P = .36-.80). Almost all simulated MRI scans (84 of 88 [95%]) were considered of diagnostic quality (ratings of excellent, acceptable, or good). Quantitative analysis demonstrated strong similarity (structural similarity index, 0.88 ± 0.05), low voxel-wise error (symmetric mean absolute percent error, 3.26%), and Dice coefficient of enhancing tumor overlap of 0.75 ± 0.25. Conclusion It is feasible to generate simulated contrast-enhanced breast MRI scans with use of deep learning. Simulated and real contrast-enhanced MRI scans demonstrated comparable tumor sizes, areas of tumor enhancement, and image quality without significant qualitative or quantitative differences. © RSNA, 2022 Online supplemental material is available for this article. See also the editorial by Slanetz in this issue. An earlier incorrect version appeared online. This article was corrected on January 17, 2023.

Electronic health records (EHRs) offer decision support in the form of alerts, which are often though not always interruptive. These alerts, though sometimes effective, can come at the cost of high cognitive burden and workflow disruption. Less well studied is the design of the EHR itself-the ordering providers choice architecture-which nudges users toward alternatives, sometimes unintentionally toward waste and misuse, but ideally intentionally toward better practice. We studied 3 different workflows at our institution where the existing choice architecture was potentially nudging providers toward erroneous decisions, waste, and misuse in the form of inappropriate laboratory work, incorrectly specified computerized tomographic imaging, and excessive benzodiazepine dosing for imaging-related sedation. We changed the architecture to nudge providers toward better practice and found that the 3 nudges were successful to varying degrees in reducing erroneous decision-making and mitigating waste and misuse.

PurposeTo create an algorithm able to accurately detect IVC filters on radiographs without human assistance, capable of being used to screen radiographs to identify patients needing IVC filter retrieval.MethodsA primary dataset of 5225 images, 30% of which included IVC filters, was assembled and annotated. 85% of the data was used to train a Cascade R-CNN (Region Based Convolutional Neural Network) object detection network incorporating a pre-trained ResNet-50 backbone. The remaining 15% of the data, independently annotated by three radiologists, was used as a test set to assess performance. The algorithm was also assessed on an independently constructed 1424-image dataset, drawn from a different institution than the primary dataset.ResultsOn the primary test set, the algorithm achieved a sensitivity of 96.2% (95% CI 92.7-98.1%) and a specificity of 98.9% (95% CI 97.4-99.5%). Results were similar on the external test set: sensitivity 97.9% (95% CI 96.2-98.9%), specificity 99.6 (95% CI 98.9-99.9%).ConclusionFully automated detection of IVC filters on radiographs with high sensitivity and excellent specificity required for an automated screening system can be achieved using object detection neural networks. Further work will develop a system for identifying patients for IVC filter retrieval based on this algorithm.

Supplemental material is available for this article. Keywords: Informatics, MR Diffusion Tensor Imaging, MR Perfusion, MR Imaging, Neuro-Oncology, CNS, Brain/Brain Stem, Oncology, Radiogenomics, Radiology-Pathology Integration © RSNA, 2022.

Technological innovation has enabled the development of machine learning (ML) tools that aim to improve the practice of radiologists. In the last decade, ML applications to neuro-oncology have expanded significantly, with the pre-operative prediction of glioma grade using medical imaging as a specific area of interest. We introduce the subject of ML models for glioma grade prediction by remarking upon the models reported in the literature as well as by describing their characteristic developmental workflow and widely used classifier algorithms. The challenges facing these models-including data sources, external validation, and glioma grade classification methods -are highlighted. We also discuss the quality of how these models are reported, explore the present and future of reporting guidelines and risk of bias tools, and provide suggestions for the reporting of prospective works. Finally, this review offers insights into next steps that the field of ML glioma grade prediction can take to facilitate clinical implementation.

On October 5, 2020, the Medical Image Computing and Computer Assisted Intervention Society (MICCAI) 2020 conference hosted a virtual panel discussion with members of the Machine Learning Steering Subcommittee of the Radiological Society of North America. The MICCAI Society brings together scientists, engineers, physicians, educators, and students from around the world. Both societies share a vision to develop radiologic and medical imaging techniques through advanced quantitative imaging biomarkers and artificial intelligence. The panel elaborated on how collaborations between radiologists and machine learning scientists facilitate the creation and clinical success of imaging technology for radiology. This report presents structured highlights of the moderated dialogue at the panel. Keywords: Back-Propagation, Artificial Neural Network Algorithms, Machine Learning Algorithms © RSNA, 2021.

Coronary artery disease (CAD), the most common manifestation of cardiovascular disease, remains the most common cause of mortality in the United States. Risk assessment is key for primary prevention of coronary events and coronary artery calcium (CAC) scoring using computed tomography (CT) is one such non-invasive tool. Despite the proven clinical value of CAC, the current clinical practice implementation for CAC has limitations such as the lack of insurance coverage for the test, need for capital-intensive CT machines, specialized imaging protocols, and accredited 3D imaging labs for analysis (including personnel and software). Perhaps the greatest gap is the millions of patients who undergo routine chest CT exams and demonstrate coronary artery calcification, but their presence is not often reported or quantitation is not feasible. We present two deep learning models that automate CAC scoring demonstrating advantages in automated scoring for both dedicated gated coronary CT exams and routine non-gated chest CTs performed for other reasons to allow opportunistic screening. First, we trained a gated coronary CT model for CAC scoring that showed near perfect agreement (mean difference in scores = -2.86; Cohen's Kappa = 0.89, P

The coronavirus disease 2019 (COVID-19) pandemic is a global health care emergency. Although reverse-transcription polymerase chain reaction testing is the reference standard method to identify patients with COVID-19 infection, chest radiography and CT play a vital role in the detection and management of these patients. Prediction models for COVID-19 imaging are rapidly being developed to support medical decision making. However, inadequate availability of a diverse annotated data set has limited the performance and generalizability of existing models. To address this unmet need, the RSNA and Society of Thoracic Radiology collaborated to develop the RSNA International COVID-19 Open Radiology Database (RICORD). This database is the first multi-institutional, multinational, expert-annotated COVID-19 imaging data set. It is made freely available to the machine learning community as a research and educational resource for COVID-19 chest imaging. Pixel-level volumetric segmentation with clinical annotations was performed by thoracic radiology subspecialists for all COVID-19-positive thoracic CT scans. The labeling schema was coordinated with other international consensus panels and COVID-19 data annotation efforts, the European Society of Medical Imaging Informatics, the American College of Radiology, and the American Association of Physicists in Medicine. Study-level COVID-19 classification labels for chest radiographs were annotated by three radiologists, with majority vote adjudication by board-certified radiologists. RICORD consists of 240 thoracic CT scans and 1000 chest radiographs contributed from four international sites. It is anticipated that RICORD will ideally lead to prediction models that can demonstrate sustained performance across populations and health care systems.

PurposeThe coronavirus disease 2019 (COVID-19) pandemic has led to significant disruptions in the healthcare system including surges of infected patients exceeding local capacity, closures of primary care offices, and delays of non-emergent medical care. Government-initiated measures to decrease healthcare utilization (i.e., "flattening the curve") have included shelter-in-place mandates and social distancing, which have taken effect across most of the USA. We evaluate the immediate impact of the Public Health Messaging and shelter-in-place mandates on Emergency Department (ED) demand for radiology services.MethodsWe analyzed ED radiology volumes from the five University of California health systems during a 2-week time period following the shelter-in-place mandate and compared those volumes with March 2019 and early April 2019 volumes.ResultsED radiology volumes declined from the 2019 baseline by 32 to 40% (p < 0.001) across the five health systems with a total decrease in volumes across all 5 systems by 35% (p < 0.001). Stratifying by subspecialty, the smallest declines were seen in non-trauma thoracic imaging, which decreased 18% (p value < 0.001), while all other non-trauma studies decreased by 48% (p < 0.001).ConclusionTotal ED radiology demand may be a marker for public adherence to shelter-in-place mandates, though ED chest radiology demand may increase with an increase in COVID-19 cases.

[This corrects the article DOI: 10.1148/ryai.2020190211.].

This dataset is composed of annotations of the five hemorrhage subtypes (subarachnoid, intraventricular, subdural, epidural, and intraparenchymal hemorrhage) typically encountered at brain CT.

BackgroundPneumothorax can precipitate a life-threatening emergency due to lung collapse and respiratory or circulatory distress. Pneumothorax is typically detected on chest X-ray; however, treatment is reliant on timely review of radiographs. Since current imaging volumes may result in long worklists of radiographs awaiting review, an automated method of prioritizing X-rays with pneumothorax may reduce time to treatment. Our objective was to create a large human-annotated dataset of chest X-rays containing pneumothorax and to train deep convolutional networks to screen for potentially emergent moderate or large pneumothorax at the time of image acquisition.Methods and findingsIn all, 13,292 frontal chest X-rays (3,107 with pneumothorax) were visually annotated by radiologists. This dataset was used to train and evaluate multiple network architectures. Images showing large- or moderate-sized pneumothorax were considered positive, and those with trace or no pneumothorax were considered negative. Images showing small pneumothorax were excluded from training. Using an internal validation set (n = 1,993), we selected the 2 top-performing models; these models were then evaluated on a held-out internal test set based on area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and positive predictive value (PPV). The final internal test was performed initially on a subset with small pneumothorax excluded (as in training; n = 1,701), then on the full test set (n = 1,990), with small pneumothorax included as positive. External evaluation was performed using the National Institutes of Health (NIH) ChestX-ray14 set, a public dataset labeled for chest pathology based on text reports. All images labeled with pneumothorax were considered positive, because the NIH set does not classify pneumothorax by size. In internal testing, our "high sensitivity model" produced a sensitivity of 0.84 (95% CI 0.78-0.90), specificity of 0.90 (95% CI 0.89-0.92), and AUC of 0.94 for the test subset with small pneumothorax excluded. Our "high specificity model" showed sensitivity of 0.80 (95% CI 0.72-0.86), specificity of 0.97 (95% CI 0.96-0.98), and AUC of 0.96 for this set. PPVs were 0.45 (95% CI 0.39-0.51) and 0.71 (95% CI 0.63-0.77), respectively. Internal testing on the full set showed expected decreased performance (sensitivity 0.55, specificity 0.90, and AUC 0.82 for high sensitivity model and sensitivity 0.45, specificity 0.97, and AUC 0.86 for high specificity model). External testing using the NIH dataset showed some further performance decline (sensitivity 0.28-0.49, specificity 0.85-0.97, and AUC 0.75 for both). Due to labeling differences between internal and external datasets, these findings represent a preliminary step towards external validation.ConclusionsWe trained automated classifiers to detect moderate and large pneumothorax in frontal chest X-rays at high levels of performance on held-out test data. These models may provide a high specificity screening solution to detect moderate or large pneumothorax on images collected when human review might be delayed, such as overnight. They are not intended for unsupervised diagnosis of all pneumothoraces, as many small pneumothoraces (and some larger ones) are not detected by the algorithm. Implementation studies are warranted to develop appropriate, effective clinician alerts for the potentially critical finding of pneumothorax, and to assess their impact on reducing time to treatment.

The purpose of this study was to objectively quantify the impact of implementing picture archiving and communication system-electronic medical record (PACS-EMR) integration on the time required to access data in the EMR and the frequency with which data are accessed by radiologists. Time to access a clinic note in the EMR was measured before and after integration with a stopwatch and compared by t test. An IRB-approved, HIPAA-compliant retrospective review of EMR access data from security audit logs was conducted for a 14-month period spanning the integration. Correlation of these data with report signatures identified the studies in which the radiologist accessed the EMR to obtain additional clinical data. Proportions of studies with EMR access were plotted and compared before and after integration using a chi-square test. Time to access the EMR decreased from 52 to 6 s (p

PurposeWe compared contrast enhanced ultrasound and fluoroscopic nephrostography in the evaluation of ureteral patency following percutaneous nephrolithotomy.Materials and methodsThis prospective cohort, noninferiority study was performed after obtaining institutional review board approval. We enrolled eligible patients with kidney and proximal ureteral stones who underwent percutaneous nephrolithotomy at our center. On postoperative day 1 patients received contrast enhanced ultrasound and fluoroscopic nephrostogram within 2 hours of each other to evaluate ureteral patency, which was the primary outcome of this study.ResultsA total of 92 pairs of imaging studies were performed in 82 patients during the study period. Five study pairs were excluded due to technical errors that prevented imaging interpretation. Females slightly predominated over males with a mean ± SD age of 50.5 ± 15.9 years and a mean body mass index of 29.6 ± 8.6 kg/m2. Of the remaining 87 sets of studies 69 (79.3%) demonstrated concordant findings regarding ureteral patency for the 2 imaging techniques and 18 (20.7%) were discordant. The nephrostomy tube was removed on the same day in 15 of the 17 patients who demonstrated antegrade urine flow only on contrast enhanced ultrasound and they had no subsequent adverse events. No adverse events were noted related to ultrasound contrast injection. While contrast enhanced ultrasound used no ionizing radiation, fluoroscopic nephrostograms provided a mean radiation exposure dose of 2.8 ± 3.7 mGy.ConclusionsA contrast enhanced ultrasound nephrostogram can be safely performed to evaluate for ureteral patency following percutaneous nephrolithotomy. This imaging technique was mostly concordant with fluoroscopic findings. Most discordance was likely attributable to the higher sensitivity for patency of contrast enhanced ultrasound compared to fluoroscopy.

Purpose To demonstrate the feasibility of contrast material-enhanced ulrasonographic (US) nephrostograms to assess ureteral patency after percutaneous nephrolithotomy (PCNL) in this proof-of-concept study. Materials and Methods For this HIPAA-compliant, institutional review board-approved prospective blinded pilot study, patients undergoing PCNL provided consent to undergo contrast-enhanced US and fluoroscopic nephrostograms on postoperative day 1. For contrast-enhanced US, 1.5 mL of Optison (GE Healthcare, Oslo, Norway) microbubble contrast agent solution (perflutren protein-type A microspheres) was injected via the nephrostomy tube. Unobstructed antegrade ureteral flow was defined by the presence of contrast material in the bladder. Contrast-enhanced US results were compared against those of fluoroscopic nephrostograms for concordance. Results Ten studies were performed in nine patients (four women, five men). Contrast-enhanced US demonstrated ureteral patency in eight studies and obstruction in two. One patient underwent two studies, one showing obstruction and the second showing patency. Concordance between US and fluoroscopic assessments of ureteral patency was evaluated by using a Clopper-Pearson exact binomial test. These results were perfectly concordant with fluoroscopic nephrostogram results, with a 95% confidence interval of 69.2% and 100%. No complications or adverse events related to contrast-enhanced US occurred. Conclusion Contrast-enhanced US nephrostograms are simple to perform and are capable of demonstrating both patency and obstruction of the ureter. The perfect concordance with fluoroscopic results across 10 studies demonstrated here is not sufficient to establish diagnostic accuracy of this technique, but motivates further, larger scale investigation. If subsequent larger studies confirm these preliminary results, contrast-enhanced US may provide a safer, more convenient way to evaluate ureteral patency than fluoroscopy. © RSNA, 2016 Online supplemental material is available for this article.

ObjectivesUltrasound-guided percutaneous nephrolithotomy (PCNL) has become increasingly utilized. Patients with nondilated collecting systems represent a challenge: the target calix is often difficult to visualize. Here we report pilot study results for retrograde ultrasound contrast injection to aid in percutaneous renal access during ultrasound-guided PCNL.Patients and methodsFrom April to July 2016, consecutive patients over the age of 18 years with nondilated collecting systems on preoperative imaging who presented for PCNL were enrolled. B-mode ultrasound imaging was compared with contrast-enhanced mode with simultaneous retrograde injection of Optison™ via an ipsilateral ureteral catheter.ResultsFive patients (four males and one female) with renal stones underwent PCNL with retrograde ultrasound contrast injection during the study period. Mean body mass index was 28.3 ± 5.6 kg/m2 and mean stone size was 24.5 ± 12.0 mm. Under B-mode ultrasound, all patients demonstrated nondilated renal collecting systems that appeared as hyperechoic areas, where it was difficult to identify a target calix for puncture. Retrograde contrast injection facilitated delineation of all renal calices initially difficult to visualize under B-mode ultrasound. Renal puncture was then performed effectively in all cases with a mean puncture time of 55.4 ± 44.8 seconds. All PCNL procedures were completed without intraoperative complications and no adverse events related to ultrasound contrast injection occurred.ConclusionRetrograde ultrasound contrast injection as an aide for renal puncture during PCNL is a feasible technique. By improving visualization of the collecting system, it facilitates needle placement in challenging patients without hydronephrosis. Future larger scale studies comparing its use to standard ultrasound-guided technique will be required to validate this concept.

The study aimed to determine if computer vision techniques rooted in deep learning can use a small set of radiographs to perform clinically relevant image classification with high fidelity. One thousand eight hundred eighty-five chest radiographs on 909 patients obtained between January 2013 and July 2015 at our institution were retrieved and anonymized. The source images were manually annotated as frontal or lateral and randomly divided into training, validation, and test sets. Training and validation sets were augmented to over 150,000 images using standard image manipulations. We then pre-trained a series of deep convolutional networks based on the open-source GoogLeNet with various transformations of the open-source ImageNet (non-radiology) images. These trained networks were then fine-tuned using the original and augmented radiology images. The model with highest validation accuracy was applied to our institutional test set and a publicly available set. Accuracy was assessed by using the Youden Index to set a binary cutoff for frontal or lateral classification. This retrospective study was IRB approved prior to initiation. A network pre-trained on 1.2 million greyscale ImageNet images and fine-tuned on augmented radiographs was chosen. The binary classification method correctly classified 100 % (95 % CI 99.73-100 %) of both our test set and the publicly available images. Classification was rapid, at 38 images per second. A deep convolutional neural network created using non-radiological images, and an augmented set of radiographs is effective in highly accurate classification of chest radiograph view type and is a feasible, rapid method for high-throughput annotation.

To assess the ability of dual-energy CT (DECT) to separate intravenous contrast of bowel wall from intraluminal contrast, we scanned 16 rabbits on a clinical DECT scanner: n = 3 using only iodinated intravenous contrast, and n = 13 double-contrast enhanced scans using iodinated intravenous contrast and experimental enteric non-iodinated contrast agents in the bowel lumen (five bismuth, four tungsten, and four tantalum based). Representative image pairs from conventional CT images and DECT iodine density maps of small bowel (116 pairs from 232 images) were viewed by four abdominal imaging attending radiologists to independently score each comparison pair on a visual analog scale (-100 to +100%) for (1) preference in small bowel wall visualization and (2) preference in completeness of intraluminal enteric contrast subtraction. Median small bowel wall visualization was scored 39 and 42 percentage points (95% CI 30-44% and 36-45%, both p