Search

Abstract Healthcare datasets are becoming larger and more complex, necessitating the development of accurate and generalizable AI models for medical applications. Unstructured datasets, including medical imaging, electrocardiograms, and natural language data, are gaining attention with advancements in deep convolutional neural networks and large language models. However, estimating the generalizability of these models to new healthcare settings without extensive validation on external data remains challenging. In experiments across 13 datasets including X-rays, CTs, ECGs, clinical discharge summaries, and lung auscultation data, our results demonstrate that model performance is frequently overestimated by up to 20% on average due to shortcut learning of hidden data acquisition biases (DAB). Shortcut learning refers to a phenomenon in which an AI model learns to solve a task based on spurious correlations present in the data as opposed to features directly related to the task itself. We propose an open source, bias-corrected external accuracy estimate, P E s t , that better estimates external accuracy to within 4% on average by measuring and calibrating for DAB-induced shortcut learning.

Background and purpose: No best practices currently exist for achieving high quality radiation therapy (RT) treatment plan adaptation during magnetic resonance (MR) guided RT of prostate cancer. This study validates the use of machine learning (ML) automated RT treatment plan adaptation and benchmarks it against current clinical RT plan adaptation methods. Materials and methods: We trained an atlas-based ML automated treatment planning model using reference MR RT treatment plans (42.7 Gy in 7 fractions) from 46 patients with prostate cancer previously treated at our institution. For a held-out test set of 38 patients, retrospectively generated ML RT plans were compared to clinical human-generated adaptive RT plans for all 266 fractions. Differences in dose-volume metrics and clinical objective pass rates were evaluated using Wilcoxon tests (p