Search

This study explores the concept of similarity in machine learning (ML) and its congruence with human judgment in medical contexts, focusing primarily on radiology. We conducted a user study involving two radiologists and two orthopedic and spine surgeons. These experts evaluated the similarity of 72 cases, selected from a larger dataset by an ML model based on Cosine and Euclidean distances, in comparison to 18 representative base cases of vertebral fractures. Our analysis focused on correlating these ML-derived distances with the experts’ assessments. The findings reveal that: (1) both Cosine and Euclidean distances had limited correlation with human judgments; (2) Cosine distances showed a marginally higher correlation than Euclidean distances; despite the limitations due to the small samples of evaluations and evaluators, our findings emphasize the necessity for ongoing research to enhance AI similarity metrics, aiming for greater human-centricity and relevance, particularly considering their critical role in ML training and inference. Our study’s implications are far-reaching, advocating for a comprehensive reevaluation of similarity assessments in AI to achieve a closer alignment with human cognitive processes, extending well beyond the realm of medical imaging.

Objectives: To assess the reliability and diagnostic accuracy of new radiographic definitions for calcium pyrophosphate deposition (CPPD) identification, developed by an international multidisciplinary working group. Methods: Patients with knee osteoarthritis scheduled for knee replacement were enrolled. Two radiologists and two rheumatologists assessed twice the images for presence/absence of CPPD on menisci, hyaline cartilage, tendons, joint capsule, synovial membrane, using the new definitions. In case of disagreement, a consensus decision was taken and considered for the assessment of diagnostic performance. Histological examination of specimens under compensated polarized light microscopy was the reference standard. Prevalence-adjusted bias-adjusted kappa (PABAK) was used to assess the reliability. Diagnostic performance statistics were calculated. Results: Sixty-seven participants were enrolled for the reliability study. The inter-observer reliability was substantial in most of the assessed structures when considering all 4 readers (kappa range 0.59 – 0.90), substantial to almost perfect among radiologists (kappa range 0.70-0.91), and moderate to almost perfect among rheumatologists (kappa range 0.46 – 0.88). The intra-observer reliability was substantial to almost perfect for all the observers (kappa range 0.70 – 1). Fifty-one patients were enrolled for the accuracy study. Radiography demonstrated to be specific for CPPD (92%), but sensitivity remained low in all sites and in the overall diagnosis (54%). Conclusion: The new imaging definitions of CPPD are highly specific against the gold standard of histological diagnosis; when described findings are present these definitions allow for a definite diagnosis of CPPD, rather than other calcium-containing crystal depositions; instead a negative finding does not exclude the diagnosis.

In this paper, we study human–AI collaboration protocols, a design-oriented construct aimed at establishing and evaluating how humans and AI can collaborate in cognitive tasks. We applied this construct in two user studies involving 12 specialist radiologists (the knee MRI study) and 44 ECG readers of varying expertise (the ECG study), who evaluated 240 and 20 cases, respectively, in different collaboration configurations. We confirm the utility of AI support but find that XAI can be associated with a “white-box paradox”, producing a null or detrimental effect. We also find that the order of presentation matters: AI-first protocols are associated with higher diagnostic accuracy than human-first protocols, and with higher accuracy than both humans and AI alone. Our findings identify the best conditions for AI to augment human diagnostic skills, rather than trigger dysfunctional responses and cognitive biases that can undermine decision effectiveness.

We aimed to demonstrate the applicability of Snyder’s arthroscopic classification of rotator cuff tears (RCT) in shoulder ultrasound (US) and to compare it with MR arthrography (MRA). Forty-six patients (34 males; mean age:34 ± 14 years) underwent shoulder US and MRA. Two radiologists (R1 = 25 years of experience; R2 = 2 years of experience) assigned A1–4, B1–4, or C1–4 values depending on the extent of RCT in both US and MRA. Inter-reader intra-modality and intra-reader inter-modality agreement were calculated using Cohen’s kappa coefficient. US sensitivity and specificity of both readers were calculated using MRA as the gold standard. Patients were divided into intact cuff vs. tears, mild (A1/B1) vs. moderate (A2–3/B2–3) tears, mild-moderate (A2/B2) vs. high-moderate (A3/B3) cuff tears, moderate (A2–3/B2–3) vs. advanced (A4/B4) and full-thickness (C) tears. The highest agreement values in inter-reader US evaluation were observed for mild-moderate vs. high-moderate RCT (K = 0.745), in inter-reader MRA evaluation for mild vs. moderate RCT (K = 0.821), in R1 inter-modality (US-MRA) for mild-moderate vs. high-moderate and moderate vs. advanced/full-thickness RCT (K = 1.000), in R2 inter-modality (US-MRA) for moderate vs. advanced/full-thickness RCT (K = 1.000). US sensitivity ranged from 88.89%(R1)–84.62%(R2) to 100% (both readers), while specificity from 77.78%(R1)–90.00%(R2) to 100% (both readers). Snyder’s classification can be used in US to ensure the correct detection and characterization of RCT.

Objectives To develop evidence-based Points to Consider (PtC) for the use of imaging modalities to guide interventional procedures in patients with rheumatic and musculoskeletal diseases (RMDs). Methods European Alliance of Associations for Rheumatology (EULAR) standardised operating procedures were followed. A systematic literature review was conducted to retrieve data on the role of imaging modalities including ultrasound (US), fluoroscopy, MRI, CT and fusion imaging to guide interventional procedures. Based on evidence and expert opinion, the task force (25 participants consisting of physicians, healthcare professionals and patients from 11 countries) developed PtC, with consensus obtained through voting. The final level of agreement was provided anonymously. Results A total of three overarching principles and six specific PtC were formulated. The task force recommends preference of imaging over palpation to guide targeted interventional procedures at peripheral joints, periarticular musculoskeletal structures, nerves and the spine. While US is the favoured imaging technique for peripheral joints and nerves, the choice of the imaging method for the spine and sacroiliac joints has to be individualised according to the target, procedure, expertise, availability and radiation exposure. All imaging guided interventions should be performed by a trained specialist using appropriate operational procedures, settings and assistance by technical personnel. Conclusion These are the first EULAR PtC to provide guidance on the role of imaging to guide interventional procedures in patients with RMDs.

We aimed to evaluate the diagnostic performance of shoulder MR arthrography (MRA) acquired in the neutral (N), internal rotation (IR), and external rotation (ER) positions of the shoulder to detect SLAP lesions. Three observers evaluated 130 MRAs to detect SLAP lesions and to calculate labral diastasis in this triple-blinded study. Sensitivity was much higher in the ER (92.5–97.5%) than in the N (60–72.5%) and IR (42.5–52.5%) positions, and the specificity of all the reviewers was 100% in all the positions. The diagnostic accuracy was higher in the ER too (97.7–99.2%). The diastasis length was significantly higher in the ER (median = 2.5–2.8 mm) than in the N (1 mm) and IR (0 mm) positions and was also significantly higher in those patients requiring surgery (p = 0.001). The highest inter-rater agreement values were observed in the ER both in SLAP detection (k = 0.982) and the diastasis length evaluation (ICC = 0.962). The diastasis length threshold in the ER that best separated the patients who did and did not require surgery was 3.1 mm (AUC = 0.833). In 14.6% of the cases, ER enabled the detection of SLAP lesions not identified in the N position. MRA with the ER improves the diagnosis of SLAP lesions and, together with the IR position, provides additional dynamic information about the diastasis of the lesions. It is recommended to perform additional ER and IR scans in the shoulder MRA protocol.

In this paper, we present and discuss a novel reliability metric to quantify the extent a ground truth, generated in multi-rater settings, as a reliable basis for the training and validation of machine learning predictive models. To define this metric, three dimensions are taken into account: agreement (that is, how much a group of raters mutually agree on a single case); confidence (that is, how much a rater is certain of each rating expressed); and competence (that is, how accurate a rater is). Therefore, this metric produces a reliability score weighted for the raters' confidence and competence, but it only requires the former information to be actually collected, as the latter can be obtained by the ratings themselves, if no further information is available. We found that our proposal was both more conservative and robust to known paradoxes than other existing agreement measures, by virtue of a more articulated notion of the agreement due to chance, which was based on an empirical estimation of the reliability of the single raters involved. We discuss the above metric within a realistic annotation task that involved 13 expert radiologists in labeling the MRNet dataset. We also provide a nomogram by which to assess the actual accuracy of a classification model, given the reliability of its ground truth. In this respect, we also make the point that theoretical estimates of model performance are consistently overestimated if ground truth reliability is not properly taken into account.

Purpose: The integration of Artificial Intelligence into medical practices has recently been advocated for the promise to bring increased efficiency and effectiveness to these practices. Nonetheless, little research has so far been aimed at understanding the best human-AI interaction protocols in collaborative tasks, even in currently more viable settings, like independent double-reading screening tasks. Methods: To this aim, we report about a retrospective case–control study, involving 12 board-certified radiologists, in the detection of knee lesions by means of Magnetic Resonance Imaging, in which we simulated the serial combination of two Deep Learning models with humans in eight double-reading protocols. Inspired by the so-called Kasparov’s Laws, we investigate whether the combination of humans and AI models could achieve better performance than AI models alone, and whether weak reader, when supported by fit-for-use interaction protocols, could out-perform stronger readers. Results: We discuss two main findings: groups of humans who perform significantly worse than a state-of-the-art AI can significantly outperform it if their judgements are aggregated by majority voting (in concordance with the first part of the Kasparov’s law); small ensembles of significantly weaker readers can significantly outperform teams of stronger readers, supported by the same computational tool, when the judgments of the former ones are combined within “fit-for-use” protocols (in concordance with the second part of the Kasparov’s law). Conclusion: Our study shows that good interaction protocols can guarantee improved decision performance that easily surpasses the performance of individual agents, even of realistic super-human AI systems. This finding highlights the importance of focusing on how to guarantee better co-operation within human-AI teams, so to enable safer and more human sustainable care practices.

Gait control becomes more demanding in healthy older adults, yet what cognitive or motor process leads to this age-related change is unknown. The present study aimed to investigate whether it might depend on specific decay in the quality of gait motor representation and/or a more general reduction in the efficiency of lower limb motor control. Younger and older healthy participants performed in fMRI a virtual walking paradigm that combines motor imagery (MI) of walking and standing on the spot with the presence (Dynamic Motor Imagery condition, DMI) or absence (pure MI condition) of overtly executed ankle dorsiflexion. Gait imagery was aided by the concomitant observation of moving videos simulating a stroll in the park from a first-person perspective. Behaviorally, older participants showed no sign of evident depletion in the quality of gait motor representations, and absence of between-group differences in the neural correlates of MI. However, while younger participants showed increased frontoparietal activity during DMI, older participants displayed stronger activation of premotor areas when controlling the pure execution of ankle dorsiflexion, regardless of the imagery task. These data suggest that reduced automaticity of lower limb motor control in healthy older subjects leads to the recruitment of additional premotor resources even in the absence of basic gait functional disabilities.

Only few studies have reported muscle involvement in spinal muscular atrophy using muscle MRI but this has not been systematically investigated in a large cohort of both pediatric and adult patients with type 2 and type 3 spinal muscular atrophy. The aim of the present study was to define possible patterns of muscle involvement on MRI, assessing both fatty replacement and muscle atrophy, in a cohort of type 2 and type 3 spinal muscular atrophy children and adults (age range 2–45 years), including both ambulant and non-ambulant patients. Muscle MRI protocol consisted in T1-weighted sequences acquired on axial plane covering the pelvis, the thigh, and the leg with contiguous slices. Each muscle was examined through its whole extension using a grading system that allows a semiquantitative evaluation of fatty infiltration. Thigh muscles were also grouped in anterior, posterior, and medial compartment for classification of global atrophy. The results showed a large variability in both type 2 and type 3 spinal muscular atrophy, with a various degree of proximal to distal gradient. Some muscles, such us the adductor longus and gracilis were always selectively spared. In all patients, the involvement was a combination of muscle atrophy and muscle infiltration. The variability observed may help to better understand both natural history and response to new treatments.

Purpose: To compare contrast material enhancement of glioblastoma multiforme (GBM) with intraoperative contrast-enhanced ultrasonography (US) versus that with preoperative gadolinium- enhanced T1-weighted magnetic resonance (MR) imaging by using real-time fusion imaging. Materials and Methods: Ten patients with GBM were retrospectively identified by using routinely collected, anonymized data. Navigated contrastenhanced US was performed after intravenous administration of contrast material before tumor resection. All patients underwent tumor excision with navigated intraoperative US guidance with use of fusion imaging between real-time intraoperative US and preoperative MR imaging. With use of fusion imaging, glioblastoma contrast enhancement at contrast-enhanced US (regarding location, morphologic features, margins, dimensions, and pattern) was compared with that at gadolinium-enhanced T1-weighted MR imaging. Results: Fusion imaging for virtual navigation enabled matching of realtime contrast-enhanced US scans to corresponding coplanar preoperative gadolinium-enhanced T1-weighted MR images in all cases, with a positional discrepancy of less than 2 mm. Contrast enhancement of gadolinium-enhanced T1-weighted MR imaging and contrast-enhanced US was superimposable in all cases with regard to location, margins, dimensions, and morphologic features. The qualitative analysis of contrast enhancement pattern demonstrated a similar distribution in contrast-enhanced US and gadolinium-enhanced T1-weighted MR imaging in nine patients: Seven lesions showed peripheral inhomogeneous ring enhancement, and two lesions showed a prevalent nodular pattern. In one patient, the contrast enhancement pattern differed between the two modalities: Contrast- enhanced US showed enhancement of the entire bulk of the tumor, whereas gadolinium-enhanced T1-weighted MR imaging demonstrated peripheral contrast enhancement. Conclusion: Glioblastoma contrast enhancement with contrast-enhanced US is supe

Objective: Ultrasonography (US) demonstrated to be a promising tool for the diagnosis of calcium pyrophosphate dihydrate deposition disease (CPPD). The aim of this systematic literature review (SLR) was to collect the definitions for the US elementary lesions and to summarize the available data about US diagnostic accuracy in CPPD. Methods: We systematically reviewed all the studies that considered US as the index test for CPPD diagnosis without restrictions about the reference test or that provided definitions about US identification of CPPD. Sensitivity and specificity were calculated for each study and definitions were extrapolated. Subgroup analyses were planned by anatomical site included in the index text and different reference standards. Results: Thirty-seven studies were included in this review. All the studies were eligible for the collection of US findings and all definitions were summarized. US description of elementary lesions appeared heterogeneous among the studies. Regarding US accuracy, 13 articles entered in the meta-analysis. Considering each joint structure, the sensitivity ranged between 0.77 (0.63-0.87) and 0.34 (0.16-0.58) while the specificity varies between 1.00 (0.89-1.00) and 0.92 (0.16-1.00). Considering the reference standards used, the sensibility ranged between 0.34 (0.02-0.65) and 0.87 (0.76-0.99) while specificity ranged between 0.84 (0.52-1.00) and 1.00 (0.99-1.00). Conclusion: US is potentially a useful tool for the diagnosis of CPPD but universally accepted definitions and further testing are necessary in order to assess the role of the technique in the diagnostic process.

Objective To assess the diagnostic performance of ultrasound (US), x-rays, and microscopic analysis of synovial fluid (SF) for calcium pyrophosphate dihydrate crystal deposition disease (CPPD) using histology as a reference standard Methods We enrolled consecutive patients with osteoarthritis waiting to undergo knee replacement surgery. Each patient underwent US of the knee, focusing on menisci and the hyaline cartilage, the day before surgery. During surgery, SF, menisci and condyles were retrieved and examined microscopically. For the meniscus and cartilage microscopic analysis, 8 samples were collected from each specimen and knee radiographs, performed up to 3 months before surgery, were also assessed. A dichotomous score was given for the presence/absence of CPP for each method. Microscopic findings of the specimens were considered the reference standard. All the procedures followed were in accordance with the ethical standards of the local responsible committee. Results 42 patients (14 males) were enrolled. All patients underwent US, 34 had eligible radiographs and 32 had SF analysis. 25 patients (59.5%) were positive for CPP at US, 15 (44.1%) at X-ray and 14 (43.7%) at SF. Sensitivity and specificity values were 96% and 87% for US, 75% and 93% for radiography and 77% and 100% for SF respectively. There were no statistically significant differences between the diagnostic performance across single tests. Conclusion US proved to be at least as accurate as SF analysis for the diagnosis of CPPD. US, which is feasible and harmless, could be considered the first exam of choice for CPPD diagnosis.