Search

Purpose: Controversy continues regarding the effect of screening mammography on breast cancer outcomes. We evaluated late-stage cancer rate and overall survival (OS) for different screening intervals using a real-world institutional research data mart., Methods: Patients having both a cancer registry record of new breast cancer diagnosis and prediagnosis screening history between 2004 and 2019 were identified from our institutional research breast data mart. Time interval between the two screening mammograms immediately preceding diagnosis and the time to cancer diagnosis were determined. Screening interval was deemed annual if ≤15 months, biennial if >15 and ≤27 months, intermittent if >27 months, and baseline if only one prediagnosis screen was known. The primary end point was late-stage cancer (TNM stage IIB or worse), and the secondary end point was OS. The association of screening interval and late-stage cancer was analyzed using multivariable logistic regression adjusting for prediagnosis characteristics. Proportional hazards regression was used for survival analysis. Potential lead time was analyzed using survival from a uniform fixed time point., Results: In total, 8,145 patients with breast cancer had prediagnosis screening mammography in the timeframe. The percentage of late-stage cancers diagnosed increased significantly with screening interval with 9%, 14%, and 19% late stages for annual, biennial, and intermittent groups ( P < .001), respectively. The trend persisted regardless of age, race, and menopausal status. Biennial and intermittent groups had substantially worse OS than the annual screened group, with relative hazards of 1.42 (95% CI, 1.11 to 1.82) and 2.69 (95% CI, 2.11 to 3.43), respectively, and 1.39 (95% CI, 1.08 to 1.78) and 2.01 (95% CI, 1.58 to 2.55) after adjustment for potential lead time., Conclusion: Annual mammographic screening was associated with lower risk of late-stage cancer and better OS across clinical and demographic subgroups. Our study suggests benefit of annual screening for women 40 years and older.

Background Studies suggest that readers experience perceptual adaptation when interpreting batched screening mammograms, which may serve as a mechanism for improved performance. Purpose To analyze clinical digital breast tomosynthesis (DBT) screening data to evaluate changes in reader performance during sequential batch reading. Materials and Methods This observational retrospective study used data from the radiology information system collected for screening DBT examinations performed from January 2018 to December 2019. The reference standard was established based on pathology results, if applicable, or imaging findings at 1-year follow-up. Examinations were aggregated into batches, defined as a sequence of cases for a given reader with differences in interexamination time of 10 minutes or less. Mixed-effect models were used to evaluate performance and timing across batch positions. Statistical adjustments accounted for potential confounders, including patient characteristics (age, breast density), reading factors (day of week, time of day), and between-reader heterogeneity. Results The dataset included 121 652 examinations (median patient age, 61 years [IQR, 53-69 years]), including 1081 cancers interpreted by 15 radiologists. Unadjusted false-positive rates decreased from an averaged 15.5% at the first within-batch examination to 10.5% after three sequentially read examinations ( P < .001), without a significant change in sensitivity (82.6% vs 84.2%; P = .15). The interpretation time consistently decreased, achieving a substantial reduction in longer batches (average 2.8 to 2.2 minutes for noncancer examinations; P < .001). Adjustment for sampling bias and confounders (patient age, breast density, day of week, time of day; all P < .005) reduced the effect on false-positive rates (11.5% to 9.4% after three examinations; P < .001) and interpretation time (3.2 to 2.7 minutes for noncancer examinations; P < .001). Conclusion Radiologists showed improved recall rates and interpretation times without change in sensitivity over the course of batch reading of DBT screening examinations. © RSNA, 2024 See also the editorial by Iima and Satake in this issue.

Background Digital breast tomosynthesis (DBT) is often inadequate for screening women with a personal history of breast cancer (PHBC). The ongoing prospective Tomosynthesis or Contrast-Enhanced Mammography, or TOCEM, trial includes three annual screenings with both DBT and contrast-enhanced mammography (CEM). Purpose To perform interim assessment of cancer yield, stage, and recall rate when CEM is added to DBT in women with PHBC. Materials and Methods From October 2019 to December 2022, two radiologists interpreted both examinations: Observer 1 reviewed DBT first and then CEM, and observer 2 reviewed CEM first and then DBT. Effects of adding CEM to DBT on incremental cancer detection rate (ICDR), cancer type and node status, recall rate, and other performance characteristics of the primary radiologist decisions were assessed. Results Among the participants (mean age at entry, 63.6 years ± 9.6 [SD]), 1273, 819, and 227 women with PHBC completed year 1, 2, and 3 screening, respectively. For observer 1, year 1 cancer yield was 20 of 1273 (15.7 per 1000 screenings) for DBT and 29 of 1273 (22.8 per 1000 screenings; ICDR, 7.1 per 1000 screenings [95% CI: 3.2, 13.4]) for DBT plus CEM ( P < .001). Year 2 plus 3 cancer yield was four of 1046 (3.8 per 1000 screenings) for DBT and eight of 1046 (7.6 per 1000 screenings; ICDR, 3.8 per 1000 screenings [95% CI: 1.0, 7.6]) for DBT plus CEM ( P = .001). Year 1 recall rate for observer 1 was 103 of 1273 (8.1%) for (incidence) DBT alone and 187 of 1273 (14.7%) for DBT plus CEM (difference = 84 of 1273, 6.6% [95% CI: 5.3, 8.1]; P < .001). Year 2 plus 3 recall rate was 40 of 1046 (3.8%) for DBT and 92 of 1046 (8.8%) for DBT plus CEM (difference = 52 of 1046, 5.0% [95% CI: 3.7, 6.3]; P < .001). In 18 breasts with cancer detected only at CEM after integration of both observers, 13 (72%) cancers were invasive (median tumor size, 0.6 cm) and eight of nine (88%) with staging were N0. Among 1883 screenings with adequate reference standard, there were three interval cancers (one at the scar, two in axillae). Conclusion CEM added to DBT increased early breast cancer detection each year in women with PHBC, with an accompanying approximately 5.0%-6.6% recall rate increase. Clinical trial registration no. NCT04085510 © RSNA, 2024 Supplemental material is available for this article.

Purpose: In the United States, a comprehensive national breast cancer registry (CR) does not exist. Thus, care and coverage decisions are based on data from population subsets, other countries, or models. We report a prototype real-world research data mart to assess mortality, morbidity, and costs for breast cancer diagnosis and treatment., Methods: With institutional review board approval and Health Insurance Portability and Accountability Act (HIPPA) compliance, a multidisciplinary clinical and research data warehouse (RDW) expert group curated demographic, risk, imaging, pathology, treatment, and outcome data from the electronic health records (EHR), radiology (RIS), and CR for patients having breast imaging and/or a diagnosis of breast cancer in our institution from January 1, 2004, to December 31, 2020. Domains were defined by prebuilt views to extract data denormalized according to requirements from the existing RDW using an export, transform, load pattern. Data dictionaries were included. Structured query language was used for data cleaning., Results: Five-hundred eighty-nine elements (EHR 311, RIS 211, and CR 67) were mapped to 27 domains; all, except one containing CR elements, had cancer and noncancer cohort views, resulting in a total of 53 views (average 12 elements/view; range, 4-67). EHR and RIS queries returned 497,218 patients with 2,967,364 imaging examinations and associated visit details. Cancer biology, treatment, and outcome details for 15,619 breast cancer cases were imported from the CR of our primary breast care facility for this prototype mart., Conclusion: Institutional real-world data marts enable comprehensive understanding of care outcomes within an organization. As clinical data sources become increasingly structured, such marts may be an important source for future interinstitution analysis and potentially an opportunity to create robust real-world results that could be used to support evidence-based national policy and care decisions for breast cancer.

Sensory systems continuously recalibrate their responses according to the current stimulus environment. As a result, perception is strongly affected by the current and recent context. These adaptative changes affect both sensitivity (e.g., habituating to noise, seeing better in the dark) and appearance (e.g. how things look, what catches attention) and adjust to many perceptual properties (e.g. from light level to the characteristics of someone's face). They therefore have a profound effect on most perceptual experiences, and on how and how well the senses work in different settings. Characterizing the properties of adaptation, how it manifests, and when it influences perception in modern environments can provide insights into the diversity of human experience . Adaptation could also be leveraged both to optimize perceptual abilities (e.g. in visual inspection tasks like radiology) and to mitigate unwanted consequences (e.g. exposure to potentially unhealthy stimulus environments).

Objective: To guide implementation of supplemental breast screening by assessing patient preferences for contrast-enhanced mammography (CEM) versus MRI using analytic hierarchy process (AHP) methodology., Methods: In an institutional review board-approved, HIPAA-compliant protocol, from March 23 to June 3, 2022, we contacted 579 women who had both CEM screening and MRI. Women were e-mailed an invitation to complete an online survey developed using an AHP-based model to elicit preferences for CEM or MRI. Methods for categorical data analysis were used to evaluate factors affecting preferences, under the Bonferroni correction for multiplicity., Results: Complete responses were received from 222 (38.3%) women; the 189 women with a personal history of breast cancer had a mean age 61.8 years, and the 34 women without a personal history of breast cancer had a mean age of 53.6 years. Of 222 respondents, 157 (70.7%, confidence interval [CI]: 64.7-76.7) were determined to prefer CEM to MRI. Breast positioning was the most important criterion for 74 of 222 (33.3%) respondents, with claustrophobia, intravenous line placement, and overall stress most important for 38, 37, and 39 women (17.1%, 16.7%, and 17.6%), respectively, and noise level, contrast injection, and indifference being emphasized least frequently (by 10 [4.5%], 11 [5.0%], and 13 [5.9%] women, respectively). CEM preference was most prevalent (MRI least prevalent) for respondents emphasizing claustrophobia (37 of 38 [97%], CI: 86.2-99.9); CEM preference was least prevalent (MRI most prevalent) for respondents emphasizing breast positioning (40 of 74 [54%], CI: 42.1-65.7)., Conclusions: AHP-based modeling reveals strong patient preferences for CEM over MRI, with claustrophobia favoring preference for CEM and breast positioning relatively favoring preference for MRI. Our results should help guide implementation of screening CEM and MRI., (Copyright © 2023 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

Purpose: To assess diagnostic performance of digital breast tomosynthesis (DBT) alone or combined with technologist-performed handheld screening ultrasound (US) in women with dense breasts., Methods: In an institutional review board-approved, Health Insurance Portability and Accountability Act-compliant multicenter protocol in western Pennsylvania, 6,179 women consented to three rounds of annual screening, interpreted by two radiologist observers, and had appropriate follow-up. Primary analysis was based on first observer results., Results: Mean participant age was 54.8 years (range, 40-75 years). Across 17,552 screens, there were 126 cancer events in 125 women (7.2/1,000; 95% CI, 5.9 to 8.4). In year 1, DBT-alone cancer yield was 5.0/1,000, and of DBT+US, 6.3/1,000, difference 1.3/1,000 (95% CI, 0.3 to 2.1; P = .005). In years 2 + 3, DBT cancer yield was 4.9/1,000, and of DBT+US, 5.9/1,000, difference 1.0/1,000 (95% CI, 0.4 to 1.5; P < .001). False-positive rate increased from 7.0% for DBT in year 1 to 11.5% for DBT+US and from 5.9% for DBT in year 2 + 3 to 9.7% for DBT+US ( P < .001 for both). Nine cancers were seen only by double reading DBT and one by double reading US. Ten interval cancers (0.6/1,000 [95% CI, 0.2 to 0.9]) were identified. Despite reduction in specificity, addition of US improved receiver operating characteristic curves, with area under receiver operating characteristic curve increasing from 0.83 for DBT alone to 0.92 for DBT+US in year 1 ( P = .01), with smaller improvements in subsequent years. Of 6,179 women, across all 3 years, 172/6,179 (2.8%) unique women had a false-positive biopsy because of DBT as did another 230/6,179 (3.7%) women because of US ( P < .001)., Conclusion: Overall added cancer detection rate of US screening after DBT was modest at 19/17,552 (1.1/1,000; CI, 0.5- to 1.6) screens but potentially overcomes substantial increases in false-positive recalls and benign biopsies.

Objectives: To evaluate the impact of body composition derived from computed tomography (CT) scans on postoperative lung cancer recurrence., Methods: We created a retrospective cohort of 363 lung cancer patients who underwent lung resections and had verified recurrence, death, or at least 5-year follow-up without either event. Five key body tissues and ten tumor features were automatically segmented and quantified based on preoperative whole-body CT scans (acquired as part of a PET-CT scan) and chest CT scans, respectively. Time-to-event analysis accounting for the competing event of death was performed to analyze the impact of body composition, tumor features, clinical information, and pathological features on lung cancer recurrence after surgery. The hazard ratio (HR) of normalized factors was used to assess individual significance univariately and in the combined models. The 5-fold cross-validated time-dependent receiver operating characteristics analysis, with an emphasis on the area under the 3-year ROC curve (AUC), was used to characterize the ability to predict lung cancer recurrence., Results: Body tissues that showed a standalone potential to predict lung cancer recurrence include visceral adipose tissue (VAT) volume (HR = 0.88, p = 0.047), subcutaneous adipose tissue (SAT) density (HR = 1.14, p = 0.034), inter-muscle adipose tissue (IMAT) volume (HR = 0.83, p = 0.002), muscle density (HR = 1.27, p < 0.001), and total fat volume (HR = 0.89, p = 0.050). The CT-derived muscular and tumor features significantly contributed to a model including clinicopathological factors, resulting in an AUC of 0.78 (95% CI: 0.75-0.83) to predict recurrence at 3 years., Conclusions: Body composition features (e.g., muscle density, or muscle and inter-muscle adipose tissue volumes) can improve the prediction of recurrence when combined with clinicopathological factors., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier B.V. All rights reserved.)

Purpose: Radiologists and other image readers spend prolonged periods inspecting medical images. The visual system can rapidly adapt or adjust sensitivity to the images that an observer is currently viewing, and previous studies have demonstrated that this can lead to pronounced changes in the perception of mammogram images. We compared these adaptation effects for images from different imaging modalities to explore both general and modality-specific consequences of adaptation in medical image perception., Approach: We measured perceptual changes induced by adaptation to images acquired by digital mammography (DM) or digital breast tomosynthesis (DBT), which have both similar and distinct textural properties. Participants (nonradiologists) adapted to images from the same patient acquired from each modality or for different patients with American College of Radiology-Breast Imaging Reporting and Data System (BI-RADS) classification of dense or fatty tissue. The participants then judged the appearance of composite images formed by blending the two adapting images (i.e., DM versus DBT or dense versus fatty in each modality)., Results: Adaptation to either modality produced similar significant shifts in the perception of dense and fatty textures, reducing the salience of the adapted component in the test images. In side-by-side judgments, a modality-specific adaptation effect was not observed. However, when the images were directly fixated during adaptation and testing, so that the textural differences between the modalities were more visible, significantly different changes in the sensitivity to the noise in the images were observed., Conclusions: These results confirm that observers can readily adapt to the visual properties or spatial textures of medical images in ways that can bias their perception of the images, and that adaptation can also be selective for the distinctive visual features of images acquired by different modalities., (© 2023 Society of Photo-Optical Instrumentation Engineers (SPIE).)

Background and Purpose: A high-intensity zone identified on preprocedural MR imaging is known to correlate with pain at provocation lumbar discography. The correlation between enhancing annular fissures and pain at provocation lumbar discography has not been comprehensively evaluated. The purpose of this study was to assess the pain response and imaging features at enhancing annular fissure nonoperated disc levels identified on preprocedural MR imaging with comparison with the high-intensity zone and nonenhancing disc levels in patients referred for provocation lumbar discography., Materials and Methods: One-hundred nonoperated discs in 44 patients were retrospectively evaluated for an enhancing annular fissure on sagittal postcontrast T1-weighted pre-discogram MR imaging. Enhancing annular fissure discs were graded on the sagittal T2-weighted sequence (Grade 4: like CSF to Grade 1: negative/barely visible) for high-intensity-zone conspicuity. High-intensity-zone detection was performed independently. In the primary assessment, enhancing annular fissure and high-intensity zones were associated with pain response at provocation lumbar discography. Additional analysis included intradiscal anesthetic response and postdiscogram CT appearance., Results: Thirty-nine discs demonstrated an enhancing annular fissure, with 23/39 demonstrating a high-intensity zone. The presence of a high-intensity zone predicted severe pain (concordant + nonconcordant; P = .005, sensitivity of 40%, specificity of 94%) and concordant pain ( P = .007, sensitivity of 39%, specificity of 86%) at provocation lumbar discography. Enhancing annular fissures without a detected high-intensity zone were more frequently observed among severely painful (50%) and concordant (36%) discs than among discs negative for pain (9%; P = .01). This finding resulted in a substantially greater overall sensitivity of enhancing annular fissures for severe ( P  < .001, 64%) and concordant pain ( P = .008, 61%), significantly improving the overall predictive ability of a high-intensity zone alone. A high-intensity zone went undetected in 9/11 Grade 1 disc levels with concordant pain present in 7/9., Conclusions: Consideration of enhancing annular fissures on preprocedural MR imaging substantially improves the prediction of severe/concordant pain in provocation lumbar discography., (© 2023 by American Journal of Neuroradiology.)

Objective: While provocation lumbar discography has been used to identify discs responsible for low back pain, the biomechanical effects of disc injection have received little attention. The purpose of this study was to assess the motion of the functional spinal unit including the endplate and facet/pedicle region during disc injection including comparison between normal and degenerative discs., Subjects: Subjects represent 91 consecutive patients referred for discography with chronic low back pain., Methods: Lateral projection vertebral motion was retrospectively analyzed at 232 levels (normal: 76 [32.8%], degenerative: 156 [67.2%]). Pre- and postinjection fluoroscopic images were size scaled, and lower endplates were superimposed on separate PowerPoint images. Upper endplate and facet/pedicle motion was separately and independently analyzed on toggled PowerPoint images, subjectively graded as prominent, intermediate, questionable/uncertain, or no motion. Disc morphology was graded using the anteroposterior/lateral postinjection disc appearance (Adams criteria)., Results: Prominent or intermediate endplate and facet/pedicle motion was identified at most lumbar levels with substantial overall agreement (degenerative: κ = 0.93, 95% confidence intervals [CI] = 0.87-1.00; normal: κ = 0.80, 95% CI = 0.61-1.00). Degenerative levels were strongly associated with a lower degree of endplate and facet/pedicle motion compared with normal: ("prominent" motion grade: endplate: 61% [95/156] vs 89% [68/76], P < 0.001; facet/pedicle: 60% [93/156] vs 88% [67/76], P < 0.001)., Conclusion: Disc injection expands the disc space inducing endplate motion, pedicle motion, and facet translation in almost all normal and most degenerate levels. Disc injection therefore biomechanically "provokes" the entire functional spinal unit. When provoked pain is encountered during lumbar discography, contribution from the associated facet joint and myotendinous insertions should be considered., (© The Author(s) 2021. Published by Oxford University Press on behalf of the American Academy of Pain Medicine. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Objective: For breast US interpretation, to assess impact of computer-aided diagnosis (CADx) in original mode or with improved sensitivity or specificity., Methods: In this IRB approved protocol, orthogonal-paired US images of 319 lesions identified on screening, including 88 (27.6%) cancers (median 7 mm, range 1-34 mm), were reviewed by 9 breast imaging radiologists. Each observer provided BI-RADS assessments (2, 3, 4A, 4B, 4C, 5) before and after CADx in a mode-balanced design: mode 1, original CADx (outputs benign, probably benign, suspicious, or malignant); mode 2, artificially-high-sensitivity CADx (benign or malignant); and mode 3, artificially-high-specificity CADx (benign or malignant). Area under the receiver operating characteristic curve (AUC) was estimated under each modality and for standalone CADx outputs. Multi-reader analysis accounted for inter-reader variability and correlation between same-lesion assessments., Results: AUC of standalone CADx was 0.77 (95% CI: 0.72-0.83). For mode 1, average reader AUC was 0.82 (range 0.76-0.84) without CADx and not significantly changed with CADx. In high-sensitivity mode, all observers' AUCs increased: average AUC 0.83 (range 0.78-0.86) before CADx increased to 0.88 (range 0.84-0.90), P < 0.001. In high-specificity mode, all observers' AUCs increased: average AUC 0.82 (range 0.76-0.84) before CADx increased to 0.89 (range 0.87-0.92), P < 0.0001. Radiologists responded more frequently to malignant CADx cues in high-specificity mode (42.7% vs 23.2% mode 1, and 27.0% mode 2, P = 0.008)., Conclusion: Original CADx did not substantially impact radiologists' interpretations. Radiologists showed improved performance and were more responsive when CADx produced fewer false-positive malignant cues., (© Society of Breast Imaging 2021. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Objective: Using terms adapted from the BI-RADS Mammography and MRI lexicons, we trained radiologists to interpret contrast-enhanced mammography (CEM) and assessed reliability of their description and assessment., Methods: A 60-minute presentation on CEM and terminology was reviewed independently by 21 breast imaging radiologist observers. For 21 CEM exams with 31 marked findings, observers recorded background parenchymal enhancement (BPE) (minimal, mild, moderate, marked), lesion type (oval/round or irregular mass, or non-mass enhancement), intensity of enhancement (none, weak, medium, strong), enhancement quality (none, homogeneous, heterogeneous, rim), and BI-RADS assessment category (2, 3, 4A, 4B, 4C, 5). "Expert" consensus of 3 other radiologists experienced in CEM was developed. Kappa statistic was used to assess agreement between radiologists and expert consensus, and between radiologists themselves, on imaging feature categories and final assessments. Reproducibility of specific feature descriptors was assessed as fraction of consensus-concordant responses., Results: Radiologists demonstrated moderate agreement for BPE, (mean kappa, 0.43; range, 0.05-0.69), and lowest reproducibility for "minimal." Agreement was substantial for lesion type (mean kappa, 0.70; range, 0.47-0.93), moderate for intensity of enhancement (mean kappa, 0.57; range, 0.44-0.76), and moderate for enhancement quality (mean kappa, 0.59; range, 0.20-0.78). Agreement on final assessment was fair (mean kappa, 0.26; range, 0.09-0.44), with BI-RADS category 3 the least reproducible. Decision to biopsy (BI-RADS 2-3 vs 4-5) showed moderate agreement with consensus (mean kappa, 0.54; range, -0.06-0.87)., Conclusion: With minimal training, agreement for description of CEM findings by breast imaging radiologists was comparable to other BI-RADS lexicons., (© Society of Breast Imaging 2021. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Rationale and Objectives: To preliminarily asses if Contrast Enhanced Digital Mammography (CEDM) can accurately reduce biopsy rates for soft tissue BI-RADS 4A or 4B lesions., Materials and Methods: Eight radiologists retrospectively and independently reviewed 60 lesions in 54 consenting patients who underwent CEDM under Health Insurance Portability and Accountability Act compliant institutional review board-approved protocols. Readers provided Breast Imaging Reporting & Data System ratings sequentially for digital mammography/digital breast tomosynthesis (DM/DBT), then with ultrasound, then with CEDM for each lesion. Area under the curve (AUC), true positive rates and false positive rates, positive predictive values and negative predictive values were calculated. Statistical analysis accounting for correlation between lesion-examinations and between-reader variability was performed using OR/DBM (for SAS v.3.0), generalized linear mixed model for binary data (proc glimmix, SAS v.9.4, SAS Institute, Cary North Carolina), and bootstrap., Results: The cohort included 49 benign, two high-risk and nine cancerous lesions in 54 women aged 34-74 (average 50) years. Reader-averaged AUC for CEDM was significantly higher than DM/DBT alone (0.85 versus 0.66, p < 0.001) or with US (0.85 versus 0.75, p = 0.001). CEDM increased true positive rates from 0.74 under DB/DBT, and 0.89 with US, to 0.90 with CEDM, (p = 0.019 DM/DBT versus CEDM, p = 0.78 DM/DBT + US versus CEDM) and decreased false positive rates from 0.47 using DM/DBT and 0.61 with US to 0.39 with CEDM (p = 0.017 DM/DBT versus CEDM, p = 0.001 DM/DBT+ US versus CEDM). For an expected cancer rate of 10%, CEDM positive predictive values was 20.5% (95% CI: 16%-27%) and negative predictive values 98.3% (95% CI: 96%-100%)., Conclusion: Addition of CEDM for evaluation of low-moderate suspicion soft tissue breast lesions can substantially reduce biopsy of benign lesions without compromising cancer detection., (Copyright © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Objective: To assess prospectively the interpretative performance of automated breast ultrasound (ABUS) as a supplemental screening after digital breast tomosynthesis (DBT) or as a standalone screening of women with dense breast tissue., Methods: Under an IRB-approved protocol (written consent required), women with dense breasts prospectively underwent concurrent baseline DBT and ABUS screening. Examinations were independently evaluated, in opposite order, by two of seven Mammography Quality Standards Act-qualified radiologists, with the primary radiologist arbitrating disagreements and making clinical management recommendations. We report results for 1111 screening examinations (598 first year and 513 second year) for which all diagnostic workups are complete. Imaging was also retrospectively reviewed for all cancers. Statistical assessments used a 0.05 significance level and accounted for correlation between participants' examinations., Results: Of 1111 women screened, primary radiologists initially "recalled" based on DBT alone (6.6%, 73/1111, CI: 5.2%-8.2%), of which 20 were biopsied, yielding 6/8 total cancers. Automated breast ultrasound increased recalls overall to 14.4% (160/1111, CI: 12.4%-16.6%), with 27 total biopsies, yielding 1 additional cancer. Double reading of DBT alone increased the recall rate to 10.7% (119/1111), with 21 biopsies, with no improvement in cancer detection. Double reading ABUS increased the recall rate to 15.2% (169/1111, CI: 13.2%-17.5%) of women, of whom 22 were biopsied, yielding the detection of 7 cancers, including one seen only on double reading ABUS. Inter-radiologist agreement was similar for recall recommendations from DBT (κ = 0.24, CI: 0.14-0.34) and ABUS (κ = 0.23, CI: 0.15-0.32). Integrated assessments from both readers resulted in a recall rate of 15.1% (168/1111, CI: 13.1%-17.4%)., Conclusion: Supplemental or standalone ABUS screening detected cancers not seen on DBT, but substantially increased noncancer recall rates., (© Society of Breast Imaging 2020. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

Background Staging newly diagnosed breast cancer by using dynamic contrast material-enhanced MRI is limited by access, high cost, and false-positive findings. The utility of contrast-enhanced mammography (CEM) and 99m Tc sestamibi-based molecular breast imaging (MBI) in this setting is largely unknown. Purpose To compare extent-of-disease assessments by using MRI, CEM, and MBI versus pathology in women with breast cancer. Materials and Methods In this HIPAA-compliant prospective study, women with biopsy-proven breast cancer underwent MRI, CEM, and MBI between October 2014 and April 2018. Eight radiologists independently interpreted each examination result prospectively and were blinded to interpretations of findings with the other modalities. Visibility of index malignancies, lesion size, and additional suspicious lesions (malignant or benign) were compared during pathology review. Accuracy of index lesion sizing and detection of additional lesions in women without neoadjuvant chemotherapy were compared. Results A total of 102 women were enrolled and 99 completed the study protocol (mean age, 51 years ± 11 [standard deviation]; range, 32-77 years). Lumpectomy or mastectomy was performed in 71 women (79 index malignancies) without neoadjuvant chemotherapy and in 28 women (31 index malignancies) with neoadjuvant chemotherapy. Of the 110 index malignancies, MRI, CEM, and MBI depicted 102 (93%; 95% confidence interval [CI]: 86%, 97%), 100 (91%; 95% CI: 84%, 96%), and 101 (92%; 95% CI: 85%, 96%) malignancies, respectively. In patients without neoadjuvant chemotherapy, pathologic size of index malignancies was overestimated with all modalities ( P = .02). MRI led to overestimation of 24% (17 of 72) of malignancies by more than 1.5 cm compared with 11% (eight of 70) with CEM and 15% (11 of 72) with MBI. MRI depicted more ( P = .007) nonindex lesions, with sensitivity similar to that of CEM or MBI, resulting in lower positive predictive value of additional biopsies (13 of 46 [28%; 95% CI: 17%, 44%] for MRI; 14 of 27 [52%; 95% CI: 32%, 71%] for CEM; and 11 of 25 [44%; 95% CI: 24%, 65%] for MBI (overall P = .01). Conclusion Contrast-enhanced mammography, molecular breast imaging, and MRI showed similar detection of all malignancies. MRI depicted more nonindex suspicious benign lesions than did contrast-enhanced mammography or molecular breast imaging, leading to lower positive predictive value of additional biopsies. All three modalities led to overestimation of index tumor size, particularly MRI. © RSNA, 2019 Online supplemental material is available for this article.

Diagnostic systems designed to detect possibly multiple lesions per patient (e.g. multiple polyps during CT colonoscopy) are often evaluated in "free-response" studies that allow for diagnostic responses unconstrained in their number and locations. Analysis of free-response studies requires extensions of the traditional receiver operating characteristic (ROC) analysis, which are termed free-response ROC (FROC) methodology. Despite substantial developments in this area, FROC tools and approaches are much more cumbersome than traditional ROC methods. Alternative approaches that use well-known ROC tools (e.g. ROI-ROC) require defining and physically delineating regions of interest (ROI) and combine FROC data within ROIs. We propose an approach that allows analyzing FROC data using conventional ROC tools without delineating the actual ROIs or reducing data. The design parameters of FROC study are used to make FROC data analyzable using ROC tools and to calibrate the corresponding FROC and ROC curves on both conceptual and numerical levels. Differences in the performance indices of the nonparametric FROC and the new approach are shown to be asymptotically negligible and typically rather small in practice. Data from a large multi-reader study of colon cancer detection are used to illustrate the new approach.

Background Digital breast tomosynthesis (DBT) is replacing digital mammography (DM) in the clinical workflow. Currently, there are limited prospective studies comparing the diagnostic accuracy of both examinations and the role of synthetic mammography (SM) and computer-aided detection (CAD). Purpose To compare the accuracy of DM versus DM + DBT in population-based breast cancer screening. Materials and Methods This prospective study, performed from November 2010 to December 2012, included 24 301 women (mean age, 59.1 years ± 5.7 [standard deviation]) with 281 cancers, of which 51 were interval cancers. Each examination was independently interpreted with four reading modes: DM, DM + CAD, DM + DBT, and SM + DBT. Sensitivity and specificity were compared for DM versus DM + DBT, DM versus DM + CAD, DM + DBT versus SM + DBT, and DM versus DM + DBT at double reading. Reader-adjusted performance characteristics of reading modes were evaluated on the basis of pre-arbitration (initial interpretation) scores. Statistical analysis was based on cluster bootstrap analysis using 10 000 random resamples. Results Sensitivity was 54.1% (152 of 281) for DM and 70.5% (198 of 281) for DM + DBT. Reader-adjusted difference was 12.6% (95% confidence interval [CI]: 5.2%, 19.7%; P = .001). Specificity was 94.2% (false-positive fraction [FPF], 5.8%; 1388 of 24 020) for DM and 95.0% (FPF, 5.0%; 1209/24 020) for DM + DBT, with a reader-adjusted difference in FPF of -1.2% (95% CI: -1.7%, -0.7%; P < .001). Sensitivity was 69.0% (194 of 281) for SM + DBT and 70.5% (198 of 281) for DM + DBT, with a reader-adjusted difference of 1.0% (95% CI: -6.2%, 8.5%; P = .77). Specificity was 95.4% (FPF, 4.6%; 1111 of 24 020) for SM + DBT and 95.0% (FPF, 5.0%;1209 of 24 020) for DM + DBT, with reader-adjusted 95% CIs for FPF of 4.7%, 5.4% and 5.0%, 5.7%, respectively, and a difference of -0.3% (95% CI: -0.8%, 0.2%; P = .23). Differences in sensitivity and specificity with the addition of CAD were small and not significant (P > .2). Conclusion Addition of digital breast tomosynthesis to digital mammography resulted in significant gains in sensitivity and specificity. Synthetic mammography in combination with digital breast tomosynthesis had similar sensitivity and specificity to digital mammography in combination with digital breast tomosynthesis. © RSNA, 2019 See also the editorial by Lång in this issue.

Purpose To determine rate of malignancy at stereotactic biopsy of amorphous calcifications with different distributions using current imaging, clinical, and histopathologic criteria. Materials and Methods From January 2009 to September 2013, this retrospective study reviewed a large set of stereotactic biopsies to identify amorphous calcifications and their clinical, imaging, and histopathologic characteristics. Calcification distribution was correlated with malignancy rate after adjusting for known risk factors using logistic regression. Results Of 1903 sequential biopsies, 546 (28.7%) were for amorphous calcifications. After excluding atypical lesions not excised and patients with more than one biopsy in the same year, 497 lesions from 494 women (median age, 52 years; age range, 30-81 years) remained. Fifty-two (10.5%; 95% confidence interval [CI]: 7.9, 13.5) lesions proved malignant, with 17 of 52 (42.7%) being invasive cancers (median, 0.3 cm; range, 0.1-1.3 cm) and all 17 of them being estrogen and progesterone receptor positive and node negative. Malignancy rates in a segmental (six of 21 [28.6%]), linear (eight of 32 [25.0%]), or multiple group same quadrant (nine of 36 [25.0%]) distribution were significantly higher than malignancy rate in a solitary group of amorphous calcifications (25 of 356 [7.0%]) (P = .004, P = .003, and P = .002, respectively). Of 356 grouped amorphous calcifications, 102 (28.7%) yielded atypical results prompting excision, with three of 102 (2.9%) upgraded to ductal carcinoma in situ at excision. In women younger than 50 years without a personal history of cancer, grouped amorphous calcifications showed four of 127 (3.1%) (95% CI: 0.9, 7.9) were malignant and 39 of 127 (30.7%) were atypical at final histopathology. Conclusion Biopsy of amorphous calcifications remains necessary, with an overall malignancy rate of 10.5%; only 17 of 497 (3.4%) biopsies showed invasive cancer, and all of these were estrogen and progesterone receptor positive. Grouped amorphous calcifications in women younger than 50 years without history of breast or ovarian cancer showed a low malignancy rate of 3.1% (four of 127)., (© RSNA, 2018 Online supplemental material is available for this article.)

Purpose: Digital breast tomosynthesis (DBT) has the potential to overcome limitations of conventional mammography. This study investigated the effects of addition of DBT on interval and detected cancers in population-based screening., Methods: Oslo Tomosynthesis Screening Trial (OTST) was a prospective, independent double-reading trial inviting women 50-69 years biennially, comparing full-field digital mammography (FFDM) plus DBT with FFDM alone. Performance indicators and characteristics of screen-detected and interval cancers were compared with two previous FFDM rounds., Results: 24,301 consenting women underwent FFDM + DBT screening over a 2-year period. Results were compared with 59,877 FFDM examinations during prior rounds. Addition of DBT resulted in a non-significant increase in sensitivity (76.2%, 378/496, vs. 80.8%, 227/281, p = 0.151) and a significant increase in specificity (96.4%, 57229/59381 vs. 97.5%, 23427/24020, p < .001). Number of recalls per screen-detected cancer decreased from 6.7 (2530/378) to 3.6 (820/227) with DBT (p < .001). Cancer detection per 1000 women screened increased (6.3, 378/59877, vs. 9.3, 227/24301, p < .001). Interval cancer rate per 1000 screens for FFDM + DBT remained similar to previous FFDM rounds (2.1, 51/24301 vs. 2.0, 118/59877, p = 0.734). Interval cancers post-DBT were comparable to prior rounds but significantly different in size, grade, and node status from cancers detected only using DBT. 39.6% (19/48) of interval cancers had positive nodes compared with only 3.9% (2/51) of additional DBT-only-detected cancers., Conclusions: DBT-supplemented screening resulted in significant increases in screen-detected cancers and specificity. However, no significant change was observed in the rate, size, node status, or grade of interval cancers. ClinicalTrials.gov: NCT01248546.

Purpose To assess the accuracy of staging positron emission tomography (PET)/computed tomography (CT) in detecting distant metastasis in patients with local-regionally advanced cervical and high-risk endometrial cancer in the clinical trial by the American College of Radiology Imaging Network (ACRIN) and the Gynecology Oncology Group (GOG) (ACRIN 6671/GOG 0233) and to compare central and institutional reader performance. Materials and Methods In this prospective multicenter trial, PET/CT and clinical data were reviewed for patients enrolled in ACRIN 6671/GOG 0233. Two central readers, blinded to site read and reference standard, reviewed PET/CT images for distant metastasis. Central review was then compared with institutional point-of-care interpretation. Reference standard was pathologic and imaging follow-up. Test performance for central and site reviews of PET/CT images was calculated and receiver operating characteristic analysis was performed. Generalized estimating equations and nonparametric bootstrap procedure for clustered data were used to assess statistical significance. Results There were 153 patients with cervical cancer and 203 patients with endometrial cancer enrolled at 28 sites. Overall prevalence of distant metastasis was 13.7% (21 of 153) for cervical cancer and 11.8% (24 of 203) for endometrial cancer. Central reader PET/CT interpretation demonstrated sensitivity, specificity, positive predictive value (PPV), and negative predictive value of 54.8%, 97.7%, 79.3%, and 93.1% for cervical cancer metastasis versus 64.6%, 98.6%, 86.1%, and 95.4% for endometrial cancer, respectively. By comparison, local institutional review demonstrated sensitivity, specificity, PPV, and negative predictive value of 47.6%, 93.9%, 55.6%, and 91.9% for cervical cancer metastasis and 66.7%, 93.9%, 59.3%, and 95.5% for endometrial cancer, respectively. For central readers, the specificity and PPV of PET/CT detection of cervical and endometrial cancer metastases were all significantly higher compared with that of local institutional review (P < .05). Central reader area under the receiver operating characteristic curve (AUC) values were 0.78 and 0.89 for cervical and endometrial cancer, respectively; these were not significantly different from local institutional AUC values (0.75 and 0.84, respectively; P > .05 for both). Conclusion FDG PET/CT demonstrates high specificity and PPV for detecting distant metastasis in cervical and endometrial cancer and should be included in the staging evaluation. Blinded central review of imaging provides improved specificity and PPV for the detection of metastases and should be considered for future oncologic imaging clinical trials. © RSNA, 2017.

Assessing performance of diagnostic markers is a necessary step for their use in decision making regarding various conditions of interest in diagnostic medicine and other fields. Globally useful markers could, however, have ranges of values that are " diagnostically non-informative" . This paper demonstrates that the presence of marker values from diagnostically non-informative ranges could lead to a loss in statistical efficiency during nonparametric evaluation and shows that grouping non-informative values provides a natural resolution to this problem. These points are theoretically proven and an extensive simulation study is conducted to illustrate the possible benefits of using grouped marker values in a number of practically reasonable scenarios. The results contradict the common conjecture regarding the detrimental effect of grouped marker values during performance assessments. Specifically, contrary to the common assumption that grouped marker values lead to bias, grouping non-informative values does not introduce bias and could substantially reduce sampling variability. The proven concept that grouped marker values could be statistically beneficial without detrimental consequences implies that in practice, tied values do not always require resolution whereas the use of continuous diagnostic results without addressing diagnostically non-informative ranges could be statistically detrimental. Based on these findings, more efficient methods for evaluating diagnostic markers could be developed.

Receiver operating characteristic analysis provides an important methodology for assessing traditional (e.g., imaging technologies and clinical practices) and new (e.g., genomic studies, biomarker development) diagnostic problems. The area under the clinically/practically relevant part of the receiver operating characteristic curve (partial area or partial area under the receiver operating characteristic curve) is an important performance index summarizing diagnostic accuracy at multiple operating points (decision thresholds) that are relevant to actual clinical practice. A robust estimate of the partial area under the receiver operating characteristic curve is provided by the area under the corresponding part of the empirical receiver operating characteristic curve. We derive a closed-form expression for the jackknife variance of the partial area under the empirical receiver operating characteristic curve. Using the derived analytical expression, we investigate the differences between the jackknife variance and a conventional variance estimator. The relative properties in finite samples are demonstrated in a simulation study. The developed formula enables an easy way to estimate the variance of the empirical partial area under the receiver operating characteristic curve, thereby substantially reducing the computation burden, and provides important insight into the structure of the variability. We demonstrate that when compared with the conventional approach, the jackknife variance has substantially smaller bias, and leads to a more appropriate type I error rate of the Wald-type test. The use of the jackknife variance is illustrated in the analysis of a data set from a diagnostic imaging study.

Rationale and Objectives: The "binormal" model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have "improper" (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve., Materials and Methods: We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1, we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data., Results: We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage., Conclusions: The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences., (Copyright © 2017 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Assess results of a prospective, single-site clinical study evaluating digital breast tomosynthesis (DBT) during baseline screening mammography., Materials and Methods: Under an institutional review board-approved Health Insurance Portability and Accountability Act (HIPAA)-compliant protocol, consenting women between ages 34 and 56 years scheduled for their initial and/or baseline screening mammogram underwent both full field digital mammography (FFDM) and DBT. The FFDM and the FFDM plus DBT images were interpreted independently in a reader by mode balanced approach by two of 14 participating radiologists. A woman was recalled for a diagnostic work-up if either radiologist recommended a recall. We report overall recall rates and related diagnostic outcome from the 1080 participants. Proportion of recommended recalls (Breast Imaging Reporting and Data System 0) were compared using a generalized linear mixed model (SAS 9.3) with a significance level of P = .0294., Results: The fraction of women without breast cancer recommended for recall using FFDM alone and FFDM plus DBT were 412 of 1074 (38.4%) and 274 of 1074 (25.5%), respectively (P < .001). Large inter-reader variability in terms of recall reduction was observed among the 14 readers; however, 11 of 14 readers recalled fewer women using FFDM plus DBT (5 with P < .015). Six cancers (four ductal carcinomas in situ [DCIS] and two invasive ductal carcinomas [IDC]) were detected. One IDC was detected only on DBT and one DCIS cancer was detected only on FFDM, whereas the remaining cancers were detected on both modalities., Conclusions: The use of FFDM plus DBT resulted in a significant decrease in recall rates during baseline screening mammography with no reduction in sensitivity., (Copyright © 2015 AUR. Published by Elsevier Inc. All rights reserved.)

Purpose: To assess the effect of and interaction between the availability of prior images and digital breast tomosynthesis (DBT) images in decisions to recall women during mammogram interpretation., Materials and Methods: Verbal informed consent was obtained for this HIPAA-compliant institutional review board-approved protocol. Eight radiologists independently interpreted twice deidentified mammograms obtained in 153 women (age range, 37-83 years; mean age, 53.7 years ± 9.3 [standard deviation]) in a mode by reader by case-balanced fully crossed study. Each study consisted of current and prior full-field digital mammography (FFDM) images and DBT images that were acquired in our facility between June 2009 and January 2013. For one reading, sequential ratings were provided by using (a) current FFDM images only, (b) current FFDM and DBT images, and (c) current FFDM, DBT, and prior FFDM images. The other reading consisted of (a) current FFDM images only, (b) current and prior FFDM images, and (c) current FFDM, prior FFDM, and DBT images. Fifty verified cancer cases, 60 negative and benign cases (clinically not recalled), and 43 benign cases (clinically recalled) were included. Recall recommendations and interaction between the effect of prior FFDM and DBT images were assessed by using a generalized linear model accounting for case and reader variability., Results: Average recall rates in noncancer cases were significantly reduced with the addition of prior FFDM images by 34% (145 of 421) and 32% (106 of 333) without and with DBT images, respectively (P < .001). However, this recall reduction was achieved at the cost of a corresponding 7% (23 of 345) and 4% (14 of 353) reduction in sensitivity (P = .006). In contrast, availability of DBT images resulted in a smaller reduction in recall rates (false-positive interpretations) of 19% (76 of 409) and 26% (71 of 276) without and with prior FFDM images, respectively (P = .001). Availability of DBT images resulted in 4% (15 of 338) and 8% (25 of 322) increases in sensitivity, respectively (P = .007). The effects of the availability of prior FFDM images or DBT images did not significantly change regardless of the sequence in presentation (P = .81 and P = .47 for specificity and sensitivity, respectively)., Conclusion: The availability of prior FFDM or DBT images is a largely independent contributing factor in reducing recall recommendations during mammographic interpretation.

Rationale and Objectives: To assess radiologists' perceptions of how the new Breast Density Notification Act (BDNA) of Pennsylvania would affect their breast density reporting and their actual reporting patterns after implementation., Materials and Methods: Under an institutional review board-approved protocol, we surveyed 21 radiologists about how they believe the new law affected their breast density reporting patterns and analyzed actual changes for 16 respondents before and after the law took effect. Three hundred consecutive reports were assessed for each radiologist before and after the effective date. The distributions of reported density Breast Imaging Reporting and Data System (BI-RADS) (1-4) were compared using a type III test in the context of an ordinal mixed model accounting for between-reader variability and adjusting for age (PROC GLIMMIX, SAS, version 9.3) using a two-sided .05 significance level., Results: Seventeen radiologists responded to the survey; however, one retired shortly after responding. Of the 16 respondents, 56% (nine of 16) did not favor the law, 13% (two of 16) were in favor, and 31% (five of 16) were neutral. The fraction perceived that after implementation, they rated more, equally, or less frequently breasts as scattered fibroglandular densities (BI-RADS 2) versus heterogeneously dense rating (BI-RADS 3) was 50% (eight of 16), 44% (seven of 16), and 6% (one of 16), respectively. In practice, 44% (seven of 16) performed differently than their survey answers. Fourteen of 16 radiologists increased the frequency of reported BI-RADS 2 scores after BDNA implementation with seven having statistically significant (P < .05) increases after adjusting for age differences., Conclusions: Radiologists' reporting patterns changed, at least for a short duration, after the new density reporting law and for some of the radiologists in an unexpected way., (Copyright © 2015 AUR. Published by Elsevier Inc. All rights reserved.)

Evaluation of diagnostic performance is typically based on the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as its summary index. The partial area under the curve (pAUC) is an alternative index focusing on the range of practical/clinical relevance. One of the problems preventing more frequent use of the pAUC is the perceived loss of efficiency in cases of noncrossing ROC curves. In this paper, we investigated statistical properties of comparisons of two correlated pAUCs. We demonstrated that outside of the classic model there are practically reasonable ROC types for which comparisons of noncrossing concave curves would be more powerful when based on a part of the curve rather than the entire curve. We argue that this phenomenon stems in part from the exclusion of noninformative parts of the ROC curves that resemble straight-lines. We conducted extensive simulation studies in families of binormal, straight-line, and bigamma ROC curves. We demonstrated that comparison of pAUCs is statistically more powerful than comparison of full AUCs when ROC curves are close to a "straight line". For less flat binormal ROC curves an increase in the integration range often leads to a disproportional increase in pAUCs' difference, thereby contributing to an increase in statistical power. Thus, efficiency of differences in pAUCs of noncrossing ROC curves depends on the shape of the curves, and for families of ROC curves that are nearly straight-line shaped, such as bigamma ROC curves, there are multiple practical scenarios in which comparisons of pAUCs are preferable., (© 2014 WILEY-VCH Verlag GmbH & Co. KGaA, Weinheim.)

Purpose: To assess interpretation performance and radiation dose when two-dimensional synthesized mammography (SM) images versus standard full-field digital mammography (FFDM) images are used alone or in combination with digital breast tomosynthesis images., Materials and Methods: A fully crossed, mode-balanced multicase (n = 123), multireader (n = 8), retrospective observer performance study was performed by using deidentified images acquired between 2008 and 2011 with institutional review board approved, HIPAA-compliant protocols, during which each patient signed informed consent. The cohort included 36 cases of biopsy-proven cancer, 35 cases of biopsy-proven benign lesions, and 52 normal or benign cases (Breast Imaging Reporting and Data System [BI-RADS] score of 1 or 2) with negative 1-year follow-up results. Accuracy of sequentially reported probability of malignancy ratings and seven-category forced BI-RADS ratings was evaluated by using areas under the receiver operating characteristic curve (AUCs) in the random-reader analysis., Results: Probability of malignancy-based mean AUCs for SM and FFDM images alone was 0.894 and 0.889, respectively (difference, -0.005; 95% confidence interval [CI]: -0.062, 0.054; P = .85). Mean AUC for SM with tomosynthesis and FFDM with tomosynthesis was 0.916 and 0.939, respectively (difference, 0.023; 95% CI: -0.011, 0.057; P = .19). In terms of the reader-specific AUCs, five readers performed better with SM alone versus FFDM alone, and all eight readers performed better with combined FFDM and tomosynthesis (absolute differences from 0.003 to 0.052). Similar results were obtained by using a nonparametric analysis of forced BI-RADS ratings., Conclusion: SM alone or in combination with tomosynthesis is comparable in performance to FFDM alone or in combination with tomosynthesis and may eliminate the need for FFDM as part of a routine clinical study.

Purpose: To compare the performance of two versions of reconstructed two-dimensional (2D) images in combination with digital breast tomosynthesis (DBT) versus the performance of standard full-field digital mammography (FFDM) plus DBT., Materials and Methods: This trial had ethical committee approval, and all participants gave written informed consent. Examinations (n = 24 901) in women between the ages of 50 and 69 years (mean age, 59.2 years) were interpreted prospectively as part of a screening trial that included independent interpretations of FFDM plus DBT and reconstructed 2D images plus DBT. Reconstructed 2D images do not require radiation exposure. Using analyses for binary data that accounted for correlated interpretations and were adjusted for reader-specific volume, two versions (initial and current) of reconstructed 2D images used during trial periods 1 (from November 22, 2010, to December 21, 2011; 12 631 women) and 2 (from January 20, 2012, to December 19, 2012; 12 270 women) were compared in terms of cancer detection and false-positive rates with the corresponding FFDM plus DBT interpretations., Results: Cancer detection rates were 8.0, 7.4, 7.8, and 7.7 per 1000 screening examinations for FFDM plus DBT in period 1, initial reconstructed 2D images plus DBT in period 1, FFDM plus DBT in period 2, and current reconstructed 2D images plus DBT in period 2, respectively. False-positive scores were 5.3%, 4.6%, 4.6%, and 4.5%, respectively. Corresponding reader-adjusted paired comparisons of false-positive scores revealed significant differences for period 1 (P = .012) but not for period 2 (ratio = 0.99; 95% confidence interval: 0.88, 1.11; P = .85)., Conclusion: The combination of current reconstructed 2D images and DBT performed comparably to FFDM plus DBT and is adequate for routine clinical use when interpreting screening mammograms.

Rationale and Objectives: To assess the interaction between the availability of prior examinations and digital breast tomosynthesis (DBT) in decisions to recall a woman during interpretation of mammograms., Materials and Methods: Eight radiologists independently interpreted twice 36 mammography examinations, each of which had current and prior full-field digital mammography images (FFDM) and DBT under a Health Insurance Portability and Accountability Act-compliant, institutional review board-approved protocol (written consent waived). During the first reading, three sequential ratings were provided using FFDM only, followed by FFDM + DBT, and then followed by FFDM + DBT + priors. The second reading included FFDM only, then FFDM + priors, and then FFDM + priors + DBT. Twenty-two benign cases clinically recalled, 12 negative/benign examinations (not recalled), and two verified cancer cases were included. Recall recommendations and interaction between the effect of priors and DBT on decisions were assessed (P = .05 significance level) using generalized linear model (PROC GLIMMIX, SAS, version 9.3; SAS Institute, Cary, NC) accounting for case and reader variability., Results: Average recall rates in noncancer cases were significantly reduced (51%; P < .001) with the addition of DBT and with addition of priors (23%; P = .01). In absolute terms, the addition of DBT to FFDM reduced the recall rates from 0.67 to 0.42 and from 0.54 to 0.27 when DBT was available before and after priors, respectively. Recall reductions were from 0.64 to 0.54 and from 0.42 to 0.33 when priors were available before and after DBT, respectively. Regardless of the sequence in presentation, there were no statistically significant interactions between the effect of availability of DBT and priors (P = .80)., Conclusions: Availability of both priors and DBT are independent primary factors in reducing recall recommendations during mammographic interpretations., (Copyright © 2014 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives. To compare the sensitivities of ultrasound guided core biopsy and fine needle aspiration (FNA) for detection of axillary lymph node metastases in patients with a current diagnosis of ipsilateral breast cancer. Materials and Methods. From December 2008 to December 2010, 105 patients with breast cancer and abnormal appearing lymph nodes in the ipsilateral axilla consented to undergo FNA of an axillary node immediately followed by core biopsy of the same node, both with ultrasound guidance. Experienced pathologists evaluated the aspirate cytology without knowledge of the core histology. Cytology and core biopsy results were compared to sentinel node excision or axillary dissection pathology. Sensitivities were compared using McNemar's test. Results. Of 70 patients with axillary node metastases, FNA was positive in 55/70 (78.6%) and core was positive in 61/70 (87.1%) (P = 0.18). The FNA and core results were discordant in 14/70 (20%) patients. Ten cases were FNA negative/core positive. Four cases were FNA positive/core negative. Conclusion. Core biopsy detected six (8.6%) more cases of metastatic lymphadenopathy than FNA but the difference in sensitivities was not statistically significant. Core biopsy should be considered if the node is clearly imaged and readily accessible. FNA is a good alternative when a smaller needle is desired due to node location or other patient factors. This trial is registered with NCT01920139.

Evaluation of diagnostic performance is a necessary component of new developments in many fields including medical diagnostics and decision making. The methodology for statistical analysis of diagnostic performance continues to develop, offering new analytical tools for conventional inferences and solutions for novel and increasingly more practically relevant questions. In this paper, we focus on the partial area under the Receiver Operating Characteristic (ROC) curve or pAUC. This summary index is considered to be more practically relevant than the area under the entire ROC curve (AUC), but because of several perceived limitations, it is not used as often. To improve interpretation, results for pAUC analysis are frequently reported using a rescaled index such as the standardized partial AUC proposed by McClish (1989). We derive two important properties of the relationship between the 'standardized' pAUC and the defined range of interest, which could facilitate a wider and more appropriate use of this important summary index. First, we mathematically prove that the 'standardized' pAUC increases with increasing range of interest for practically common ROC curves. Second, using comprehensive numerical investigations, we demonstrate that, contrary to common belief, the uncertainty about the estimated standardized pAUC can either decrease or increase with an increasing range of interest. Our results indicate that the partial AUC could frequently offer advantages in terms of statistical uncertainty of the estimation. In addition, selection of a wider range of interest will likely lead to an increased estimate even for standardized pAUC., (Copyright © 2013 John Wiley & Sons, Ltd.)

Objectives: To compare double readings when interpreting full field digital mammography (2D) and tomosynthesis (3D) during mammographic screening., Methods: A prospective, Ethical Committee approved screening study is underway. During the first year 12,621 consenting women underwent both 2D and 3D imaging. Each examination was independently interpreted by four radiologists under four reading modes: Arm A-2D; Arm B-2D + CAD; Arm C-2D + 3D; Arm D-synthesised 2D + 3D. Examinations with a positive score by at least one reader were discussed at an arbitration meeting before a final management decision. Paired double reading of 2D (Arm A + B) and 2D + 3D (Arm C + D) were analysed. Performance measures were compared using generalised linear mixed models, accounting for inter-reader performance heterogeneity (P < 0.05)., Results: Pre-arbitration false-positive scores were 10.3 % (1,286/12,501) and 8.5 % (1,057/12,501) for 2D and 2D + 3D, respectively (P < 0.001). Recall rates were 2.9 % (365/12,621) and 3.7 % (463/12,621), respectively (P = 0.005). Cancer detection was 7.1 (90/12,621) and 9.4 (119/12,621) per 1,000 examinations, respectively (30 % increase, P < 0.001); positive predictive values (detected cancer patients per 100 recalls) were 24.7 % and 25.5 %, respectively (P = 0.97). Using 2D + 3D, double-reading radiologists detected 27 additional invasive cancers (P < 0.001)., Conclusion: Double reading of 2D + 3D significantly improves the cancer detection rate in mammography screening., Key Points: • Tomosynthesis-based screening was successfully implemented in a large prospective screening trial. • Double reading of tomosynthesis-based examinations significantly reduced false-positive interpretations. • Double reading of tomosynthesis significantly increased the detection of invasive cancers.

Purpose: To develop an approach of estimating subject-centered free-response receiver operating characteristic (FROC) curve for providing patient-centered inferences regarding detection-localization characteristics of a diagnostic system., Methods: The authors examine properties of the conventional, target-centered, FROC curve and demonstrate that in scenarios where the diagnostic performance correlates with the total number of targets on a subject, it may lead to inadequate inferences from the perspective of possible benefits to a patient. Following solutions to patient-centered approaches in other applications, the authors define a subject-centered FROC curve and develop its formulation as a covariate-adjusted FROC curve. The authors also conduct a numerical study illustrating the relative properties of the conventional and subject-centered approach and provide an example., Results: A simple-to-implement approach for estimating the subject-centered FROC curve and its overall index can be formulated as a type of stratified FROC analysis. The authors demonstrate that when diagnostic performance is associated with the number of targets, the diagnostic system with apparently superior target-centered characteristics (conventional approach) can be actually inferior from the subject-centered perspective. The authors show that under some clinically reasonable conditions the magnitude of disagreement in results could be substantial. An example from an actual observer performance study illustrates the natural setting where the developed approach would be relevant and lead to conclusions that are contradictory to those obtained from conventional analysis., Conclusions: The authors developed a subject-centered FROC curve and its overall index provides tools for inferences that may be relevant from a perspective of potential benefits to a patient.

Purpose: To assess cancer detection rates, false-positive rates before arbitration, positive predictive values for women recalled after arbitration, and the type of cancers detected with use of digital mammography alone and combined with tomosynthesis in a large prospective screening trial., Materials and Methods: A prospective, reader- and modality-balanced screening study of participants undergoing combined mammography plus tomosynthesis, the results of which were read independently by four different radiologists, is under way. The study was approved by a regional ethics committee, and all participants provided written informed consent. The authors performed a preplanned interim analysis of results from 12,631 examinations interpreted by using mammography alone and mammography plus tomosynthesis from November 22, 2010, to December 31, 2011. Analyses were based on marginal log-linear models for binary data, accounting for correlated interpretations and adjusting for reader-specific performance levels by using a two-sided significance level of .0294., Results: Detection rates, including those for invasive and in situ cancers, were 6.1 per 1000 examinations for mammography alone and 8.0 per 1000 examinations for mammography plus tomosynthesis (27% increase, adjusted for reader; P = .001). False-positive rates before arbitration were 61.1 per 1000 examinations with mammography alone and 53.1 per 1000 examinations with mammography plus tomosynthesis (15% decrease, adjusted for reader; P < .001). After arbitration, positive predictive values for recalled patients with cancers verified later were comparable (29.1% and 28.5%, respectively, with mammography alone and mammography plus tomosynthesis; P = .72). Twenty-five additional invasive cancers were detected with mammography plus tomosynthesis (40% increase, adjusted for reader; P < .001). The mean interpretation time was 45 seconds for mammography alone and 91 seconds for mammography plus tomosynthesis (P < .001)., Conclusion: The use of mammography plus tomosynthesis in a screening environment resulted in a significantly higher cancer detection rate and enabled the detection of more invasive cancers. Clinical trial registration no. NCT01248546., (RSNA, 2013)

Purpose: To compare the diagnostic performance of breast tomosynthesis versus supplemental mammography views in classification of masses, distortions, and asymmetries., Materials and Methods: Eight radiologists who specialized in breast imaging retrospectively reviewed 217 consecutively accrued lesions by using protocols that were HIPAA compliant and institutional review board approved in 182 patients aged 31-60 years (mean, 50 years) who underwent diagnostic mammography and tomosynthesis. The lesions in the cohort included 33% (72 of 217) cancers and 67% (145 of 217) benign lesions. Eighty-four percent (182 of 217) of the lesions were masses, 11% (25 of 217) were asymmetries, and 5% (10 of 217) were distortions that were initially detected at clinical examination in 8% (17 of 217), at mammography in 80% (173 of 217), at ultrasonography (US) in 11% (25 of 217), or at magnetic resonance imaging in 1% (2 of 217). Histopathologic examination established truth in 191 lesions, US revealed a cyst in 12 lesions, and 14 lesions had a normal follow-up. Each lesion was interpreted once with tomosynthesis and once with supplemental mammographic views; both modes included the mediolateral oblique and craniocaudal views in a fully crossed and balanced design by using a five-category Breast Imaging Reporting and Data System (BI-RADS) assessment and a probability-of-malignancy score. Differences between modes were analyzed with a generalized linear mixed model for BI-RADS-based sensitivity and specificity and with modified Obuchowski-Rockette approach for probability-of-malignancy-based area under the receiver operating characteristic (ROC) curve., Results: Average probability-of-malignancy-based area under the ROC curve was 0.87 for tomosynthesis versus 0.83 for supplemental views (P < .001). With tomosynthesis, the false-positive rate decreased from 85% (989 of 1160) to 74% (864 of 1160) (P < .01) for cases that were rated BI-RADS category 3 or higher and from 57% (663 of 1160) to 48% (559 of 1160) for cases rated BI-RADS category 4 or 5 (P < .01), without a meaningful change in sensitivity. With tomosynthesis, more cancers were classified as BI-RADS category 5 (39% [226 of 576] vs 33% [188 of 576]; P = .017) without a decrease in specificity., Conclusion: Tomosynthesis significantly improved diagnostic accuracy for noncalcified lesions compared with supplemental mammographic views., (RSNA, 2012)

Rationale and Objectives: The aim of this study was to retrospectively compare the interpretive performance of synthetically reconstructed two-dimensional images in combination with digital breast tomosynthesis (DBT) versus full-field digital mammography (FFDM) plus DBT., Materials and Methods: Ten radiologists trained in reading tomosynthesis examinations interpreted retrospectively, under two modes, 114 mammograms. One mode included the directly acquired full-field digital mammograms combined with DBT, and the other included synthetically reconstructed projection images combined with DBT. The reconstructed images do not require additional radiation exposure. The two modes were compared with respect to sensitivity, namely, recommendation to recall a breast with either a pathology-proven cancer (n = 48) or a high-risk lesion (n = 6), and specificity, namely, no recommendation to recall a breast not depicting an abnormality (n = 144) or depicting only benign abnormalities (n = 30)., Results: The average sensitivity for FFDM with DBT was 0.826, compared to 0.772 for synthetic FFDM with DBT (difference, 0.054; P = .017 and P = .053 for fixed and random reader effects, respectively). The proportions of breasts with no or benign abnormalities recommended to be recalled were virtually the same: 0.298 and 0.297 for the two modalities, respectively (95% confidence intervals for the difference, -0.028 to 0.036 and -0.070 to 0.066 for fixed and random reader effects, respectively). Sixteen additional clusters of microcalcifications ("positive" breasts) were missed by all readers combined when interpreting the mode with synthesized images versus FFDM., Conclusions: Lower sensitivity with comparable specificity was observed with the tested version of synthetically generated images compared to FFDM, both combined with DBT. Improved synthesized images with experimentally verified acceptable diagnostic quality will be needed to eliminate double exposure during DBT-based screening., (Copyright © 2012 AUR. Published by Elsevier Inc. All rights reserved.)

Objective: The purpose of our study was to assess diagnostic performance when retrospectively interpreting full-field digital mammography (FFDM) and breast tomosynthesis examinations under a free-response receiver operating characteristic (FROC) paradigm., Materials and Methods: We performed FROC analysis of a previously reported study in which eight experienced radiologists interpreted 125 examinations, including 35 with verified cancers. The FROC paradigm involves detecting, locating, and rating each suspected abnormality. Radiologists reviewed and rated both FFDM alone and a combined display mode of FFDM and digital breast tomosynthesis (DBT) (combined). Observer performance levels were assessed and compared with respect to the fraction of correctly identified abnormalities, the number of reported location-specific findings (both true and false), and their associated ratings. The analysis accounts for the number and locations of findings and the location-based ratings using a summary performance index (Λ), which is the FROC analog of the area between the receiver operating characteristic curve and the diagonal (chance) line., Results: Under the FROC paradigm, each reader detected more true abnormalities associated with cancer, or a higher true-positive fraction, under the combined mode. In an analysis focused on both the number of findings and associated location-based ratings, each of the radiologists performed better under the combined mode compared with FFDM alone, with increases in Λ ranging from 5% to 34%. On average, under the combined mode radiologists achieved a 16% improvement in Λ compared with the FFDM alone mode (95% CI, 7-26%; p < 0.01)., Conclusion: We showed that DBT-based breast imaging in combination with FFDM could result in better performance under the FROC paradigm.

Purpose: When comparing binary test results from two diagnostic systems, superiority in both "sensitivity" and "specificity" also implies differences in all conventional summary indices and locally in the underlying receiver operating characteristics (ROC) curves. However, when one of the two binary tests has higher sensitivity and lower specificity (or vice versa), comparisons of their performance levels are nontrivial and the use of different summary indices may lead to contradictory conclusions. A frequently used approach that is free of subjectivity associated with summary indices is based on the comparison of the underlying ROC curves that requires the collection of rating data using multicategory scales, whether natural or experimentally imposed. However, data for reliable estimation of ROC curves are frequently unavailable. The purpose of this article is to develop an approach of using "diagnostic likelihood ratios", namely, likelihood ratios of "positive" or "negative" responses, to make simple inferences regarding the underlying ROC curves and associated areas in the absence of reliable rating data or regarding the relative binary characteristics, when these are of primary interest., Methods: For inferences related to underlying curves, the authors exploit the assumption of concavity of the true underlying ROC curve to describe conditions under which these curves have to be different and under which the curves have different areas. For scenarios when the binary characteristics are of primary interest, the authors use characteristics of "chance performance" to demonstrate that the derived conditions provide strong evidence of superiority of one binary test as compared to another. By relating these derived conditions to hypotheses about the true likelihood ratios of two binary diagnostic tests being compared, the authors enable a straightforward statistical procedure for the corresponding inferences., Results: The authors derived simple algebraic and graphical methods for describing the conditions for superiority of one of two diagnostic tests with respect to their binary characteristics, the underlying ROC curves, or the areas under the curves. The graphical regions are useful for identifying potential differences between two systems, which then have to be tested statistically. The simple statistical tests can be performed with well known methods for comparison of diagnostic likelihood ratios. The developed approach offers a solution for some of the more difficult to analyze scenarios, where diagnostic tests do not demonstrate concordant differences in terms of both sensitivity and specificity. In addition, the resulting inferences do not contradict the conclusions that can be obtained using conventional and reasonably defined summary indices., Conclusions: When binary diagnostic tests are of primary interest, the proposed approach offers an objective and powerful method for comparing two binary diagnostic tests. The significant advantage of this method is that it enables objective analyses when one test has higher sensitivity but lower specificity, while ensuring agreement with study conclusions based on other reasonable and widely acceptable summary indices. For truly multicategory diagnostic tests, the proposed method can help in concluding inferiority of one of the diagnostic tests based on binary data, thereby potentially saving the need for conducting a more expensive multicategory ROC study.

Rationale and Objectives: The aim of this study was to assess similarities and differences between methods of performance comparisons under binary (yes or no) and receiver-operating characteristic (ROC)-type pseudocontinuous (0-100) rating data ascertained during an observer performance study of interpretation of full-field digital mammography (FFDM) versus FFDM plus digital breast tomosynthesis., Materials and Methods: Rating data consisted of ROC-type pseudocontinuous and binary ratings generated by eight radiologists evaluating 77 digital mammographic examinations. Overall performance levels were summarized with a conventionally used probability of correct discrimination or, equivalently, the area under the ROC curve (AUC), which under a binary scale is related to Youden's index. Magnitudes of differences in the reader-averaged empirical AUCs between FFDM alone and FFDM plus digital breast tomosynthesis were compared in the context of fixed-reader and random-reader variability of the estimates., Results: The absolute differences between modes using the empirical AUCs were larger on average for the binary scale (0.12 vs 0.07) and for the majority of individual readers (six of eight). Standardized differences were consistent with this finding (2.32 vs 1.63 on average). Reader-averaged differences in AUCs standardized by fixed-reader and random-reader variances were also smaller under the binary rating paradigm. The discrepancy between AUC differences depended on the location of the reader-specific binary operating points., Conclusions: The human observer's operating point should be a primary consideration in designing an observer performance study. Although in general, the ROC-type rating paradigm provides more detailed information on the characteristics of different modes, it does not reflect the actual operating point adopted by human observers. There are application-driven scenarios in which analysis based on binary responses may provide statistical advantages., (Copyright 2010 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To compare time to interpretation and diagnostic performance levels during repeat readings of full-field digital mammography (FFDM) and digital breast tomosynthesis (DBT) in a retrospective study., Materials and Methods: Three experienced radiologists twice interpreted 125 selected examinations, 35 with verified cancers and 90 negative for cancer during a period of 22 months using FFDM alone followed by a combined FFDM + DBT mode. Changes in time to "review and rate" these examinations as well as in diagnostic performance levels where assessed. A fixed-effect analysis accounting for cross-correlation due to the review of the same examinations by the same readers was performed., Results: The total (combined) time to review and rate an examination increased on average by 33% between the first and second readings of the same examinations (P < .001). Radiologists reduced their time to review FFDM before making the DBT available for viewing. However, they spent more time reviewing the combined FFDM + DBT mode. The recall rates for examinations depicting cancer remained largely unchanged. Among the groups of examinations with concordant and discordant recall recommendations during the two readings only the group examinations that were "newly recalled" during repeat reading, took significantly longer (P < .01)., Conclusion: DBT-based breast imaging may ultimately result in a substantial increase in performance; however, without efficiency improvements DBT may take longer to interpret. Addition of "false-positive recalls" was most strongly associated with increase in interpretation time while elimination of "false-positive recalls" did not require longer interpretation time., (Copyright 2010 AUR. Published by Elsevier Inc. All rights reserved.)

Objective: The purpose of this study was to compare in a retrospective observer study the diagnostic performance of full-field digital mammography (FFDM) with that of digital breast tomosynthesis., Materials and Methods: Eight experienced radiologists interpreted images from 125 selected examinations, 35 with verified findings of cancer and 90 with no finding of cancer. The four display conditions included FFDM alone, 11 low-dose projections, reconstructed digital breast tomosynthesis images, and a combined display mode of FFDM and digital breast tomosynthesis images. Observers rated examinations using the screening BI-RADS rating scale and the free-response receiver operating characteristic paradigm. Observer performance levels were measured as the proportion of examinations prompting recall of patients for further diagnostic evaluation. The results were presented in terms of true-positive fraction and false-positive fraction. Performance levels were compared among the acquisitions and reading modes. Time to view and interpret an examination also was evaluated., Results: Use of the combination of digital breast tomosynthesis and FFDM was associated with 30% reduction in recall rate for cancer-free examinations that would have led to recall if FFDM had been used alone (p < 0.0001 for the participating radiologists, p = 0.047 in the context of a generalized population of radiologists). Use of digital breast tomosynthesis alone also tended to reduce recall rates, an average of 10%, although the observed decrease was not statistically significant (p = 0.09 for the participating radiologists). There was no convincing evidence that use of digital breast tomosynthesis alone or in combination with FFDM results in a substantial improvement in sensitivity., Conclusion: Use of digital breast tomosynthesis for breast imaging may result in a substantial decrease in recall rate.

Free-response assessment of diagnostic systems continues to gain acceptance in areas related to the detection, localization, and classification of one or more "abnormalities" within a subject. A free-response receiver operating characteristic (FROC) curve is a tool for characterizing the performance of a free-response system at all decision thresholds simultaneously. Although the importance of a single index summarizing the entire curve over all decision thresholds is well recognized in ROC analysis (e.g., area under the ROC curve), currently there is no widely accepted summary of a system being evaluated under the FROC paradigm. In this article, we propose a new index of the free-response performance at all decision thresholds simultaneously, and develop a nonparametric method for its analysis. Algebraically, the proposed summary index is the area under the empirical FROC curve penalized for the number of erroneous marks, rewarded for the fraction of detected abnormalities, and adjusted for the effect of the target size (or "acceptance radius"). Geometrically, the proposed index can be interpreted as a measure of average performance superiority over an artificial "guessing" free-response process and it represents an analogy to the area between the ROC curve and the "guessing" or diagonal line. We derive the ideal bootstrap estimator of the variance, which can be used for a resampling-free construction of asymptotic bootstrap confidence intervals and for sample size estimation using standard expressions. The proposed procedure is free from any parametric assumptions and does not require an assumption of independence of observations within a subject. We provide an example with a dataset sampled from a diagnostic imaging study and conduct simulations that demonstrate the appropriateness of the developed procedure for the considered sample sizes and ranges of parameters.

Rationale and Objectives: To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms., Materials and Methods: We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations., Results: All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733)., Conclusion: The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

Purpose: To compare radiologists' performance during interpretation of screening mammograms in the clinic with their performance when reading the same mammograms in a retrospective laboratory study., Materials and Methods: This study was conducted under an institutional review board-approved, HIPAA-compliant protocol; the need for informed consent was waived. Nine experienced radiologists rated an enriched set of mammograms that they had personally read in the clinic (the "reader-specific" set) mixed with an enriched "common" set of mammograms that none of the participants had previously read in the clinic by using a screening Breast Imaging Reporting and Data System (BI-RADS) rating scale. The original clinical recommendations to recall the women for a diagnostic work-up, for both reader-specific and common sets, were compared with their recommendations during the retrospective experiment. The results are presented in terms of reader-specific and group-averaged sensitivity and specificity levels and the dispersion (spread) of reader-specific performance estimates., Results: On average, the radiologists' performance was significantly better in the clinic than in the laboratory (P = .035). Interreader dispersion of the computed performance levels was significantly lower during the clinical interpretations (P < .01)., Conclusion: Retrospective laboratory experiments may not represent either expected performance levels or interreader variability during clinical interpretations of the same set of mammograms in the clinical environment well., ((c) RSNA, 2008.)

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

The task of searching and detecting multiple abnormalities depicted on an image, or a series of images, is a common problem in different areas such as military target detection or diagnostic medical imaging. A free response receiver operating characteristic (FROC) approach for assessing performance in many of these scenarios entails marking the locations of suspected abnormalities and indicating a level of suspicion at each of the marked locations. One of the important characteristics of a system being evaluated under the FROC paradigm is its performance in the conventional ROC domain, namely classifying a subject (or a unit of interest) as "negative" or "positive" in regard to the presence of the abnormality (or any of the abnormalities) of interest. With FROC data we can compare subjects by specifying a function of multiple scores within a subject. This approach allows formulating subject-based ROC type indices that can be estimated using existing ROC concepts. In this article we focus on indices that reflect the ability of the system to discriminate between actually negative and actually positive subjects. We consider a previously proposed index that is based on the comparison of the highest scores on subjects and two new indices that are based on potentially more stable comparison functions, namely comparison of average scores and stochastic dominance. Based on these indices we develop nonparametric procedures for comparing subject-based discriminative ability of diagnostic systems being evaluated under the FROC paradigm. We also investigate the properties of the statistical procedures in a simulation study.

A specific issue related to the selection of the analytic tool used when comparing the estimated performance of systems within the receiver operating characteristic (ROC) paradigm is reviewed. This issue is related to the possible effect of the last experimentally ascertained ROC data point in terms of highest true-positive and false-positive fractions. An example of a case is presented where the selection of a specific analysis approach could affect the study conclusion from being not statistically significant for parametric analysis and significant for nonparametric analysis. This is followed by recommendations that should help prevent misinterpretation of results.