Search

Rationale and Objectives: To preliminarily asses if Contrast Enhanced Digital Mammography (CEDM) can accurately reduce biopsy rates for soft tissue BI-RADS 4A or 4B lesions., Materials and Methods: Eight radiologists retrospectively and independently reviewed 60 lesions in 54 consenting patients who underwent CEDM under Health Insurance Portability and Accountability Act compliant institutional review board-approved protocols. Readers provided Breast Imaging Reporting & Data System ratings sequentially for digital mammography/digital breast tomosynthesis (DM/DBT), then with ultrasound, then with CEDM for each lesion. Area under the curve (AUC), true positive rates and false positive rates, positive predictive values and negative predictive values were calculated. Statistical analysis accounting for correlation between lesion-examinations and between-reader variability was performed using OR/DBM (for SAS v.3.0), generalized linear mixed model for binary data (proc glimmix, SAS v.9.4, SAS Institute, Cary North Carolina), and bootstrap., Results: The cohort included 49 benign, two high-risk and nine cancerous lesions in 54 women aged 34-74 (average 50) years. Reader-averaged AUC for CEDM was significantly higher than DM/DBT alone (0.85 versus 0.66, p < 0.001) or with US (0.85 versus 0.75, p = 0.001). CEDM increased true positive rates from 0.74 under DB/DBT, and 0.89 with US, to 0.90 with CEDM, (p = 0.019 DM/DBT versus CEDM, p = 0.78 DM/DBT + US versus CEDM) and decreased false positive rates from 0.47 using DM/DBT and 0.61 with US to 0.39 with CEDM (p = 0.017 DM/DBT versus CEDM, p = 0.001 DM/DBT+ US versus CEDM). For an expected cancer rate of 10%, CEDM positive predictive values was 20.5% (95% CI: 16%-27%) and negative predictive values 98.3% (95% CI: 96%-100%)., Conclusion: Addition of CEDM for evaluation of low-moderate suspicion soft tissue breast lesions can substantially reduce biopsy of benign lesions without compromising cancer detection., (Copyright © 2019 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: The "binormal" model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have "improper" (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve., Materials and Methods: We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1, we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data., Results: We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage., Conclusions: The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences., (Copyright © 2017 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: Assess results of a prospective, single-site clinical study evaluating digital breast tomosynthesis (DBT) during baseline screening mammography., Materials and Methods: Under an institutional review board-approved Health Insurance Portability and Accountability Act (HIPAA)-compliant protocol, consenting women between ages 34 and 56 years scheduled for their initial and/or baseline screening mammogram underwent both full field digital mammography (FFDM) and DBT. The FFDM and the FFDM plus DBT images were interpreted independently in a reader by mode balanced approach by two of 14 participating radiologists. A woman was recalled for a diagnostic work-up if either radiologist recommended a recall. We report overall recall rates and related diagnostic outcome from the 1080 participants. Proportion of recommended recalls (Breast Imaging Reporting and Data System 0) were compared using a generalized linear mixed model (SAS 9.3) with a significance level of P = .0294., Results: The fraction of women without breast cancer recommended for recall using FFDM alone and FFDM plus DBT were 412 of 1074 (38.4%) and 274 of 1074 (25.5%), respectively (P < .001). Large inter-reader variability in terms of recall reduction was observed among the 14 readers; however, 11 of 14 readers recalled fewer women using FFDM plus DBT (5 with P < .015). Six cancers (four ductal carcinomas in situ [DCIS] and two invasive ductal carcinomas [IDC]) were detected. One IDC was detected only on DBT and one DCIS cancer was detected only on FFDM, whereas the remaining cancers were detected on both modalities., Conclusions: The use of FFDM plus DBT resulted in a significant decrease in recall rates during baseline screening mammography with no reduction in sensitivity., (Copyright © 2015 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To assess radiologists' perceptions of how the new Breast Density Notification Act (BDNA) of Pennsylvania would affect their breast density reporting and their actual reporting patterns after implementation., Materials and Methods: Under an institutional review board-approved protocol, we surveyed 21 radiologists about how they believe the new law affected their breast density reporting patterns and analyzed actual changes for 16 respondents before and after the law took effect. Three hundred consecutive reports were assessed for each radiologist before and after the effective date. The distributions of reported density Breast Imaging Reporting and Data System (BI-RADS) (1-4) were compared using a type III test in the context of an ordinal mixed model accounting for between-reader variability and adjusting for age (PROC GLIMMIX, SAS, version 9.3) using a two-sided .05 significance level., Results: Seventeen radiologists responded to the survey; however, one retired shortly after responding. Of the 16 respondents, 56% (nine of 16) did not favor the law, 13% (two of 16) were in favor, and 31% (five of 16) were neutral. The fraction perceived that after implementation, they rated more, equally, or less frequently breasts as scattered fibroglandular densities (BI-RADS 2) versus heterogeneously dense rating (BI-RADS 3) was 50% (eight of 16), 44% (seven of 16), and 6% (one of 16), respectively. In practice, 44% (seven of 16) performed differently than their survey answers. Fourteen of 16 radiologists increased the frequency of reported BI-RADS 2 scores after BDNA implementation with seven having statistically significant (P < .05) increases after adjusting for age differences., Conclusions: Radiologists' reporting patterns changed, at least for a short duration, after the new density reporting law and for some of the radiologists in an unexpected way., (Copyright © 2015 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To assess the interaction between the availability of prior examinations and digital breast tomosynthesis (DBT) in decisions to recall a woman during interpretation of mammograms., Materials and Methods: Eight radiologists independently interpreted twice 36 mammography examinations, each of which had current and prior full-field digital mammography images (FFDM) and DBT under a Health Insurance Portability and Accountability Act-compliant, institutional review board-approved protocol (written consent waived). During the first reading, three sequential ratings were provided using FFDM only, followed by FFDM + DBT, and then followed by FFDM + DBT + priors. The second reading included FFDM only, then FFDM + priors, and then FFDM + priors + DBT. Twenty-two benign cases clinically recalled, 12 negative/benign examinations (not recalled), and two verified cancer cases were included. Recall recommendations and interaction between the effect of priors and DBT on decisions were assessed (P = .05 significance level) using generalized linear model (PROC GLIMMIX, SAS, version 9.3; SAS Institute, Cary, NC) accounting for case and reader variability., Results: Average recall rates in noncancer cases were significantly reduced (51%; P < .001) with the addition of DBT and with addition of priors (23%; P = .01). In absolute terms, the addition of DBT to FFDM reduced the recall rates from 0.67 to 0.42 and from 0.54 to 0.27 when DBT was available before and after priors, respectively. Recall reductions were from 0.64 to 0.54 and from 0.42 to 0.33 when priors were available before and after DBT, respectively. Regardless of the sequence in presentation, there were no statistically significant interactions between the effect of availability of DBT and priors (P = .80)., Conclusions: Availability of both priors and DBT are independent primary factors in reducing recall recommendations during mammographic interpretations., (Copyright © 2014 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: The aim of this study was to retrospectively compare the interpretive performance of synthetically reconstructed two-dimensional images in combination with digital breast tomosynthesis (DBT) versus full-field digital mammography (FFDM) plus DBT., Materials and Methods: Ten radiologists trained in reading tomosynthesis examinations interpreted retrospectively, under two modes, 114 mammograms. One mode included the directly acquired full-field digital mammograms combined with DBT, and the other included synthetically reconstructed projection images combined with DBT. The reconstructed images do not require additional radiation exposure. The two modes were compared with respect to sensitivity, namely, recommendation to recall a breast with either a pathology-proven cancer (n = 48) or a high-risk lesion (n = 6), and specificity, namely, no recommendation to recall a breast not depicting an abnormality (n = 144) or depicting only benign abnormalities (n = 30)., Results: The average sensitivity for FFDM with DBT was 0.826, compared to 0.772 for synthetic FFDM with DBT (difference, 0.054; P = .017 and P = .053 for fixed and random reader effects, respectively). The proportions of breasts with no or benign abnormalities recommended to be recalled were virtually the same: 0.298 and 0.297 for the two modalities, respectively (95% confidence intervals for the difference, -0.028 to 0.036 and -0.070 to 0.066 for fixed and random reader effects, respectively). Sixteen additional clusters of microcalcifications ("positive" breasts) were missed by all readers combined when interpreting the mode with synthesized images versus FFDM., Conclusions: Lower sensitivity with comparable specificity was observed with the tested version of synthetically generated images compared to FFDM, both combined with DBT. Improved synthesized images with experimentally verified acceptable diagnostic quality will be needed to eliminate double exposure during DBT-based screening., (Copyright © 2012 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: The aim of this study was to assess similarities and differences between methods of performance comparisons under binary (yes or no) and receiver-operating characteristic (ROC)-type pseudocontinuous (0-100) rating data ascertained during an observer performance study of interpretation of full-field digital mammography (FFDM) versus FFDM plus digital breast tomosynthesis., Materials and Methods: Rating data consisted of ROC-type pseudocontinuous and binary ratings generated by eight radiologists evaluating 77 digital mammographic examinations. Overall performance levels were summarized with a conventionally used probability of correct discrimination or, equivalently, the area under the ROC curve (AUC), which under a binary scale is related to Youden's index. Magnitudes of differences in the reader-averaged empirical AUCs between FFDM alone and FFDM plus digital breast tomosynthesis were compared in the context of fixed-reader and random-reader variability of the estimates., Results: The absolute differences between modes using the empirical AUCs were larger on average for the binary scale (0.12 vs 0.07) and for the majority of individual readers (six of eight). Standardized differences were consistent with this finding (2.32 vs 1.63 on average). Reader-averaged differences in AUCs standardized by fixed-reader and random-reader variances were also smaller under the binary rating paradigm. The discrepancy between AUC differences depended on the location of the reader-specific binary operating points., Conclusions: The human observer's operating point should be a primary consideration in designing an observer performance study. Although in general, the ROC-type rating paradigm provides more detailed information on the characteristics of different modes, it does not reflect the actual operating point adopted by human observers. There are application-driven scenarios in which analysis based on binary responses may provide statistical advantages., (Copyright 2010 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To compare time to interpretation and diagnostic performance levels during repeat readings of full-field digital mammography (FFDM) and digital breast tomosynthesis (DBT) in a retrospective study., Materials and Methods: Three experienced radiologists twice interpreted 125 selected examinations, 35 with verified cancers and 90 negative for cancer during a period of 22 months using FFDM alone followed by a combined FFDM + DBT mode. Changes in time to "review and rate" these examinations as well as in diagnostic performance levels where assessed. A fixed-effect analysis accounting for cross-correlation due to the review of the same examinations by the same readers was performed., Results: The total (combined) time to review and rate an examination increased on average by 33% between the first and second readings of the same examinations (P < .001). Radiologists reduced their time to review FFDM before making the DBT available for viewing. However, they spent more time reviewing the combined FFDM + DBT mode. The recall rates for examinations depicting cancer remained largely unchanged. Among the groups of examinations with concordant and discordant recall recommendations during the two readings only the group examinations that were "newly recalled" during repeat reading, took significantly longer (P < .01)., Conclusion: DBT-based breast imaging may ultimately result in a substantial increase in performance; however, without efficiency improvements DBT may take longer to interpret. Addition of "false-positive recalls" was most strongly associated with increase in interpretation time while elimination of "false-positive recalls" did not require longer interpretation time., (Copyright 2010 AUR. Published by Elsevier Inc. All rights reserved.)

Rationale and Objectives: To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms., Materials and Methods: We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic ("individualized") mixed with a set that none of them had read in the clinic ("common set"). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations., Results: All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval -0.078 to 0.733)., Conclusion: The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

We have observed that a very large fraction of responses for several detection tasks during the performance of observer studies are in the extreme ranges of lower than 11% or higher than 89% regardless of the actual presence or absence of the abnormality in question or its subjectively rated "subtleness." This observation raises questions regarding the validity and appropriateness of using multicategory rating scales for such detection tasks. Monte Carlo simulation of binary and multicategory ratings for these tasks demonstrate that the use of the former (binary) often results in a less biased and more precise summary index and hence may lead to a higher statistical power for determining differences between modalities.

Rationale and Objectives: We sought to assess whether or not prevalence levels affected the confidence ratings of readers during the interpretation of cases in a laboratory receiver operating characteristic-type observer performance study., Materials and Methods: We reanalyzed a previously conducted observer performance study that included 14 readers and 5 different levels of prevalence. The previous study yielded the observation that in the laboratory we could not detect a "prevalence effect" in terms of differences in areas under the receiver operating characteristic curves. The detection ratings (for presence or absence) of lung nodules, interstitial disease, and pneumothorax for the five prevalence levels were compared, and a test for trend in averaged ratings as a function of abnormality prevalence was performed within a mixed-model setting that accounts for different sources of variability and correlations induced by the study design., Results: The ratings of the cases in terms of confidence that the specific abnormality in question is present tend, on average, to be larger when actual disease prevalence is lower. The rate of the increase of the average confidence ratings with the decreasing prevalence of a specific abnormality is very similar for actually positive and actually negative cases for every considered abnormality. The observed trend in the changes of the average confidence ratings as a function of prevalence levels was statistically significant (p < 0.01)., Conclusion: Expectations of disease prevalence in the case mix during a laboratory observer performance study may systematically affect the behavior of observers in terms of their actual confidence ratings.

Rationale and Objectives: To investigate the variability between discriminative performances of readers as a function of average performance levels during receiver operating characteristic (ROC) studies., Materials and Methods: Four subsets of cases from previously ascertained ROC rating data by 12 observers when detecting interstitial disease and pneumothorax on posteroanterior chest films were selected for each abnormality and reanalyzed to assess changes in "reader" variance component. The subsets were selected based on a prestudy subjective assessment of the subtleness of depicted abnormality (positive cases) and the difficulty in determining its absence (negative cases). Reader variance component was estimated using a bootstrap approach for each subset and the results were used to assess a general relationship between variability and average performance level., Results: The reader variance component decreased substantially (from 0.007704 to 0.000426), as expected, when the areas under the ROC curves (AUC) for detecting pneumothoraces increased from 84% to 97%. On the other hand, reader variance component increased substantially (from 0.000890 to 0.005181) when AUC for detecting interstitial disease increased from 59% to 87%. The large magnitude of and changes in the reader variance component resulted in a consistent nonmonotone relationship as a function of AUC when other related variance components were included in addition to the reader component., Conclusion: Among several factors affecting generalizability of ROC results to the population of readers, the reader variance component depended nonmonotonically on the average diagnostic performance and is lowest at both very high and very low levels of performance.

Rationale and Objectives: The aim of the study is to develop a permutation test to compare receiver operating characteristic (ROC) curves of two diagnostic modalities in a multireader paired design., Materials and Methods: A statistical test for comparing two diagnostic modalities is developed based on all possible exchanges of the set of reader-ratings between the two modalities. An exact permutation test is formed by determining the frequency of the most extreme values of the statistic estimating the average difference in the areas under the ROC curves (AUCs). An asymptotic version of the test is constructed by obtaining the exact permutation variance and appealing to the asymptotic normality of the nonparametric estimator of the average difference in areas. Computer simulations were conducted to validate the type I error for small sample sizes., Results: The new test provides a permutation approach for comparing ROC curves in a multireader paired-design setting in which effects of the readers are considered to be fixed. The type I error of the asymptotic test is close to the true value, even for samples as small as 20 normal and 20 abnormal cases. The test is designed to be sensitive to alternatives in which the AUCs of the two diagnostic modalities differ., Conclusions: The proposed test provides a powerful method for comparing two diagnostic modalities in a multireader paired-study design when the primary interest is to detect difference in average AUCs.

Rationale and Objectives: The aim of the study is to assess variance components in observer performance studies and the possible impact on study results and conclusions., Materials and Methods: Two previously performed retrospective receiver operating characteristic-type observer performance studies to evaluate the performance of seven radiologists in detecting interstitial disease on conventional posteroanterior chest films and nine radiologists in detecting interstitial disease on a high-resolution workstation were reanalyzed by using the Beiden, Wagner, and Campbell nine-component model to estimate the different variance components. We estimated case-, reader-, and mode-related components of the variance for the group as a whole and after excluding (round robin) each reader. Overall variance was evaluated, and the effect of individual readers on overall study conclusions was assessed., Results: Overall results and conclusions of the reanalysis agreed with the original one in that, as a group, radiologists performed significantly better when using conventional films (P < .05) in both studies. Reader variability was large compared with all other components, and in one study, it was substantially larger for the workstation reading mode. Reader variability was affected substantially by one observer in each study, and in one study, reader-by-mode variability was affected by another reader who performed better on the workstation., Conclusion: Estimates of variance components can shed light on the appropriateness of study design, as well as the sensitivity of results to the inclusion (or exclusion) of individual observers.

Rationale and Objectives: We sought to develop a new index that incorporates utility-weights when assessing the overall performance of a diagnostic system and to provide a statistical test for comparing two indices in a paired study design., Materials and Methods: The area under the receiver operating characteristic (ROC) curve (AUC) was used as the basis for constructing a new index. The index we propose represents a weighted average of class-specific AUCs each of which relates to a class of pairs of actually negative (normal) and actually positive (abnormal) cases with a specific predetermined utility (or clinical importance). For each pair of normal-abnormal cases, the utility is defined a priori and based on external (covariate) information. In the proposed approach utility-weights represent the relative importance (utility) of discriminating between different types of normal and abnormal cases (pairs of the same type are combined in the classes termed utility-classes). We also describe a simple nonparametric procedure for comparing the proposed indices as computed from paired data. Computer simulations were conducted to evaluate the behavior of the type I error of the proposed test in the simple albeit important instance of two utility-classes., Results: The new index provides an extension of the commonly used area under the ROC curve. It allows for incorporation of utility-weights into the analysis and reduces to the conventional AUC index when all assigned utility-weights are equal to unity. Computer simulations indicate that in the considered scenario of two utility-classes, the type I error of the proposed test is comparable to that of the conventional nonparametric test for equality of AUC indices., Conclusions: The proposed index and the statistical test provide a practical approach of incorporating utilities when comparing diagnostic systems.

Rationale and Objectives: To develop a conditional nonparametric procedure for comparing two correlated areas under receiver operating characteristic (ROC) curves (AUC)., Materials and Methods: A nonparametric conditional test to compare areas under two ROC curves was developed using the distribution of the elements of the nonparametric AUC estimators in a permutation space. The conditioning is made on the observed discordances between the relative orderings of ratings of the normal and abnormal cases for the two modalities taken over all possible pairs. The type I error of the procedure was verified using computer simulations. The power of the test was compared with an existing unconditional procedure on simulated datasets from binormal distributions as well as from a mixture of binormal distributions of ratings., Results: The proposed test is conservative for low sample sizes, large AUC, and high correlation between modalities. It possesses a reasonable type I error for sample sizes as low as 20 actually positive and 20 actually negative cases. In plausible situations in which the sample in observer performance studies can not be monotonically transformed into a binormal distribution, this approach may have modest power advantages over the conventional nonparametric test., Conclusion: The conditional nonparametric test presented here is an alternative approach to existing unconditional procedures and may offer advantages in certain types of observer performance studies.