Search

PURPOSE To assess diagnostic performance of digital breast tomosynthesis (DBT) alone or combined with technologist-performed handheld screening ultrasound (US) in women with dense breasts. METHODS In an institutional review board–approved, Health Insurance Portability and Accountability Act–compliant multicenter protocol in western Pennsylvania, 6,179 women consented to three rounds of annual screening, interpreted by two radiologist observers, and had appropriate follow-up. Primary analysis was based on first observer results. RESULTS Mean participant age was 54.8 years (range, 40-75 years). Across 17,552 screens, there were 126 cancer events in 125 women (7.2/1,000; 95% CI, 5.9 to 8.4). In year 1, DBT-alone cancer yield was 5.0/1,000, and of DBT+US, 6.3/1,000, difference 1.3/1,000 (95% CI, 0.3 to 2.1; P = .005). In years 2 + 3, DBT cancer yield was 4.9/1,000, and of DBT+US, 5.9/1,000, difference 1.0/1,000 (95% CI, 0.4 to 1.5; P < .001). False-positive rate increased from 7.0% for DBT in year 1 to 11.5% for DBT+US and from 5.9% for DBT in year 2 + 3 to 9.7% for DBT+US ( P < .001 for both). Nine cancers were seen only by double reading DBT and one by double reading US. Ten interval cancers (0.6/1,000 [95% CI, 0.2 to 0.9]) were identified. Despite reduction in specificity, addition of US improved receiver operating characteristic curves, with area under receiver operating characteristic curve increasing from 0.83 for DBT alone to 0.92 for DBT+US in year 1 ( P = .01), with smaller improvements in subsequent years. Of 6,179 women, across all 3 years, 172/6,179 (2.8%) unique women had a false-positive biopsy because of DBT as did another 230/6,179 (3.7%) women because of US ( P < .001). CONCLUSION Overall added cancer detection rate of US screening after DBT was modest at 19/17,552 (1.1/1,000; CI, 0.5- to 1.6) screens but potentially overcomes substantial increases in false-positive recalls and benign biopsies.

Objective For breast US interpretation, to assess impact of computer-aided diagnosis (CADx) in original mode or with improved sensitivity or specificity. Methods In this IRB approved protocol, orthogonal-paired US images of 319 lesions identified on screening, including 88 (27.6%) cancers (median 7 mm, range 1–34 mm), were reviewed by 9 breast imaging radiologists. Each observer provided BI-RADS assessments (2, 3, 4A, 4B, 4C, 5) before and after CADx in a mode-balanced design: mode 1, original CADx (outputs benign, probably benign, suspicious, or malignant); mode 2, artificially-high-sensitivity CADx (benign or malignant); and mode 3, artificially-high-specificity CADx (benign or malignant). Area under the receiver operating characteristic curve (AUC) was estimated under each modality and for standalone CADx outputs. Multi-reader analysis accounted for inter-reader variability and correlation between same-lesion assessments. Results AUC of standalone CADx was 0.77 (95% CI: 0.72–0.83). For mode 1, average reader AUC was 0.82 (range 0.76–0.84) without CADx and not significantly changed with CADx. In high-sensitivity mode, all observers’ AUCs increased: average AUC 0.83 (range 0.78–0.86) before CADx increased to 0.88 (range 0.84–0.90), P < 0.001. In high-specificity mode, all observers’ AUCs increased: average AUC 0.82 (range 0.76–0.84) before CADx increased to 0.89 (range 0.87–0.92), P < 0.0001. Radiologists responded more frequently to malignant CADx cues in high-specificity mode (42.7% vs 23.2% mode 1, and 27.0% mode 2, P = 0.008). Conclusion Original CADx did not substantially impact radiologists’ interpretations. Radiologists showed improved performance and were more responsive when CADx produced fewer false-positive malignant cues.

Objective Using terms adapted from the BI-RADS Mammography and MRI lexicons, we trained radiologists to interpret contrast-enhanced mammography (CEM) and assessed reliability of their description and assessment. Methods A 60-minute presentation on CEM and terminology was reviewed independently by 21 breast imaging radiologist observers. For 21 CEM exams with 31 marked findings, observers recorded background parenchymal enhancement (BPE) (minimal, mild, moderate, marked), lesion type (oval/round or irregular mass, or non-mass enhancement), intensity of enhancement (none, weak, medium, strong), enhancement quality (none, homogeneous, heterogeneous, rim), and BI-RADS assessment category (2, 3, 4A, 4B, 4C, 5). “Expert” consensus of 3 other radiologists experienced in CEM was developed. Kappa statistic was used to assess agreement between radiologists and expert consensus, and between radiologists themselves, on imaging feature categories and final assessments. Reproducibility of specific feature descriptors was assessed as fraction of consensus-concordant responses. Results Radiologists demonstrated moderate agreement for BPE, (mean kappa, 0.43; range, 0.05–0.69), and lowest reproducibility for “minimal.” Agreement was substantial for lesion type (mean kappa, 0.70; range, 0.47–0.93), moderate for intensity of enhancement (mean kappa, 0.57; range, 0.44–0.76), and moderate for enhancement quality (mean kappa, 0.59; range, 0.20–0.78). Agreement on final assessment was fair (mean kappa, 0.26; range, 0.09–0.44), with BI-RADS category 3 the least reproducible. Decision to biopsy (BI-RADS 2–3 vs 4–5) showed moderate agreement with consensus (mean kappa, 0.54; range, −0.06–0.87). Conclusion With minimal training, agreement for description of CEM findings by breast imaging radiologists was comparable to other BI-RADS lexicons.

Objective To develop and test computer software to detect, quantify, and monitor progression of pneumonia associated with COVID-19 using chest CT scans. Methods One hundred twenty chest CT scans from subjects with lung infiltrates were used for training deep learning algorithms to segment lung regions and vessels. Seventy-two serial scans from 24 COVID-19 subjects were used to develop and test algorithms to detect and quantify the presence and progression of infiltrates associated with COVID-19. The algorithm included (1) automated lung boundary and vessel segmentation, (2) registration of the lung boundary between serial scans, (3) computerized identification of the pneumonitis regions, and (4) assessment of disease progression. Agreement between radiologist manually delineated regions and computer-detected regions was assessed using the Dice coefficient. Serial scans were registered and used to generate a heatmap visualizing the change between scans. Two radiologists, using a five-point Likert scale, subjectively rated heatmap accuracy in representing progression. Results There was strong agreement between computer detection and the manual delineation of pneumonic regions with a Dice coefficient of 81% (CI 76–86%). In detecting large pneumonia regions (> 200 mm3), the algorithm had a sensitivity of 95% (CI 94–97%) and specificity of 84% (CI 81–86%). Radiologists rated 95% (CI 72 to 99) of heatmaps at least “acceptable” for representing disease progression. Conclusion The preliminary results suggested the feasibility of using computer software to detect and quantify pneumonic regions associated with COVID-19 and to generate heatmaps that can be used to visualize and assess progression. Key Points • Both computer vision and deep learning technology were used to develop computer software to quantify the presence and progression of pneumonia associated with COVID-19 depicted on CT images. • The computer software was tested using both quantitative experiments and subjective assessment. • The computer software has the potential to assist in the detection of the pneumonic regions, monitor disease progression, and assess treatment efficacy related to COVID-19. Electronic supplementary material The online version of this article (10.1007/s00330-020-07156-2) contains supplementary material, which is available to authorized users.

To define the uniqueness of chest CT infiltrative features associated with COVID-19 image characteristics as potential diagnostic biomarkers. We retrospectively collected chest CT exams including n = 498 on 151 unique patients RT-PCR positive for COVID-19 and n = 497 unique patients with community-acquired pneumonia (CAP). Both COVID-19 and CAP image sets were partitioned into three groups for training, validation, and testing respectively. In an attempt to discriminate COVID-19 from CAP, we developed several classifiers based on three-dimensional (3D) convolutional neural networks (CNNs). We also asked two experienced radiologists to visually interpret the testing set and discriminate COVID-19 from CAP. The classification performance of the computer algorithms and the radiologists was assessed using the receiver operating characteristic (ROC) analysis, and the nonparametric approaches with multiplicity adjustments when necessary. One of the considered models showed non-trivial, but moderate diagnostic ability overall (AUC of 0.70 with 99% CI 0.56–0.85). This model allowed for the identification of 8–50% of CAP patients with only 2% of COVID-19 patients. Professional or automated interpretation of CT exams has a moderately low ability to distinguish between COVID-19 and CAP cases. However, the automated image analysis is promising for targeted decision-making due to being able to accurately identify a sizable subsect of non-COVID-19 cases. • Both human experts and artificial intelligent models were used to classify the CT scans. • ROC analysis and the nonparametric approaches were used to analyze the performance of the radiologists and computer algorithms. • Unique image features or patterns may not exist for reliably distinguishing all COVID-19 from CAP; however, there may be imaging markers that can identify a sizable subset of non-COVID-19 cases.

Objective To assess prospectively the interpretative performance of automated breast ultrasound (ABUS) as a supplemental screening after digital breast tomosynthesis (DBT) or as a standalone screening of women with dense breast tissue. Methods Under an IRB-approved protocol (written consent required), women with dense breasts prospectively underwent concurrent baseline DBT and ABUS screening. Examinations were independently evaluated, in opposite order, by two of seven Mammography Quality Standards Act–qualified radiologists, with the primary radiologist arbitrating disagreements and making clinical management recommendations. We report results for 1111 screening examinations (598 first year and 513 second year) for which all diagnostic workups are complete. Imaging was also retrospectively reviewed for all cancers. Statistical assessments used a 0.05 significance level and accounted for correlation between participants’ examinations. Results Of 1111 women screened, primary radiologists initially “recalled” based on DBT alone (6.6%, 73/1111, CI: 5.2%–8.2%), of which 20 were biopsied, yielding 6/8 total cancers. Automated breast ultrasound increased recalls overall to 14.4% (160/1111, CI: 12.4%–16.6%), with 27 total biopsies, yielding 1 additional cancer. Double reading of DBT alone increased the recall rate to 10.7% (119/1111), with 21 biopsies, with no improvement in cancer detection. Double reading ABUS increased the recall rate to 15.2% (169/1111, CI: 13.2%–17.5%) of women, of whom 22 were biopsied, yielding the detection of 7 cancers, including one seen only on double reading ABUS. Inter-radiologist agreement was similar for recall recommendations from DBT (κ = 0.24, CI: 0.14–0.34) and ABUS (κ = 0.23, CI: 0.15–0.32). Integrated assessments from both readers resulted in a recall rate of 15.1% (168/1111, CI: 13.1%–17.4%). Conclusion Supplemental or standalone ABUS screening detected cancers not seen on DBT, but substantially increased noncancer recall rates.

Background Staging newly diagnosed breast cancer by using dynamic contrast material-enhanced MRI is limited by access, high cost, and false-positive findings. The utility of contrast-enhanced mammography (CEM) and 99mTc sestamibi-based molecular breast imaging (MBI) in this setting is largely unknown. Purpose To compare extent-of-disease assessments by using MRI, CEM, and MBI versus pathology in women with breast cancer. Materials and Methods In this HIPAA-compliant prospective study, women with biopsy-proven breast cancer underwent MRI, CEM, and MBI between October 2014 and April 2018. Eight radiologists independently interpreted each examination result prospectively and were blinded to interpretations of findings with the other modalities. Visibility of index malignancies, lesion size, and additional suspicious lesions (malignant or benign) were compared during pathology review. Accuracy of index lesion sizing and detection of additional lesions in women without neoadjuvant chemotherapy were compared. Results A total of 102 women were enrolled and 99 completed the study protocol (mean age, 51 years ± 11 [standard deviation]; range, 32-77 years). Lumpectomy or mastectomy was performed in 71 women (79 index malignancies) without neoadjuvant chemotherapy and in 28 women (31 index malignancies) with neoadjuvant chemotherapy. Of the 110 index malignancies, MRI, CEM, and MBI depicted 102 (93%; 95% confidence interval [CI]: 86%, 97%), 100 (91%; 95% CI: 84%, 96%), and 101 (92%; 95% CI: 85%, 96%) malignancies, respectively. In patients without neoadjuvant chemotherapy, pathologic size of index malignancies was overestimated with all modalities (P = .02). MRI led to overestimation of 24% (17 of 72) of malignancies by more than 1.5 cm compared with 11% (eight of 70) with CEM and 15% (11 of 72) with MBI. MRI depicted more (P = .007) nonindex lesions, with sensitivity similar to that of CEM or MBI, resulting in lower positive predictive value of additional biopsies (13 of 46 [28%; 95% CI: 17%, 44%] for MRI; 14 of 27 [52%; 95% CI: 32%, 71%] for CEM; and 11 of 25 [44%; 95% CI: 24%, 65%] for MBI (overall P = .01). Conclusion Contrast-enhanced mammography, molecular breast imaging, and MRI showed similar detection of all malignancies. MRI depicted more nonindex suspicious benign lesions than did contrast-enhanced mammography or molecular breast imaging, leading to lower positive predictive value of additional biopsies. All three modalities led to overestimation of index tumor size, particularly MRI. © RSNA, 2019 Online supplemental material is available for this article.

Objective While provocation lumbar discography has been used to identify discs responsible for low back pain, the biomechanical effects of disc injection have received little attention. The purpose of this study was to assess the motion of the functional spinal unit including the endplate and facet/pedicle region during disc injection including comparison between normal and degenerative discs. Subjects Subjects represent 91 consecutive patients referred for discography with chronic low back pain. Methods Lateral projection vertebral motion was retrospectively analyzed at 232 levels (normal: 76 [32.8%], degenerative: 156 [67.2%]). Pre- and postinjection fluoroscopic images were size scaled, and lower endplates were superimposed on separate PowerPoint images. Upper endplate and facet/pedicle motion was separately and independently analyzed on toggled PowerPoint images, subjectively graded as prominent, intermediate, questionable/uncertain, or no motion. Disc morphology was graded using the anteroposterior/lateral postinjection disc appearance (Adams criteria). Results Prominent or intermediate endplate and facet/pedicle motion was identified at most lumbar levels with substantial overall agreement (degenerative: κ = 0.93, 95% confidence intervals [CI] = 0.87–1.00; normal: κ = 0.80, 95% CI = 0.61–1.00). Degenerative levels were strongly associated with a lower degree of endplate and facet/pedicle motion compared with normal: (“prominent” motion grade: endplate: 61% [95/156] vs 89% [68/76], P Conclusion Disc injection expands the disc space inducing endplate motion, pedicle motion, and facet translation in almost all normal and most degenerate levels. Disc injection therefore biomechanically “provokes” the entire functional spinal unit. When provoked pain is encountered during lumbar discography, contribution from the associated facet joint and myotendinous insertions should be considered.

Background Digital breast tomosynthesis (DBT) is replacing digital mammography (DM) in the clinical workflow. Currently, there are limited prospective studies comparing the diagnostic accuracy of both examinations and the role of synthetic mammography (SM) and computer-aided detection (CAD). Purpose To compare the accuracy of DM versus DM + DBT in population-based breast cancer screening. Materials and Methods This prospective study, performed from November 2010 to December 2012, included 24 301 women (mean age, 59.1 years ± 5.7 [standard deviation]) with 281 cancers, of which 51 were interval cancers. Each examination was independently interpreted with four reading modes: DM, DM + CAD, DM + DBT, and SM + DBT. Sensitivity and specificity were compared for DM versus DM + DBT, DM versus DM + CAD, DM + DBT versus SM + DBT, and DM versus DM + DBT at double reading. Reader-adjusted performance characteristics of reading modes were evaluated on the basis of pre-arbitration (initial interpretation) scores. Statistical analysis was based on cluster bootstrap analysis using 10 000 random resamples. Results Sensitivity was 54.1% (152 of 281) for DM and 70.5% (198 of 281) for DM + DBT. Reader-adjusted difference was 12.6% (95% confidence interval [CI]: 5.2%, 19.7%; P = .001). Specificity was 94.2% (false-positive fraction [FPF], 5.8%; 1388 of 24 020) for DM and 95.0% (FPF, 5.0%; 1209/24 020) for DM + DBT, with a reader-adjusted difference in FPF of -1.2% (95% CI: -1.7%, -0.7%; P.001). Sensitivity was 69.0% (194 of 281) for SM + DBT and 70.5% (198 of 281) for DM + DBT, with a reader-adjusted difference of 1.0% (95% CI: -6.2%, 8.5%; P = .77). Specificity was 95.4% (FPF, 4.6%; 1111 of 24 020) for SM + DBT and 95.0% (FPF, 5.0%;1209 of 24 020) for DM + DBT, with reader-adjusted 95% CIs for FPF of 4.7%, 5.4% and 5.0%, 5.7%, respectively, and a difference of -0.3% (95% CI: -0.8%, 0.2%; P = .23). Differences in sensitivity and specificity with the addition of CAD were small and not significant (P.2). Conclusion Addition of digital breast tomosynthesis to digital mammography resulted in significant gains in sensitivity and specificity. Synthetic mammography in combination with digital breast tomosynthesis had similar sensitivity and specificity to digital mammography in combination with digital breast tomosynthesis. © RSNA, 2019 See also the editorial by Lång in this issue.

Our ultimate goal is to develop a personalized breastscreening model. This model will provide information to an individual woman for planning her own screening options based on her imaging and other characteristics. To do so, we are developing a computerized method to estimate a near term risk of screen detectable breast cancer (sBCa) by analyzing lateral and temporal differences in screening mammograms using the Radon Cumulative Distribution Transform (RCDT), a deep convolutional neural network (dCNN). The RCDT can highlight subtle signals associated with future sBCa by exploiting lateral and temporal differences in screening mammograms of women. We used a dCNN to build a robust classification marker of having sBCa within a year from a negative screening. We explored the potential for incremental improvement over known risk factors by combining the dCNN with age and showed consistent gains in identifying asymptomatic women with a low risk of having sBCa (who could use the personalized screening option to safely skip their next screening). We used 4296 screening mammogram images of 537 women with 151 sBCa cases and fine-tuned the VGG16 network using a data split ratio of 0.64:0.16:0.2 for training, validation, and testing. The final model on the independent testing dataset achieved an AUC of 0.82, and a specificity at a sensitivity level of 98% (Sp@Se98) of 0.51 with a 95% confidence interval of 0.40 to 0.65. This indicates that the model could reliably identify at least 40% of low-risk women with a 2% error for a false negative rate.

Purpose To determine rate of malignancy at stereotactic biopsy of amorphous calcifications with different distributions using current imaging, clinical, and histopathologic criteria. Materials and Methods From January 2009 to September 2013, this retrospective study reviewed a large set of stereotactic biopsies to identify amorphous calcifications and their clinical, imaging, and histopathologic characteristics. Calcification distribution was correlated with malignancy rate after adjusting for known risk factors using logistic regression. Results Of 1903 sequential biopsies, 546 (28.7%) were for amorphous calcifications. After excluding atypical lesions not excised and patients with more than one biopsy in the same year, 497 lesions from 494 women (median age, 52 years; age range, 30-81 years) remained. Fifty-two (10.5%; 95% confidence interval [CI]: 7.9, 13.5) lesions proved malignant, with 17 of 52 (42.7%) being invasive cancers (median, 0.3 cm; range, 0.1-1.3 cm) and all 17 of them being estrogen and progesterone receptor positive and node negative. Malignancy rates in a segmental (six of 21 [28.6%]), linear (eight of 32 [25.0%]), or multiple group same quadrant (nine of 36 [25.0%]) distribution were significantly higher than malignancy rate in a solitary group of amorphous calcifications (25 of 356 [7.0%]) (P = .004, P = .003, and P = .002, respectively). Of 356 grouped amorphous calcifications, 102 (28.7%) yielded atypical results prompting excision, with three of 102 (2.9%) upgraded to ductal carcinoma in situ at excision. In women younger than 50 years without a personal history of cancer, grouped amorphous calcifications showed four of 127 (3.1%) (95% CI: 0.9, 7.9) were malignant and 39 of 127 (30.7%) were atypical at final histopathology. Conclusion Biopsy of amorphous calcifications remains necessary, with an overall malignancy rate of 10.5%; only 17 of 497 (3.4%) biopsies showed invasive cancer, and all of these were estrogen and progesterone receptor positive. Grouped amorphous calcifications in women younger than 50 years without history of breast or ovarian cancer showed a low malignancy rate of 3.1% (four of 127).

BackgroundThe performance of PET/MRI in the determination of distant metastases (M stage) in rectal cancer relative to the current practice with contrast material-enhanced CT is largely unknown.PurposeTo compare the staging of clinical M stage rectal cancer with fluorine 18 fluorodeoxyglucose (FDG) PET/MRI (including dedicated liver and rectal MRI) to that of chest and abdominopelvic CT and dedicated rectal MRI.Materials and MethodsFrom January 2016 to August 2017, patients with newly diagnosed advanced mid to low rectal cancers were recruited for this prospective study (clinicaltrials.gov identifier: NCT0265170). Participants underwent both FDG PET/MRI with dedicated liver and rectal MRI and chest and abdominopelvic CT (the standard-of-care protocol) within 3 weeks of each other. Thereafter, M stage assessment performance was determined by using findings from 6-month clinical follow-up or biopsy as the reference standard. Performance was compared between protocols. Agreement in M stage classification was also assessed. Nonparametric statistical analyses were performed, and P .99).ConclusionPET/MRI with dedicated rectal and liver MRI can facilitate the staging work-up of newly diagnosed advanced rectal cancers by helping assess indeterminate lesions, metastases, and incidental findings better than contrast-enhanced CT, obviating for additional imaging work-up.© RSNA, 2019Online supplemental material is available for this article.Clinical trial registration no. NCT02651701.

Diagnostic systems designed to detect possibly multiple lesions per patient (e.g. multiple polyps during CT colonoscopy) are often evaluated in “free-response” studies that allow for diagnostic responses unconstrained in their number and locations. Analysis of free-response studies requires extensions of the traditional receiver operating characteristic (ROC) analysis, which are termed free-response ROC (FROC) methodology. Despite substantial developments in this area, FROC tools and approaches are much more cumbersome than traditional ROC methods. Alternative approaches that use well-known ROC tools (e.g. ROI-ROC) require defining and physically delineating regions of interest (ROI) and combine FROC data within ROIs. We propose an approach that allows analyzing FROC data using conventional ROC tools without delineating the actual ROIs or reducing data. The design parameters of FROC study are used to make FROC data analyzable using ROC tools and to calibrate the corresponding FROC and ROC curves on both conceptual and numerical levels. Differences in the performance indices of the nonparametric FROC and the new approach are shown to be asymptotically negligible and typically rather small in practice. Data from a large multi-reader study of colon cancer detection are used to illustrate the new approach.

PURPOSE: To assess the accuracy of staging positron emission tomography (PET)/computed tomography (CT) in detecting distant metastasis in patients with local-regionally advanced cervical and high-risk endometrial cancer in the clinical trial by the American College of Radiology Imaging Network (ACRIN) and the Gynecology Oncology Group (GOG) (ACRIN 6671/GOG 0233) and to compare central and institutional reader performance. MATERIALS AND METHODS: In this prospective multicenter trial, PET/CT and clinical data were reviewed for patients enrolled in ACRIN 6671/GOG 0233. Two central readers, blinded to site read and reference standard, reviewed PET/CT images for distant metastasis. Central review was then compared with institutional point-of-care interpretation. Reference standard was pathologic and imaging follow-up. Test performance for central and site reviews of PET/CT images was calculated and receiver operating characteristic analysis was performed. Generalized estimating equations and nonparametric bootstrap procedure for clustered data were used to assess statistical significance. RESULTS: There were 153 patients with cervical cancer and 203 patients with endometrial cancer enrolled at 28 sites. Overall prevalence of distant metastasis was 13.7% (21 of 153) for cervical cancer and 11.8% (24 of 203) for endometrial cancer. Central reader PET/CT interpretation demonstrated sensitivity, specificity, positive predictive value (PPV), and negative predictive value of 54.8%, 97.7%, 79.3%, and 93.1% for cervical cancer metastasis versus 64.6%, 98.6%, 86.1%, and 95.4% for endometrial cancer, respectively. By comparison, local institutional review demonstrated sensitivity, specificity, PPV, and negative predictive value of 47.6%, 93.9%, 55.6%, and 91.9% for cervical cancer metastasis and 66.7%, 93.9%, 59.3%, and 95.5% for endometrial cancer, respectively. For central readers, the specificity and PPV of PET/CT detection of cervical and endometrial cancer metastases were all significantly higher compared with that of local institutional review (P < .05). Central reader area under the receiver operating characteristic curve (AUC) values were 0.78 and 0.89 for cervical and endometrial cancer, respectively; these were not significantly different from local institutional AUC values (0.75 and 0.84, respectively; P > .05 for both). CONCLUSION: FDG PET/CT demonstrates high specificity and PPV for detecting distant metastasis in cervical and endometrial cancer and should be included in the staging evaluation. Blinded central review of imaging provides improved specificity and PPV for the detection of metastases and should be considered for future oncologic imaging clinical trials. (©) RSNA, 2017

Digital breast tomosynthesis (DBT) has the potential to overcome limitations of conventional mammography. This study investigated the effects of addition of DBT on interval and detected cancers in population-based screening. Oslo Tomosynthesis Screening Trial (OTST) was a prospective, independent double-reading trial inviting women 50–69 years biennially, comparing full-field digital mammography (FFDM) plus DBT with FFDM alone. Performance indicators and characteristics of screen-detected and interval cancers were compared with two previous FFDM rounds. 24,301 consenting women underwent FFDM + DBT screening over a 2-year period. Results were compared with 59,877 FFDM examinations during prior rounds. Addition of DBT resulted in a non-significant increase in sensitivity (76.2%, 378/496, vs. 80.8%, 227/281, p = 0.151) and a significant increase in specificity (96.4%, 57229/59381 vs. 97.5%, 23427/24020, p

Assessing performance of diagnostic markers is a necessary step for their use in decision making regarding various conditions of interest in diagnostic medicine and other fields. Globally useful markers could, however, have ranges of values that are “diagnostically non-informative”. This paper demonstrates that the presence of marker values from diagnostically non-informative ranges could lead to a loss in statistical efficiency during nonparametric evaluation and shows that grouping non-informative values provides a natural resolution to this problem. These points are theoretically proven and an extensive simulation study is conducted to illustrate the possible benefits of using grouped marker values in a number of practically reasonable scenarios. The results contradict the common conjecture regarding the detrimental effect of grouped marker values during performance assessments. Specifically, contrary to the common assumption that grouped marker values lead to bias, grouping non-informative values does not introduce bias and could substantially reduce sampling variability. The proven concept that grouped marker values could be statistically beneficial without detrimental consequences implies that in practice, tied values do not always require resolution whereas the use of continuous diagnostic results without addressing diagnostically non-informative ranges could be statistically detrimental. Based on these findings, more efficient methods for evaluating diagnostic markers could be developed.

Rationale and Objectives The “binormal” model is the most frequently used tool for parametric receiver operating characteristic (ROC) analysis. The binormal ROC curves can have “improper” (non-concave) shapes that are unrealistic in many practical applications, and several tools (eg, PROPROC) have been developed to address this problem. However, due to the general robustness of binormal ROCs, the improperness of the fitted curves might carry little consequence for inferences about global summary indices, such as the area under the ROC curve (AUC). In this work, we investigate the effect of severe improperness of fitted binormal ROC curves on the reliability of AUC estimates when the data arise from an actually proper curve. Materials and Methods We designed theoretically proper ROC scenarios that induce severely improper shape of fitted binormal curves in the presence of well-distributed empirical ROC points. The binormal curves were fitted using maximum likelihood approach. Using simulations, we estimated the frequency of severely improper fitted curves, bias of the estimated AUC, and coverage of 95% confidence intervals (CIs). In Appendix S1 , we provide additional information on percentiles of the distribution of AUC estimates and bias when estimating partial AUCs. We also compared the results to a reference standard provided by empirical estimates obtained from continuous data. Results We observed up to 96% of severely improper curves depending on the scenario in question. The bias in the binormal AUC estimates was very small and the coverage of the CIs was close to nominal, whereas the estimates of partial AUC were biased upward in the high specificity range and downward in the low specificity range. Compared to a non-parametric approach, the binormal model led to slightly more variable AUC estimates, but at the same time to CIs with more appropriate coverage. Conclusions The improper shape of the fitted binormal curve, by itself, ie, in the presence of a sufficient number of well-distributed points, does not imply unreliable AUC-based inferences.

To preliminarily asses if Contrast Enhanced Digital Mammography (CEDM) can accurately reduce biopsy rates for soft tissue BI-RADS 4A or 4B lesions.Eight radiologists retrospectively and independently reviewed 60 lesions in 54 consenting patients who underwent CEDM under Health Insurance Portability and Accountability Act compliant institutional review board-approved protocols. Readers provided Breast Imaging ReportingData System ratings sequentially for digital mammography/digital breast tomosynthesis (DM/DBT), then with ultrasound, then with CEDM for each lesion. Area under the curve (AUC), true positive rates and false positive rates, positive predictive values and negative predictive values were calculated. Statistical analysis accounting for correlation between lesion-examinations and between-reader variability was performed using OR/DBM (for SAS v.3.0), generalized linear mixed model for binary data (proc glimmix, SAS v.9.4, SAS Institute, Cary North Carolina), and bootstrap.The cohort included 49 benign, two high-risk and nine cancerous lesions in 54 women aged 34-74 (average 50) years. Reader-averaged AUC for CEDM was significantly higher than DM/DBT alone (0.85 versus 0.66, p0.001) or with US (0.85 versus 0.75, p = 0.001). CEDM increased true positive rates from 0.74 under DB/DBT, and 0.89 with US, to 0.90 with CEDM, (p = 0.019 DM/DBT versus CEDM, p = 0.78 DM/DBT + US versus CEDM) and decreased false positive rates from 0.47 using DM/DBT and 0.61 with US to 0.39 with CEDM (p = 0.017 DM/DBT versus CEDM, p = 0.001 DM/DBT+ US versus CEDM). For an expected cancer rate of 10%, CEDM positive predictive values was 20.5% (95% CI: 16%-27%) and negative predictive values 98.3% (95% CI: 96%-100%).Addition of CEDM for evaluation of low-moderate suspicion soft tissue breast lesions can substantially reduce biopsy of benign lesions without compromising cancer detection.

Rationale and Objectives Assess results of a prospective, single-site clinical study evaluating digital breast tomosynthesis (DBT) during baseline screening mammography. Materials and Methods Under an institutional review board–approved Health Insurance Portability and Accountability Act (HIPAA)-compliant protocol, consenting women between ages 34 and 56 years scheduled for their initial and/or baseline screening mammogram underwent both full field digital mammography (FFDM) and DBT. The FFDM and the FFDM plus DBT images were interpreted independently in a reader by mode balanced approach by two of 14 participating radiologists. A woman was recalled for a diagnostic work-up if either radiologist recommended a recall. We report overall recall rates and related diagnostic outcome from the 1080 participants. Proportion of recommended recalls (Breast Imaging Reporting and Data System 0) were compared using a generalized linear mixed model (SAS 9.3) with a significance level of P = .0294. Results The fraction of women without breast cancer recommended for recall using FFDM alone and FFDM plus DBT were 412 of 1074 (38.4%) and 274 of 1074 (25.5%), respectively ( P P Conclusions The use of FFDM plus DBT resulted in a significant decrease in recall rates during baseline screening mammography with no reduction in sensitivity.

Rationale and Objectives To assess radiologists' perceptions of how the new Breast Density Notification Act (BDNA) of Pennsylvania would affect their breast density reporting and their actual reporting patterns after implementation. Materials and Methods Under an institutional review board–approved protocol, we surveyed 21 radiologists about how they believe the new law affected their breast density reporting patterns and analyzed actual changes for 16 respondents before and after the law took effect. Three hundred consecutive reports were assessed for each radiologist before and after the effective date. The distributions of reported density Breast Imaging Reporting and Data System (BI-RADS) (1–4) were compared using a type III test in the context of an ordinal mixed model accounting for between-reader variability and adjusting for age (PROC GLIMMIX, SAS, version 9.3) using a two-sided .05 significance level. Results Seventeen radiologists responded to the survey; however, one retired shortly after responding. Of the 16 respondents, 56% (nine of 16) did not favor the law, 13% (two of 16) were in favor, and 31% (five of 16) were neutral. The fraction perceived that after implementation, they rated more, equally, or less frequently breasts as scattered fibroglandular densities (BI-RADS 2) versus heterogeneously dense rating (BI-RADS 3) was 50% (eight of 16), 44% (seven of 16), and 6% (one of 16), respectively. In practice, 44% (seven of 16) performed differently than their survey answers. Fourteen of 16 radiologists increased the frequency of reported BI-RADS 2 scores after BDNA implementation with seven having statistically significant (P Conclusions Radiologists' reporting patterns changed, at least for a short duration, after the new density reporting law and for some of the radiologists in an unexpected way.

Evaluation of diagnostic performance is typically based on the receiver operating characteristic (ROC) curve and the area under the curve (AUC) as its summary index. The partial area under the curve (pAUC) is an alternative index focusing on the range of practical/clinical relevance. One of the problems preventing more frequent use of the pAUC is the perceived loss of efficiency in cases of noncrossing ROC curves. In this paper, we investigated statistical properties of comparisons of two correlated pAUCs. We demonstrated that outside of the classic model there are practically reasonable ROC types for which comparisons of noncrossing concave curves would be more powerful when based on a part of the curve rather than the entire curve. We argue that this phenomenon stems in part from the exclusion of noninformative parts of the ROC curves that resemble straight-lines. We conducted extensive simulation studies in families of binormal, straight-line, and bigamma ROC curves. We demonstrated that comparison of pAUCs is statistically more powerful than comparison of full AUCs when ROC curves are close to a "straight line". For less flat binormal ROC curves an increase in the integration range often leads to a disproportional increase in pAUCs' difference, thereby contributing to an increase in statistical power. Thus, efficiency of differences in pAUCs of noncrossing ROC curves depends on the shape of the curves, and for families of ROC curves that are nearly straight-line shaped, such as bigamma ROC curves, there are multiple practical scenarios in which comparisons of pAUCs are preferable.

Free-response ROC (FROC) data are typically collected when primary question of interest is focused on the proportions of the correct detection-localization of known targets and frequencies of false positive responses, which can be multiple per subject (image). These studies are particularly relevant for CAD and related applications. The fundamental tool of the location-level FROC analysis is the FROC curve. Although there are many methods of FROC analysis, as we describe in this work, some of the standard and popular approaches, while important, are not suitable for analyzing specifically the location-level FROC performance as summarized by the FROC curve. Analysis of the FROC curve, on the other hand, might not be straightforward. Recently we developed an approach for the location-level analysis of the FROC data using the well-known tools for clustered ROC analysis. In the current work, based on previously developed concepts, and using specific examples, we demonstrate the key reasons why specifically location-level FROC performance cannot be fully addressed by the common approaches as well as illustrate the proposed solution. Specifically, we consider the two most salient FROC approaches, namely JAFROC and the area under the exponentially transformed FROC curve (AFE) and show that clearly superior FROC curves can have lower values for these indices. We describe the specific features that make these approaches inconsistent with FROC curves. This work illustrates some caveats for using the common approaches for location-level FROC analysis and provides guidelines for the appropriate assessment or comparison of FROC systems.

Receiver operating characteristic analysis provides an important methodology for assessing traditional (e.g., imaging technologies and clinical practices) and new (e.g., genomic studies, biomarker development) diagnostic problems. The area under the clinically/practically relevant part of the receiver operating characteristic curve (partial area or partial area under the receiver operating characteristic curve) is an important performance index summarizing diagnostic accuracy at multiple operating points (decision thresholds) that are relevant to actual clinical practice. A robust estimate of the partial area under the receiver operating characteristic curve is provided by the area under the corresponding part of the empirical receiver operating characteristic curve. We derive a closed-form expression for the jackknife variance of the partial area under the empirical receiver operating characteristic curve. Using the derived analytical expression, we investigate the differences between the jackknife variance and a conventional variance estimator. The relative properties in finite samples are demonstrated in a simulation study. The developed formula enables an easy way to estimate the variance of the empirical partial area under the receiver operating characteristic curve, thereby substantially reducing the computation burden, and provides important insight into the structure of the variability. We demonstrate that when compared with the conventional approach, the jackknife variance has substantially smaller bias, and leads to a more appropriate type I error rate of the Wald-type test. The use of the jackknife variance is illustrated in the analysis of a data set from a diagnostic imaging study.

RATIONALE AND OBJECTIVES To assess the interaction between the availability of prior examinations and digital breast tomosynthesis (DBT) in decisions to recall a woman during interpretation of mammograms. MATERIALS AND METHODS Eight radiologists independently interpreted twice 36 mammography examinations, each of which had current and prior full-field digital mammography images (FFDM) and DBT under a Health Insurance Portability and Accountability Act-compliant, institutional review board-approved protocol (written consent waived). During the first reading, three sequential ratings were provided using FFDM only, followed by FFDM + DBT, and then followed by FFDM + DBT + priors. The second reading included FFDM only, then FFDM + priors, and then FFDM + priors + DBT. Twenty-two benign cases clinically recalled, 12 negative/benign examinations (not recalled), and two verified cancer cases were included. Recall recommendations and interaction between the effect of priors and DBT on decisions were assessed (P = .05 significance level) using generalized linear model (PROC GLIMMIX, SAS, version 9.3; SAS Institute, Cary, NC) accounting for case and reader variability. RESULTS Average recall rates in noncancer cases were significantly reduced (51%; P < .001) with the addition of DBT and with addition of priors (23%; P = .01). In absolute terms, the addition of DBT to FFDM reduced the recall rates from 0.67 to 0.42 and from 0.54 to 0.27 when DBT was available before and after priors, respectively. Recall reductions were from 0.64 to 0.54 and from 0.42 to 0.33 when priors were available before and after DBT, respectively. Regardless of the sequence in presentation, there were no statistically significant interactions between the effect of availability of DBT and priors (P = .80). CONCLUSIONS Availability of both priors and DBT are independent primary factors in reducing recall recommendations during mammographic interpretations.

The idiopathic inflammatory myopathies (IIM) are a group of acquired, heterogeneous, systemic connective tissue diseases (CTD) that include polymyositis (PM), adult dermatomyositis (DM), childhood myositis (predominantly juvenile DM), myositis associated with cancer or another connective tissue disease, and inclusion body myositis (IBM) (1, 2). Over the last few decades, survival has improved in IIM, with patients experiencing less cumulative damage and better health related quality of life. Despite an improvement in survival, our knowledge about clinical and serological predictors of clinical improvement in IIM is limited by the lack of well-designed, long–term epidemiological studies and clinical trials. IIM patients have heterogeneous features from a mild rash to life threatening muscle weakness or lung involvement. Their course can be self-limited or may require long-term glucocorticoids and multiple immunosuppressive medications. The response to immunosuppressive drugs is quite variable and current data do not allow the accurate prediction of clinical improvement, which poses a significant challenge to treating physicians, as well as investigators. The varying clinical features of myositis are closely linked to myositis autoantibodies, some of which may contribute to the pathogenesis of IIM (3). Although these autoantibodies provide useful prognostic information on patient outcomes (4–6), this relationship has not been established in prospective cohorts with uniform treatment. Previous evidence suggested that patients possessing anti-Mi-2 autoantibodies had a better prognosis, while patients with anti-SRP fared worse and those with anti-synthetase autoantibodies had intermediate outcomes (6, 7). In addition, there is a paucity of literature regarding predictors of clinical improvement by IIM disease subgroups. IBM is associated with poor treatment responses but studies differentiating responses between PM, DM and JDM are lacking (6). Treatment delay, muscle damage and longer disease duration have also been shown to be associated with poor prognosis (7–10). However, published studies are limited by small sample sizes, retrospective design and a limited assessment of prognostic factors. The availability of targeted therapies and validated outcome measures (11–13) spearheaded the recently completed Rituximab in Myositis (RIM) trial that was designed to evaluate the safety and efficacy of B cell depletion in adult and pediatric myositis patients (14). Rituximab has been studied in a wide variety of autoimmune diseases, as B cells play a critical role in the initiation and propagation of the immune response and are specifically implicated in the pathogenesis of myositis (15). As biologic agents are increasingly utilized in autoimmune diseases, it is important to elucidate the factors that predict a favorable outcome so that clinical trials can be designed with stratification of patients with a good and poor likelihood of improvement. The aim of this study was to identify the clinical and laboratory predictors of clinical improvement in a trial of refractory myositis subjects treated with B cell depletion. This is the first comprehensive study in myositis to evaluate factors associated with clinical improvement in a large prospective cohort.

To assess cancer detection rates, false-positive rates before arbitration, positive predictive values for women recalled after arbitration, and the type of cancers detected with use of digital mammography alone and combined with tomosynthesis in a large prospective screening trial.A prospective, reader- and modality-balanced screening study of participants undergoing combined mammography plus tomosynthesis, the results of which were read independently by four different radiologists, is under way. The study was approved by a regional ethics committee, and all participants provided written informed consent. The authors performed a preplanned interim analysis of results from 12,631 examinations interpreted by using mammography alone and mammography plus tomosynthesis from November 22, 2010, to December 31, 2011. Analyses were based on marginal log-linear models for binary data, accounting for correlated interpretations and adjusting for reader-specific performance levels by using a two-sided significance level of .0294.Detection rates, including those for invasive and in situ cancers, were 6.1 per 1000 examinations for mammography alone and 8.0 per 1000 examinations for mammography plus tomosynthesis (27% increase, adjusted for reader; P = .001). False-positive rates before arbitration were 61.1 per 1000 examinations with mammography alone and 53.1 per 1000 examinations with mammography plus tomosynthesis (15% decrease, adjusted for reader; P.001). After arbitration, positive predictive values for recalled patients with cancers verified later were comparable (29.1% and 28.5%, respectively, with mammography alone and mammography plus tomosynthesis; P = .72). Twenty-five additional invasive cancers were detected with mammography plus tomosynthesis (40% increase, adjusted for reader; P.001). The mean interpretation time was 45 seconds for mammography alone and 91 seconds for mammography plus tomosynthesis (P.001).The use of mammography plus tomosynthesis in a screening environment resulted in a significantly higher cancer detection rate and enabled the detection of more invasive cancers. Clinical trial registration no. NCT01248546.

Performance changes in a binary environment when using additional information is affected only when changes in recommendations are made due to the additional information in question. In a recent study, we have shown that, contrary to general expectation, introducing prior examinations improved recall rates, but not sensitivity. In this study, we assessed cancer detection differences when prior examinations and/or digital breast tomosynthesis (DBT) were made available to the radiologist. We identified a subset of 21 cancer cases with differences in the number of radiologists who recalled these cases after reviewing either a prior examination or DBT. For the cases with differences in recommendations after viewing either priors or DBT, separately, we evaluated the total number of readers that changed their recommendations, regardless of the specific radiologist in question. Confidence intervals for the number of readers and a test for the hypothesis of no difference was performed using the non-parameteric bootstrap approach addressing both case and reader-related sources of variability by resampling cases and readers. With the addition of priors, there were 14 cancer cases (out of 15) where the number of “recalling radiologists” decreased. With the addition of DBT, the number of “recalling radiologists” decreased in only five cases (out of 15) while increasing in the remaining 9 cases. Unlike most new approaches to breast imaging DBT seems to improve both recall rates and cancer detection rates. Changes in recommendations were noted by all radiologists for all cancers by type, size, and breast density.

Rationale and Objectives The aim of this study was to retrospectively compare the interpretive performance of synthetically reconstructed two-dimensional images in combination with digital breast tomosynthesis (DBT) versus full-field digital mammography (FFDM) plus DBT. Materials and Methods Ten radiologists trained in reading tomosynthesis examinations interpreted retrospectively, under two modes, 114 mammograms. One mode included the directly acquired full-field digital mammograms combined with DBT, and the other included synthetically reconstructed projection images combined with DBT. The reconstructed images do not require additional radiation exposure. The two modes were compared with respect to sensitivity, namely, recommendation to recall a breast with either a pathology-proven cancer ( n = 48) or a high-risk lesion ( n = 6), and specificity, namely, no recommendation to recall a breast not depicting an abnormality ( n = 144) or depicting only benign abnormalities ( n = 30). Results The average sensitivity for FFDM with DBT was 0.826, compared to 0.772 for synthetic FFDM with DBT (difference, 0.054; P = .017 and P = .053 for fixed and random reader effects, respectively). The proportions of breasts with no or benign abnormalities recommended to be recalled were virtually the same: 0.298 and 0.297 for the two modalities, respectively (95% confidence intervals for the difference, −0.028 to 0.036 and −0.070 to 0.066 for fixed and random reader effects, respectively). Sixteen additional clusters of microcalcifications (“positive” breasts) were missed by all readers combined when interpreting the mode with synthesized images versus FFDM. Conclusions Lower sensitivity with comparable specificity was observed with the tested version of synthetically generated images compared to FFDM, both combined with DBT. Improved synthesized images with experimentally verified acceptable diagnostic quality will be needed to eliminate double exposure during DBT-based screening.

Rationale and Objectives To compare time to interpretation and diagnostic performance levels during repeat readings of full-field digital mammography (FFDM) and digital breast tomosynthesis (DBT) in a retrospective study. Materials and Methods Three experienced radiologists twice interpreted 125 selected examinations, 35 with verified cancers and 90 negative for cancer during a period of 22 months using FFDM alone followed by a combined FFDM + DBT mode. Changes in time to “review and rate” these examinations as well as in diagnostic performance levels where assessed. A fixed-effect analysis accounting for cross-correlation due to the review of the same examinations by the same readers was performed. Results The total (combined) time to review and rate an examination increased on average by 33% between the first and second readings of the same examinations (P Conclusion DBT-based breast imaging may ultimately result in a substantial increase in performance; however, without efficiency improvements DBT may take longer to interpret. Addition of “false-positive recalls” was most strongly associated with increase in interpretation time while elimination of “false-positive recalls” did not require longer interpretation time.

In this article, we analyze the three-way bootstrap estimate of the variance of the reader-averaged nonparametric area under the receiver operating characteristic (ROC) curve. The setting for this work is medical imaging, and the experimental design involves sampling from three distributions: a set of normal and diseased cases (patients), and a set of readers (doctors). The experiment we consider is fully crossed in that each reader reads each case. A reading generates a score that indicates the reader's level of suspicion that the patient is diseased. The distribution of scores for the normal patients is compared to the distribution of scores for the diseased patients via an ROC curve, and the area under the ROC curve (AUC) summarizes the reader's diagnostic ability to separate the normal patients from the diseased ones. We find that the bootstrap estimate of the variance of the reader-averaged AUC is biased, and we represent this bias in terms of moments of success outcomes. This representation helps un...

Rationale and Objectives To investigate consistency of the orders of performance levels when interpreting mammograms under three different reading paradigms. Materials and Methods We performed a retrospective observer study in which nine experienced radiologists rated an enriched set of mammography examinations that they personally had read in the clinic (“individualized”) mixed with a set that none of them had read in the clinic (“common set”). Examinations were interpreted under three different reading paradigms: binary using screening Breast Imaging Reporting and Data System (BI-RADS), receiver-operating characteristic (ROC), and free-response ROC (FROC). The performance in discriminating between cancer and noncancer findings under each of the paradigms was summarized using Youden's index/2+0.5 (Binary), nonparameteric area under the ROC curve (AUC), and an overall FROC index (JAFROC-2). Pearson correlation coefficients were then computed to assess consistency in the ordering of observers' performance levels. Statistical significance of the computed correlation coefficients was assessed using bootstrap confidence intervals obtained by resampling sets of examination-specific observations. Results All but one of the computed pair-wise correlation coefficients were larger than 0.66 and were significantly different from zero. The correlation between the overall performance measures under the Binary and ROC paradigms was the lowest (0.43) and was not significantly different from zero (95% confidence interval −0.078 to 0.733). Conclusion The use of different evaluation paradigms in the laboratory tends to lead to consistent ordering of the overall performance levels of observers. However, one should recognize that conceptually similar performance indexes resulting from different paradigms often measure different performance characteristics and thus disagreements are not only possible but frequently quite natural.

The authors investigated radiologists, performances during retrospective interpretation of screening mammograms when using a binary decision whether to recall a woman for additional procedures or not and compared it with their receiver operating characteristic (ROC) type performance curves using a semi-continuous rating scale. Under an Institutional Review Board approved protocol nine experienced radiologists independently rated an enriched set of 155 examinations that they had not personally read in the clinic, mixed with other enriched sets of examinations that they had individually read in the clinic, using both a screening BI-RADS rating scale (recall/not recall) and a semi-continuous ROC type rating scale (0 to 100). The vertical distance, namely the difference in sensitivity levels at the same specificity levels, between the empirical ROC curve and the binary operating point were computed for each reader. The vertical distance averaged over all readers was used to assess the proximity of the performance levels under the binary and ROC-type rating scale. There does not appear to be any systematic tendency of the readers towards a better performance when using either of the two rating approaches, namely four readers performed better using the semi-continuous rating scale, four readers performed better with the binary scale, and one reader had the point exactly on the empirical ROC curve. Only one of the nine readers had a binary "operating point" that was statistically distant from the same reader's empirical ROC curve. Reader-specific differences ranged from -0.046 to 0.128 with an average width of the corresponding 95% confidence intervals of 0.2 and p-values ranging for individual readers from 0.050 to 0.966. On average, radiologists performed similarly when using the two rating scales in that the average distance between the run in individual reader's binary operating point and their ROC curve was close to zero. The 95% confidence interval for the fixed-reader average (0.016) was (-0.0206, 0.0631) (two-sided p-value 0.35). In conclusion the authors found that in retrospective observer performance studies the use of a binary response or a semi-continuous rating scale led to consistent results in terms of performance as measured by sensitivity-specificity operating points.

The task of searching and detecting multiple abnormalities depicted on an image, or a series of images, is a common problem in different areas such as military target detection or diagnostic medical imaging. A free response receiver operating characteristic (FROC) approach for assessing performance in many of these scenarios entails marking the locations of suspected abnormalities and indicating a level of suspicion at each of the marked locations. One of the important characteristics of a system being evaluated under the FROC paradigm is its performance in the conventional ROC domain, namely classifying a subject (or a unit of interest) as “negative” or “positive” in regard to the presence of the abnormality (or any of the abnormalities) of interest. With FROC data we can compare subjects by specifying a function of multiple scores within a subject. This approach allows formulating subject-based ROC type indices that can be estimated using existing ROC concepts. In this article we focus on indices that reflect the ability of the system to discriminate between actually negative and actually positive subjects. We consider a previously proposed index that is based on the comparison of the highest scores on subjects and two new indices that are based on potentially more stable comparison functions, namely comparison of average scores and stochastic dominance. Based on these indices we develop nonparametric procedures for comparing subject-based discriminative ability of diagnostic systems being evaluated under the FROC paradigm. We also investigate the properties of the statistical procedures in a simulation study.

The area under the Receiver Operating Characteristic (ROC) curve (AUC) and related summary indices are widely used for assessment of accuracy of an individual and comparison of performances of several diagnostic systems in many areas including studies of human perception, decision making, and the regulatory approval process for new diagnostic technologies. Many investigators have suggested implementing the bootstrap approach to estimate variability of AUC-based indices. Corresponding bootstrap quantities are typically estimated by sampling a bootstrap distribution. Such a process, frequently termed Monte Carlo bootstrap, is often computationally burdensome and imposes an additional sampling error on the resulting estimates. In this article, we demonstrate that the exact or ideal (sampling error free) bootstrap variances of the nonparametric estimator of AUC can be computed directly, i.e., avoiding resampling of the original data, and we develop easy-to-use formulas to compute them. We derive the formulas ...

Rationale and Objectives We have observed that a very large fraction of responses for several detection tasks during the performance of observer studies are in the extreme ranges of lower than 11% or higher than 89% regardless of the actual presence or absence of the abnormality in question or its subjectively rated “subtleness.” This observation raises questions regarding the validity and appropriateness of using multicategory rating scales for such detection tasks. Monte Carlo simulation of binary and multicategory ratings for these tasks demonstrate that the use of the former (binary) often results in a less biased and more precise summary index and hence may lead to a higher statistical power for determining differences between modalities.

Assess results of a prospective, single-site clinical study evaluating digital breast tomosynthesis (DBT) during baseline screening mammography.Under an institutional review board-approved Health Insurance Portability and Accountability Act (HIPAA)-compliant protocol, consenting women between ages 34 and 56 years scheduled for their initial and/or baseline screening mammogram underwent both full field digital mammography (FFDM) and DBT. The FFDM and the FFDM plus DBT images were interpreted independently in a reader by mode balanced approach by two of 14 participating radiologists. A woman was recalled for a diagnostic work-up if either radiologist recommended a recall. We report overall recall rates and related diagnostic outcome from the 1080 participants. Proportion of recommended recalls (Breast Imaging Reporting and Data System 0) were compared using a generalized linear mixed model (SAS 9.3) with a significance level of P = .0294.The fraction of women without breast cancer recommended for recall using FFDM alone and FFDM plus DBT were 412 of 1074 (38.4%) and 274 of 1074 (25.5%), respectively (P.001). Large inter-reader variability in terms of recall reduction was observed among the 14 readers; however, 11 of 14 readers recalled fewer women using FFDM plus DBT (5 with P.015). Six cancers (four ductal carcinomas in situ [DCIS] and two invasive ductal carcinomas [IDC]) were detected. One IDC was detected only on DBT and one DCIS cancer was detected only on FFDM, whereas the remaining cancers were detected on both modalities.The use of FFDM plus DBT resulted in a significant decrease in recall rates during baseline screening mammography with no reduction in sensitivity.

To assess radiologists' perceptions of how the new Breast Density Notification Act (BDNA) of Pennsylvania would affect their breast density reporting and their actual reporting patterns after implementation.Under an institutional review board-approved protocol, we surveyed 21 radiologists about how they believe the new law affected their breast density reporting patterns and analyzed actual changes for 16 respondents before and after the law took effect. Three hundred consecutive reports were assessed for each radiologist before and after the effective date. The distributions of reported density Breast Imaging Reporting and Data System (BI-RADS) (1-4) were compared using a type III test in the context of an ordinal mixed model accounting for between-reader variability and adjusting for age (PROC GLIMMIX, SAS, version 9.3) using a two-sided .05 significance level.Seventeen radiologists responded to the survey; however, one retired shortly after responding. Of the 16 respondents, 56% (nine of 16) did not favor the law, 13% (two of 16) were in favor, and 31% (five of 16) were neutral. The fraction perceived that after implementation, they rated more, equally, or less frequently breasts as scattered fibroglandular densities (BI-RADS 2) versus heterogeneously dense rating (BI-RADS 3) was 50% (eight of 16), 44% (seven of 16), and 6% (one of 16), respectively. In practice, 44% (seven of 16) performed differently than their survey answers. Fourteen of 16 radiologists increased the frequency of reported BI-RADS 2 scores after BDNA implementation with seven having statistically significant (P.05) increases after adjusting for age differences.Radiologists' reporting patterns changed, at least for a short duration, after the new density reporting law and for some of the radiologists in an unexpected way.