Search

The purpose of this work is to describe the LUNGx Challenge for the computerized classification of lung nodules on diagnostic computed tomography (CT) scans as benign or malignant and report the performance of participants' computerized methods along with that of six radiologists who participated in an observer study performing the same Challenge task on the same dataset. The Challenge provided sets of calibration and testing scans, established a performance assessment process, and created an infrastructure for case dissemination and result submission. Ten groups applied their own methods to 73 lung nodules (37 benign and 36 malignant) that were selected to achieve approximate size matching between the two cohorts. Area under the receiver operating characteristic curve (AUC) values for these methods ranged from 0.50 to 0.68; only three methods performed statistically better than random guessing. The radiologists' AUC values ranged from 0.70 to 0.85; three radiologists performed statistically better than the best-performing computer method. The LUNGx Challenge compared the performance of computerized methods in the task of differentiating benign from malignant lung nodules on CT scans, placed in the context of the performance of radiologists on the same task. The continued public availability of the Challenge cases will provide a valuable resource for the medical imaging research community.

To retrospectively evaluate whether a difference-image computer-aided detection (CAD) scheme can help radiologists detect peripheral lung cancers missed at low-dose computed tomography (CT).Institutional review board approval and informed patient and observer consent were obtained. Seventeen patients (eight men and nine women; mean age, 60 years) with a missed peripheral lung cancer and 10 control subjects (five men and five women; mean age, 63 years) without cancer at low-dose CT were included in an observer study. Fourteen radiologists were divided into two groups on the basis of different image display formats: Six radiologists (group 1) reviewed CT scans with a multiformat display, and eight radiologists (group 2) reviewed images with a "stacked" cine-mode display. The radiologists, first without and then with the CAD scheme, indicated their confidence level regarding the presence (or absence) of cancer and the most likely position of a lesion on each CT scan. Receiver operating characteristic (ROC) curves were calculated without and with localization to evaluate the observers' performance.With the CAD scheme, the average area under the ROC curve improved from 0.763 to 0.854 for all radiologists (P = .002), from 0.757 to 0.862 for group 1 (P = .04), and from 0.768 to 0.848 for group 2 (P = .01). The average sensitivity in the detection of 17 cancers improved from 52% (124 of 238 observations) to 68% (163 of 238 observations) for all radiologists (P.001), from 49% (50 of 102 observations) to 71% (72 of 102 observations) for group 1 (P = .02), and from 54% (74 of 136 observations) to 67% (91 of 136 observations) for group 2 (P = .006). The localization ROC curve also improved.Lung cancers missed at low-dose CT were very difficult to detect, even in an observer study. The use of CAD, however, can improve radiologists' performance in the detection of these subtle cancers.

A computer-aided diagnosis (CAD) scheme for determination of the likelihood of malignancy of 244 nodules on high-resolution CT (HRCT) was developed. The performance (Az) for 16 radiologists was improved from 0.785 to 0.853 (P=0.02) with the aid of the CAD scheme by use of 56 nodules, including 28 cancerous and 28 benign nodules which were matched in size and pattern to the cancers. Our purpose in this study was to investigate further whether a CAD scheme can assist radiologists in distinguishing benign from malignant nodules in different groups. The results indicated that Az values for radiologists without and with the CAD scheme were improved from 0.770 to 0.855 for general radiologists (P=0.01) and from 0.805 to 0.850 for chest radiologists (P=0.12); from 0.717 to 0.821 for nodules at 6–10 mm (P=0.04) and from 0.837 to 0.901 for nodules at 11–20 mm (P=0.04); and from 0.812 to 0.892 for nodules with pure ground-glass opacity (GGO) (P=0.149), from 0.819 to 0.863 for nodules with mixed GGO (P=0.196), and from 0.784 to 0.844 for solid nodules (P=0.334). CAD has the potential to improve the diagnostic accuracy in distinguishing benign nodules from malignant ones in different groups on HRCT.

RATIONALE AND OBJECTIVES: The Lung Image Database Consortium (LIDC) is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource to promote the development of computer-aided detection or characterization of pulmonary nodules. To obtain the best estimate of the location and spatial extent of lung nodules, expert thoracic radiologists reviewed and annotated each scan. Because a consensus panel approach was neither feasible nor desirable, a unique two-phase, multicenter data collection process was developed to allow multiple radiologists at different centers to asynchronously review and annotate each CT scan. This data collection process was also intended to capture the variability among readers.MATERIALS AND METHODS: Four radiologists reviewed each scan using the following process. In the first or "blinded" phase, each radiologist reviewed the CT scan independently. In the second or "unblinded" review phase, results from all four blinded reviews were compiled and presented to each radiologist for a second review, allowing the radiologists to review their own annotations together with the annotations of the other radiologists. The results of each radiologist's unblinded review were compiled to form the final unblinded review. An XML-based message system was developed to communicate the results of each reading.RESULTS: This two-phase data collection process was designed, tested, and implemented across the LIDC. More than 500 CT scans have been read and annotated using this method by four expert readers; these scans either are currently publicly available at http://ncia.nci.nih.gov or will be in the near future.CONCLUSIONS: A unique data collection process was developed, tested, and implemented that allowed multiple readers at distributed sites to asynchronously review CT scans multiple times. This process captured the opinions of each reader regarding the location and spatial extent of lung nodules.

RATIONALE AND OBJECTIVES: The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured.MATERIALS AND METHODS: Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.RESULTS: After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.CONCLUSION: The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.