Search

Purpose: The development of computer-aided diagnostic (CAD) methods for lung nodule detection, classification, and quantitative assessment can be facilitated through a well-characterized repository of computed tomography (CT) scans. The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI) completed such a database, establishing a publicly available reference for the medical imaging research community. Initiated by the National Cancer Institute (NCI), further advanced by the Foundation for the National Institutes of Health (FNIH), and accompanied by the Food and Drug Administration (FDA) through active participation, this public-private partnership demonstrates the success of a consortium founded on a consensus-based process. Methods: Seven academic centers and eight medical imaging companies collaborated to identify, address, and resolve challenging organizational, technical, and clinical issues to provide a solid foundation for a robust database. The LIDC/IDRI Database contains 1018 cases, each of which includes images from a clinical thoracic CT scan and an associated XML file that records the results of a two-phase image annotation process performed by four experienced thoracic radiologists. In the initial blinded-read phase, each radiologist independently reviewed each CT scan and marked lesions belonging to one of three categories (" nodule�3 mm," " nodule

Purpose: The current linear method to track tumor progression and evaluate treatment efficacy is insufficient for malignant pleural mesothelioma (MPM). A volumetric method for tumor measurement could improve the evaluation of novel treatments, but a fully manual implementation of volume measurement is too tedious and time-consuming. This manuscript presents a computerized method for the three-dimensional segmentation and volumetric analysis of MPM. Methods: The computerized MPM segmentation method segments the lung parenchyma and hemithoracic cavities to define the pleural space. Nonlinear diffusion and a k-means classifier are then implemented to identify MPM in the pleural space. A database of 31 computed tomography scans from 31 patients with pathologically confirmed MPM was retrospectively collected. Three observers independently outlined five randomly selected sections in each scan. The Jaccard similarity coefficient (J) between each of the observers and between the observer-defined and computer-defined segmentations was calculated. The computer-defined and the observer-defined segmentation areas (averaged over all observers) were both calculated for each axial section and compared using Bland–Altman plots. Results: The median J value among observers averaged over all sections was 0.517. The median J between the computer-defined and manual segmentations was 0.484. The difference between these values was not statistically significant. The area delineated by the computerized method demonstrated variability and bias comparable to the tumor area calculated from manual delineations. Conclusions: A computerized method for segmentation and measurement of MPM was developed. This method requires minimal initialization by the user and demonstrated good agreement with manually drawn outlines and area measurements. This method will allow volumetric tracking of tumor progression and may improve the evaluation of novel MPM treatments.

Purpose: Initial outlines are often presented as an aid to reduce the time-cost associated with manual segmentation and measurement of structures in medical images. This study evaluated the influence of initial outlines on manual segmentation intraobserver and interobserver precision. Methods: Three observers independently outlined all pleural mesothelioma tumors present in five computed tomography (CT) sections in each of 30 patient scans. After a lapse of time, each observer was presented with the same series of CT sections with the outlines of each observer superimposed as initial outlines. Each observer created altered outlines by altering the initial outlines to reflect their perception of the tumor boundary. Altered outlines were compared to original outlines using the Jaccard similarity coefficient (J). Intraobserver and interobserver precision of observer outlines were calculated by applying linear mixed effects analysis of variance models to the J values. The percent of minor alterations (alterations that resulted in only slight changes in the initial outline) was also recorded. Results: The average J value between pairs of observer original outlines was 0.371. The average J value between pairs of observer outlines when altered from an identical initial outline was 0.796, indicating increased interobserver precision. The average difference between J values of an observer’s segmentation created by altering their own initial outline and when altering a different observer’s initial outline was 0.476, indicating initial outlines strongly influence intraobserver precision. Observers made minor alterations on 74.5% of initial outlines with which they were presented. Conclusions: Intraobserver and interobserver precision were strongly dependent on the initial outline. These effects are likely due to the tendency of observers to make only minor corrections to initial outlines. This finding could impact observer study design, tumor growth assessment, computer-aided diagnosis system validation, and radiation therapy target volume definition when initial outlines are used as an observer aid.

Rationale and ObjectivesStudies that evaluate the lung-nodule-detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the “truth”). The purpose of this study was to analyze (1) variability in the “truth” defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of “truth” in the context of lung nodule detection on computed tomography (CT) scans.Materials and MethodsTwenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodules ≥ 3 mm in maximum diameter. Panel “truth” sets of nodules then were derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule-detection performance of the other radiologists was evaluated based on these panel “truth” sets.ResultsThe number of “true” nodules in the different panel “truth” sets ranged from 15–89 (mean: 49.8±25.6). The mean radiologist nodule-detection sensitivities across radiologists and panel “truth” sets for different panel “truth” conditions ranged from 51.0–83.2%; mean false-positive rates ranged from 0.33–1.39 per case.ConclusionSubstantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of “truth” on which lung nodule detection studies are based must be carefully considered, since even experienced thoracic radiologists may not perform well when measured against the “truth” established by other experienced thoracic radiologists.Keywords: lung nodule, computed tomography (CT), thoracic imaging, inter-observer variability, computer-aided diagnosis (CAD)

Measurement of the size of anatomic regions of interest in medical images is used to diagnose disease, track growth, and evaluate response to therapy. The discrete nature of medical images allows for both continuous and discrete definitions of region boundary. These definitions may, in turn, support several methods of area calculation that give substantially different quantitative values. This study investigated several boundary definitions (e.g., continuous polygon, internal discrete, and external discrete) and area calculation methods (pixel counting and Green’s theorem). These methods were applied to three separate databases: A synthetic image database, the Lung Image Database Consortium database of lung nodules and a database of adrenal gland outlines. Average percent differences in area on the order of 20% were found among the different methods applied to the clinical databases. These results support the idea that inconsistent application of region boundary definition and area calculation may substantially impact measurement accuracy.

Rationale and Objectives The goal was to investigate the effects of choosing between different metrics in estimating the size of pulmonary nodules as a factor both of nodule characterization and of performance of computer aided detection systems, because the latter are always qualified with respect to a given size range of nodules. Materials and Methods This study used 265 whole-lung CT scans documented by the Lung Image Database Consortium (LIDC) using their protocol for nodule evaluation. Each inspected lesion was reviewed independently by four experienced radiologists who provided boundary markings for nodules larger than 3 mm. Four size metrics, based on the boundary markings, were considered: a unidimensional and two bidimensional measures on a single image slice and a volumetric measurement based on all the image slices. The radiologist boundaries were processed and those with four markings were analyzed to characterize the interradiologist variation, while those with at least one marking were used to examine the difference between the metrics. Results The processing of the annotations found 127 nodules marked by all of the four radiologists and an extended set of 518 nodules each having at least one observation with three-dimensional sizes ranging from 2.03 to 29.4 mm (average 7.05 mm, median 5.71 mm). A very high interobserver variation was observed for all these metrics: 95% of estimated standard deviations were in the following ranges for the three-dimensional, unidimensional, and two bidimensional size metrics, respectively (in mm): 0.49–1.25, 0.67–2.55, 0.78–2.11, and 0.96–2.69. Also, a very large difference among the metrics was observed: 0.95 probability-coverage region widths for the volume estimation conditional on unidimensional, and the two bidimensional size measurements of 10 mm were 7.32, 7.72, and 6.29 mm, respectively. Conclusions The selection of data subsets for performance evaluation is highly impacted by the size metric choice. The LIDC plans to include a single size measure for each nodule in its database. This metric is not intended as a gold standard for nodule size; rather, it is intended to facilitate the selection of unique repeatable size limited nodule subsets.

The current linear method to track tumor progression and evaluate treatment efficacy is insufficient for malignant pleural mesothelioma (MPM). A volumetric method for tumor measurement could improve the evaluation of novel treatments, but a fully manual implementation of volume measurement is too tedious and time-consuming. This manuscript presents a computerized method for the three-dimensional segmentation and volumetric analysis of MPM.The computerized MPM segmentation method segments the lung parenchyma and hemithoracic cavities to define the pleural space. Nonlinear diffusion and a k-means classifier are then implemented to identify MPM in the pleural space. A database of 31 computed tomography scans from 31 patients with pathologically confirmed MPM was retrospectively collected. Three observers independently outlined five randomly selected sections in each scan. The Jaccard similarity coefficient (J) between each of the observers and between the observer-defined and computer-defined segmentations was calculated. The computer-defined and the observer-defined segmentation areas (averaged over all observers) were both calculated for each axial section and compared using Bland-Altman plots.The median J value among observers averaged over all sections was 0.517. The median J between the computer-defined and manual segmentations was 0.484. The difference between these values was not statistically significant. The area delineated by the computerized method demonstrated variability and bias comparable to the tumor area calculated from manual delineations.A computerized method for segmentation and measurement of MPM was developed. This method requires minimal initialization by the user and demonstrated good agreement with manually drawn outlines and area measurements. This method will allow volumetric tracking of tumor progression and may improve the evaluation of novel MPM treatments.

Digital radiography and display systems have revolutionized radiologic practice in recent years and have enabled clinical application of advanced image processing techniques. These include dual energy subtraction and temporal subtraction, both of which can improve diagnostic accuracy for abnormal findings in chest radiographs, especially for subtle lesions such as early lung cancer or focal pneumonia. Dual energy radiography exploits the differential attenuation of low-energy x-ray photons by calcium to produce separate images on the bones and soft tissues, which provides improved detection and characterization of both calcified and noncalcified lung lesions. Dual energy subtraction radiography is currently available from 2 of the major vendors and is in clinical use at many institutions in the United States. Temporal subtraction is a complementary technique that enhances interval change, by using a previous radiograph as a subtraction mask, so that unchanged normal anatomy is suppressed, whereas new abnormalities are enhanced. Though it is not yet a product in the United States, temporal subtraction is available for clinical use in Japan. Temporal subtraction can be combined with energy subtraction to reduce misregistration artifacts, and also has potential to improve computer-aided detection of nodules and other types of lung disease.

Studies that evaluate the lung nodule detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection in computed tomographic (CT) scans.Twenty-five thoracic CT scans were reviewed by four thoracic radiologists, who independently marked lesions they considered to be nodulesor=3 mm in maximum diameter. Panel "truth" sets of nodules were then derived from the nodules marked by different combinations of two and three of these four radiologists. The nodule detection performance of the other radiologists was evaluated based on these panel "truth" sets.The number of "true" nodules in the different panel "truth" sets ranged from 15 to 89 (mean 49.8 +/- 25.6). The mean radiologist nodule detection sensitivities across radiologists and panel "truth" sets for different panel "truth" conditions ranged from 51.0 to 83.2%; mean false-positive rates ranged from 0.33 to 1.39 per case.Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be carefully considered, because even experienced thoracic radiologists may not perform well when measured against the "truth" established by other experienced thoracic radiologists.

RATIONALE AND OBJECTIVES: The purpose of this study was to analyze the variability of experienced thoracic radiologists in the identification of lung nodules on computed tomography (CT) scans and thereby to investigate variability in the establishment of the "truth" against which nodule-based studies are measured.MATERIALS AND METHODS: Thirty CT scans were reviewed twice by four thoracic radiologists through a two-phase image annotation process. During the initial "blinded read" phase, radiologists independently marked lesions they identified as "nodule >or=3 mm (diameter)," "nodule or=3 mm." During the subsequent "unblinded read" phase, the blinded read results of all four radiologists were revealed to each radiologist, who then independently reviewed their marks along with the anonymous marks of their colleagues; a radiologist's own marks then could be deleted, added, or left unchanged. This approach was developed to identify, as completely as possible, all nodules in a scan without requiring forced consensus.RESULTS: After the initial blinded read phase, 71 lesions received "nodule >or=3 mm" marks from at least one radiologist; however, all four radiologists assigned such marks to only 24 (33.8%) of these lesions. After the unblinded reads, a total of 59 lesions were marked as "nodule >or=3 mm" by at least one radiologist. Twenty-seven (45.8%) of these lesions received such marks from all four radiologists, three (5.1%) were identified as such by three radiologists, 12 (20.3%) were identified by two radiologists, and 17 (28.8%) were identified by only a single radiologist.CONCLUSION: The two-phase image annotation process yields improved agreement among radiologists in the interpretation of nodules >or=3 mm. Nevertheless, substantial variability remains across radiologists in the task of lung nodule identification.

The development of computer-aided diagnostic (CAD) systems requires an initial establishment of "truth" by expert human observers. Potential inconsistencies in the "truth" data must be identified and corrected before investigators can rely on this data. We developed a quality assurance model to supplement the "truth" collection process for lung nodules on CT scans. A two-phase process was established for the interpretation of CT scans by four radiologists. During the initial "blinded read," radiologists independently assigned lesions they identified into one of three categories: "nodule g 3mm," "nodule < 3mm," or "non-nodule g 3mm." During the subsequent "unblinded read," the blinded read results of all radiologists were revealed. The radiologists then independently reviewed their marks along with their colleague's marks; a radiologist's own marks could be left unchanged, deleted, switched in terms of lesion category, or additional marks could be added. The final set of marks underwent quality assurance, which consisted of identification of potential errors that occurred during the reading process and error correction. All marks were visually grouped into discrete nodules. Six categories of potential error were defined, and any nodule with a mark that satisfied the criterion for one of these categories was referred to the radiologist who assigned the mark in question. The radiologist either corrected the mark or confirmed that the mark was intentional. A total of 829 nodules were identified by at least one radiologist in 100 CT scans through the two-phase process designed to capture "truth." The quality assurance process yielded 81 nodules with potential errors. The establishment of "truth" must incorporate a quality assurance model to guarantee the integrity of the "truth" that will provide the basis for the training and testing of CAD systems.

Purpose: Measurement of the size of anatomic regions of interest is used to diagnose disease, track growth, and evaluate response to therapy. The discrete nature of medical images allows for both continuous and discrete definitions of region boundary. These definitions may, in turn, support several methods of area calculation that give substantially different values. This study investigated several boundary definitions and area calculation methods to quantify these differences. Method and Materials: Two sets of region boundaries were investigated, one defined in continuous space and one defined in discrete space. A total 1,764 manual lung nodule boundaries were obtained from the LungImageDatabase Consortium (LIDC) database.Lung nodule area was calculated for each of these discrete‐space boundaries based on four area metrics: boundary‐excluded pixel counting, boundary‐included pixel counting, and two variants of Green's Theorem applied to vertices defined by the center of each boundary pixel. Adrenal gland area was calculated for 71 manual continuous‐space adrenal gland boundaries based of four area metrics: Green's Theorem applied to the original continuous boundary, boundary‐excluded pixel counting after direct conversion to discrete space, boundary‐included pixel counting after direct conversion to discrete space, and pixel counting after pixel‐center conversion to discrete space. Results: Based on the same set of adrenal gland boundaries, mean adrenal gland area ranged from 85.1 ± 35.4 pixels to 126.2 ± 42.8 pixels, depending on the method of area calculation. Based on the same set of lung nodule boundaries, mean lung nodule area ranged from 147.0 ± 212.1 pixels to 208.8 ± 251.7 pixels, depending on the method of calculation. Conclusion: Inconsistent application of region boundary definition and area calculation may substantially impact measurement accuracy. Substantial differences exist among the various area calculation methods supporting the necessity, in both the research and clinical settings, to consistently apply boundary definition and area calculation methods.

Purpose: In radiological diagnoses, it is important to understand and interpret anatomical changes, specifically those observed in CT scans, which correlate with treatment plans. There are currently many anticancer agents, which affect tumor vasculature growth as well as normal organ vessel growth. Monitoring these effects with imaging may be useful to guide selection and dosing of different treatments. By observing perfusion of certain organs, it is possible to monitor vessel competency. In cases where perfusion is constant over time, it can be assumed that there is no significant change in vasculature, and thus no damage as an effect of chemotherapeutic agents. In contrast, significant changes in perfusion over time, may suggest decease in normal vessel growth as a result of therapy. Methods & Materials: Patients receiving the VEGF inhibitor sorafenib, underwent CTimaging every six weeks, beginning with a baseline study prior to treatment. A “jog scan” was used to track perfusion through the adrenal glands (chosen due to their significant fenestration). Sixteen pairs of adrenal images were obtained per jog scan, and manually contoured using a contouring program. Each of the sixteen scans represents different perfusion time intervals from 0–150 seconds. The mean pixel values of each gland were obtained, and these values were compared over time for any significant changes in pixel value, and thus change in vasculature perfusion over time. Results: The average change in maximum pixel values from baseline to six weeks after treatment initiation shows a change of 4.58% increase in peak pixel value for both adrenal glands. Conclusion: The manual contouring of adrenal glands in conjunction with calculated maximum pixels values shows changes in adrenal perfusion between baseline and beginning therapy. The continued monitoring of perfusion could prove beneficial to the radiologic diagnosis of significant anatomical changes as a result of continuous chemotherapy.

Purpose: When manually segmenting structures in medical images, an observer identifies the structure and then traces the structure's boundary. When an observer instead is presented with an initial segmentation boundary from a computerized method, the expectation is that the observer will alter that outline as necessary to fit their own perception of the boundary, which should not be substantially influenced by the initial, computer‐defined segmentation. This expectation is implicit when observers modify computerized outlines to establish “truth,” measure the area/volume of a structure, or use a computerized system as an initial reader. The goal of this study was to quantifying the extent to which initial outlines influence human observers. Method and Materials: A database of 30 thoracic CT scans from different mesothelioma patients was collected. For each scan, 5 sections were randomly selected for analysis, and three experienced observers independently outlined all mesothelioma tumor in each of these 150 sections to produce “truth outlines.” After a three‐month period, each observer was presented with all 450 truth outlines, which served as “initial outlines” in this second component of the study, and observers could alter these initial outlines to produce “modified outlines” that best captured their perception of tumor boundary. The area‐of‐overlap measure (AOM) between pairwise combinations of outlines was calculated. Results: The average AOM between the truth outlines of a given observer and the modified outlines derived from the initial outline of that same observer was 0.842 ± 0.009 across all observers. In comparison, the average AOM between the truth outlines of a given observer and the modified outlines derived from the initial outlines of the other two observers was 0.443 ± 0.006 across all observers. Conclusion: The substantial difference between the two mean AOM values implies that observers are strongly impacted by the presence of an initial outline.

Purpose: The measurement of mesothelioma is a crucial component of the assessment of disease response to therapy. Current standards for response assessment utilize summed linear measurements acquired on three CT sections. The purpose of this work was to evaluate area measurements as a metric for response assessment, specifically through the study of the inter‐observer variability of such measurements.Method/Materials: One CT scan from each of 21 mesothelioma patients was collected. Using a computer interface, three radiologists contoured the visible disease on three CT sections containing visible disease. The radiologists were able to inspect the full CT scan, but were only able to contour specified sections. The resulting 189 area measurements (three radiologists, three sections per patient, 21 patients) were compared using a random‐effects analysis of variance model to assess relative inter‐observer variability. The 63 sums of three section measurements per patient were also analyzed, since these sums of areas are more clinically relevant for response assessment. Results: When each radiologist's measurements were compared with the average of the other two radiologists'measurements, moderate correlation was observed (r‐values of 0.64 to 0.94). The 95% confidence interval for relative inter‐observer variability of section area measurements was [−81.3%, +433.7%], spanning a range of 515.0%. For the summed area measurements (three sections per patient), the 95% confidence interval for relative inter‐observer variability was [−68.8%, +220.7%], spanning a range of 289.3%. Conclusion: The inter‐observer variability in area measurements of mesothelioma tumor spans a range that encompasses the response categories of the original WHO bi‐dimensional response criteria, as well as extrapolations of the RECIST response categories to two dimensions. Manual area measurements may not be a robust means of response assessment in mesothelioma patients, thus motivating future research in more precise semi‐automated segmentation methods.

Purpose: In radiology it is important to understand and interpret anatomical changes which correlate with treatment plans. Several anticancer agents exist that affect tumor vasculature growth and normal vessel growth. Monitoring these effects with CTimaging may be useful to guide dosing of treatment. Observing perfusion of certain organs it possible to monitor vessel competency. If perfusion is constant over time, it can be assumed there is no significant change in vasculature, and no damage due to chemotherapeutics. In contrast, significant changes in perfusion over time may suggest a decrease in normal vessel growth as a result of therapy. Method and Materials: Patients receiving the vascular endothelial growth factor (VEGF) inhibitor sorafenib underwent CTimaging every six weeks, beginning with baseline studies. A “jog scan” tracked perfusion through the highly fenestrated adrenal glands. Sixteen pairs of adrenal images were contoured, with each of the sixteen scans representing different perfusion time intervals from 0–90 seconds. The mean pixel values of each gland were obtained and compared over time for any significant changes in pixel value that could indicate change in vascular perfusion over time. Further calculations were performed isolating the medullary component of the glands because of its high vascularity. Results: The average change in maximum pixel values from baseline to six weeks after treatment initiation demonstrated a 4.58% increase in peak pixel value for both adrenals, with a subsequent decrease of 3.2% in the third scan. Changes in the medullary region demonstrated a 6.1% increase in pixel value in comparison to the entire adrenal area. Conclusion: Manual contouring of adrenal glands in conjunction with calculated maximum pixel values revealed changes in adrenal perfusion between baseline and therapy‐monitoring CT scans. The continued monitoring of perfusion could prove beneficial to the radiologic diagnosis of significant anatomical changes as a result of chemotherapy.

Purpose: The quantification of pleural mesothelioma tumor extent is required to evaluate the efficacy of clinical trials. The manual acquisition of up to three linear tumor thickness measurements on each of three sections across a series of computed tomography(CT) scans is the current standard for tumor response assessment. The purpose of this study was to determine the correlation of response based on linear tumor thickness measurements and response based on tumor area. Method/MATERIALS: Two CT scans from each of 22 mesothelioma patients were collected. Using a computer interface, a radiologist acquired linear tumor thickness measurements on three sections of each patient's baseline scan and on the corresponding sections of each patient's follow‐up scan in accordance with our clinical protocol. These linear measurements across 132 CT sections (3 sections per scan, 2 scans per patient, 22 patients) provided the standard for comparison of area measurements. Another radiologist used a computer interface to delineate the tumor border in the same 132 CT sections to obtain tumor area and the changes in tumor area between the baseline and follow‐up scans of each patient. Results: A comparison of the sum of tumor thickness measurements and tumor area yielded a correlation coefficient of 0.59 across the 132 sections. With regard to tumor response, a comparison of change in the sum of tumor thickness measurements and change in the total tumor area between the baseline and follow‐up scans of the 22 patients yielded a correlation coefficient of 0.83. This relatively high correlation, however, does not capture the extent of variability in the data. For example, among patients with RECIST‐based “stable disease,” change in tumor area ranged from a decrease of 58% to an increase of 89%. Conclusions: Although measurements of tumor thickness and tumor area demonstrated moderate correlation, variability in this association requires further investigation.