Search

Rationale and Objectives Participation in clinical research can be both highly rewarding and logistically demanding. As highlighted by recent Food and Drug Administration guidance, imaging has become an integral part of this research. The unique technical and administrative aspects of clinical trial imaging may differ substantially from those of standard-of-care imaging and thus burden the established clinical infrastructure at investigational sites. Failure to comply with requirements can lead to unusable data, repeat imaging, or the removal of patients from the trial. It is therefore imperative that all stakeholders address these challenges to engage in clinical research successfully. Materials and Methods The authors’ experiences in managing clinical trial imaging requirements at their institution were used to identify common challenges. The impact of these challenges was assessed from an operational perspective. Results Although contract research organizations attempt to minimize these challenges, their efforts are necessarily limited and insufficient, and there is a lack of infrastructure available at investigational sites to address these issues. As such, recommendations are proposed for addressing these challenges at institutional and industry levels. Conclusion The challenges associated with clinical trial imaging require an investment of resources from all stakeholders. Investigational sites must confront these challenges to satisfy trial requirements effectively, maintain a superior level of patient care, and guarantee trial integrity. Similarly, sponsors must acknowledge the burden of clinical trial imaging and support the development of the necessary local infrastructure. The implementation of the recommendations described here will improve the conduct of clinical trial imaging.

Rationale and Objectives Managing and supervising the complex imaging examinations performed for clinical research in an academic medical center can be a daunting task. Coordinating with both radiology and research staff to ensure that the necessary imaging is performed, analyzed, and delivered in accordance with the research protocol is nontrivial. The purpose of this communication is to report on the establishment of a new Human Imaging Research Office (HIRO) at our institution that provides a dedicated infrastructure to assist with these issues and improve collaborations between radiology and research staff. Materials and Methods The HIRO was created with three primary responsibilities: 1) coordinate the acquisition of images for clinical research per the study protocol, 2) facilitate reliable and consistent assessment of disease response for clinical research, and 3) manage and distribute clinical research images in a compliant manner. Results The HIRO currently provides assistance for 191 clinical research studies from 14 sections and departments within our medical center and performs quality assessment of image-based measurements for six clinical research studies. The HIRO has fulfilled 1806 requests for medical images, delivering 81,712 imaging examinations (more than 44.1 million images) and related reports to investigators for research purposes. Conclusions The ultimate goal of the HIRO is to increase the level of satisfaction and interaction among investigators, research subjects, radiologists, and other imaging professionals. Clinical research studies that use the HIRO benefit from a more efficient and accurate imaging process. The HIRO model could be adopted by other academic medical centers to support their clinical research activities; the details of implementation may differ among institutions, but the need to support imaging in clinical research through a dedicated, centralized initiative should apply to most academic medical centers.

Rationale and Objectives The automated classification of sonographic breast lesions is generally accomplished by extracting and quantifying various features from the lesions. The selection of images to be analyzed, however, is usually left to the radiologist. Here we present an analysis of the effect that image selection can have on the performance of a breast ultrasound computer-aided diagnosis system. Materials and Methods A database of 344 different sonographic lesions was analyzed for this study (219 cysts/benign processes, 125 malignant lesions). The database was collected in an institutional review board–approved, Health Insurance Portability and Accountability Act–compliant manner. Three different image selection protocols were used in the automated classification of each lesion: all images, first image only, and randomly selected images. After image selection, two different protocols were used to classify the lesions: (a) the average feature values were input to the classifier or (b) the classifier outputs were averaged together. Both protocols generated an estimated probability of malignancy. Round-robin analysis was performed using a Bayesian neural network-based classifier. Receiver-operating characteristic analysis was used to evaluate the performance of each protocol. Significance testing of the performance differences was performed via 95% confidence intervals and noninferiority tests. Results The differences in the area under the receiver-operating characteristic curves were never more than 0.02 for the primary protocols. Noninferiority was demonstrated between these protocols with respect to standard input techniques (all images selected and feature averaging). Conclusion We have proved that our automated lesion classification scheme is robust and can perform well when subjected to variations in user input.

This chapter will focus on the robustness, or reliability, of computer-aided diagnosis, abbreviated as CADx. In CADx, computerized analysis is used to characterize abnormalities identified by a human user (e.g., a radiologist). CADx is part of the broader area of quantitative imaging (Giger, Chan, & Boone, 2008). Other examples of quantitative imaging applications include computer-aided detection (Drukker, et al., 2002; Drukker, Giger, & Mendelson, 2003), computer-aided prognosis (Bhooshan, et al., 2010), and computer-aided assessment of response to therapy (Shi, et al., 2009; Shiraishi, Li, Appelbaum, Pu, & Doi, 2007). The task of primary interest in this chapter is the diagnosis of breast cancer using sonographic images(i.e., the distinction between benign and malignant breast lesions). The only steps that require human interaction are the identification of suspicious abnormalities prior to the computerized analysis by the CADx system and the interpretation of the CADx results. It is important to note that CADx only provides the radiologist with additional information. The final decision as to whether or not a given abnormality is worrisome lies with the radiologist. In order for CADx to be of potential help it is required that a CADx system demonstrates not only good but also consistent performance. The latter aspect is the focus of this chapter in which the authors will explore causes of variability and investigate CADx performance under different realistic scenarios.

To evaluate the robustness of a breast ultrasonographic (US) computer-aided diagnosis (CAD) system in terms of its performance across different patient populations.Three US databases were analyzed for this study: one South Korean and two United States databases. All three databases were utilized in an institutional review board-approved and HIPAA-compliant manner. Round-robin analysis and independent testing were performed to evaluate the performance of a computerized breast cancer classification scheme across the databases. Receiver operating characteristic (ROC) analysis was used to evaluate performance differences.The round-robin analyses of each database demonstrated similar results, with areas under the ROC curve ranging from 0.88 (95% confidence interval [CI]: 0.820, 0.918) to 0.91 (95% CI: 0.86, 0.95). The independent testing of each database, however, indicated that although the performances were similar, the range in areas under the ROC curve (from 0.79 [95% CI: 0.730, 0.842] to 0.87 [95% CI: 0.794, 0.923]) was wider than that with the round-robin tests. However, the only instances in which statistically significant differences in performance were demonstrated occurred when the Korean database was used in a testing capacity in independent testing.The few observed statistically significant differences in performance indicated that while the US features used by the system were useful across the databases, their relative importance differed. In practice, this means that a CAD system may need to be adjusted when applied to a different population.

We investigated three classifiers for the task of distinguishing between benign and malignant breast lesions. Classification performance was measured in terms of area under the ROC curve (AUC value). We compared linear discriminant analysis (LDA), quadratic discriminant analysis (QDA) and a Bayesian neural net (BNN) with 5 hidden units. For each lesion, 46 image features were extracted and principal component analysis (PCA) of these features was used as classifier input. For each classifier, the optimal number of principal components was determined by performing PCA within each step of a leave-one-case-out protocol for the training dataset (1125 lesions, 14% cancer prevalence) and determining which number of components maximized the AUC value. Subsequently, each classifier was trained on the training dataset and applied ‘cold turkey’ to an independent test set from a different population (341 lesions, 30% cancer prevalence). The optimal number of principal components for LDA was 24, accounting for 97% of the variance in the image features. For QDA and BNN, these numbers were 5 (70%) and 15 (93%), respectively. The LDA, QDA and BNN obtained AUC values of 0.88, 0.85, and 0.91, respectively, in the leave-one-case-out analysis. In the independent test – with AUCs of 0.88, 0.76, and 0.82 – only LDA achieved performance identical to that for the training set (lower bound of 95% non-inferiority interval -.0067), while the others performed significantly worse (p-values << 0.05). While the more complex BNN classifier outperformed the others in leave-one-case-out of a large dataset, LDA was the robust best-performer in an independent test.

To evaluate the performance of a computer-aided diagnosis (CAD) workstation in classifying cancer in a realistic data set representative of a clinical diagnostic breast ultrasonography (US) practice.The database consisted of consecutive diagnostic breast US scans collected with informed consent with a protocol approved by the institutional review board and compliant with the HIPAA. Images from 508 patients with a total of 1046 distinct abnormalities were used. One hundred one patients had breast cancer. Results both for patients in whom the lesion abnormality was proved with either biopsy or aspiration (n = 183) and for all patients irrespective of biopsy status (n = 508) are presented. The ability of the CAD workstation to help differentiate malignancies from benign lesions was evaluated with a leave-one-out-by-case analysis. The clinical specificity of the radiologists for this dataset was determined according to the biopsy rate and outcome.In the task of differentiating cancer from all other lesions sent to biopsy, the CAD workstation obtained an area under the receiver operating characteristic curve (AUC) value of 0.88, with 100% sensitivity at 26% specificity (157 cancers and 362 lesions total). The radiologists' specificity at 100% sensitivity for this set was zero. When analyzing all lesions irrespective of biopsy status, which is more representative of actual clinical practice, the CAD scheme obtained an AUC of 0.90 and 100% sensitivity at 30% specificity (157 cancers and 1046 lesions total). The radiologists' specificity at 100% sensitivity for this set was 77%.Current levels of computer performance warrant a clinical evaluation of the potential of US CAD to aid radiologists in lesion work-up recommendations.

The development of image databases for CAD research is not a trivial task. The collection and management of images and their related metadata from multiple sources is a time-consuming but necessary process. By standardizing and centralizing the methods in which these data are maintained, one can generate subsets of a larger database that match the specific criteria needed for a particular research project in a quick and efficient manner. A research-oriented management system of this type is highly desirable in a multi-modality CAD research environment. An online, webbased database system for the storage and management of research-specific medical image metadata was designed for use with four modalities of breast imaging: screen-film mammography, full-field digital mammography, breast ultrasound and breast MRI. The system was designed to consolidate data from multiple clinical sources and provide the user with the ability to anonymize the data. Input concerning the type of data to be stored as well as desired searchable parameters was solicited from researchers in each modality. The backbone of the database was created using MySQL. A robust and easy-to-use interface for entering, removing, modifying and searching information in the database was created using HTML and PHP. This standardized system can be accessed using any modern web-browsing software and is fundamental for our various research projects on computer-aided detection, diagnosis, cancer risk assessment, multimodality lesion assessment, and prognosis. Our CAD database system stores large amounts of research-related metadata and successfully generates subsets of cases that match the user's desired search criteria.

To be clinically viable, computer-aided diagnosis (CAD) systems must be as automated and user-friendly as possible. CAD systems for breast ultrasound are still preliminary and are not adapted for use in a standard clinical environment. For example, computer detection and classification schemes need the pixel size of each image to operate correctly, and while the DICOM standard allows pixel size to be encoded in the image file, some equipment manufacturers neglect to utilize the encoding. As a result, the pixel size is calculated from user input. In order to increase clinical efficiency and reduce the likelihood of error due to incorrect image specifications, automating this input process is a highly desirable asset. We developed and applied a character recognition algorithm to the annotation region of each ultrasound image in our database. A set of numerical masks, which corresponded to the characters used in the annotation information, enabled the filtering of each image. Numerical masks yielding the maximum output from the comparison operation between image data and mask were output to obtain the annotation information. Each image was then automatically cropped to remove the annotation banner and leave only the image data. The cropped image matrix dimensions and character recognition output were used to determine the corresponding pixel size. The algorithm was tested on 1110 images with various pixel sizes. In every case, the value output by the algorithm corresponded exactly to the true value. Our recognition algorithm now allows for the clinical translation of our fully-automated breast ultrasound CAD system.

Purpose: The clinical imaging infrastructure has been designed for the consistent execution of standard‐of‐care imaging examinations. Many imaging protocols for clinical research, however, have unique requirements that may burden the established infrastructure. Our purpose was to develop a solution to the many imaging demands and challenges presented by clinical research.Methods: We have created the Human Imaging Research Office (HIRO) at our institution. The HIRO is responsible for coordinating the acquisition, collection, analysis, and maintenance of images and related data associated with clinical research studies involving human subjects. The purpose of the HIRO is to coordinate, harmonize, and streamline the imaging aspects of clinical research, so that these non‐standard examinations can be performed as required by the study protocol. In effect, the HIRO is becoming the institutional resource for a broad spectrum of research imaging needs. The activities and services of the HIRO include site validation and image quality verification, development of standardized research‐specific requisition and reporting forms, scheduling, implementation of research billing codes, study‐specific image‐basedmeasurements, collection of customized imagedatabases, and distribution of anonymized images to researchers and study sponsors.Results: The HIRO has helped facilitate the acquisition of images for clinical research for 117 active, pending, or recently closed clinical research studies. These clinical research studies are being conducted by 60 unique principal investigators across 14 different sections and departments within the Medical Center. The HIRO has filled over 650 requests for de‐identified images, accounting for over 42.3 million images Conclusions: Imaging for clinical research often has requirements that differ substantially from those of clinical imaging. The complex infrastructure for standard‐of‐care imaging typically is stressed by the unique technical and administrative needs of a research protocol. The HIRO is able to respond effectively to the dynamic imaging needs of clinical research. SGA, HM, and MLG receive royalties and licensing fees through the University of Chicago related to computer‐aided diagnosis. HM is a consultant for Riverain Medical.

Purpose: To determine the accuracy and variability of an automated sonographic breast lesion segmentation algorithm when it is used across different patient populations. Method and Materials: Two sonographic databases containing images of breast lesions were collected: one consisting of 456 lesions (145 malignant, 311 benign) from patients in Asia(database A), and one consisting of 433 lesions (127 malignant, 306 benign) from patients in the United States (database B). The same model of ultrasoundscanner was used to generate the images in both databases. Our average radial derivative‐based segmentation method was used to segment all the lesions in each database by using the center of each lesion as the starting seed point. The amount of overlap between the automated segmentation and an outline drawn by a radiologist was calculated for each lesion using O = area(S∩R)/area(S∪R) where S is the automated segmentation and R is the radiologist‐drawn outline. An overlap value of less than 0.4 is considered to be poor. Results: In database A, 85% of the lesions had an overlap value of 0.4 or greater. In database B, 80% of the lesions had an overlap value of 0.4 or greater. The automated segmentation performed better on the Asian database (p‐value=0.003). Conclusion: While the automated segmentation algorithm performs well on both databases, the disparity in performance raises interesting questions. The difference in performance could be the result of variation due to different radiologist‐drawn truths, differences in clinical imaging protocol, or even differences between the breast anatomy of the American and Asian populations. Further investigation is necessary to determine both the cause and magnitude of the variability. Conflict of Interest: Research supported in part by NIH. Some authors receive royalties, research funding, and/or are stock holders in Hologic.

Purpose: To determine the robustness of a breast ultrasound computer‐aided diagnosis system (CADx) when it is used across different patient populations. Method and Materials: A sonographic database consisting of 433 lesions (127 malignant, 306 benign) from patients in the United States was used in the training of our breast ultrasoundCADx system. A second sonographic database consisting of 456 lesions (145 malignant, 311 benign) was obtained from a separate patient population in Asia and used to test the trained classifier. Four sonographic features were extracted from each lesion (shape, margin sharpness, posterior acoustic behavior, and texture). These features were used to train a Bayesian neural network classifier. Both round‐robin and independent testing were used to evaluate the classifier with the two databases. Performance was assessed by calculating the area under the ROC curve (AUC) for each test. Results: The AUC of the independent test was 0.80 while the AUCs for the round‐robin tests were 0.87 and 0.88 for the Asian and American databases, respectively. The difference between the independent test AUC and the round‐robin AUC for the Asian database was statistically significant (p‐value=0.02). Conclusion: This work indicates that the breast ultrasoundCADx system is moderately robust across different patient populations. The statistically significant difference between the AUCs of the Asian database along with the similarity of the round‐robin AUCs indicate that while the sonographic features used by the system are useful in both databases, their relative importance differs. This motivates the future exploration of optimal feature sets to improve the overall performance of the CADx system. Conflict of Interest: Research supported in part by NIH. Some authors receive royalties, research funding, and/or are stock holders in Hologic.

Purpose: To investigate the performance of a computerized lesion classification scheme in a realistic testing protocol resembling clinical practice. Computerized classification of breast lesions has generally been tested on lesions of biopsy-proven pathology. In that fashion, the known pathology serves as the truth in the performance evaluation of the computer. In practice, however, many patients are never sent to biopsy because their lesions are deemed to be most likely benign. In order for a computerized classification scheme to be useful and its results believable to the radiologists, it needs to be able to classify those lesions correctly. Method and Materials: We investigated the performance of our computerized lesion segmentation and classification scheme. There were 42 images of 11 cancerous lesions, 114 images of 30 biopsy-proven benign lesions (including both cystic and solid lesions), and 243 images of 57 lesions that to date have not been sent to biopsy (including suspected cysts, benign solid lesions, and other benign breast disease). The computer was trained on the biopsy-proven set and we performed stepwise feature selection to obtain a 4-feature subset that best distinguished malignant from biopsy-proven benign lesions. The computer scheme was tested on the non-biopsy cases, and the ability to distinguish these from the cancers in the training set was assessed. Results: The area under the ROC curve (Az value) was 0.96 for the training of the scheme on biopsy-proven pathologies in the distinction between cancers and benign lesions. The Az value was 0.93 for testing in the distinction between cancers and non-biopsy-proven benign lesions. Conclusion: Our computerized classification scheme shows promising performance in a testing protocol that is more representative of its intended use in clinical practice than the typical testing on lesions with biopsy-proven pathology only. COI: grants: USPHS and U.S. Army. Shareholder: R2 Technology (last author)