Search

A loss function has two crucial roles in training a conventional discriminant deep neural network (DNN): (i) it measures the goodness of classification and (ii) generates the gradients that drive the training of the network. In this paper, we approximate the gradients of cross-entropy loss which is the most often used loss function in the classification DNNs. The proposed approximations are noise-free, which means they depend only on the labels of the training set. They have a fixed length to avoid the vanishing gradient problem of the cross-entropy loss. By skipping the forward pass, the computational complexities of the proposed approximations are reduced to O(n) where n is the batch size. Two claims are established based on the experiments of training DNNs using the proposed approximations: (i) It is possible to train a discriminant network without explicitly defining a loss function. (ii) The success of training does not imply the convergence of network parameters to fixed values. The experiments show that the proposed gradient approximations achieve comparable classification accuracy to the conventional loss functions and can accelerate the training process on multiple datasets.

Real-time semantic segmentation is playing a more important role in computer vision, due to the growing demand for mobile devices and autonomous driving. Therefore, it is very important to achieve a good trade-off among performance, model size and inference speed. In this paper, we propose a Channel-wise Feature Pyramid (CFP) module to balance those factors. Based on the CFP module, we built CFPNet for real-time semantic segmentation which applied a series of dilated convolution channels to extract effective features. Experiments on Cityscapes and CamVid datasets show that the proposed CFPNet achieves an effective combination of those factors. For the Cityscapes test dataset, CFPNet achieves 70.1% class-wise mIoU with only 0.55 million parameters and 2.5 MB memory. The inference speed can reach 30 FPS on a single RTX 2080Ti GPU with a 1024x2048-pixel image.

Although deep neural networks (DNNs) achieve state-of-the-art performances in a wide range of machine learning (ML) applications, they are vulnerable: when small intentional perturbations are added to inputs, the network would misclassify them with high confidence. This phenomenon attracts broad attention because it is a security issue. In this paper, we study the geometric properties of the decision boundaries in the representation space of DNNs and propose novel adversarial approaches by moving the representations of the inputs toward the decision boundaries and thus change the predictions of the DNN. Our experimental results show that the proposed algorithms are on par or better than state-of-the-art adversarial approaches in terms of the magnitude of perturbation and computation time.

Recently, deep learning has become much more popular in computer vision applications. The Convolutional Neural Network (CNN) has brought a breakthrough in image segmentation, especially for medical images. In this regard, the UNet is the predominant approach to the medical image segmentation task. The U-Net not only performs well in segmenting multimodal medical images generally, but also in some difficult cases. We found, however, that the classical U-Net architecture has limitations in several respects. Therefore, we applied modifications: 1) designed efficient CNN architecture to replace encoder and decoder, 2) applied residual module to replace skip connection between encoder and decoder to improve, based on the-state-of-the-art U-Net model. Following these modifications, we designed a novel architecture -- DC-UNet, as a potential successor to the U-Net architecture. We created a new effective CNN architecture and built the DC-UNet based on this CNN. We have evaluated our model on three datasets with difficult cases and have obtained a relative improvement in performance of 2.90%, 1.49%, and 11.42% respectively compared with classical UNet. In addition, we used the Tanimoto similarity measure to replace the Jaccard measure for gray-to-gray image comparisons.

Coronavirus disease 2019 (COVID-19) is an ongoing global pandemic that has spread rapidly since December 2019. Real-time reverse transcription polymerase chain reaction (rRT-PCR) and chest computed tomography (CT) imaging both play an important role in COVID-19 diagnosis. Chest CT imaging offers the benefits of quick reporting, a low cost, and high sensitivity for the detection of pulmonary infection. Recently, deep-learning-based computer vision methods have demonstrated great promise for use in medical imaging applications, including X-rays, magnetic resonance imaging, and CT imaging. However, training a deep-learning model requires large volumes of data, and medical staff faces a high risk when collecting COVID-19 CT data due to the high infectivity of the disease. Another issue is the lack of experts available for data labeling. In order to meet the data requirements for COVID-19 CT imaging, we propose a CT image synthesis approach based on a conditional generative adversarial network that can effectively generate high-quality and realistic COVID-19 CT images for use in deep-learning-based medical imaging tasks. Experimental results show that the proposed method outperforms other state-of-the-art image synthesis methods with the generated COVID-19 CT images and indicates promising for various machine learning applications including semantic segmentation and classification., Comment: Accepted by IEEE Journal of Biomedical and Health Informatics (J-BHI)

Humans can count very fast by subitizing, but slow substantially as the number of objects increases. Previous studies have shown a trained deep neural network (DNN) detector can count the number of objects in an amount of time that increases slowly with the number of objects. Such a phenomenon suggests the subitizing ability of DNNs, and unlike humans, it works equally well for large numbers. Many existing studies have successfully applied DNNs to object counting, but few studies have studied the subitizing ability of DNNs and its interpretation. In this paper, we found DNNs do not have the ability to generally count connected components. We provided experiments to support our conclusions and explanations to understand the results and phenomena of these experiments. We proposed three ML-learnable characteristics to verify learnable problems for ML models, such as DNNs, and explain why DNNs work for specific counting problems but cannot generally count connected components.

The 2019 Novel Coronavirus (COVID-19) has caused an acute shortage of personal protective equipment (PPE) globally as well as shortage in the ability to test PPE such as respirator fit testing. This limits not only the ability to fit PPE to medical practitioners, but also the ability to rapidly prototype and produce alternative sources of PPE as it is difficult to validate fit. At the George Washington University, we evaluated an easily sourced method of qualitative fit testing using a nebulizer or “atomizer” and a sodium saccharin solution in water. If aerosolized saccharin entered candidate masks due to poor fit or inadequate filtration, then a sweet taste was detected in the mouth of the user. This method was tested against previously fit tested Milwaukee N95 and 3D Printed Reusable N95 Respirator as a positive control. A Chinese sourced KN95, cotton cloth material, and surgical mask were tested as other masks of interest. Sensitivity testing was done with no mask prior to fit test. A sweet taste was detected for both the surgical mask and cotton cloth, demonstrating a lack of seal. However, there was no sweet taste detected for the Milwaukee N95, 3D Printed Reusable N95 Respirator, or Chinese KN95. These results demonstrate this could be a valuable methodology for rapid prototyping, evaluation, and validation of fit in a non-clinical environment for use in creation of PPE. This method should be not be used without confirmation in a formal qualitative or quantitative fit test but can be used to preserve those resources until developers are confident that potential new N95 comparable respirators will pass. We strongly suggest validation of masks and respirators with Occupational Safety and Health Administration (OSHA) approved fit testing prior to use in a clinical environment.

To evaluate clustering results is a significant part of cluster analysis. There are no true class labels for clustering in typical unsupervised learning. Thus, a number of internal evaluations, which use predicted labels and data, have been created. They are also named internal cluster validity indices (CVIs). Without true labels, to design an effective CVI is not simple because it is similar to create a clustering method. And, to have more CVIs is crucial because there is no universal CVI that can be used to measure all datasets, and no specific method for selecting a proper CVI for clusters without true labels. Therefore, to apply more CVIs to evaluate clustering results is necessary. In this paper, we propose a novel CVI - called Distance-based Separability Index (DSI), based on a data separability measure. We applied the DSI and eight other internal CVIs including early studies from Dunn (1974) to most recent studies CVDD (2019) as comparison. We used an external CVI as ground truth for clustering results of five clustering algorithms on 12 real and 97 synthetic datasets. Results show DSI is an effective, unique, and competitive CVI to other compared CVIs. In addition, we summarized the general process to evaluate CVIs and created a new method - rank difference - to compare the results of CVIs., Comment: 8 pages, 4 figures. Accepted by IEEE ICTAI 2020 (Long Paper & Oral Presentation)

For supervised learning models, the analysis of generalization ability (generalizability) is vital because the generalizability expresses how well a model will perform on unseen data. Traditional generalization methods, such as the VC dimension, do not apply to deep neural network (DNN) models. Thus, new theories to explain the generalizability of DNNs are required. In this study, we hypothesize that the DNN with a simpler decision boundary has better generalizability by the law of parsimony (Occam's Razor). We create the decision boundary complexity (DBC) score to define and measure the complexity of decision boundary of DNNs. The idea of the DBC score is to generate data points (called adversarial examples) on or near the decision boundary. Our new approach then measures the complexity of the boundary using the entropy of eigenvalues of these data. The method works equally well for high-dimensional data. We use training data and the trained model to compute the DBC score. And, the ground truth for model's generalizability is its test accuracy. Experiments based on the DBC score have verified our hypothesis. The DBC is shown to provide an effective method to measure the complexity of a decision boundary and gives a quantitative measure of the generalizability of DNNs., Comment: 7 pages, 11 figures. Accepted by ICMLA 2020

Automatic segmentation of infection areas in computed tomography (CT) images has proven to be an effective diagnostic approach for COVID-19. However, due to the limited number of pixel-level annotated medical images, accurate segmentation remains a major challenge. In this paper, we propose an unsupervised domain adaptation based segmentation network to improve the segmentation performance of the infection areas in COVID-19 CT images. In particular, we propose to utilize the synthetic data and limited unlabeled real COVID-19 CT images to jointly train the segmentation network. Furthermore, we develop a novel domain adaptation module, which is used to align the two domains and effectively improve the segmentation network's generalization capability to the real domain. Besides, we propose an unsupervised adversarial training scheme, which encourages the segmentation network to learn the domain-invariant feature, so that the robust feature can be used for segmentation. Experimental results demonstrate that our method can achieve state-of-the-art segmentation performance on COVID-19 CT images.

The Generative Adversarial Network (GAN) is a state-of-the-art technique in the field of deep learning. A number of recent papers address the theory and applications of GANs in various fields of image processing. Fewer studies, however, have directly evaluated GAN outputs. Those that have been conducted focused on using classification performance, e.g., Inception Score (IS) and statistical metrics, e.g., Fr\'echet Inception Distance (FID). Here, we consider a fundamental way to evaluate GANs by directly analyzing the images they generate, instead of using them as inputs to other classifiers. We characterize the performance of a GAN as an image generator according to three aspects: 1) Creativity: non-duplication of the real images. 2) Inheritance: generated images should have the same style, which retains key features of the real images. 3) Diversity: generated images are different from each other. A GAN should not generate a few different images repeatedly. Based on the three aspects of ideal GANs, we have designed the Likeness Score (LS) to evaluate GAN performance, and have applied it to evaluate several typical GANs. We compared our proposed measure with two commonly used GAN evaluation methods: IS and FID, and four additional measures. Furthermore, we discuss how these evaluations could help us deepen our understanding of GANs and improve their performance., Comment: 16 pages, 11 figures. Accepted by the Neural Computing and Applications journal

In Deep Neural Networks (DNNs), the learning process (updating the parameters) is driven by the gradients of a loss function; thus, the loss function does not only measures the "goodness" of classification, but also determines the form of representations. In this paper we propose two loss functions for Deep Neural Networks (DNNs), namely Multi-class Discriminant Analysis Loss Function and Maximum Scatter to Closest Neigh-bor Class loss function, which share the same idea, i.e., they minimize the within-class variance (scatter) and maximize the minimum between-class distance. They overcome the problem of overlapping class in the Linear Discriminant Analysis (LDA)-based loss functions and can be used to multi-class classification tasks. We show that our proposed loss functions achieve state-of-the-art accuracy in a medical image classification task. Particularly, in the Optical Coherence Tomography (OCT) image dataset, we got nearly perfect (99.6%) test accuracy.

Breast cancer is the second leading cause of death for women in the U.S. Early detection of breast cancer is key to higher survival rates to breast cancer patients. We are investigating infrared (IR) thermography as a noninvasive adjunct to mammography for breast cancer screening. IR imaging is radiation-free, pain-free, and non-contact. Automatic segmentation of the breast area from the acquired full-size breast IR images will help limit the area for tumor search, as well as reduce the time and effort costs of manual hand segmentation. Autoencoder-like convolutional and deconvolutional neural networks (C-DCNN) had been applied to automatically segment the breast area in IR images in previous studies. In this study, we applied a state-of-the-art deep-learning segmentation model, MultiResUnet, which consists of an encoder part to capture features and a decoder part for precise localization. It was used to segment the breast area by using a set of breast IR images, collected in our clinical trials by imaging breast cancer patients and normal volunteers with our infrared camera (N2 Imager). The database we used has 450 images, acquired from 14 patients and 16 volunteers. We used a thresholding method to remove interference in the raw images and remapped them from the original 16-bit to 8-bit, and then cropped and segmented the 8-bit images manually. Experiments using leave-one-out cross-validation (LOOCV) and comparison with the ground-truth images by using Tanimoto similarity show that the average accuracy of MultiResUnet is 91.47%, which is about 2% higher than that of the autoencoder. MultiResUnet offers a better approach to segment breast IR images than our previous model.

Recently, a number of papers have addressed the theory and applications of the Generative Adversarial Network (GAN) in various fields of image processing. Fewer studies, however, have directly evaluated GAN outputs. Those that have been conducted focused on using classification performance and statistical metrics. In this paper, we consider a fundamental way to evaluate GANs by directly analyzing the images they generate, instead of using them as inputs to other classifiers. We consider an ideal GAN according to three aspects: 1) Creativity: non-duplication of the real images. 2) Inheritance: generated images should have the same style, which retains key features of the real images. 3) Diversity: generated images are different from each other. Based on the three aspects, we have designed the Creativity-Inheritance-Diversity (CID) index to evaluate GAN performance. We compared our proposed measures with three commonly used GAN evaluation methods: Inception Score (IS), Frechet Inception Distance (FID) and 1-Nearest Neighbor classifier (1NNC). In addition, we discuss how the evaluation could help us deepen our understanding of GANs and improve their performance.

Purpose To use machine-learning algorithms and blur measure (BM) operators to automatically detect motion blur in mammograms. Motion blur has been reported to reduce lesion detection performance and mask small abnormalities, resulting in failure to detect them until they reach more advanced stages. Automatic detection of blur could support the clinical decision-making process during the mammography exam by allowing for an immediate retake, thereby preventing unnecessary expense, time, and patient anxiety. Methods Blur was simulated mathematically to mimic the real blur seen in clinical practice. The blur point-spread-function (PSF) mask is generated by distributing pixel intensity of an image pixel moving under random motion within the range of blur effect (the maximum amount of tissue motion allowed). The random motion trajectory vector is generated on a super-sampled image frame to accommodate smaller substeps; the vector was then sampled on a regular pixel grid using subpixel linear interpolation to generate the blur PSF mask. This randomly generated motion trajectory is constrained by several factors: the effects of variations in tissue elasticity, imaging exposure time, and size of blur effect (motion boundary in millimeters) were examined. The blur mask is convolved with a mammogram to create blur. Five motion blur magnitudes (0.1, 0.25, 0.5, 1.0, and 1.5 mm) were simulated on 244 and 434 mammograms from the INbreast and DDSM databases, respectively. Blur was quantified using nine BM operators for each mammogram and at each blur level. The mammograms were assigned to training (70%) and testing (30%) datasets to train three machine-learning classifiers: Ensemble Bagged Trees, fine Gaussian SVM, and weighted KNN, to distinguish five levels of blurred from unblurred mammograms, using six-way classification. Results For the INbreast mammograms, the average classification accuracies were 87.7%, 85.7%, and 85.7% for Ensemble Bagged Trees, fine Gaussian SVM, and weighted KNN, respectively, and the average classification accuracies for DDSM were 93.5%, 93.6%, and 92.7% for Ensemble Bagged Trees, fine Gaussian SVM, and weighted KNN, respectively. Conclusions Preliminary results show the potential to detect simulated blur automatically using those methods. Although limited work has been done to quantify the effects of motion blur on radiologists' performance, there is evidence that motion blur might not be detected visually by a human observer and could negatively affect radiologists' lesion detection performance. As of this date, no other study has investigated the ability of machine-learning classifiers and BM operators to detect motion blur in mammograms.

In the U.S., breast cancer is diagnosed in about 12% of women during their lifetime and it is the second leading reason for women’s death. Since early diagnosis could improve treatment outcomes and longer survival times for breast cancer patients, it is significant to develop breast cancer detection techniques. The Convolutional Neural Network (CNN) can extract features from images automatically and then perform classification. To train the CNN from scratch, however, requires a large number of labeled images, which is infeasible for some kinds of medical image data such as mammographic tumor images. In this paper, we proposed two solutions to the lack of training images. 1)To generate synthetic mammographic images for training by the Generative Adversarial Network (GAN). Adding GAN generated images made to train CNN from scratch successful and adding more GAN images improved CNN’s validation accuracy to at most (best) 98.85%. 2)To apply transfer learning in CNN. We used the pre-trained VGG-16 model to extract features from input mammograms and used these features to train a Neural Network (NN)-classifier. The stable average validation accuracy converged at about 91.48% for classifying abnormal vs. normal cases in the DDSM database. Then, we combined the two deep-learning based technologies together. That is to apply GAN for image augmentation and transfer learning in CNN for breast cancer detection. To the training set including real and GAN augmented images, although transfer learning model did not perform better than the CNN, the speed of training transfer learning model was about 10 times faster than CNN training. Adding GAN images can help training avoid over-fitting and image augmentation by GAN is necessary to train CNN classifiers from scratch. On the other hand, transfer learning is necessary to be applied for training on pure real images. To apply GAN to augment training images for training CNN classifier obtained the best classification performance.

One consensus in the machine learning community is that obtaining good representations of the data is crucial for the classification tasks. But establishing a clear objective for representation learning is an open question and difficult. In this paper, we propose the Discriminant Analysis Loss Function (DALF) for the representation learning in Deep Neural Networks (DNNs). The gradients of DALF explicitly minimize the within-class variances (scatter) and maximize the between-class variances. We use DALF to drive the training of DNNs and call them Discriminant Analysis Deep Neural Networks (DisAnDNNs). Compared to other Linear Discriminant Analysis (LDA)-based cost functions, the computational cost of DALF is drastically reduced by avoiding eigen-decomposition and matrix inversion. We used simple datasets to illustrate the geometric meaning of DALF and compared it with LDA, then experimented with DALF-driven Residual Learning Nets (ResNets) on the pediatric pneumonia (chest X-ray image) dataset. The experimental results show that the DisAnDNNs achieve state-of the-art accuracy in the binary classification task. Particularly, in the pediatric pneumonia dataset, we achieved the accuracy of 96.63%, with a sensitivity of 99.23% and a specificity of 92.30%, all of which are better than the results in the literature that published the dataset.

Breast cancer is the second leading cause of death for women in the U.S. Early detection of breast cancer has been shown to be the key to higher survival rates for breast cancer patients. We are investigating infrared thermography as a noninvasive adjunctive to mammography for breast screening. Thermal imaging is safe, radiation-free, pain-free, and non-contact. Segmentation of breast area from the acquired thermal images will help limit the area for tumor search and reduce the time and effort needed for manual hand segmentation. Autoencoder-like convolutional and deconvolutional neural networks (C-DCNN) are promising computational approaches to automatically segment breast areas in thermal images. In this study, we apply the C-DCNN to segment breast areas from our thermal breast images database, which we are collecting in our clinical trials by imaging breast cancer patients with our infrared camera (N2 Imager). For training the C-DCNN, the inputs are 132 gray-value thermal images and the corresponding manually-cropped breast area images (binary masks to designate the breast areas). For testing, we input thermal images to the trained C-DCNN and the output after post-processing are the binary breast-area images. Cross-validation and comparison with the ground-truth images show that the C-DCNN is a promising method to segment breast areas. The results demonstrate the capability of C-DCNN to learn essential features of breast regions and delineate them in thermal images.

The Convolutional Neural Network (CNN) is a promising technique to detect breast cancer based on mammograms. Training the CNN from scratch, however, requires a large amount of labeled data. Such a requirement usually is infeasible for some kinds of medical image data such as mammographic tumor images. Because improvement of the performance of a CNN classifier requires more training data, the creation of new training images -- image augmentation – could be one solution to this problem. In this study, we applied the Generative Adversarial Network (GAN) to generate synthetic mammographic images from the Digital Database for Screening Mammography (DDSM). From the DDSM, we cropped two sets of regions of interest (ROIs) from the images: normal and abnormal (cancer/tumor) Those ROIs were used to train the GAN, and the GAN then generated synthetic images. To compare the GAN with the affine transformation augmentation methods, such as rotation, shifting, scaling, etc., we used six groups of ROIs (three simple groups: affine augmented, GAN synthetic, real (original), and three mixture groups of each pair of the three simple groups) for each to train a CNN classifier from scratch. And, we used real ROIs that were not used in training to validate classification outcomes. Our results show that, to classify the normal ROIs and abnormal (tumor) ROIs from DDSM, adding GAN-generated ROIs to the training data can reduce overfitting of the classifier. But the affine transformations performed slightly better than GAN. Therefore, GAN could be an optional augmentation approach. The images augmented by GAN or affine transformation cannot substitute entirely for real images to train CNN classifiers because the absence of real images in the training set will cause serious over-fitting with more training.

Direct visualization of the ablated region in the left atrium during radiofrequency ablation (RFA) surgery for treating atrial fibrillation (AF) can improve therapy success rates. Our visualization approach is auto-fluorescence hyperspectral imaging (aHSI), which constructs each hypercube containing 31 auto-fluorescence images of the tissue. We wish to use the spectral information to characterize ablated lesions as being successful or not. In this paper, we reshaped one hypercube to a 2D matrix. Each row (sample) in the matrix represents a pixel in the spatial dimension, and the matrix has 31 columns corresponding to 31 spectral features. Then, we applied k-means clustering to detect ablated regions without a priori knowledge about the lesion. We introduced an accuracy index to evaluate the results of k-means by comparing with the reference truth images quantitatively. To speed-up the detection process, we implemented a grouping procedure to decrease the number of features. The 31 features were divided into four contiguous disjoint groups. In each group, the summation of values yielded a new feature. By the same evaluation method, we found the best 4-feature groups to adequately detect the lesions from all possible combinations. The average accuracy for detection by k-means (ke10) using 31 features was about 74p of reference truth images. And, for using the best grouped 4 features, it was about 95p of that using 31 features. The time cost of 4-feature clustering is about 41p of the 31-feature clustering. We expect that the reduction of time for both acquisition and processing will make possible intraoperative real-time display of ablation status.

This paper proposes the use of near infrared (900 to 1700 nm) transmitted light imaging with a hyperspectral camera to obtain watermarks from prints. Specifically, we show that principal component analysis applied to the hyperspectral image cube collected in the near infrared was able to separate the watermark from text printed in carbon black ink on both sides of a page from the Blaue Atlas Maior of 1662. The resulting principal component image of the watermark was compared with an image obtained using a low-energy x-ray source and a phosphor plate. Low-energy x-radiography is becoming the gold standard for imaging watermarks, replacing beta radiography. The watermark obtained by transmitted near infrared hyperspectral imaging was found to possess many of the key features of the watermark revealed by the phosphor plate radiography. The method proposed here offers an additional way to extract watermarks from works of art on paper.

The registration of technical art conservation images of Old Master paintings presents unique challenges. Specifically, X-radiographs and reflective infrared (1000–2500 nm) images reveal shifted, or new, compositional elements not visible on the surface of artworks. Here, we describe a new multimodal registration and mosaicking algorithm that is capable of providing accurate alignment of a variety of types of images, such as the registration of multispectral reflective infrared images, X-radiographs, hyperspectral image cubes, and X-ray fluorescence image cubes to reference color images taken at high spatial sampling (300–500 pixels per inch), even when content differences are present, and a validation study has been performed to quantify the algorithm’s accuracy. Key to the algorithm’s success is the use of subsets of wavelet images to select control points and a novel method for filtering candidate control-point pairs. The algorithm has been used to register more than 100 paintings at the National Gallery of Art, D.C. and The Art Institute of Chicago. Many of the resulting registered datasets have been published in online catalogues, providing scholars additional information to further their understanding of the paintings and the working methods of the artists who painted them.

In the U.S., breast cancer is diagnosed in about 12 % of women during their lifetime and it is the second leading reason for women's death. Since early diagnosis could improve treatment outcomes and longer survival times for breast cancer patients, it is significant to develop breast cancer detection techniques. The Convolutional Neural Network (CNN) can extract features from images automatically and then perform classification. To train the CNN from scratch, however, requires a large number of labeled images, which is infeasible for some kinds of medical image data such as mammographic tumor images. A promising solution is to apply transfer learning in CNN. In this paper, we firstly tested three training methods on the MIAS database: 1) trained a CNN from scratch, 2) applied the pre-trained VGG-16 model to extract features from input mammograms and used these features to train a Neural Network (NN)-classifier, 3) updated the weights in several final layers of the pre-trained VGG-16 model by back-propagation (fine-tuning) to detect abnormal regions. We found that method 2) is ideal for study because the classification accuracy of fine-tuning model was just 0.008 higher than that of feature extraction model but time cost of feature extraction model was only about 5% of that of the fine-tuning model. Then, we used method 2) to classify regions: benign vs. normal, malignant vs. normal and abnormal vs. normal from the DDSM database with 10-fold cross validation. The average validation accuracy converged at about 0.905 for abnormal vs. normal cases, and there was no obvious overfitting. This study shows that applying transfer learning in CNN can detect breast cancer from mammograms, and training a NN-classifier by feature extraction is a faster method in transfer learning.

A binocular tone-mapping framework can generate a binocular low-dynamic range (LDR) image pair that preserves more human-perceivable visual contents than a single LDR image. In this paper, to solve the temporal coherency problem when extending from this image-level system to a video-level system, we proposed a new binocular framework that integrates the existing image-level framework and the temporal coherency algorithm. The experimental data show that this proposed new framework can effectively solve the temporal coherency problem and generate binocular LDR videos without disturbing effects.

This research extends the Hierarchical Temporal Memory (HTM) algorithm and applies it to gait recognition. The gait sequence first is decomposed into temporal sub-sequences of spatial sub-regions. The sub-sequence are defined as the period of one step and half step, and the sub-regions are defined as the areas that correspond to body parts. Each sub-area will learn the temporal variation of the body part by constructing Markov Chains. Finally, the classification result is the concatenation of the beliefs of all sub-areas. Unlike other methods, which use gait-specific features, our method uses only image patches of sub-areas. Our extension of previous versions of HTM provides hierarchical temporal inference to cumulate the belief. This generalized new approach is evaluated on a dataset of 151 subjects and two walking conditions. It compares favorably to other current methods used with those data, without requiring problem-specific inputs.

In vivo autofluorescence hyperspectral imaging of moving objects can be challenging due to motion artifacts and to the limited amount of acquired photons. To address both limitations, we selectively reduced the number of spectral bands while maintaining accurate target identification. Several downsampling approaches were applied to data obtained from the atrial tissue of adult pigs with sites of radiofrequency ablation lesions. Standard image qualifiers such as the mean square error, the peak signal-to-noise ratio, the structural similarity index map, and an accuracy index of lesion component images were used to quantify the effects of spectral binning, an increased spectral distance between individual bands, as well as random combinations of spectral bands. Results point to several quantitative strategies for deriving combinations of a small number of spectral bands that can successfully detect target tissue. Insights from our studies can be applied to a wide range of applications.

Feature analysis for classification is based on the discriminatory power of features. In previous research, we have presented a metric called relative feature importance (RFI) for measuring the non-parametric discriminatory power (NPDP) of features. RFI has been shown to correctly rank features for a variety of artificial data sets. In this work, we validate RFI on natural data, using several natural data sets.

In this paper, we present an automated multi-modality registration algorithm based on hierarchical feature extraction. The approach, which has not been used previously, can be divided into two distinct stages: feature extraction (edge detection, surface extraction), and geometric matching. Two kinds of corresponding features — edge and surface — are extracted hierarchically from various image modalities. The registration then is performed using least-squares matching of the automatically extracted features. Both the robustness and accuracy of feature extraction and geometric matching steps are evaluated using simulated and patient images. The preliminary results show the error is on the average of one voxel. We have shown the proposed 3D registration algorithm provides a simple and fast method for automatic registering of MR-to-CT and MR-to-PET image modalities. Our results are comparable to other techniques and require no user interaction.

To improve the spatial sampling of scanning hyperspectral cameras, it is often necessary to capture numerous overlapping image cubes and later mosaic them to form the overall image cube. For hyperspectral camera systems having broad-area coverage, whisk-broom scanning using an external mirror is often employed. Creating the final image cube mosaic requires sub-pixel correction of the scan-mirror distortion, as well as alignment of the individual image cubes. For systems lacking geo-positional information that relates sensor to scene, alignment of the image scans is nontrivial. Here we present a novel algorithm that removes scan distortion and aligns hyperspectral image cubes based on correlation of the cubes’ image content with a reference image. The algorithm is able to provide robust results by recognizing that the cubes’ image content will not always match identically with that of the reference image. For example, in cultural heritage applications, the reference color image of the finished painting need not match the under-painting seen in the SWIR. Our approach is to identify a corresponding set of points between the cubes and the reference image, using a subset of wavelet scales, and then filtering out matches that are inconsistent with a map of the distortion. The filtering is performed by removing points iteratively according to their proximity to a function fit to their disparity (distance between the matched points). Our method will be demonstrated and our results validated using hyperspectral image cubes (976-1680 nm) and visible reference images from the fields of remote sensing and cultural heritage preservation.

We present an algorithm for automatically selecting and matching control points for the purpose of registering images acquired using different imaging modalities. The modulus maxima of the wavelet transform were used to define a criterion for identifying control points. This criterion is capable of selecting points based on the size of features in the image. This technique can be tailored, by adjusting the scale of the filters in the modulus calculation, to the specific objects or structures known to occur in each image being registered. The control-point matching technique includes an iterative method for reducing the set of control-point pairs using the horizontal and vertical disparities between the matched pairs of points. Least-squares planes are fit to the horizontal and vertical disparity data, and control-point pairings are deleted based on their distances from those planes. The remaining points are used to recompute the planes. The process is iterated until the remaining points fall within a certain distance from the planes. Finally, a spatial transformation is performed on the template image to bring it into alignment with the reference image. The result of the control-point pair reduction is a more accurate alignment than what would have been produced using the initial control-point pairs. These techniques are applicable to medical images, but examples are given using images of paintings.

A new method has been developed for interpreting the shadows of arbitrarily shaped surfaces by segmenting and labeling the shadow boundary. The method is based on the fact that a linear projection of any light ray (the ray is assumed to originate at a single, distant source) across a shadow either enters or exits the shadow at its boundary. Hence, junctures of entry and exit segments form vertices that can be found directly for any given direction of illumination and view. Entry-exit vertices that are extremes of the boundary (which is normal to the axis of light) can be identified as junctures of specific profiles of the shadow-making object. These junctures, in turn identify the segments connected to them. The method assumes successful lower level extraction of shadow boundaries. When one object occludes part of another object's shadow, critical junctures occur, but these sometimes are not entry-exit vertices. These hidden junctures create ambiguities that must be dealt with in the context of neighboring segments. Certain a priori knowledge is helpful in this situation. The method may require knowledge of the surface or the object. The entry-exit method also provides a new link between the tasks of shadow boundary extraction and shape inference in the overall process of shadow interpretation. In conjunction with existing methods for the other tasks, the entry-exit method makes it possible to interpret arbitrarily shaped shadows.

The abnormal thermogram has been shown to be a reliable indicator of a high risk of breast cancer. Nevertheless, a major weakness of current infrared breast thermography is its poor sensitivity for deeper tumors. Numerical modeling for breast thermography provides an effective tool to investigate the complex relationships between the breast thermal behaviors and the underlying patho-physiological conditions. We have developed a set of new modeling techniques to take into account some subtle factors usually ignored in previous studies, such as gravity-induced elastic deformations of the breast, nonlinear elasticity of soft tissues, and dynamic behavior of thermograms. Conventional "forward problem" modeling cannot be used directly to improve tumor detectability, however, because the underlying tissue thermal properties are generally unknown. Therefore, we propose an "inverse problem" modeling technique that aims to estimate the tissue thermal properties from the breast surface thermogram. Our data suggest that the estimation of the tumor-induced thermal contrast can be improved significantly by using the proposed inverse problem solving techniques to provide the individual-specific thermal background, especially for deeper tumors. We expect the proposed new methods, taken together, to provide a stronger foundation for, and greater specificity and precision in, thermographic diagnosis, and treatment, of breast cancer.

As high-resolution images of paintings, acquired using various imaging modalities (e.g. X-ray, luminescence, visible and infrared reflection) become more available, it is increasingly useful to have accurate registration between them. Accurate registration allows new information to be compiled from the several multimodal images. This leads to a better understanding of how the painting was constructed and of any compositional changes that have occurred. To that end, we have produced an automatic image registration algorithm that is capable of aligning X-ray, color, and infrared images, as well as multispectral luminescence and reflectance image sets, or cubes. The key steps of the algorithm include identifying large sets of candidate control points in the reference image, then pairing them with potential points in a second image using cross-correlation. Finally, after selecting the best set of control point pairs, the second image is transformed to be in register with the reference image. Tests show the algorithm to be capable of achieving sub-pixel registration across these various image modalities.

The abnormal thermogram has been shown to be a reliable indicator of a high risk of breast cancer. Nevertheless, a major weakness of current infrared breast thermography is its poor sensitivity for deeper tumors. Numerical modeling for breast thermography provides an effective tool to investigate the complex relationships between the breast thermal behaviors and the underlying pathophysiological conditions. Conventional “forward problem” modeling cannot be used to directly improve the tumor detectability, however, because the underlying tissue thermal properties are generally unknown. Based on our new comprehensive forward modeling, we propose an “inverse problem” modeling technique that aims to estimate tissue thermal properties from the breast surface thermogram. Our data suggest that the estimation of tumor-induced thermal contrast can be significantly improved by using the proposed inverse problem solving techniques to provide the individual-specific thermal background, especially for deeper tumors.

Texture analysis for tissue characterization is a current area of optical coherence tomography (OCT) research. We discuss some of the differences between OCT systems and the effects those differences have on the resulting images and subsequent image analysis. In addition, as an example, two algorithms for the automatic recognition of bladder cancer are compared: one that was developed on a single system with no consideration for system differences, and one that was developed to address the issues associated with system differences. The first algorithm had a sensitivity of 73% and specificity of 69% when tested using leave-one-out cross-validation on data taken from a single system. When tested on images from another system with a different central wavelength, however, the method classified all images as cancerous regardless of the true pathology. By contrast, with the use of wavelet analysis and the removal of system-dependent features, the second algorithm reported sensitivity and specificity values of 87 and 58%, respectively, when trained on images taken with one imaging system and tested on images taken with another.

Publisher Summary This chapter provides an introduction to various biomedical data processing and analysis methods. Computerized medical image processing and analysis involves specific methods such as image enhancement, segmentation, feature extraction, and image interpretation. Biomedical data processing and analysis has become a major component of biomedical research and clinical applications. Image enhancement is the procedure used to alter the appearance of an image or the subset of the image for better contrast or visualization of certain features and to facilitate the subsequent image-based medical diagnosis. Image enhancement algorithms are categorized into two types: spatial domain– and transform-domain–based methods. Image segmentation is the process of partitioning an image into sets of pixels corresponding to regions of physiologic interest. The segmented image can be used to make measurements such as brain volume, to detect abnormalities, or to visualize areas such as the brain surface. Classification, comparison, or analysis of images is performed almost always in terms of a set of features extracted from the images. This is necessary for one or more of the following reasons: reduction of dimensionality, incorporation of cues from human perception, transcendence of the limits of human perception, or the need for invariance. Medical image interpretation involves how radiologists work and how they interpret a medical image. There are four processes in image interpretation: search, detection/rejection, description, and diagnosis. Diagnoses based on images vary in their certainty, and the degree of uncertainty may be included in the radiologist’s report on an image.

We report on the reproducibility of human observers' vanishing detection thresholds for visual targets in contrast-detail (C/D) analysis of ultrasound B-mode images. The images used in this study contain visual targets which are circular cross sections of constant-contrast conical structures in the C/D phantom. The vanishing threshold diameters for these targets vary as a function of the perceived size of the imaged target, target-to-background contrast, image noise content, and reproducibility of the decision levels of human observers for repeated observations. Our study indicates that the determination of absolute vanishing threshold diameter values for several targets of different contrast by human observers yields a high degree of error that is not predicted by existing theoretical assumptions based on a static threshold detector. We find that systematic error is introduced by the observers during the course of the experiment and that the levels of sensitivity of the observers differ widely at all times, and increase the amount of total observer error. These results suggest that, due to the large total observer error, C/D analysis may be impractical in a clinical environment, unless there is access to a team of observers specifically and extensively trained in this task. We suggest that a computer-based observer may be more reliable for the objective performance of contrast-detail analysis as a method for evaluating ultrasound image quality and comparison of imaging systems.

Model observers have been used successfully to predict human observer performance and to evaluate image quality for detection tasks on various backgrounds in medical applications. This paper will apply the closed-form compression noise statistics in analytic form to model observers and the derived channelized Hotelling observer (CHO) for decompressed images. The performance of CHO on decompressed images is validated using JPEG compression algorithm and lumpy background images. The results show that the derived CHO performance predicts closely its simulated performance.

The lining of the bladder is comprised of well defined layers that are clearly visible in optical coherence tomography (OCT) images of healthy bladder tissue. These layers are disturbed when cancerous cells are present. Consequently, recognition and classification of the layers of the bladder is very important in recognizing and staging bladder cancer. We present an algorithm that uses texture analysis, the k-means clustering algorithm, and edge detection software to segment OCT images of the lining of the bladder. We visually segmented 101 OCT images of bladder tissue, ran our algorithm on the images, and compared the results. The segmentation matched 59% of the time, matched reasonably well, with clear sources of error 29% of the time, and was incorrect 12% of the time. The combination of this algorithm with future texture analysis of the segmented layers will provide the tools required to perform a real-time optical biopsy using OCT

Compression methods based on Gabor functions are implemented for simulated nuclear medicine liver images with and without lesions. The performance of the compression schemes is assessed objectively by comparing the original images to the compressed/reconstructed images through calculation of the Hotelling trace, an index that has been shown to correlate well with performance for images from this imaging modality. Gabor-based compression has not previously been implemented on medical images, nor has any rigorous task-based measure of quality been used to assess the compression. For compression based on thresholding the complex Gabor coefficients, a better than 2∶1 compression is obtained without appreciable reduction in image quality, which when combined with gains expected from bit reduction schemes, corresponds to an overall approximate 8∶1 compression. A large number of nuclear medicine liver images with and without space-occupying lesions were simulated. Then a compression scheme based on transformation of the images into the “information space” proposed by Gabor [1] was implemented. Two tasks were examined: 1) determination of the presence or absence of the lesion in a given location, and 2) determination of the presence or absence of the lesion in one of several locations. The task-based performance using the compressed/reconstructed images is compared to that using the original images according to the Hotelling trace criterion.

The most common metric to assess a classifier's performance is the classification error rate, or the probability of misclassification (PMC). Receiver operating characteristic (ROC) analysis is a more general way to measure the performance. Some metrics that summarize the ROC curve are the two normal-deviate-axes parameters, i.e., a and b, and the area under the curve (AUC). The parameters "a" and "b" represent the intercept and slope, respectively, for the ROC curve if plotted on normal-deviate-axes scale. AUC represents the average of the classifier TPF over FPF resulting from considering different threshold values. In the present work, we used Monte-Carlo simulations to compare different bootstrap-based estimators, e.g., leave-one-out, .632, and .632+ bootstraps, to estimate the AUC. The results show the comparable performance of the different estimators in terms of RMS, while the .632+ is the least biased.

This paper provides the fundamental basis for model observers on decompressed images by the understanding of compression noise statistics. In medical applications, model observers have been successfully used to predict human observer performance and to empirically evaluate image quality for detection tasks on various backgrounds. To derive closed-form expressions for model observers, however, requires closed-form expressions for noise statistics. This paper views a decompressed image as the sum of the original image and compression noise. The statistics of compression noise depend on the compression algorithms. One of the most efficient image compression techniques is transform coding, on which the JPEG image compression standard is based. By analyzing transform coding, this paper derives probability density functions (PDF), and the first and second moments of compression noise. Those statistics are used to derive closed-form representations for the ideal and channelized Hotelling observers on decompressed images. It provides the closed-form decompressed image quality measurements in terms of model observer performance.

The US Army RDECOM CERDEC Night Vision & Electronic Sensors Directorate (NVESD) has a dynamic applied research program in sensor fusion for a wide variety of defense & defense related applications. This paper highlights efforts under the NVESD Sensor Fusion Testbed (SFTB) in the area of detection of moving vehicles with a network of image and acoustic sensors. A sensor data collection was designed and conducted using a variety of vehicles. Data from this collection included signature data of the vehicles as well as moving scenarios. Sensor fusion for detection and classification is performed at both the sensor level and the feature level, providing a basis for making tradeoffs between performance desired and resources required. Several classifier types are examined (parametric, nonparametric, learning). The combination of their decisions is used to make the final decision.

Estimation of volume or the ratio of volumes in an image requires both mensuration and classification. The former is achieved through stereology - a set of techniques that estimate such parameters as area, volume, surface area, length, and number. Classification is achieved by extracting features that capture discriminating information (e.g., about tissue type). Both stereology and classification can be performed either manually or by computer. Manual techniques for the combination are based on coarse point counting (low resolution), but assumed perfect pixel classification. Computer-based methods, on the other hand, rely on very fine point counting but in general suffer from imperfect pixel classification. This paper examines the interaction between manual and image processing-based approaches; in particular, we present a measure that combines the classification and measurement errors. Estimation of the variance is used to define the conditions under which each method is and is not advantageous despite its underlying error. This allows the user to choose a method that optimizes overall performance, given the human and machine capabilities available. Illustrations are given of cases in which each method can be preferable, as measured by the variance of the estimate of the performance that was inferred from the measurement.

Estimation of ratio or volume of tissue types in an image requires both mensuration and classification. The former is achieved through stereology - a set of techniques that estimate such parameters as length, area, surface area, volume, number and ratio. Classification is achieved by extracting features that capture the discriminating information about tissue type. Typically, manual stereological methods are based on uniform sampling. Nonuniform sampling, however, can yield better results for the same manual effort (hence, more-efficient) if we have prior knowledge of the spatial distribution or location of the parameter of interest. Both stereology and classification can be performed either manually or by computer. Manual techniques for the combination are based on coarse point counting (low resolution), but assumed perfect pixel classification. Computer-based methods, on the other hand, rely on very fine point counting but in general suffer from imperfect pixel classification. This paper examines the interaction between manual (nonuniform-sampling) and uniform-sampling image processing-based approaches; in particular, we present a measure that combines the classification and measurement errors. Analysis of the variance is used to define the conditions under which each method and its sampling design is and is not advantageous despite its underlying error. This allows the user to choose a method that optimizes overall performance, given the human and machine capabilities available. Illustrations are given of cases in which each method can be preferable, as measured by the variance of the estimate of the performance that was inferred from the measurement.

An efficient subband implementation of the channelized Hotelling observer is presented, which can be used to assess the image quality of images compressed with wavelet-based techniques. The channelized Hotelling model observer has been shown to predict human performance in detecting signals in noise-limited images. The model observer can also predict degradation of human performance due to lossy compressed images. This provides a more relevant image quality assessment for medical images, where the image's value is in supporting clinical decisions, than metrics such as mean square error. The subband implementation shown is unique in that it operates on channel responses of the wavelet subbands rather than on the entire image itself. The technique is extendable to operate on the channel response of the subband bitmaps, which would permit bit ordering optimized for human performance.

Estimation of volume or area of tissue types in an image requires both mensuration and classification. The former is achieved through stereology -- a set of techniques that estimate such parameters as area, volume, surface area, length, and number. Classification is achieved by extracting features that capture the discriminating information about tissue type. Both stereology and classification can be performed either manually or by computer. Manual techniques for the combination are based on coarse point counting (low resolution), but assumed perfect pixel classification. Computer-based methods, on the other hand, rely on very fine point counting but in general suffer from imperfect pixel classification. This paper examines the interaction between manual and image processing-based approaches; in particular, we present a measure that combines the classification and measurement errors. Estimation of the variance is used to define the conditions under which each method is and is not advantageous despite its underlying error. This allows the user to choose a method that optimizes overall performance, given the human and machine capabilities available. Illustrations are given of cases in which each method can be preferable, as measured by the variance of the estimate of the performance that was inferred from the measurement.

The US Army RDECOM CERDEC Night Vision & Electronic Sensors Directorate has a dynamic applied research program in sensor fusion for a wide variety of defense & defense related applications. This paper provides an overview of the on going research at NVESD related to fusing a mixture of active and passive sensors for countermine, dismounted & mounted soldiers, aviation and unattended ground sensor applications. Highlighted are new techniques in image registration and sensor fusion ion that enable the detection of moving vehicles with a network of image and acoustic sensors. A set of experiments was designed and conducted using a variety of vehicles and scenarios. The incremental value (in terms of error probability) added to the acoustic information by the image sensors (visible and infrared wavelength cameras) is assessed in combination with the fusion techniques themselves. The approach specifically accounts for the effects of location, speed, weather, and background (acoustic, visible, and infrared). Sensor fusion for detection and classification is preformed at both the sensor level and the feature level providing a basis for making tradeoffs between performance desired and resources required. Several classifier types are examined (parametric, nonparametric, learning). The combination of their decisions is used to make the final decision. Keywords: Tracking, fusion, sensor fusion, vehicle detection, detection, location.