Search

The core idea of Multi-Task Learning (MTL) is to develop neural networks with a shared feature extraction backbone and multiple prediction heads, each capable of inferring a different task simultaneously. Parameters in the backbone contribute to all tasks while those in the prediction heads contribute to only one or fewer tasks. Challenges arise when multiple tasks compete for resource. Existing methods focus on resource competition in shared parameters and proposed explanatory factors of task conflicts, task dominance, and gradient stability. However the fundamental nature of MTL is still understudied. In this paper, instead of following the existing methodology research directions, we carry out large-scale empirical study and provide deeper insight on understanding MTL. In particular, instead of focusing on resource competition in the shared parameters in the backbone, we shift our attention to resource competition in the backbone output, which is the embedded representation that is shared by all prediction heads. We show that the existing explanatory factors display weak causal relationship with model performance. We propose a novel measurement, which we term Feature Disentanglement, and show that understanding MTL problems from the perspective of how the shared representation is leveraged by different prediction heads, is a more faithful and reliable way than that from the perspective of how supervision signals from different tasks are interfering in the shared parameters. Additionally, it has been a commonly employed technique to replace gradients w.r.t. shared parameters with gradients w.r.t. shared representation for reduced computation. We conduct a comprehensive study and show that unless a theoretical analysis could be developed, there is not general guarantee that this fast approximation technique would work in practice.

Significant advances have been made in recent years in improving the robustness of deep neural networks, particularly under adversarial machine learning scenarios where the data has been contaminated to fool networks into making undesirable predictions. However, such improvements in adversarial robustness has often come at a significant cost in model accuracy when dealing with uncontaminated data (i.e., clean data), making such defense mechanisms challenging to adapt for real-world practical scenarios where data is primarily clean and accuracy needs to be high. Motivated to find a better balance between adversarial robustness and clean data accuracy, we propose a new model-agnostic adversarial defense mechanism named Dual-model Bounded Divergence (DBD), driven by a theoretical and empirical analysis of the bias-variance trade-off within an adversarial machine learning context. More specifically, the proposed DBD mechanism is premised on the observation that the variance in deep neural networks tends to increase in the presence of adversarial perturbations in the input data. As such, DBD employs a gating mechanism to decide on the final model prediction output based on a novel dual-model variance measure (coined DBD Variance), which is a bounded version of KL-Divergence between models. Not only is the proposed DBD mechanism itself training-free, but it can be combined with existing adversarial defense mechanisms to boost the balance between clean data accuracy and adversarial robustness. Comprehensive experimental results across over 10 different state-of-the-art adversarial defense mechanisms using both CIFAR-10 and ImageNet benchmark datasets across different adversarial attacks (e.g., APGD, AutoAttack) demonstrates that the integration of DBD can lead to as much as a 6% improvement on clean data accuracy without compromising much on adversarial robustness.

Ocular surface temperature (OST) is affected by changes in eye physiology caused by normal homeostasis, environmental changes, or systemic and local disease. OST can help a physician diagnose eye disease with improved accuracy and provide useful information for eye research. This paper presents a novel system, including novel hardware design and novel algorithms, capable of automatically measuring and tracking OST from the cornea over any period of time. The system uses an infrared (IR) camera and a visible (VIS) camera to capture synchronous thermal and visible videos, respectively, from the eye surface. The frames for each camera video sequence are then registered together (video registration) using two sets of control points. The points are manually selected on the first pair of timestamped IR/VIS frames and tracked over all subsequent frames using the Lucas–Kanade (LK) optical flow algorithm (point tracking). A mean square error (MSE) of 5.43±2.01 pixels was reported for salient point tracking of the IR video and 6.81±2.32 pixels for tracking of the VIS video. Overall MSE for registration was 5.03 ±1.82 pixels. The corneal area was segmented in the VIS images and localized on the IR images using the semantic segmentation method (corneal segmentation). A mean Intersection over Union (IoU) of 94.6% was found, representing the accuracy of corneal segmentation. A system for measuring and tracking eye surface temperature over time was developed. The system is able to localize the cornea on both VIS and IR images, and report temperature profiles of the cornea over the period of measurement. Experimental results show that the system can work as a tool for measuring and tracking OST over time.

In this study, we investigated whether self-supervised pretraining could produce a neural network feature extractor applicable to multiple classification tasks in B-mode lung ultrasound analysis. When fine-tuning on three lung ultrasound tasks, pretrained models resulted in an improvement of the average across-task area under the receiver operating characteristic curve (AUC) by 0.032 and 0.061 on local and external test sets respectively. Compact nonlinear classifiers trained on features outputted by a single pretrained model did not improve performance across all tasks; however, they reduced inference time by 49% compared to the serial execution of separate fine-tuned models. When training using 1% of the available labels, pretrained models consistently outperformed fully supervised models, with a maximum observed test AUC increase of 0.396 for the task of view classification. Overall, the results indicate that self-supervised pretraining is a useful strategy for producing initial weights for lung ultrasound classifiers.

While significant research advances have been made in the field of deep reinforcement learning, there have been no concrete adversarial attack strategies in literature tailored for studying the vulnerability of deep reinforcement learning algorithms to membership inference attacks. In such attacking systems, the adversary targets the set of collected input data on which the deep reinforcement learning algorithm has been trained. To address this gap, we propose an adversarial attack framework designed for testing the vulnerability of a state-of-the-art deep reinforcement learning algorithm to a membership inference attack. In particular, we design a series of experiments to investigate the impact of temporal correlation, which naturally exists in reinforcement learning training data, on the probability of information leakage. Moreover, we compare the performance of collective and individual membership attacks against the deep reinforcement learning algorithm. Experimental results show that the proposed adversarial attack framework is surprisingly effective at inferring data with an accuracy exceeding 84% in individual and 97% in collective modes in three different continuous control Mujoco tasks, which raises serious privacy concerns in this regard. Finally, we show that the learning state of the reinforcement learning algorithm influences the level of privacy breaches significantly.

Neural architecture search has proven to be highly effective in the design of efficient convolutional neural networks that are better suited for mobile deployment than hand-designed networks. Hypothesizing that neural architecture search holds great potential for human pose estimation, we explore the application of neuroevolution, a form of neural architecture search inspired by biological evolution, in the design of 2D human pose networks for the first time. Additionally, we propose a new weight transfer scheme that enables us to accelerate neuroevolution in a flexible manner. Our method produces network designs that are more efficient and more accurate than state-of-the-art hand-designed networks. In fact, the generated networks process images at higher resolutions using less computation than previous hand-designed networks at lower resolutions, allowing us to push the boundaries of 2D human pose estimation. Our base network designed via neuroevolution, which we refer to as EvoPose2D-S, achieves comparable accuracy to SimpleBaseline while being 50% faster and $12.7\times $ smaller in terms of file size. Our largest network, EvoPose2D-L, achieves new state-of-the-art accuracy on the Microsoft COCO Keypoints benchmark, is $4.3\times $ smaller than its nearest competitor, and has similar inference speed. The code is publicly available at https://github.com/wmcnally/evopose2d.

Advance knowledge of soil water content (SWC) in the soil wetting layer of crop irrigation can help develop more reasonable irrigation plans and improve the efficiency of agricultural irrigation water use. To improve the accuracy of predicting SWC at multiple depths, the ResBiLSTM model was proposed, in which continuous meteorological and SWC data were gridded and transformed as model inputs, and then high-dimensional spatial and time series features were extracted by ResNet and BiLSTM, respectively, and integrated by a meta-learner. Meteorological, SWC and growth stage records data from seven typical maize monitoring stations in Hebei Province, China, during the 2016-2018 summer maize planting process were utilized for the training, evaluation and testing of the ResBiLSTM model, with model prediction targets set at 20cm, 30cm, 40cm and 50cm depths. Experimental results showed that: 1) ResBiLSTM model could achieve better model fit and prediction of meteorological and SWC data at all growth stages, with R2 within [0.818, 0.991], average MAE within [0.79%, 2.00%], and the overall prediction accuracy ranked as follows: anthesis maturity stage > seedling stage > tassel stage; 2) The average MSE of the ResBiLSTM model for the prediction of SWC in the next 1-6 days was within [3.91%, 15.82%], and the prediction accuracy decreased with the extension of the prediction time; 3) Compared with the classical machine learning model and related deep learning models, the ResBiLSTM model was able to obtain better prediction accuracy performance.

Radiomics-driven computer aided diagnosis (CAD) has shown considerable promise in recent years as a potential tool for improving clinical decision support in medical oncology, particularly those based around the concept of discovery radiomics, where radiomic sequencers are discovered through the analysis of medical imaging data. One of the main limitations, with current CAD approaches, is that it is very difficult to gain insight or rationale as to how decisions are made, thus limiting their utility to clinicians. In this paper, we propose CLEAR-DR, a novel interpretable CAD system based on the notion of CLass-Enhanced Attentive Response Discovery Radiomics for the purpose of clinical decision support for diabetic retinopathy. In addition to disease grading via the discovered deep radiomic sequencer, the CLEAR-DR system also produces a visual interpretation of the decision-making process to provide better insight and understanding of the decision-making process of the system. We demonstrate the effectiveness and utility of the proposed CLEAR-DR system of enhancing the interpretability of diagnostic grading results for the application of diabetic retinopathy grading. CLEAR-DR can act as a potentially powerful tool to address the uninterpretability issue of current CAD systems, thus improving their utility to clinicians.

Harmful algae blooms are a growing global concern since they negatively affect the quality of drinking water. The gold-standard process to identify and enumerate algae requires highly trained professionals to manually observe algae under a microscope. Therefore, an automated approach to identify and enumerate these micro-organisms is needed. This research investigates the feasibility of leveraging machine learning and fluorescence-based spectral-morphological features to enable the identification of six different algae types in an automated fashion. A custom multi-band fluorescence imaging microscope is used to capture fluorescence data of water samples at six different excitation wavelengths ranging from 405 to 530 nm. Automated data processing and segmentation were performed on the captured data to isolate different micro-organisms from the water sample. Different morphological and spectral fluorescence features are then extracted from the isolated micro-organism imaging data and is used to train neural network classification models. The experimental results using three different neural network classification models (one trained on morphological features, one trained on fluorescence-based spectral features, and one trained on fluorescence-based spectral-morphological features) showed that the use of either fluorescence-based spectral features or fluorescence-based spectral-morphological features to train neural network classification models led to statistically significant improvements in identification accuracy when compared to the use of morphological features (with average identification accuracies of 95.7% ± 3.5% and 96.1% ± 1.5%, respectively). These preliminary results are promising and illustrate the feasibility of leveraging machine learning and fluorescence-based spectral-morphological features as a viable method for automated identification of different algae types.

Lung cancer is the leading cause of cancer-related death worldwide. Computer-aided diagnosis (CAD) systems have shown significant promise in recent years for facilitating the effective detection and classification of abnormal lung nodules in computed tomography (CT) scans. While hand-engineered radiomic features have been traditionally used for lung cancer prediction, there have been significant recent successes achieving state-of-the-art results in the area of discovery radiomics. Here, radiomic sequencers comprising of highly discriminative radiomic features are discovered directly from archival medical data. However, the interpretation of predictions made using such radiomic sequencers remains a challenge. A novel end-to-end interpretable discovery radiomics-driven lung cancer prediction pipeline has been designed, build, and tested. The radiomic sequencer being discovered possesses a deep architecture comprised of stacked interpretable sequencing cells (SISC). The SISC architecture is shown to outperform previous approaches while providing more insight in to its decision making process. The SISC radiomic sequencer is able to achieve state-of-the-art results in lung cancer prediction, and also offers prediction interpretability in the form of critical response maps. The critical response maps are useful for not only validating the predictions of the proposed SISC radiomic sequencer, but also provide improved radiologist-machine collaboration for effective diagnosis.

Traffic sign recognition is a very important computer vision task for a number of real-world applications such as intelligent transportation surveillance and analysis. While deep neural networks have been demonstrated in recent years to provide the state-of-the-art performance traffic sign recognition, a key challenge for enabling the widespread deployment of deep neural networks for embedded traffic sign recognition is the high computational and memory requirements of such networks. As a consequence, there are significant benefits in investigating compact deep neural network architectures for traffic sign recognition that are better suited for embedded devices. In this paper, we introduce MicronNet, a highly compact deep convolutional neural network for real-time embedded traffic sign recognition designed based on macroarchitecture design principles (e.g., spectral macroarchitecture augmentation, parameter precision optimization, etc.) as well as numerical microarchitecture optimization strategies. The resulting overall architecture of MicronNet is thus designed with as few parameters and computations as possible while maintaining recognition performance, leading to optimized information density of the proposed network. The resulting MicronNet possesses a model size of just 1 MB and 5 10000 parameters ( 27× fewer parameters than state-of-the-art) while still achieving a human performance level top-1 accuracy of 98.9% on the German traffic sign recognition benchmark. Furthermore, the MicronNet requires just 10 million multiply-accumulate operations to perform inference, and has a time-to-compute of just 32.19 ms on a Cortex-A53 high efficiency processor. These experimental results show that the highly compact, optimized deep neural network architectures can be designed for real-time traffic sign recognition that are well-suited for embedded scenarios.

The use of Markov random fields (MRFs) is a common approach for performing image segmentation, where the problem is modeled using MRFs that incorporate priors on neighborhood nodes to allow for efficient Maximum a Posteriori inference. These local MRF models often result in smoothed segmentation boundaries, since they penalize the assignment of different labels to neighboring pixels and are limited in the use of long-range interactions. Although recent work on fully connected random fields and deep random fields has shown to be very promising in addressing these issues, both streams of approaches face certain limitations, which could affect inference performance and computational tractability. In this paper, we introduce the concept of deep randomly connected conditional random fields DRCRF, which fuse fully-connected random fields and deep random fields together to obtain benefits from long-range interactions while allowing for efficient inference using arbitrary potential functions. Leveraging random graph theory, the concept of stochastic cliques is incorporated into a deep CRF structure to take better advantage of long-range interactions while maintaining computational tractability. The experimental results demonstrate that the proposed DRCRF framework outperforms existing fully connected CRF frameworks and provides results comparable to the principled deep random field framework, which is among the state of the art in random field frameworks for image segmentation.

Automated analysis of broadcast soccer game video is a challenging computer vision problem. Prior to performing high-level analysis (such as event detection), accurate classification of shot views and play-break segmentation are required to analyze the structure of soccer video. A novel deep network called parallel feature fusion network (PFF-Net) combines local and full-scene features to produce accurate shot view classification based on camera zoom and out-of-field status. Then, a novel hidden-to-observable Markov model (H2O-MM) is introduced to determine play/break status of the shots. Testing is performed using a variety of professional broadcast soccer videos. Variations of the PFF-Net are considered and compared with four existing methods where the PFF-Net demonstrates superior performance (92.6%). The H2O-MM has the accuracy of 98.7% for play-break segmentation, which is an improvement over two existing hidden Markov models. The new methods provide improved temporal labeling of broadcast soccer videos, which can be used to further improve overall automated event detection.

We propose an effective framework for salient region detection in natural images based on the concept of self-guided statistical non-redundancy (SGNR). Salient regions are unique, because they have low information redundancy within a given image, while the rest of the scene may highly be redundant. We first analyze the structural characteristics of the image using structured image elements (samples) and classify them as being non-redundant or redundant based on textural compactness and overall non-redundancy. This guides saliency detection toward regions with low information redundancy by considering explicitly high information redundancy of samples potentially belonging to the background. We then compute the saliency map by determining the statistical non-redundancy of each sample using a conditional graph model. Experimental results based on publicly available data sets show that SGNR provides promising results when compared with existing saliency approaches.

Deep neural networks are a branch in machine learning that has seen a meteoric rise in popularity due to its powerful abilities to represent and model high-level abstractions in highly complex data. One area in deep neural networks that are ripe for exploration is neural connectivity formation. A pivotal study on the brain tissue of rats found that synaptic formation for specific functional connectivity in neocortical neural microcircuits can be surprisingly well modeled and predicted as a random formation. Motivated by this intriguing finding, we introduce the concept of StochasticNet where deep neural networks are formed via stochastic connectivity between neurons. As a result, any type of deep neural networks can be formed as a StochasticNet by allowing the neuron connectivity to be stochastic. Stochastic synaptic formations in a deep neural network architecture can allow for efficient utilization of neurons for performing specific tasks. To evaluate the feasibility of such a deep neural network architecture, we train a StochasticNet using four different image datasets (CIFAR-10, MNIST, SVHN, and STL-10). Experimental results show that a StochasticNet using less than half the number of neural connections as a conventional deep neural network achieves comparable accuracy and reduces overfitting on the CIFAR-10, MNIST, and SVHN data sets. Interestingly, StochasticNet with less than half the number of neural connections, achieved a higher accuracy (relative improvement in test error rate of ~6% compared to ConvNet) on the STL-10 data set than a conventional deep neural network. Finally, the StochasticNets have faster operational speeds while achieving better or similar accuracy performances.

The broadband spectrum contains significantly more information than what the human eye can detect, with different wavelengths providing unique information about the intrinsic properties of an object. Recently, compressive sensing-based strategies for multi-spectral imaging via wavelength filtering at the pixel level on the imaging detector have been proposed for simultaneous acquisition of multi-spectral imaging data greatly reducing the acquisition times. To utilize such compressive sensing strategies for multi-spectral imaging, strong reconstruction algorithms that can reconstruct dense multi-spectral image cubes from the sparse compressively sensed observations are required. This paper proposes a comprehensive inter-spectral multi-layered conditional random field (IS-MCRF) sparse reconstruction framework for multi-spectral compressively sensed data captured using such acquisition strategies. The IS-MCRF framework leverages the information between neighboring spectral bands to better utilize the available information for reconstruction. The proposed framework was evaluated using compressively sensed multi-spectral acquisitions ranging from visible to near infrared spectral bands obtained by a simulated compressive sensing-based multi-spectral imaging system. Results show noticeable improvement over the existing sparse reconstruction techniques for compressive sensing-based multi-spectral imaging systems in preserving spatial and spectral fidelity.

There has been significant interest in the use of fully connected graphical models and deep-structured graphical models for the purpose of structured inference. However, fully connected and deep-structured graphical models have been largely explored independently, leaving the unification of these two concepts ripe for exploration. A fundamental challenge with unifying these two types of models is in dealing with computational complexity. In this paper, we investigate the feasibility of unifying fully connected and deep-structured models in a computationally tractable manner for the purpose of structured inference. To accomplish this, we introduce a deep-structured fully connected random field (DFRF) model that integrates a series of intermediate sparse autoencoding layers placed between state layers to significantly reduce the computational complexity. The problem of image segmentation was used to illustrate the feasibility of using the DFRF for structured inference in a computationally tractable manner. Results in this paper show that it is feasible to unify fully connected and deep-structured models in a computationally tractable manner for solving structured inference problems such as image segmentation.

Multiscale decomposition has been an invaluable tool for the processing of physiological signals. Much focus on multiscale decomposition for processing such signals have been based on scale-space theory and wavelet transforms. In this paper, we take a different perspective on multiscale decomposition by investigating the feasibility of utilizing a Bayesian-based method for multiscale signal decomposition called Bayesian residual transform (BRT) for the purpose of physiological signal processing. In BRT, a signal is modeled as the summation of residual signals, each characterizing information from the signal at different scales. A deep cascading framework is introduced as a realization of the BRT. Signal-to-noise ratio analysis using electrocardiography signals was used to illustrate the feasibility of using the BRT for suppressing the noise in physiological signals. Results in this paper show that it is feasible to utilize the BRT for processing physiological signals for tasks, such as noise suppression.

The use of high-volume quantitative radiomics features extracted from multi-parametric magnetic resonance imaging (MP-MRI) is gaining attraction for the autodetection of prostate tumors, since it provides a plethora of mineable data, which can be used for both detection and prognosis of prostate cancer. While current voxel-resolution radiomics-driven prostate tumor detection approaches utilize quantitative radiomics features associated with individual voxels on an independent basis, the incorporation of additional information regarding the spatial and radiomics feature relationships between voxels has significant potential for achieving a more reliable detection performance. Motivated by this, we present a novel approach for automatic prostate cancer detection using a radiomics-driven conditional random field (RD-CRF) framework. In addition to the high-throughput extraction and utilization of a comprehensive set of voxel-level quantitative radiomics features, the proposed RD-CRF framework leverages inter-voxel spatial and radiomics feature relationships to ensure that the autodetected tumor candidates exhibit interconnected tissue characteristics reflective of cancerous tumors. We evaluated the performance of the proposed framework using clinical prostate MP-MRI data of 20 patients, and the results of RD-CRF framework demonstrated a clear improvement with respect to the state-of-the-art in quantitative radiomics for automatic voxel-resolution prostate cancer detection.