Search

Saliency detection by human refers to the ability to identify pertinent information using our perceptive and cognitive capabilities. While human perception is attracted by visual stimuli, our cognitive capability is derived from the inspiration of constructing concepts of reasoning. Saliency detection has gained intensive interest with the aim of resembling human ‘perceptual’ system. However, saliency related to human ‘cognition’, particularly the analysis of complex salient regions (‘cogitating’ process), is yet to be fully exploited. We propose to resemble human cognition, coupled with human perception, to improve saliency detection. We recognize saliency in three phases (‘Seeing’ - ‘Perceiving’ - ‘Cogitating), mimicking human's perceptive and cognitive thinking of an image. In our method, ‘Seeing’ phase is related to human perception, and we formulate the ‘Perceiving’ and ‘Cogitating’ phases related to the human cognition systems via deep neural networks (DNNs) to construct a new module (Cognitive Gate) that enhances the DNN features for saliency detection. To the best of our knowledge, this is the first work that established DNNs to resemble human cognition for saliency detection. In our experiments, our approach outperformed 17 benchmarking DNN methods on six well-recognized datasets, demonstrating that resembling human cognition improves saliency detection. [ABSTRACT FROM AUTHOR]

With an increasing number of remote sensing images (RSIs), the automatic region of interest (ROI) extraction based on convolutional neural networks (CNNs) has attracted much interest in recent years. Although fully supervised CNN-based methods have shown superiority in the field of object extraction, pixelwise annotations are expensive and time-consuming. Moreover, due to the unstable distribution of ROIs in complex landscapes, the ratios of foreground and background areas are quite different in RSIs. Training CNNs with such unbalanced data sets lead to over-fitting and low accuracy. In this article, we propose a framework that combines multiview learning and attention mechanism (MLAM) to solve the above mentioned problems. First, we develop a CNN-based weakly supervised method with a weight-balanced loss function to solve the problems caused by an unbalanced data set. It also helps to generate imagewise saliency maps by computing the gradient maps with respect to the input images. Then, we design a multiview strategy to dramatically reduce the missing inspection. Finally, we design a feedback attention mechanism based on the stage neighbor binary pattern to further modify the extraction result. In summary, the proposed framework achieves pixelwise ROI extraction under imagewise annotations through data-driven ML and a knowledge-driven visual attention mechanism. We evaluate the performance of the MLAM framework on two challenging data sets with complex backgrounds. The experimental results indicate that the proposed framework can achieve better performance than other eight ROI extraction models for unbalanced remote sensing data sets. [ABSTRACT FROM AUTHOR]

Image annotation aims to annotate a given image with a variable number of class labels corresponding to diverse visual concepts. In this paper, we address two main issues in large-scale image annotation: 1) how to learn a rich feature representation suitable for predicting a diverse set of visual concepts ranging from object, scene to abstract concept and 2) how to annotate an image with the optimal number of class labels. To address the first issue, we propose a novel multi-scale deep model for extracting rich and discriminative features capable of representing a wide range of visual concepts. Specifically, a novel two-branch deep neural network architecture is proposed, which comprises a very deep main network branch and a companion feature fusion network branch designed for fusing the multi-scale features computed from the main branch. The deep model is also made multi-modal by taking noisy user-provided tags as model input to complement the image input. For tackling the second issue, we introduce a label quantity prediction auxiliary task to the main label prediction task to explicitly estimate the optimal label number for a given image. Extensive experiments are carried out on two large-scale image annotation benchmark datasets, and the results show that our method significantly outperforms the state of the art. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have recently shown excellent performance on the task of fine-grained vehicle classification, where the motivation is to identify the fine-grained categories of the given vehicles. Generally speaking, the main motivation of the conventional back-propagation algorithm is to optimize the loss function. The algorithm itself does not guarantee if the extracted features are discriminative for the task of classification. Intuitively, if we can learn more discriminative features with a relatively small number of feature maps, the generalization ability of the CNNs will be significantly improved. Therefore, we propose a channel max pooling (CMP) scheme, where a new layer is inserted between the fully connected layers and the convolutional layers. The proposed CMP scheme divides the feature maps into to several sub-groups. Then, it compresses the feature maps within each sub-group into a new one. The compression is carried out by selecting the maximum value among the same locations from different feature maps. Moreover, the proposed CMP layer has the advantage that it can reduce the number of parameters via reducing the number of channels in the CNNs. Experimental results on two fine-grained vehicle datasets demonstrate that the CMP modified CNNs can improve the classification accuracies on the task of fine-grained vehicle classification while a massive amount of parameters are reduced. Moreover, it has competitive performance when comparing with the-state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Remote sensing image scene classification is a fundamental problem, which aims to label an image with a specific semantic category automatically. Recently, deep learning methods have achieved competitive performance for remote sensing image scene classification, especially the methods based on a convolutional neural network (CNN). However, most of the existing CNN methods only use feature vectors of the last fully connected layer. They give more importance to global information and ignore local information of images. It is common that some images belong to different categories, although they own similar global features. The reason is that the category of an image may be highly related to local features, other than the global feature. To address this problem, a method based on rearranged local features is proposed in this paper. First, outputs of the last convolutional layer and the last fully connected layer are employed to depict the local and global information, respectively. After that, the remote sensing images are clustered to several collections using their global features. For each collection, local features of an image are rearranged according to their similarities with local features of the cluster center. In addition, a fusion strategy is proposed to combine global and local features for enhancing the image representation. The proposed method surpasses the state of the arts on four public and challenging data sets: UC-Merced, WHU-RS19, Sydney, and AID. [ABSTRACT FROM AUTHOR]

The main challenges of visual object tracking arise from the arbitrary appearance of the objects that need to be tracked. Most existing algorithms try to solve this problem by training a new model to regenerate or classify each tracked object. As a result, the model needs to be initialized and retrained for each new object. In this paper, we propose to track different objects in an object-independent approach with a novel two-flow convolutional neural network (YCNN). The YCNN takes two inputs (one is an object image patch, the other is a larger searching image patch), then outputs a response map which predicts how likely and where the object would appear in the search patch. Unlike the object-specific approaches, the YCNN is actually trained to measure the similarity between the two image patches. Thus, this model will not be limited to any specific object. Furthermore, the network is end-to-end trained to extract both shallow and deep dedicated convolutional features for visual tracking. And once properly trained, the YCNN can be used to track all kinds of objects without further training and updating. As a result, our algorithm is able to run at a very high speed of 45 frames-per-second. The effectiveness of the proposed algorithm can also be proved by the experiments on two popular data sets: OTB-100 and VOT-2014. [ABSTRACT FROM AUTHOR]

Emotion recognition is challenging due to the emotional gap between emotions and audio–visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio–visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio–visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio–visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio–visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio–visual emotion recognition. [ABSTRACT FROM AUTHOR]

Classification of plants based on a multi-organ approach is very challenging. Although additional data provide more information that might help to disambiguate between species, the variability in shape and appearance in plant organs also raises the degree of complexity of the problem. Despite promising solutions built using deep learning enable representative features to be learned for plant images, the existing approaches focus mainly on generic features for species classification, disregarding the features representing plant organs. In fact, plants are complex living organisms sustained by a number of organ systems. In our approach, we introduce a hybrid generic-organ convolutional neural network (HGO-CNN), which takes into account both organ and generic information, combining them using a new feature fusion scheme for species classification. Next, instead of using a CNN-based method to operate on one image with a single organ, we extend our approach. We propose a new framework for plant structural learning using the recurrent neural network-based method. This novel approach supports classification based on a varying number of plant views, capturing one or more organs of a plant, by optimizing the contextual dependencies between them. We also present the qualitative results of our proposed models based on feature visualization techniques and show that the outcomes of visualizations depict our hypothesis and expectation. Finally, we show that by leveraging and combining the aforementioned techniques, our best network outperforms the state of the art on the PlantClef2015 benchmark. The source code and models are available at https://github.com/cs-chan/Deep-Plant. [ABSTRACT FROM AUTHOR]

In this paper, we propose a nonlinear structural hashing approach to learn compact binary codes for scalable video search. Unlike most existing video hashing methods which consider image frames within a video separately for binary code learning, we develop a multi-layer neural network to learn compact and discriminative binary codes by exploiting both the structural information between different frames within a video and the nonlinear relationship between video samples. To be specific, we learn these binary codes under two different constraints at the output of our network: 1) the distance between the learned binary codes for frames within the same scene is minimized and 2) the distance between the learned binary matrices for a video pair with the same label is less than a threshold and that for a video pair with different labels is larger than a threshold. To better measure the structural information of the scenes from videos, we employ a subspace clustering method to cluster frames into different scenes. Moreover, we design multiple hierarchical nonlinear transformations to preserve the nonlinear relationship between videos. Experimental results on three video data sets show that our method outperforms state-of-the-art hashing approaches on the scalable video search task. [ABSTRACT FROM AUTHOR]

We present a framework to automatically detect and remove shadows in real world scenes from a single image. Previous works on shadow detection put a lot of effort in designing shadow variant and invariant hand-crafted features. In contrast, our framework automatically learns the most relevant features in a supervised manner using multiple convolutional deep neural networks (ConvNets). The features are learned at the super-pixel level and along the dominant boundaries in the image. The predicted posteriors based on the learned features are fed to a conditional random field model to generate smooth shadow masks. Using the detected shadow masks, we propose a Bayesian formulation to accurately extract shadow matte and subsequently remove shadows. The Bayesian formulation is based on a novel model which accurately models the shadow generation process in the umbra and penumbra regions. The model parameters are efficiently estimated using an iterative optimization procedure. Our proposed framework consistently performed better than the state-of-the-art on all major shadow databases collected under a variety of conditions. [ABSTRACT FROM PUBLISHER]