Search

Facial Attribute Classification (FAC) has attracted increasing attention in computer vision and pattern recognition. However, state-of-the-art FAC methods perform face detection/alignment and FAC independently. The inherent dependencies between these tasks are not fully exploited. In addition, most methods predict all facial attributes using the same CNN network architecture, which ignores the different learning complexities of facial attributes. To address the above problems, we propose a novel deep multi-task multi-label CNN, termed DMM-CNN, for effective FAC. Specifically, DMM-CNN jointly optimizes two closely-related tasks (i.e., facial landmark detection and FAC) to improve the performance of FAC by taking advantage of multi-task learning. To deal with the diverse learning complexities of facial attributes, we divide the attributes into two groups: objective attributes and subjective attributes. Two different network architectures are respectively designed to extract features for two groups of attributes, and a novel dynamic weighting scheme is proposed to automatically assign the loss weight to each facial attribute during training. Furthermore, an adaptive thresholding strategy is developed to effectively alleviate the problem of class imbalance for multi-label learning. Experimental results on the challenging CelebA and LFWA datasets show the superiority of the proposed DMM-CNN method compared with several state-of-the-art FAC methods.

In recent years, deep learning-based person re-identification (Re-ID) methods have made significant progress. However, the performance of these methods substantially decreases when dealing with occlusion, which is ubiquitous in realistic scenarios. In this article, we propose a novel semantic-aware occlusion-robust network (SORN) that effectively exploits the intrinsic relationship between the tasks of person Re-ID and semantic segmentation for occluded person Re-ID. Specifically, the SORN is composed of three branches, including a local branch, a global branch, and a semantic branch. In particular, the local branch extracts part-based local features, and the global branch leverages a novel spatial-patch contrastive loss (SPC) to extract occlusion-robust global features. Meanwhile, the semantic branch generates a foreground-background mask for a pedestrian image, which indicates the non-occluded areas of the human body. The three branches are jointly trained in a unified multi-task learning network. Finally, pedestrian matching is performed based on the local features extracted from the non-occluded areas and the global features extracted from the whole pedestrian image. Extensive experimental results on a large-scale occluded person Re-ID dataset (i.e., Occluded-DukeMTMC) and two partial person Re-ID datasets (i.e., Partial-REID and Partial-iLIDS) show the superiority of the proposed method compared with several state-of-the-art methods for occluded and partial person Re-ID. We also demonstrate the effectiveness of the proposed method on two general person Re-ID datasets (i.e., Market-1501 and DukeMTMC-reID).

Disparity estimation is a popular topic in computer vision and has drawn increasing attention in recent years. In this article, we propose a new multi-stage network for the purpose of two to three-dimensional video conversion that contains two training stages: an initial disparity estimation as the first training stage and depth-image-based rendering (DIBR) as an extra component to form the second training stage. In the first training stage, we propose a revised end-to-end feature pyramid stereo network, in which the original non-pyramid structure is replaced by a bottom-up convolutional neural network pyramid for disparity regression. It utilizes the spatial information by concatenating different scale features to boost the performance on boundary consistency. Mirror connections between feature extraction and disparity regression on the corresponding layers are also added to improve the quality of the results. In the second stage, we propose an improved disocclusion filling technique in the DIBR branch and connect the non-neural-network method to the disparity estimation network. This two-stage training strategy can work effectively to generate the improved disparity estimation for two to three-dimensional video conversion. Extensive experiments are conducted and some selected state-of-the-art algorithms are compared with our proposed approach on the popular KITTI2015 and Scene Flow datasets. The results demonstrate that our estimated disparity map can generate high quality 3D images.

In this paper, we propose a novel continuous latent semantic analysis fitting method, to efficiently and effectively estimate the parameters of model instances in data, based on latent semantic analysis and continuous preference analysis. Specifically, we construct a new latent semantic space (LSS): where inliers of different model instances are mapped into several independent directions, while gross outliers are distributed close to the origin of LSS. After that, we analyze the data distribution to effectively remove gross outliers in LSS, and propose an improved clustering algorithm to segment the remaining data points. On the one hand, the proposed fitting method is able to achieve excellent fitting results; due to the effective continuous preference analysis in LSS. On the other hand, the proposed method can efficiently obtain final fitting results due to the dimensionality reduction in LSS. Experimental results on both synthetic data and real images demonstrate that the proposed method achieves significant superiority over several state-of-the-art model fitting methods on both fitting accuracy and computational speed.

In this paper, a new robust model fitting method is proposed to efficiently segment multistructure data even when they are heavily contaminated by outliers. The proposed method is composed of three steps: first, a conventional greedy search strategy is employed to generate (initial) model hypotheses based on the sequential "fit-and-remove" procedure because of its computational efficiency. Second, to efficiently generate accurate model hypotheses close to the true models, a novel global greedy search strategy initially samples from the inliers of the obtained model hypotheses and samples subsequent data subsets from the whole input data. Third, mutual information theory is applied to fuse the model hypotheses of the same model instance. The conventional greedy search strategy is used to generate model hypotheses for the remaining model instances, if the number of retained model hypotheses is less than that of the true model instances after fusion. The second and the third steps are performed iteratively until an adequate solution is obtained. Experimental results demonstrate the effectiveness and efficiency of the proposed method for model fitting.

Current face or object detection methods via convolutional neural network (such as OverFeat, R-CNN and DenseNet) explicitly extract multi-scale features based on an image pyramid. However, such a strategy increases the computational burden for face detection. In this paper, we propose a fast face detection method based on discriminative complete features (DCFs) extracted by an elaborately designed convolutional neural network, where face detection is directly performed on the complete feature maps. DCFs have shown the ability of scale invariance, which is beneficial for face detection with high speed and promising performance. Therefore, extracting multi-scale features on an image pyramid employed in the conventional methods is not required in the proposed method, which can greatly improve its efficiency for face detection. Experimental results on several popular face detection datasets show the efficiency and the effectiveness of the proposed method for face detection., 11 figures, 30 pages, To appear in Neurocomputing

In this paper, we propose a two-step hypergraph reduction based fitting for unbalanced data. A hypergraph effectively characterizes the relationship between model hypotheses and data points for model fitting. However, a hypergraph-based fitting method often suffers from the problem of high computational cost due to the complex relationship between hyperedges and vertices. Hypergraph reduction algorithms are used to alleviate this problem, but they cannot work well for unbalanced data. To deal with the unbalanced model fitting problem, we first locally remove hyperedges corresponding to the same model instances in data, and then globally remove hyperedges corresponding to the bad model hypotheses. Moreover, we extend the binary incident matrix of a normal hypergraph to a continuous (soft) generalization, to improve the accuracy of hypergraph partition for model fitting. Experimental results on both synthetic data and real images demonstrate that the proposed method has significant superiority over several other state-of-the-art fitting methods.

When different target surfaces, in three-dimensional space, are mapped onto an image plane, they have different projections. These projections vary with the viewpoint. These local differences have influence on the accuracy of image stitching. Most of the existing image stitching methods divide an input image into a number of fixed-size cells, and the pixels within the same cell are then warped using the same local transformation model for the alignment. These methods are based on the hypothesis that the transformation models in one cell are consistent. However, this hypothesis does not hold in general. In this paper, we propose a novel projective-consistent plane based image stitching method (termed PCPS). It divides the overlapping regions of an input image into some projective-consistent planes according to the normal vectors’ orientations of local regions and the reprojection errors of aligned images. The local projective transformation model is estimated for each projective-consistent plane. And then, a hybrid warping model is estimated. For the pixels in overlapping regions, the local projective transformation models are adopted to achieve a better alignment. While for the pixels in non-overlapping regions, a global projective transformation model is estimated by using the inliers uniformly distributed in the projective-consistent planes to avoid distortion. Compared with the state-of-the-art image stitching methods, the experimental results on a number of challenging image sequences show that the projective transformation model estimated by the proposed PCPS method for each projective-consistent plane is more accurate, and the achieved stitching results have less seams and projective distortion.

Extending image-based object detectors into video domain suffers from immense inadaptability due to the deteriorated frames caused by motion blur, partial occlusion or strange poses. Therefore, the generated features of deteriorated frames encounter the poor quality of misalignment, which degrades the overall performance of video object detectors. How to capture valuable information locally or globally is of importance to feature alignment but remains quite challenging. In this paper, we propose a Global and Local Feature Alignment (abbreviated as GLFA) module for video object detection, which can distill both global and local information to excavate the deep relationship between features for feature alignment. Specifically, GLFA can model the spatial-temporal dependencies over frames based on propagating global information and capture the interactive correspondences within the same frame based on aggregating valuable local information. Moreover, we further introduce a Self-Adaptive Calibration (SAC) module to strengthen the semantic representation of features and distill valuable local information in a dual local-alignment manner. Experimental results on the ImageNet VID dataset show that the proposed method achieves high performance as well as a good trade-off between real-time speed and competitive accuracy.

Existing regression-based deep trackers usually localize a target based on a response map, where the highest peak response corresponds to the predicted target location. Nevertheless, when the background distractors appear or the target scale changes frequently, the response map is prone to produce multiple sub-peak responses to interfere with model prediction. In this paper, we propose a robust online tracking method via Scale-Aware localization and Peak Response strength (SAPR), which can learn a discriminative model predictor to estimate a target state accurately. Specifically, to cope with large scale variations, we propose a Scale-Aware Localization (SAL) module to provide multi-scale response maps based on the scale pyramid scheme. Furthermore, to focus on the target response, we propose a simple yet effective Peak Response Strength (PRS) module to fuse the multi-scale response maps and the response maps generated by a correlation filter. According to the response map with the maximum classification score, the model predictor iteratively updates its filter weights for accurate target state estimation. Experimental results on three benchmark datasets, including OTB100, VOT2018 and LaSOT, demonstrate that the proposed SAPR accurately estimates the target state, achieving the favorable performance against several state-of-the-art trackers.

Video object detection is a challenging task due to the appearance deterioration problems in video frames. Thus, object features extracted from different frames of a video are usually deteriorated in varying degrees. Currently, some state-of-the-art methods enhance the deteriorated object features in a reference frame by aggregating the undeteriorated object features extracted from other frames, simply based on their learned appearance relation among object features. In this paper, we propose a novel intra-inter semantic aggregation method (ISA) to learn more effective intra and inter relations for semantically aggregating object features. Specifically, in the proposed ISA, we first introduce an intra semantic aggregation module (Intra-SAM) to enhance the deteriorated spatial features based on the learned intra relation among the features at different positions of an individual object. Then, we present an inter semantic aggregation module (Inter-SAM) to enhance the deteriorated object features in the temporal domain based on the learned inter relation among object features. As a result, by leveraging Intra-SAM and Inter-SAM, the proposed ISA can generate discriminative features from the novel perspective of intra-inter semantic aggregation for robust video object detection. We conduct extensive experiments on the ImageNet VID dataset to evaluate ISA. The proposed ISA obtains 84.5% mAP and 85.2% mAP with ResNet-101 and ResNeXt-101, and it achieves superior performance compared with several state-of-the-art video object detectors.

Recently, some correlation filter based trackers with detection proposals have achieved state-of-the-art tracking results. However, a large number of redundant proposals given by the proposal generator may degrade the performance and speed of these trackers. In this paper, we propose an adaptive proposal selection algorithm which can generate a small number of high-quality proposals to handle the problem of scale variations for visual object tracking. Specifically, we firstly utilize the color histograms in the HSV color space to represent the instances (i.e., the initial target in the first frame and the predicted target in the previous frame) and proposals. Then, an adaptive strategy based on the color similarity is formulated to select high-quality proposals. We further integrate the proposed adaptive proposal selection algorithm with coarse-to-fine deep features to validate the generalization and efficiency of the proposed tracker. Experiments on two benchmark datasets demonstrate that the proposed algorithm performs favorably against several state-of-the-art trackers., 6 pages, 14 figures

In this paper, we propose a novel fitting method, called the Conceptual Space based Model Fitting (CSMF), to fit and segment multi-structure data contaminated with a large number of outliers. CSMF includes two main parts: an outlier removal algorithm and a model selection algorithm. Specifically, we firstly construct a novel conceptual space to measure data points by only considering the good model hypotheses. Then we analyze the conceptual space to effectively remove the gross outliers. Based on the results of outlier removal, we propose to search center points (representing the estimated model instances) in the conceptual space for model selection. CSMF is able to efficiently and effectively remove gross outliers in data, and simultaneously estimate the number and the parameters of model instances without using prior information. Experimental results on both synthetic data and real images demonstrate the advantages of the proposed method over several state-of-the-art fitting methods.

In this paper, we propose a novel expression-targeted feature learning (ETFL) method for effective facial expression recognition, which takes advantage of multi-task learning for discriminative feature learning. Specifically, the common features are firstly extracted from the lower layers of CNN. Then, based on the common features, the expression-specific features (ESF) are respectively learned for each facial expression via multi-task learning. In order to enhance the discriminability of ESF, we develop a joint loss (the combination of the center loss and a novel inter-class loss) to explicitly reduce intra-class variations while enlarging inter-class differences. Furthermore, we introduce the sample-sensitive weights and the soft-expression weights to balance the joint loss for better performance. Finally, all ESFs are combined for final classification. ETFL effectively exploits the relationship among all facial expressions, which leads to superiority feature discriminability. Experiments on public facial expression databases demonstrate the effectiveness of ETFL compared with several state-of-the-art methods.

Deep Neural Network (DNN) has recently achieved outstanding performance in a variety of computer vision tasks, including facial attribute classification. The great success of classifying facial attributes with DNN often relies on a massive amount of labelled data. However, in real-world applications, labelled data are only provided for some commonly used attributes (such as age, gender); whereas, unlabelled data are available for other attributes (such as attraction, hairline). To address the above problem, we propose a novel deep transfer neural network method based on multi-label learning for facial attribute classification, termed FMTNet, which consists of three sub-networks: the Face detection Network (FNet), the Multi-label learning Network (MNet) and the Transfer learning Network (TNet). Firstly, based on the Faster Region-based Convolutional Neural Network (Faster R-CNN), FNet is fine-tuned for face detection. Then, MNet is fine-tuned by FNet to predict multiple attributes with labelled data, where an effective loss weight scheme is developed to explicitly exploit the correlation between facial attributes based on attribute grouping. Finally, based on MNet, TNet is trained by taking advantage of unsupervised domain adaptation for unlabelled facial attribute classification. The three sub-networks are tightly coupled to perform effective facial attribute classification. A distinguishing characteristic of the proposed FMTNet method is that the three sub-networks (FNet, MNet and TNet) are constructed in a similar network structure. Extensive experimental results on challenging face datasets demonstrate the effectiveness of our proposed method compared with several state-of-the-art methods.

In this paper, we propose a simple but effective method for fast image segmentation. We re-examine the locality-preserving character of spectral clustering by constructing a graph over image regions with both global and local connections. Our novel approach to build graph connections relies on two key observations: 1) local region pairs that co-occur frequently will have a high probability to reside on a common object; 2) spatially distant regions in a common object often exhibit similar visual saliency, which implies their neighborship in a manifold. We present a novel energy function to efficiently conduct graph partitioning. Based on multiple high quality partitions, we show that the generated eigenvector histogram based representation can automatically drive effective unary potentials for a hierarchical random field model to produce multi-class segmentation. Sufficient experiments, on the BSDS500 benchmark, large-scale PASCAL VOC and COCO datasets, demonstrate the competitive segmentation accuracy and significantly improved efficiency of our proposed method compared with other state of the arts.

Color and intensity are two important components in an image. Usually, groups of image pixels, which are similar in color or intensity, are an informative representation for an object. They are therefore particularly suitable for computer vision tasks, such as saliency detection and object proposal generation. However, image pixels, which share a similar real-world color, may be quite different since colors are often distorted by intensity. In this paper, we reinvestigate the affinity matrices originally used in image segmentation methods based on spectral clustering. A new affinity matrix, which is robust to color distortions, is formulated for object discovery. Moreover, a Cohesion Measurement (CM) for object regions is also derived based on the formulated affinity matrix. Based on the new Cohesion Measurement, a novel object discovery method is proposed to discover objects latent in an image by utilizing the eigenvectors of the affinity matrix. Then we apply the proposed method to both saliency detection and object proposal generation. Experimental results on several evaluation benchmarks demonstrate that the proposed CM based method has achieved promising performance for these two tasks., 14 pages, 14 figures

Motion segmentation is an important task for intelligent transportation systems. In this paper, inspired by the fact that a feature point trajectory can be sparsely represented as a combination of several feature point trajectories that share coherent transformations, an efficient and effective motion segmentation method with a sparsity constraint is proposed. Specifically, we first propose an accumulated scheme to efficiently integrate motion information from all the frames of a video sequence to construct a correlation matrix. Then, a sparse affinity matrix is built on the correlation matrix by using information-theoretic principles, where the nonzero elements in the same row of the sparse affinity matrix correspond to the feature point trajectories more likely belonging to the same motion. Thereafter, a segment and merge procedure is proposed to effectively estimate the number of motions via the sparse affinity matrix. Finally, by applying spectral clustering on the sparse affinity matrix, different motions in the video sequence are accurately segmented based on the estimated number of motions. Experimental results on the Hopkins 155 and the 62-clip datasets demonstrate that the proposed method achieves superior performance compared with several state-of-the-art methods.

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in a variety of situations, such as fast motion and low illumination scenes. However, most of the event-based object tracking methods are designed for scenarios with untextured objects and uncluttered backgrounds. There are few event-based object tracking methods that support bounding box-based object tracking. The main idea behind this work is to propose an asynchronous Event-based Tracking-by-Detection (ETD) method for generic bounding box-based object tracking. To achieve this goal, we present an Adaptive Time-Surface with Linear Time Decay (ATSLTD) event-to-frame conversion algorithm, which asynchronously and effectively warps the spatio-temporal information of asynchronous retinal events to a sequence of ATSLTD frames with clear object contours. We feed the sequence of ATSLTD frames to the proposed ETD method to perform accurate and efficient object tracking, which leverages the high temporal resolution property of event cameras. We compare the proposed ETD method with seven popular object tracking methods, that are based on conventional cameras or event cameras, and two variants of ETD. The experimental results show the superiority of the proposed ETD method in handling various challenging environments., 9 pages, 5 figures

Video object detection is a tough task due to the deteriorated quality of video sequences captured under complex environments. Currently, this area is dominated by a series of feature enhancement based methods, which distill beneficial semantic information from multiple frames and generate enhanced features through fusing the distilled information. However, the distillation and fusion operations are usually performed at either frame level or instance level with external guidance using additional information, such as optical flow and feature memory. In this work, we propose a dual semantic fusion network (abbreviated as DSFNet) to fully exploit both frame-level and instance-level semantics in a unified fusion framework without external guidance. Moreover, we introduce a geometric similarity measure into the fusion process to alleviate the influence of information distortion caused by noise. As a result, the proposed DSFNet can generate more robust features through the multi-granularity fusion and avoid being affected by the instability of external guidance. To evaluate the proposed DSFNet, we conduct extensive experiments on the ImageNet VID dataset. Notably, the proposed dual semantic fusion network achieves, to the best of our knowledge, the best performance of 84.1\% mAP among the current state-of-the-art video object detectors with ResNet-101 and 85.4\% mAP with ResNeXt-101 without using any post-processing steps., Comment: 9 pages,6 figures

Recently, the Siamese network based visual tracking methods have shown great potentials in balancing the tracking accuracy and computational efficiency. These methods use two-branch convolutional neural networks (CNNs) to generate a response map between the target exemplar and each of candidate patches in the search region. However, since these methods have not fully exploit the target-specific information contained in the CNN features during the computation of the response map, they are less effective to cope with target appearance variations and background clutters. In this paper, we propose a Target-Specific Response Attention (TSRA) module to enhance the discriminability of these methods. In TSRA, a channel-wise cross-correlation operation is used to produce a multi-channel response map, where different channels correspond to different semantic information. Then, TSRA uses an attention network to dynamically re-weight the multi-channel response map at every frame. Moreover, we introduce a shortcut connection strategy to generate a residual multi-channel response map for more discriminative tracking. Finally, we integrate the proposed TSRA into the classical Siamese based tracker (i.e., SiamFC) to propose a new tracker (called TSRA-Siam). Experimental results on three popular benchmark datasets show that the proposed TSRA-Siam outperforms the baseline tracker (i.e., SiamFC) by a large margin and obtains competitive performance compared with several state-of-the-art trackers.

In recent years, Convolutional Neural Network (CNN) based trackers have achieved state-of-the-art performance on multiple benchmark datasets. Most of these trackers train a binary classifier to distinguish the target from its background. However, they suffer from two limitations. Firstly, these trackers cannot effectively handle significant appearance variations due to the limited number of positive samples. Secondly, there exists a significant imbalance of gradient contributions between easy and hard samples, where the easy samples usually dominate the computation of gradient. In this paper, we propose a robust tracking method via Statistical Positive sample generation and Gradient Aware learning (SPGA) to address the above two limitations. To enrich the diversity of positive samples, we present an effective and efficient statistical positive sample generation algorithm to generate positive samples in the feature space. Furthermore, to handle the issue of imbalance between easy and hard samples, we propose a gradient sensitive loss to harmonize the gradient contributions between easy and hard samples. Extensive experiments on three challenging benchmark datasets including OTB50, OTB100 and VOT2016 demonstrate that the proposed SPGA performs favorably against several state-of-the-art trackers., Comment: 6 pages

State-of-the-art face recognition methods have achieved excellent performance on the clean datasets. However, in real-world applications, the captured face images are usually contaminated with noise, which significantly decreases the performance of these face recognition methods. In this paper, we propose a cascaded noise-robust deep convolutional neural network (CNR-CNN) method, consisting of two sub-networks, i.e., a denoising sub-network and a face recognition sub-network, for face recognition under noise. Instead of separately training the two sub-networks, we jointly train them in a cascaded manner. As a result, the images generated from the denoising sub-network are beneficial to the training of the face recognition sub-network. Furthermore, the dense connectivity is used to concatenate the feature maps layer-by-layer in the denoising sub-network, which can effectively exploit the shallow and deep features of CNN. Experimental results on public face datasets demonstrate the superior performance of the proposed method over several state-of-the-art methods.

In recent years, Discriminative Correlation Filter (DCF) based tracking methods have achieved impressive performance in visual tracking. However, their excellent performance usually comes at the cost of sacrificing the computational speed. Furthermore, training correlation filters using high dimensional raw features may introduce the risk of severe over-fitting. To address the above issues, we propose Spatio-Temporal adaptive and Channel selective Correlation Filters (STCCF) for robust tracking. Specifically, we first select a set of target-specific features from high dimensional features via an effective channel selective scheme based on the Taylor expansion. Then, we reformulate the filter learning problem from ridge regression to elastic net regression to adaptively select the discriminative features inside the target bounding box at the spatial level. Moreover, we constrain the filters to be adaptive across temporal frames by learning a transformation matrix from the initial filters to the previous filters. In particular, with a specific spatio-temporal-channel constraint, STCCF can not only alleviate the over-fitting problem and reduce the computational cost, but also enhance the discriminability and interpretability of the learned filters. The proposed STCCF can be optimized by using a few iterations of Alternating Direction Method of Multipliers (ADMM). Experiments on six challenging datasets show that STCCF can achieve promising performance with fast running speed.

Generating promising hypotheses plays a critical role in the success of multi-structure model fitting methods. However, conventional multi-structure hypothesis generation strategies do not exploit the information derived from the results of model selection to guide the subsequent hypothesis generation process. This leads to the problem that these hypothesis generation strategies are often computationally expensive for generating promising hypotheses, especially for heavily contaminated multi-structure data. To address this problem, we first propose a guided sampling strategy to accelerate promising hypothesis generation process by using information derived from the results of model selection on the fly. Then we present a Unified Hypothesis Generation (UHG) framework, which effectively combines the conventional multi-structure hypothesis generation strategy with the proposed guided sampling strategy by using a Markov Chain Monte Carlo process based on a cooling schedule. Experimental results on public databases demonstrate that the proposed UHG achieves significant superiority over several state-of-the-art sampling methods in terms of accuracy and efficiency, especially on multi-structure data. HighlightsA unified sampling framework UHG is proposed for multi-structure model fitting.UHG combines the merits of a global search strategy and a local search strategy.The global search strategy is guided by the conventional sampling strategy.The local search strategy is guided by using the results of model selection.Experimental results show the effectiveness of UHG on public datasets.

A sampling method EGHG is proposed for multi-structure geometry estimation.EGHG combines the benefits of a global and a local sampling strategy.The global sampling strategy is designed to rapidly obtain promising solutions.The local sampling strategy is designed to efficiently achieve accurate solutions.Experimental results show the effectiveness of EGHG on public real image pairs. We propose an Efficient Guided Hypothesis Generation (EGHG) method for multi-structure epipolar geometry estimation. Based on the Markov Chain Monte Carlo process, EGHG combines two guided sampling strategies: a global sampling strategy and a local sampling strategy. The global sampling strategy, guided by using both spatial sampling probabilities and keypoint matching scores, rapidly obtains promising solutions. The spatial sampling probabilities are computed by using a normalized exponential loss function. The local sampling strategy, guided by using both Joint Feature Distributions (JFDs) and keypoint matching scores, efficiently achieves accurate solutions. In the local sampling strategy, EGHG updates a set of current best hypothesis candidates on the fly, and then computes JFDs between the input data and these candidates. Experimental results on public real image pairs show that EGHG significantly outperforms several state-of-the-art sampling methods on multi-structure data.

We propose a graph-based mode-seeking method to fit multi-structural data.The proposed method combines mode-seeking with preference analysis.The proposed method exploits the global structure of graphs by random walks.Experiments show the proposed method is superior to some other fitting methods. In this paper, we propose a novel graph-based mode-seeking fitting method to fit and segment multiple-structure data. Mode-seeking is a simple and effective data analysis technique for clustering and filtering. However, conventional mode-seeking based fitting methods are very sensitive to the proportion of good/bad hypotheses, while most of sampling techniques may generate a large proportion of bad hypotheses. In this paper, we show that the proposed graph-based mode-seeking method has significant superiority for geometric model fitting. We intrinsically combine mode seeking with preference analysis. This enables mode seeking to be beneficial for reducing the influence of bad hypotheses since bad hypotheses usually have larger residual values than good ones. In addition, the proposed method exploits the global structure of graphs by random walks to alleviate the sensitivity to unbalanced data. Experimental results on both synthetic data and real images demonstrate that the proposed method outperforms several other competing fitting methods especially for complex data.

A superpixel-wise model based on homology-similarity is proposed.The model uses two saliency principles, spatial compactness and color contrast.The homology similarity considers both color similarity and spatial connectivity.Extensive experiments are provided to evaluate the proposed model. In recent years, visual saliency detection has become a popular research topic. It can provide useful prior knowledge for high-level vision tasks, such as object detection and image classification. In this paper, a graph-based superpixel-wise similarity called "homology similarity" is proposed, which describes how likely two superpixels belong to the same object or background region. A saliency detection model is then developed based on the combination of homology distribution and improved color contrast. The homology distribution represents spatial compactness, while the color contrast characterizes color conspicuity. By combining these two saliency cues, the proposed model obtains more uniformly highlighted object-level saliency maps with fewer false positive noises. In the experiments, we evaluate our model and 14 competing models (including traditional and state-of-the-art models) on the most popular dataset MSRA-1000 and 4 other publicly available datasets. Experimental results show that, compared with these competing models, our model yields promising results.

Vector of locally aggregated descriptor (VLAD) is a popular image encoding approach for its simplicity and better scalability over conventional bag-of-visual-word approach. In order to enhance its distinctiveness and geometric invariance, covariant VLAD (CVLAD) is proposed to pool local features based on their dominant orientations/characteristic scales, which leads to a geometric-aware representation. This representation achieves rotation/scale invariance when being associated with circular matching. However, the circular matching induces several times of computation overhead, which makes CVLAD hardly suitable for large-scale retrieval tasks. In this paper, the issue of computation overhead is alleviated by performing the circular matching in CVLAD's frequency domain. In addition, by operating PCA on CVLAD in its frequency domain, much better scalability is achieved than when it is undertaken in the original feature space. Furthermore, the high-dimensional CVLAD subvectors are converted to dozens of very low-dimensional subvectors, which is possible when transforming the feature into its frequency domain. Nearest neighbor search is therefore undertaken on very low-dimensional subspaces, which becomes easily tractable. The effectiveness of our approach is demonstrated in the retrieval scenario on popular benchmarks comprising up to 1 million database images.

Channel feature detectors have shown great advantages in human detection. However, a large pool of channel features extracted for human detection usually contains many redundant and irrelevant features. To address this issue, we propose a robust discriminative weighted sparse partial least square approach for feature selection and apply it to human detection. Unlike partial least squares (PLS), which is a straightforward dimensionality reduction technique, we propose using sparse PLS to achieve feature selection. Furthermore, in order to obtain a robust latent matrix, we formulate a discriminative regularized weighted least square problem, where a discriminative term is incorporated to effectively distinguish positive samples from negative samples. A robust sparse weight matrix is trained based on the latent matrix and used for feature selection. Finally, we use the selected channel features to train the boosted decision trees and incorporate the weights of selected features with each tree. The human detector trained by the selected features can preserve high robustness and discriminativeness. Experimental results on some challenging human data sets demonstrate that the proposed approach is effective and achieves state-of-the-art performance.

Efficient hypothesis generation plays an important role in robust model fitting. In this study, based on the combination of residual sorting and local constraints, we propose an efficient guided hypothesis generation method, called Rapid Hypothesis Generation (RHG). By exploiting the local constraints to guide the hypothesis generation process, RHG raises the probability of generating promising hypotheses and reduces the computational cost during hypotheses generation. Experimental results on homography and fundamental matrix estimation show that RHG can effectively guide hypothesis generation process and rapidly generate promising hypotheses for heavily contaminated multi-structure data.

The deep convolutional neural networks have achieved significant improvements in terms of effectiveness and efficiency for single image super-resolution. However, as the depth of neural network grows, the information flow is weakened. On the other hand, most of the models adopt a single-stream structure with which integrating complementary information under different receptive fields is difficult. To improve information flow and capture sufficient knowledge for reconstructing high-frequency details, we propose a multi-scale information cross-fusion network (MSICF) in which a sequence of subnetworks is cascaded to infer high resolution features in a coarse-to-fine manner. In each cascaded subnetwork, we stack multiple multi-scale cross (MSC) modules to fuse multi-scale complementary information in an efficient way as well as to improve information flow across the layers. Meanwhile, by introducing residual-feature learning in each stage, the relative information between features at different levels is fully utilized to further boost reconstruction performance. We train the proposed network with cascaded-supervision and then assemble the intermediate predictions of the cascade to achieve high quality image. Extensive quantitative and qualitative evaluations on benchmark datasets illustrate the superiority of our proposed method over state-of-the-art super-resolution methods.

The performance of many robust model fitting techniques is largely dependent on the quality of the generated hypotheses. In this paper, we propose a novel guided sampling method, called accelerated guided sampling (AGS), to efficiently generate the accurate hypotheses for multistructure model fitting. Based on the observations that residual sorting can effectively reveal the data relationship (i.e., determine whether two data points belong to the same structure), and keypoint matching scores can be used to distinguish inliers from gross outliers, AGS effectively combines the benefits of residual sorting and keypoint matching scores to efficiently generate accurate hypotheses via information theoretic principles. Moreover, we reduce the computational cost of residual sorting in AGS by designing a new residual sorting strategy, which only sorts the top-ranked residuals of input data, rather than all input data. Experimental results demonstrate the effectiveness of the proposed method in computer vision tasks, such as homography matrix and fundamental matrix estimation.

Single image fog removal is important for surveillance applications, and, recently, many defogging methods have been proposed. Due to the adverse atmospheric conditions, the scattering properties of foggy images depend on not only the depth information of scene but also the atmospheric aerosol model, which has a more prominent influence on illumination in a fog scene than that in a haze scene. However, the recent defogging methods confuse haze and fog, and they fail to consider fully the scattering properties. Thus, these methods are not sufficient to remove fog effects, especially for images in maritime surveillance. Therefore, this paper proposes a single image defogging method for visual maritime surveillance. First, a comprehensive scattering model is proposed to formulate a fog image in the glow-shaped environmental illumination. Then, an illumination decomposition algorithm is proposed to eliminate the glow effect on the airlight radiance and recover a fog layer, in which the objects at the infinite distance have uniform luminance. Second, a transmission-map estimation based on the non-local haze-lines prior is utilized to constrain the transmission map into a reasonable range for the input fog image. Finally, the proposed illumination compensation algorithm enables the defogging image to preserve the natural illumination information of the input image. In addition, a fog image dataset is established for the visual maritime surveillance. The experimental results based on the established dataset demonstrate that the proposed method can outperform the state-of-the-art methods in terms of both the subjective and objective evaluation criteria. Moreover, the proposed method can effectively remove fog and maintain naturalness for fog images.

In recent years, Discriminative Correlation Filter (DCF) based tracking methods have achieved great success in visual tracking. However, the multi-resolution convolutional feature maps trained from other tasks like image classification, cannot be naturally used in the conventional DCF formulation. Furthermore, these high-dimensional feature maps significantly increase the tracking complexity and thus limit the tracking speed. In this paper, we present a deep and shallow feature learning network, namely DSNet, to learn the multi-level same-resolution compressed (MSC) features for efficient online tracking, in an end-to-end offline manner. Specifically, the proposed DSNet compresses multi-level convolutional features to uniform spatial resolution features. The learned MSC features effectively encode both appearance and semantic information of objects in the same-resolution feature maps, thus enabling an elegant combination of the MSC features with any DCF-based methods. Additionally, a channel reliability measurement (CRM) method is presented to further refine the learned MSC features. We demonstrate the effectiveness of the MSC features learned from the proposed DSNet on two DCF tracking frameworks: the basic DCF framework and the continuous convolution operator framework. Extensive experiments show that the learned MSC features have the appealing advantage of allowing the equipped DCF-based tracking methods to perform favorably against the state-of-the-art methods while running at high frame rates.

Automatic facial expression recognition has attracted increasing attention for a variety of applications. However, the problem of low-resolution generally causes the performance degradation of facial expression recognition methods under real-life environments. In this paper, we propose to perform low-resolution facial expression recognition from the filter learning perspective. More specifically, a novel image filter based subspace learning (IFSL) method is developed to derive an effective facial image representation. The proposed IFSL method mainly includes three steps: Firstly, we embed the image filter learning into the optimization process of linear discriminant analysis (LDA). By optimizing the cost function of LDA, a set of discriminative image filters (DIFs) corresponding to different facial expressions is learned. Secondly, the images filtered by the learned DIFs are added together to generate the combined images. Finally, a regression learning technique is leveraged for subspace learning, where an expression-aware transformation matrix is obtained using the combined images. Based on the transformation matrix, IFSL effectively removes irrelevant information while preserving useful information in the facial images. Experimental results on several facial expression datasets, including CK+, MMI, JAFFE, SFEW and RAF-DB, show the superior performance of the proposed IFSL method for low-resolution facial expression recognition, compared with several state-of-the-art methods.

In this paper, we propose a novel method, simple iterative clustering on graphs (SICG), to deal with robust model fitting problems. Specifically, we first construct a graph, where each vertex denotes a model hypothesis and each edge represents the similarity between two model hypotheses, for model fitting. We then propose a simple iterative clustering algorithm, which adapts the k-medoids clustering algorithm, to intuitively estimate model instances in data. The proposed SICG method is able to effectively fit and segment multiple-structure data contaminated with a large number of outliers and noises. Experimental results show that SICG achieves superior fitting results over several state-of-the-art model fitting methods on real images.

Convolutional Neural Networks (CNNs) have shown outstanding performance in visual object tracking. However, most of classification-based tracking methods using CNNs are timeconsuming due to expensive computation of complex online finetuning and massive feature extractions. Besides, these methods suffer from the problem of over-fitting since the training and testing stages of CNN models are based on the videos from the same domain. Recently, matching-based tracking methods (such as Siamese networks) have shown remarkable speed superiority, while they cannot well address target appearance variations and complex scenes for inherent lack of online adaptability and background information. In this paper, we propose a novel object-adaptive LSTM network, which can effectively exploit sequence dependencies and dynamically adapt to the temporal object variations via constructing an intrinsic model for object appearance and motion. In addition, we develop an efficient strategy for proposal selection, where the densely sampled proposals are firstly pre-evaluated using the fast matching-based method and then the well-selected high-quality proposals are fed to the sequence-specific learning LSTM network. This strategy enables our method to adaptively track an arbitrary object and operate faster than conventional CNN-based classification tracking methods. To the best of our knowledge, this is thefirst work to apply an LSTM network for classification in visual object tracking. Experimental results on OTB and TC-128 benchmarks show that the proposed method achieves state-ofthe-art performance, which exhibits great potentials of recurrent structures for visual object tracking.

In recent years, convolutional neural network (CNN) based correlation filter trackers have achieved state-of-the-art results on the benchmark datasets. However, the CNN based correlation filters cannot effectively handle large scale variation and distortion (such as fast motion, background clutter, occlusion, etc.), leading to the sub-optimal performance. In this paper, we propose a novel CNN based correlation filter tracker with shepherded instance-aware proposals, namely DeepCFIAP, which automatically estimates the target scale in each frame and re-detects the target when distortion happens. DeepCFIAP is proposed to take advantage of the merits of both instance-aware proposals and CNN based correlation filters. Compared with the CNN based correlation filter trackers, DeepCFIAP can successfully solve the problems of large scale variation and distortion via the shepherded instance-aware proposals, resulting in more robust tracking performance. Specifically, we develop a novel proposal ranking algorithm based on the similarities between proposals and instances. In contrast to the detection proposal based trackers, DeepCFIAP shepherds the instance-aware proposals towards their optimal positions via the CNN based correlation filters, resulting in more accurate tracking results. Extensive experiments on two challenging benchmark datasets demonstrate that the proposed DeepCFIAP performs favorably against state-of-the-art trackers and it is especially feasible for long-term tracking.

Recently, the correlation filter based trackers have achieved very good tracking performance. However, due to the boundary effects of the circulant matrix and the usage of cosine window, the lack of effective negative samples becomes a challenging problem for the correlation filter based trackers. This problem may cause overfitting so that these trackers become very sensitive to deformation and occlusion. In this paper, we propose a novel object tracker (i.e., STAPLE_HNM), which can effectively select hard negative samples and assign adaptive weights to these samples to train the correlation filter. Experimental results demonstrate that the proposed STAPLE_HNM tracker effectively improves the performance of the baseline STAPLE_CA tracker on the OTB-50 and OTB-100 datasets. Moreover, the proposed STAPLE_HNM tracker also achieves superior performance among several state-of-the-art trackers.

Channel feature detectors are the most popular approaches for pedestrian detection recently. However, most of these approaches train the boosted decision trees by selecting a single feature at each node, which does not effectively exploit the multi-feature cues and spatial information. To address this issue, this paper proposes to construct the co-occurrence of multiple channel features in local image neighborhoods for pedestrian detection. In our approach, a binary pattern of feature co-occurrence is represented by combining the binary variables quantized from each channel feature, and the spatial information is incorporated by selecting the neighbors to jointly represent the feature co-occurrence in a local image block. However, feature co-occurrence selection leads to many possible feature combinations, which significantly increase the computational cost at the training stage. Therefore, in order to reduce the number of candidate features and obtain the most discriminative features effectively, a partial least squares-based feature selection approach called variable importance on projection is exploited. Comprehensive experiments are conducted on several challenging pedestrian data sets, and superior performances are achieved by the proposed approach in comparison with some state-of-the-art pedestrian detection approaches.

In recent years, a large number of background subtraction methods have been proposed. Among these methods, the visual background subtraction method (ViBe) receives much attention due to its high efficiency and good performance. However, it can not work well in complex environments. Therefore, in this paper, we propose a novel background subtraction method based on ViBe, including a new foreground object detection strategy, a bilateral aperture detection strategy, and two effective strategies for foreground noise detection and ghost region detection. The proposed method can effectively alleviate the problems caused by foreground aperture, dynamic backgrounds and ghosts in background subtraction. Experiments on the benchmark dataset show that, the proposed method not only obtains better results compared with a couple of ViBe-based variants, but also achieves competitive results against several state-of-the-art methods.

In recent years, the correlation filter based algorithms show impressive performance for visual tracking. However, the object representations (i.e., feature descriptors) are still not robust. In addition, the models in existing correlation filter based algorithms may be updated by using corrupted samples when the tracking targets are occluded, thus leading to the drifting problem. In this paper, we present a weighted foreground appearance feature descriptor which effectively characterizes the appearance of objects. Moreover, we propose an adaptive model updating strategy to mitigate the problem that the models are updated by using corrupted samples. Our works are based on a recently proposed correlation filter based algorithm, i.e., Staple. By effectively combining the proposed feature descriptor with the adaptively updated Staple framework, the proposed algorithm is highly robust and it can achieve promising performance under complex conditions, such as deformation, rotation and scale variation. Experimental results on the OTB-50 and OTB-100 datasets demonstrate the effectiveness of the proposed tracking algorithm, compared with several other state-of-the-art algorithms.

Geometric model fitting is a fundamental research topic in computer vision and it aims to fit and segment multiple-structure data. In this paper, we propose a novel superpixel-guided two-view geometric model fitting method (called SDF), which can obtain reliable and consistent results for real images. Specifically, SDF includes three main parts: a deterministic sampling algorithm, a model hypothesis updating strategy and a novel model selection algorithm. The proposed deterministic sampling algorithm generates a set of initial model hypotheses according to the prior information of superpixels. Then the proposed updating strategy further improves the quality of model hypotheses. After that, by analyzing the properties of the updated model hypotheses, the proposed model selection algorithm extends the conventional "fit-and-remove" framework to estimate model instances in multiple-structure data. The three parts are tightly coupled to boost the performance of SDF in both speed and accuracy, and SDF has the deterministic nature. Experimental results show that the proposed SDF has significant advantages over several state-of-the-art fitting methods when it is applied to real images with single-structure and multiple-structure data.

In this paper, we propose a simple and effective {geometric} model fitting method to fit and segment multi-structure data even in the presence of severe outliers. We cast the task of geometric model fitting as a representative mode-seeking problem on hypergraphs. Specifically, a hypergraph is firstly constructed, where the vertices represent model hypotheses and the hyperedges denote data points. The hypergraph involves higher-order similarities (instead of pairwise similarities used on a simple graph), and it can characterize complex relationships between model hypotheses and data points. {In addition, we develop a hypergraph reduction technique to remove "insignificant" vertices while retaining as many "significant" vertices as possible in the hypergraph}. Based on the {simplified hypergraph, we then propose a novel mode-seeking algorithm to search for representative modes within reasonable time. Finally, the} proposed mode-seeking algorithm detects modes according to two key elements, i.e., the weighting scores of vertices and the similarity analysis between vertices. Overall, the proposed fitting method is able to efficiently and effectively estimate the number and the parameters of model instances in the data simultaneously. Experimental results demonstrate that the proposed method achieves significant superiority over {several} state-of-the-art model fitting methods on both synthetic data and real images.

Recent years have witnessed the widespread success of convolutional neural networks (CNNs) in computer vision and multimedia. The CNNs based background subtraction methods, which are effective for addressing the challenges (such as shadows, dynamic backgrounds, illumination changes) existing in real-world applications, have attracted much attention. However, these methods usually require a large amount of densely labeled video training data, which are hardly collected in the real-world. To address this problem, in this paper, we propose a multiscale cascaded scene-specific CNNs based background subtraction method equipped with a novel training strategy, which takes advantage of the balance of positive and negative training samples. The proposed method can rely on a small number of training samples to effectively train the robust neural network models. Experimental results on the CDnet-2014 dataset show that the proposed method obtains better performance with much less training samples compared with the state-of-the-art methods.

Despite recent progress, computational visual aesthetic is still challenging. Image cropping, which refers to the removal of unwanted scene areas, is an important step to improve the aesthetic quality of an image. However, it is challenging to evaluate whether cropping leads to aesthetically pleasing results because the assessment is typically subjective. In this paper, we propose a novel cascaded cropping regression (CCR) method to perform image cropping by learning the knowledge from professional photographers. The proposed CCR method improves the convergence speed of the cascaded method, which directly uses random-ferns regressors. In addition, a two-step learning strategy is proposed and used in the CCR method to address the problem of lacking labelled cropping data. Specifically, a deep convolutional neural network (CNN) classifier is first trained on large-scale visual aesthetic datasets. The deep CNN model is then designed to extract features from several image cropping datasets, upon which the cropping bounding boxes are predicted by the proposed CCR method. Experimental results on public image cropping datasets demonstrate that the proposed method significantly outperforms several state-of-the-art image cropping methods., 13 pages, 13 figures, To appear in IEEE Transactions on Multimedia, 2017

In this paper, we deal with geometric model fitting problems on graphs, where each vertex represents a model hypothesis, and each edge represents the similarity between two model hypotheses. Conventional median-shift methods are very efficient and they can automatically estimate the number of clusters. However, they assign the same weighting scores to all vertices of a graph, which can not show the discriminability on different vertices. Therefore, we propose a novel weighted median-shift on graphs method (WMSG) to fit and segment multiple-structure data. Specifically, we assign a weighting score to each vertex according to the distribution of the corresponding inliers. After that, we shift vertices towards the weighted median vertices iteratively to detect modes. The proposed method can adaptively estimate the number of model instances and deal with data contaminated with a large number of outliers. Experimental results on both synthetic data and real images show the advantages of the proposed method over several state-of-the-art model fitting methods.

In recent years, the triplet loss-based deep neural networks (DNN) are widely used in the task of face recognition and achieve the state-of-the-art performance. However, the complexity of training the triplet loss-based DNN is significantly high due to the difficulty in generating high-quality training samples. In this paper, we propose a novel DNN training framework to accelerate the training process of the triplet loss-based DNN and meanwhile to improve the performance of face recognition. More specifically, the proposed framework contains two stages: 1) The DNN initialization. A deep architecture based on the softmax loss function is designed to initialize the DNN. 2) The adaptive fine-tuning. Based on the trained model, a set of high-quality triplet samples is generated and used to fine-tune the network, where an adaptive triplet loss function is introduced to improve the discriminative ability of DNN. Experimental results show that, the model obtained by the proposed DNN training framework achieves 97.3% accuracy on the LFW benchmark with low training complexity, which verifies the efficiency and effectiveness of the proposed framework.

In this paper, we propose a novel model fitting method to recover multiple geometric structures from data corrupted by noises and outliers. Instead of analyzing each model hypothesis or each data point separately, the proposed method combines both the consensus information in all model hypotheses and the preference information in all data points into a two-layer network, in which the vertices in the first layer represent the data points and the vertices in the second layer represent the model hypotheses. Based on this formulation, the clusters in the second layer of the network, corresponding to the true structures, are detected by using an effective Two-Stage Message Passing (TSMP) algorithm. TSMP can not only accurately detect multiple structures in data without specifying the number of structures, but also handle data even with a large number of outliers. Experimental results on both synthetic data and real images further demonstrate the superiority of the proposed method over several state-of-the-art fitting methods.