Search

Multiview learning (MVL), which enhances the learners' performance by coordinating complementarity and consistency among different views, has attracted much attention. The multiview generalized eigenvalue proximal support vector machine (MvGSVM) is a recently proposed effective binary classification method, which introduces the concept of MVL into the classical generalized eigenvalue proximal support vector machine (GEPSVM). However, this approach cannot guarantee good classification performance and robustness yet. In this article, we develop multiview robust double-sided twin SVM (MvRDTSVM) with SVM-type problems, which introduces a set of double-sided constraints into the proposed model to promote classification performance. To improve the robustness of MvRDTSVM against outliers, we take L1-norm as the distance metric. Also, a fast version of MvRDTSVM (called MvFRDTSVM) is further presented. The reformulated problems are complex, and solving them are very challenging. As one of the main contributions of this article, we design two effective iterative algorithms to optimize the proposed nonconvex problems and then conduct theoretical analysis on the algorithms. The experimental results verify the effectiveness of our proposed methods.

Gait features have been widely applied in human identification. The commonly-used representations for gait recognition can be roughly classified into two categories: model-free features and model-based features. However, due to the view variances and clothes changes, model-free features are sensitive to the appearance changes. For model-based features, there is great difficulty in extracting the underlying models from gait sequences. Based on the confidence maps and the part affinity fields produced by a two-branch multi-stage CNN network, a new model-based representation, Skeleton Gait Energy Image (SGEI), has been proposed in this paper. Another contribution is that a hybrid representation has been produced, which uses SGEI to remedy the deficiency of model-free features, Gait Energy Image (GEI) for instance. The experimental performances indicate that our proposed methods are more robust to the cloth changes, and contribute to increasing the robustness of gait recognition in the unconstrained environments with view variances and clothes changes.

Traditional methods for unsupervised domain adaptation often leverage a projection matrix or a neural network as the feature extractor or classifier, where the feature extractor shared by the source and target domains enables the sample distributions to be aligned in the feature space, and simultaneously makes the source domain features separability enough for the classifier. However, only the alignment of both domains is not enough because the inter-class distance of some categories in the target domain may be too small, i.e., the feature separability is poor, which often leads to the bad condition that some samples are projected to the classification boundaries and thus misclassified. To solve this problem, we propose a pluggable generic auxiliary distribution (GAD) module for target domain in this paper. The proposed GAD module can iteratively refine the prediction of the target domain samples to increase the separability of the learned features, thereby increasing the distance between features of different categories. This operation can finally reduce the possibility of the target domain samples falling near the classification boundary, and leads to the improvement of classification accuracy for the target domain. Extensive experiments on several popular datasets are conducted, and the results demonstrate the effectiveness of the proposed method.

Face super-resolution (FSR) is an effective way to deal with low-resolution (LR) face images, which can infer the latent high-resolution (HR) face images from the LR inputs. In contrast with traditional FSR methods such as interpolation, learning-based methods generate more realistic HR images of LR faces by exploiting the relationship between HR and LR images . In this paper, we propose a novel FSR learning approach based on orthonormalized partial least squares referred to as OPLS-SR. It first learns a latent coherent feature space of low-dimensional HR and LR face embeddings via a recursive optimization, and then super-resolves the LR face images through global face reconstruction and facial detail compensation. Experimental results on the CAS-PEAL-R1 and FERET face databases have demonstrated the effectiveness of the proposed OPLS-SR method in terms of quantitative and qualitative evaluations.

Although deep features have achieved the state-of-the-art performance in action recognition recently, the hand-crafted shallow features still play a critical role in characterizing human actions for taking advantage of visual contents in an intuitive way such as edge features. Therefore, the shallow features can serve as auxiliary visual cues supplementary to deep representations. In this paper, we propose a discriminative subspace learning model (DSLM) to explore the complementary properties between the hand-crafted shallow feature representations and the deep features. As for the RGB action recognition, this is the first work attempting to mine multi-level feature complementaries by the multi-view subspace learning scheme. To sufficiently capture the complementary information among heterogeneous features, we construct the DSLM by integrating the multi-view reconstruction error and classification error into an unified objective function. To be specific, we first use Fisher Vector to encode improved dense trajectories (iDT+FV) for shallow representations and two-stream convolutional neural network models (T-CNN) for generating deep features. Moreover, the presented DSLM algorithm projects multi-level features onto a shared discriminative subspace with the complementary information and discriminating capacity simultaneously incorporated. Finally, the action types of test samples are identified by the margins from the learned compact representations to the decision boundary. The experimental results on three datasets demonstrate the effectiveness of the proposed method.

In accurate action recognition, discriminative human-region representation as auxiliary information is critical for fusing multiple visual clues in a video and further improving the recognition performance. To this end, in this paper we propose integrating a novel representation named multilayer deep features (MDF) of both the human region and whole image area into an extended region-aware multiple kernel learning (ER-MKL) framework. To be specific, we first exploit the human cues with the help of the off-the-shelf semantic segmentation models. Then more powerful representations MDF are constructed by concatenating activations at the last convolutional layer and fully connected layer. Finally, the proposed framework termed ER-MKL is presented to learn a robust classifier for fusing human-region MDF and whole-region MDF. In addition to combining multiple kernels derived from features of heterogeneous image regions, ER-MKL also considers the sets of pre-learned classifiers and incorporates prior knowledge of different regions. Extensive evaluations on the JHMDB and UCF Sports datasets validate the effectiveness and the superiority of our proposed approach.

In this paper, we address the problem of multiple features fusion in instance-level image retrieval. Achieving tremendous success in recent retrieval task, convolutional neural network (CNN) features are capable of encoding high-level image contents and demonstrate unrivaled superiority to the hand-crafted shallow image signatures. However, the shallow features still play a beneficial role in visual matching particularly when dramatic variances in viewpoint and scale are present, since they inherit certain invariance from the local robust descriptor, e.g., scale-invariant feature transform (SIFT). Thus, it is important to leverage the mutual correlation between these two heterogeneous signatures for effective visual representation. Since it is still an open problem, in this paper, we propose a subspace-based multi-view fusion strategy where a shared subspace is uncovered from the original high-dimensional features yielding a compact latent representation. Experiments on six public benchmark datasets reveal the proposed method works better than other classical fusion approaches and achieve the state-of-the-art performance.

In recent years, visual question answering (VQA) has become topical. The premise of VQA's significance as a benchmark in AI, is that both the image and textual question need to be well understood and mutually grounded in order to infer the correct answer. However, current VQA models perhaps `understand' less than initially hoped, and instead master the easier task of exploiting cues given away in the question and biases in the answer distribution. In this paper we propose the inverse problem of VQA (iVQA). The iVQA task is to generate a question that corresponds to a given image and answer pair. We propose a variational iVQA model that can generate diverse, grammatically correct and content correlated questions that match the given answer. Based on this model, we show that iVQA is an interesting benchmark for visuo-linguistic understanding, and a more challenging alternative to VQA because an iVQA model needs to understand the image better to be successful. As a second contribution, we show how to use iVQA in a novel reinforcement learning framework to diagnose any existing VQA model by way of exposing its belief set: the set of question-answer pairs that the VQA model would predict true for a given image. This provides a completely new window into what VQA models `believe' about images. We show that existing VQA models have more erroneous beliefs than previously thought, revealing their intrinsic weaknesses. Suggestions are then made on how to address these weaknesses going forward., arXiv admin note: text overlap with arXiv:1710.03370

Human perception of a signal depends on the ratio of the change of stimulus to the stimulus itself while the change of stimulus to the stimulus itself is usually ignored in hand-crafted feature descriptors. However, it is important for extracting discriminant feature. To address this problem, we firstly develop a local multiple patterns (LMP) feature descriptor based on the Weber's law for feature extraction and face recognition. In LMP, (1) the Weber's ratio is modified to contain the change direction, and thus the modified Weber's ratio is quantized into several intervals to generate multiple feature maps for describing different changes; (2) LMP concatenates the histograms of the non-overlapping regions of the feature maps for image representation. Secondly, since LMP could only capture small-scale structures, a multi-scale block LMP (MB-LMP) is presented to generate more discriminative and robust visual feature. Thus, MB-LMP could capture both the small-scale and large-scale structures. In implementation, MB-LMP is computationally efficient using integral image. In experiments, LMP and MB-LMP are evaluated on four public datasets for face recognition. The experimental results demonstrate the promise of the proposed LMP and MB-LMP descriptors with desirable efficiency.

Conventional multi-view re-ranking methods usually perform asymmetrical matching between the region of interest (ROI) in the query image and the whole target image for similarity computation. Due to the inconsistency in the visual appearance, this practice tends to degrade the retrieval accuracy particularly when the image ROI, which is usually interpreted as the image objectness, accounts for a smaller region in the image. Since Privileged Information (PI), which can be viewed as the image prior, enables well characterizing the image objectness, we are aiming at leveraging PI for further improving the performance of the multi-view re-ranking accuracy in this paper. Towards this end, we propose a discriminative multi-view re-ranking approach in which both the original global image visual contents and the local auxiliary PI features are simultaneously integrated into a unified training framework for generating the latent subspaces with sufficient discriminating power. For the on-the-fly re-ranking, since the multi-view PI features are unavailable, we only project the original multi-view image representations onto the latent subspace, and thus the re-ranking can be achieved by computing and sorting the distances from the multi-view embeddings to the separating hyperplane. Extensive experimental evaluations on the two public benchmarks Oxford5k and Paris6k reveal our approach provides further performance boost for accurate image re-ranking, whilst the comparative study demonstrates the advantage of our method against other multi-view re-ranking methods.

The Visual Object Tracking challenge VOT2021 is the ninth annual tracker benchmarking activity organized by the VOT initiative. Results of 71 trackers are presented; many are state-of-the-art trackers published at major computer vision conferences or in journals in recent years. The VOT2021 challenge was composed of four sub-challenges focusing on different tracking domains: (i) VOT-ST2021 challenge focused on short-term tracking in RGB, (ii) VOT-RT2021 challenge focused on "real-time" short-term tracking in RGB, (iii) VOT-LT2021 focused on long-term tracking, namely coping with target disappearance and reappearance and (iv) VOT-RGBD2021 challenge focused on long-term tracking in RGB and depth imagery. The VOT-ST2021 dataset was refreshed, while VOT-RGBD2021 introduces a training dataset and sequestered dataset for winner identification. The source code for most of the trackers, the datasets, the evaluation kit and the results along with the source code for most trackers are publicly available at the challenge website 1 .

We present a Siamese-like Dual-branch network based on solely Transformers for tracking. Given a template and a search image, we divide them into non-overlapping patches and extract a feature vector for each patch based on its matching results with others within an attention window. For each token, we estimate whether it contains the target object and the corresponding size. The advantage of the approach is that the features are learned from matching, and ultimately, for matching. So the features are aligned with the object tracking task. The method achieves better or comparable results as the best-performing methods which first use CNN to extract features and then use Transformer to fuse them. It outperforms the state-of-the-art methods on the GOT-10k and VOT2020 benchmarks. In addition, the method achieves real-time inference speed (about $40$ fps) on one GPU. The code and models will be released., Comment: ICCV21 Workshops

Existing visual object tracking usually learns a bounding-box based template to match the targets across frames, which cannot accurately learn a pixel-wise representation, thereby being limited in handling severe appearance variations. To address these issues, much effort has been made on segmentation-based tracking, which learns a pixel-wise object-aware template and can achieve higher accuracy than bounding-box template based tracking. However, existing segmentation-based trackers are ineffective in learning the spatio-temporal correspondence across frames due to no use of the rich temporal information. To overcome this issue, this paper presents a novel segmentation-based tracking architecture, which is equipped with a spatio-appearance memory network to learn accurate spatio-temporal correspondence. Among it, an appearance memory network explores spatio-temporal non-local similarity to learn the dense correspondence between the segmentation mask and the current frame. Meanwhile, a spatial memory network is modeled as discriminative correlation filter to learn the mapping between feature map and spatial map. The appearance memory network helps to filter out the noisy samples in the spatial memory network while the latter provides the former with more accurate target geometrical center. This mutual promotion greatly boosts the tracking performance. Without bells and whistles, our simple-yet-effective tracking architecture sets new state-of-the-arts on the VOT2016, VOT2018, VOT2019, GOT-10K, TrackingNet, and VOT2020 benchmarks, respectively. Besides, our tracker outperforms the leading segmentation-based trackers SiamMask and D3S on two video object segmentation benchmarks DAVIS16 and DAVIS17 by a large margin. The source codes can be found at https://github.com/phiphiphi31/DMB., 12 pages

Unsupervised deep hash functions are complicated due to the challenges of learning discriminative clusters and the absence of similarity-sensitive objectives. Existing approaches predominately handle these challenges independently, while neglecting the fact that cluster centroids and similarity-sensitive binary codes are correlated and can be learned simultaneously. In this paper, we propose a novel end-to-end deep framework for image retrieval, namely Clustering-driven Unsupervised Deep Hashing (CUDH), to recursively learn discriminative clusters by soft clustering model and produce binary code with high similarity responds. We employ the aggregated clusters as an auxiliary distribution to generate hashing codes. With imposing binary constraints loss and reconstruction loss of auto-encoder, our CUDH can be jointly optimized by standard stochastic gradient descent (SGD). Comprehensive experiments on three popular datasets are conducted and the results show that our CUDH can outperform the state-of-the-art methods by large margins.

In deep convolutional neural networks (DCNN), the pooling operation is usually adopted to produce condensed and transformation invariant feature maps for the input image. However, it will inevitably induce information loss, which has not be addressed yet in designing filters of DCNNs. In this paper, we propose Gaussian transfer convolutional neural networks (GT-CNN), which introduce Gaussian filters to pool convolutional filters of DCNNs. In our GT-CNN, the pooling on features can be transferred to the pooling on filters, which are achieved in the same end-to-end framework. More importantly, the Gaussian filters of multiple scales and orientations further improve the capability of GT-CNN, leading to a more robust feature representation for the input image. We evaluate our GT-CNN on various datasets, including MNIST, CIFAR-10, and CIFAR-100, and achieve the best performance compared with the state of the arts. Moreover, we also apply GT-CNN to the remote sensing image dataset, NWPU-RESISC45 dataset, and validate the superiority of GT-CNN on the task.

Differential features are popularly used in computer vision tasks, such as object detection. In this paper, we revisit these features from a functional approximation perspective. In particular, we view an image as a 2-D functional and investigate its Taylor series approximation. Differential features are derived from the approximation coefficients and, therefore, are naturally collected for appearance representation. Thus motivated, we propose to use the zeroth-, first-, and second-order differential features for pedestrian detection and call such features Taylor feature transform (TAFT). In practice, the TAFT features are computed by discrete sampling to address scale issues and meanwhile achieve computational efficiency. In addition, orientation insensitivity is handled by using directional versions of differentials. When applied to pedestrian detection, the TAFT is sampled on grid pixels and calculated from multiple channels following previous solutions. In our extensive experiments on the INRIA, Caltech, TUD-Brussel, and KITTI data sets, the TAFT achieves state-of-the-art results. It outperforms all handcrafted features and performs on par with many deep-learning solutions. Moreover, when a low false-positive rate is requested, the TAFT generates results that are better than or comparable to the state-of-the-art deep learning-based methods. Meanwhile, our implementation runs at 33 fps for $640\times 480$ images without GPU, making TAFT favorable in many practical scenarios.

We design a pedestrian proposal and refining system tailored for fast pedestrian detection. The pedestrian proposal is based on pixel differential feature (PDF), which is a light weighted feature with a high recall rate. For the pedestrian refining, we propose an aggregated region feature (ARF) to distill the co-existing dominant pixel differential patterns in a local region to reject hard false positives. Albeit discriminative, ARF largely relies on the size of anchored regions and the scale of the PDF, which hinders its performance in real-world applications. Although multi-scale PDF with spatial pyramid somewhat alleviates this problem, it is computationally expensive and thus infeasible in practice. To address this issue, we further propose a directional radius pooling method to extract discriminative information in each orientation of PDF while reducing the feature dimensionality with a more compact size. The pedestrian proposal and refining framework is built on the shared pixel differential feature map which is very computationally efficient. More specifically, a set of pedestrian proposals generated from the single-scale PDF are first obtained in images. Second, multi-scale ARF in spatial pyramid is used to fuse information from different scales and spatial resolutions for anchor regions. Third, the directional radius pooling method is proposed to extract dominant information of each orientation in the anchor regions. The pedestrian proposal and refining are finally integrated for accurate pedestrian detection. The extensive experimental evaluations on five public benchmarks show that our method achieves state-of-the-art results while running at 18 fps for $480\times640$ images.

Cascade has been widely used in face detection where classifier with low computational cost can be firstly used to shrink most of the background while keeping the recall. In this paper, a new cascaded convolutional neural network method consisting of two main steps is proposed. During the first stage, low-pixel candidate window is used as an input such that the shallow convolutional neural network quickly extracts the candidate window. In the second stage, the window from the former stage is resized and used as an input to the corresponding network layer respectively. During the training period, joint online training is conducted for hard samples and the soft non-maximum suppression algorithm is used to test on the dataset. The whole network achieves improved performance on the FDDB and PASCAL face datasets.

To mitigate the problems of visual ambiguity and domain shift in conventional zero-shot learning (ZSL), in this paper, we propose a novel method, namely, dual-verification network (DVN), which accepts features and attributes in a pairwise manner as input and verifies the result in both the attribute and feature spaces. First, the DVN projects a feature onto an orthogonal space, where the projected feature has maximum correlation with its corresponding attribute and is orthogonal to all the other attributes. Second, we adopt the concept of semantic feature representation, which computes the relationship between the semantic feature and class labels. Based on this concept, we project the attributes onto the feature space by extending the attributes and labels from the class level to instance level. In addition, we employ a deep architecture and utilize the cross entropy loss to train an end-to-end network for dual verification. Extensive experiments in ZSL and generalized ZSL are performed on four well-known datasets, and the results show that the proposed DVN exhibits a competitive performance relative to the state-of-the-art methods.

Studies show that refining real-world categories into semantic subcategories contributes to better image modeling and classification. Previous image sub-categorization work relying on labeled images and WordNet’s hierarchy is labor-intensive. To tackle this problem, in this work, we extract textual and visual features to automatically select and subsequently classify web images into semantic rich categories. The following two major challenges are well studied: (1) noise in the labels of subcategories derived from the general corpus; (2) noise in the labels of images retrieved from the web. Specifically, we first obtain the semantic refinement subcategories from the text perspective and remove the noise by using the relevance-based approach. To suppress the search error induced noisy images, we then formulate image selection and classifier learning as a multi-instance learning problem and propose to solve the employed problem by the cutting-plane algorithm. The experiments show significant performance gains by using the generated data of our approach on image categorization tasks. The proposed approach also consistently outperforms existing weakly supervised and web-supervised approaches.

Many scenarios such as edge and mobile scenarios are very sensitive to computing complexity and parameter size. Most object detection models cannot be directly deployed without specific modifications. In this paper, we introduce CrossNet, a compute-friendly lightweight anchor-free detector based on CenterNet. We abandon DCN module and design the network structure according to lightweight design principles. A new sample matching strategy based on neighbor points is adopted for improving its performance greatly. Weighted Focal Loss and Balanced L1 Loss further reduce the performance gap between CrossNet and CenterNet. CrossNet possesses a model size of 4.3MB (12.8× smaller than original CenterNet, respectively) and requires 1.51B operations for inference (11.7× smaller than original CenterNet, respectively) while still achieving mAP of 74.2% on VOC dataset. We hope our work could provide some inspiration for lightweight detector design.

Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses, and sunglasses, and recovering appropriate eyes. Due to the significant visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called the eyeglasses removal generative adversarial network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, that is, removing eyeglasses and wearing eyeglasses. Given two face images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a preprocessing method for these tasks.

Multispectral pedestrian detection with feature fusion of thermal and visible images has achieved great success in all-day scenarios, but few attentions have been paid to the quality of feature fusion. In this paper, we have proposed an improved attention aware dual-stream Faster R-CNN algorithm to ameliorate the feature quality. Quantitative analysis of three different feature fusion methods demonstrate that the performance of detector is highly related to the quality of feature fusion. Therefore, an improved multi-step spatial and channel-wise attention method is proposed to improve the feature fusion quality step by step based on the dual-stream Faster R-CNN framework. With the aid of attention mechanism, the model learning is optimized in a more smooth way, which can effectively enhance the response of fused feature map in each stage. Experiments based on the KAIST and CVC-14 dataset demonstrate that the proposed method has achieved better performance compare to thebaseline method with a decrease of 4% and 8% in AMR respectively.

Recently, there are many works on discriminant analysis, which promote the robustness of models against outliers by using L₁- or L2,1-norm as the distance metric. However, both of their robustness and discriminant power are limited. In this article, we present a new robust discriminant subspace (RDS) learning method for feature extraction, with an objective function formulated in a different form. To guarantee the subspace to be robust and discriminative, we measure the within-class distances based on L2,s-norm and use L2,p-norm to measure the between-class distances. This also makes our method include rotational invariance. Since the proposed model involves both L2,p-norm maximization and L2,s-norm minimization, it is very challenging to solve. To address this problem, we present an efficient nongreedy iterative algorithm. Besides, motivated by trace ratio criterion, a mechanism of automatically balancing the contributions of different terms in our objective is found. RDS is very flexible, as it can be extended to other existing feature extraction techniques. An in-depth theoretical analysis of the algorithm's convergence is presented in this article. Experiments are conducted on several typical databases for image classification, and the promising results indicate the effectiveness of RDS.

Due to the limitation of computational cost, one of the remaining open challenges in face detection is designing an efficient face detector for mobile platforms. Some of the previous methods obtain superior performance on the WIDER FACE benchmark, but they are computationally unfriendly for mobile devices with limited computing power and memory resource. Inspired by recent anchor-free detection methods, we propose a lightweight anchor-free face detector named FlashNet, which is built with only 150k parameters. Comprehensive and extensive experiments conducted on popular benchmarks show that the proposed lightweight face detector achieves impressive detection accuracy(78.87% AP on WIDER FACE hard validation set and 97.30% TPR discontinuous score on FDDB). Without bells and whistles, the proposed lightweight detector can achieve impressive runtime efficiency that outperforms existing methods by a large margin. Our method achieves 163.93 FPS on NVIDIA TX2 GPU and 30.51 FPS on NVIDIA TX2 CPU at VGA resolution, separately.

After the two-stage detector is first introduced and popularized by R-CNN, two-stage detectors have achieved great performance, but there are still many problems. 1. FPN tries to use different level features to deal with the scale variance problem in object detection, but lacks the screening of information, leading to important information not protruding and introducing interference; 2. Although CNN realizes the automation of feature extraction, there are still many components that need manual design, such as loss functions, etc., how to choose suitable loss function for different stages are still to be explored. To overcome these issues, we introduce attention mechanism to FPN and propose a more effective feature fusion method for it. Besides, we explore the selection criteria about choosing loss function in each stage and find combining Smooth L1 loss function with the new loss function focused on inliers such as Balanced Smooth L1 yield better results than only using a single loss function. Based on them, we propose Attention Feature Pyramid Networks(AFPN) Detector and train with different loss functions. Experiments show that our method achieves 1.1 points AP improvement than FPN Faster R-CNN on MS-COCO.

With the development of Internet, the forms of web data are rapidly increasing. However, existing cross-media retrieval methods mainly focus on coarse-grained, which is far from being satisfied in practical application. In addition, the heterogeneity gap among different types of media tends to result in inconsistent data representation, so the measuring similarity is quite challenging. In this work, we propose a novel multi-modal network for fine-grained cross-media retrieval. Specifically, our model consists of two networks, including proprietary networks and the common network. The proprietary network is designed as a single feature extraction network for each media to extract unique features for obtaining precise media feature representation. The common network is designed to extract common features of four different types of media. Comprehensive experiments demonstrate the effectiveness of our proposed approach. The source code and models of this work have been made public available at: https://github.com/fgcmr/fgcmr.

To ameliorate the problems of disorder, sparseness, and floating occur for 3D LiDAR point cloud in the road environment, we propose a novel deep CNN architecture for real-time point cloud features extraction. Specifically, we first code the 3D position of point cloud by the index of vertical and horizontal directions. In this way, the 3D point cloud can be converted into a multi-channel point feature map. Then, through multi-level features extraction and fusion of the point feature map, the semantic segmentation of the point cloud scene is finally realized. Comprehensive experiments and ablation studies on public available point cloud datasets demonstrate the superiority of our approach. More importantly, our approach has been successfully applied to the perception of the real-world self-driving system. The source code has been made public available at: https://github.com/Lab1028-19/A-Novel-CNN.

In visual tracking task, accuracy and robustness are critical issues for achieveing remarkable performance. In this paper, we propose a novel dual path network with discriminative meta-filters and hierachical representations to solve these issues. We first design geometrically sensitivity pathway (GESP) and geographical sensitivity pathway (GASP) as two subtasks for target classification and scale estimation. GASP mainly includes powerful discriminative meta-filters to find coarse location of target and GESP can refine region of interests online while adapt the appearance model to the target swiftly. Then, a dual path network is developed in a online and offline framework. Specifically, meta-filters are trained offline in order to gain meta-knowledge of similar tracking scenes. Finally, we present three suggestions on deigning modern tracker. Extensive experiments on VOT2018 datasets verify the superior performance of proposed method compared with other state-of-the-arts, achieving expected average overlap (EAO) of 0.467.

Human activity prediction aims to recognize an unfinished activity with limited appearance and motion information. In this paper, we propose to predict an incomplete activity by combining the mid-level action units and the discriminative key frames exploited from each activity class. Specifically, we extract a great deal of action-related volumes from activity videos. Based on a set of low-level powerful features, similar volumes are aggregated into a mid-level feature, named action unit. Then, we detect these action units in each activity video and generate the frame feature by computing the distribution of concurrent action units in a single frame. Notice that human can easily recognize an incomplete activity using scanty key frames composed of representative interrelated action units together. The key frames in each activity class are selected by computing the entropy of each single frame feature. Finally, a structured SVM is trained to recognize activities with different observation ratios. The proposed approach is evaluated on several publicly available datasets in comparison with state-of-the-art approaches. The experimental results and analysis clearly demonstrate the effectiveness of the proposed approach.

In real applications, the observed low-resolution face images usually have pose variations. Conventional learning-based methods ignore these variations; thus, the hallucinated high-resolution faces are not reasonable for the following recognition task. For recognition purpose, we prefer to obtain near-frontal faces. To this end, we propose a nuclear norm regularized structural orthogonal Procrustes regression (N2SOPR) approach in this work to acquire pose-robust feature representations for face hallucination with pose. The orthogonal Procrustes regression is used to seek an appropriate transformation between two data matrixes. Additionally, the nuclear norm regularization is imposed on the representation residual to preserve image structural property. We also impose a low-rank restraint on the combination weight to automatically cluster each input into the same subspace with the training samples. Both hallucination and recognition experiments conducted on common face databases have verified that our N2SOPR can obtain reasonable performance than some related methods.

Crowd counting is an important task in computer vision, which has many applications in video surveillance. Although the regression-based framework has achieved great improvements for crowd counting, how to improve the discriminative power of image representation is still an open problem. Conventional holistic features used in crowd counting often fail to capture semantic attributes and spatial cues of the image. In this paper, we propose integrating semantic information into learning locality-aware feature (LAF) sets for accurate crowd counting. First, with the help of a convolutional neural network, the original pixel space is mapped onto a dense attribute feature map, where each dimension of the pixelwise feature indicates the probabilistic strength of a certain semantic class. Then, LAF built on the idea of spatial pyramids on neighboring patches is proposed to explore more spatial context and local information. Finally, the traditional vector of locally aggregated descriptor (VLAD) encoding method is extended to a more generalized form weighted-VLAD (W-VLAD) in which diverse coefficient weights are taken into consideration. Experimental results validate the effectiveness of our presented method.

Urine sediment examination refers to the use of microscopes to examine various tangible components in urine sediment, e.g. red blood cells (RBCs), white blood cells (WBCs), tube, and crystal, etc., having a very important role in infectious diseases and circulatory diseases diagnosis. The traditional method about urine sediment analysis depends on the observation of medical staffs. So the workload is particularly large and inefficient, and relevant staff need to own some experience. Recently, the automation of urine sediment analysis can be realized. However, due to the complexity of the urine sediment microscopic image, the accuracy and efficiency of the automatic recognition for the tangible components are still very low. To solve this problem, we investigate channel features to urine sediment detection which include diverse feature types like color channel features and gradient magnitude, etc. We propose aggregate channel features plus (ACF+) detector which is based on aggregate channel features (ACF) for urine sediment detection. We adopt improved Adaboost classifier. The input image does not require any preprocessing and the specific ingredients such as RBCs can be detected directly with a high precision and efficiency. On the testing set, our proposed ACF+ detector suppresses several competitive baselines e.g. Support Vector Machine (SVM) combined with Histogram of Oriented Gradient (HOG), vanilla ACF, and ACDS. In terms of speed, it runs 3FPS on 2592 × 2048 images.

© 2013 IEEE. In the traditional graph embedding framework, the graph is usually built by k-NN or r-ball. Since it is difficult to manually set the parameters k and r in the high-dimensional space, sparse representation-based methods are usually introduced to automatically build the graphs. In recent years, nuclear norm-based matrix regression (NMR) has been proposed for face recognition using the low rank structural information (i.e., the image matrix-based error model). Inspired by NMR, we give a NMR-based projections (NMRP) method for feature extraction and recognition. The experiments on FERET and extended Yale B face databases show that NMR can be used to build the graph while NMRP is an effective feature extraction method.

Graph-based methods have been widely adopted to detect salient objects in images. However, there are two limitations of these methods. First, only one kind of query is employed for saliency propagation on the graph. Second, these methods only represent pairwise relations between vertices and thus give an incomplete representation of the relationships between image regions. In this paper, we propose a foreground- and background-queries-based hypergraph optimization framework for salient region detection. In this framework, both foreground queries and background queries are explicitly exploited to uniformly highlight the salient foreground and suppress the non-salient background. Furthermore, to include both the pairwise and the higher-order relations among two or more vertices, a probabilistic hypergraph is constructed based on local spatial correlation, global spatial correlation, and color correlation to represent the relations among image regions from different views. Extensive experimental results demonstrate the effectiveness of the proposed framework.

Human action recognition in visual data is one of the most fundamental challenges in computer vision. Existing approaches for this primary goal have been based on video data, often incorporating both color and dynamic flow information. Nevertheless, the majority of the visual data constitute still images, and for this reason, being able to recognize actions in still image is an ultimate objective of visual understanding with an extended list of applications. In this paper, we present a novel method that transfers the knowledge learned from action videos onto images to allow recognition of the principal action depicted in still image. Our intuition is that a generative model for knowledge transfer can be learned by taking advantage of the available action videos in the training stage to bridge images to videos. Based on this, we propose two complementary knowledge-transfer models utilizing fully connected networks to deliver the knowledge extracted from color and motion flow sequences to still images. We introduce a weighted reconstruction and classification loss to steer the generation procedure of the networks. In addition, we describe and analyze the influence of different data augmentation techniques, initialization strategies, and weighting coefficients for improving the performance. We observe that: both the transferred knowledge from color sequences and motion flow sequences can improve the performance of still image based human action recognition; the latter one which provides complementary dynamic information improves the performance a lot. We evaluate our models on two publicly available video based human action recognition datasets: UCF101 and HMDB51. To further validate the generalization ability of the proposed solution, we test the learned models from UCF101 dataset on two still image based human action recognition benchmarks: Willow7 Actions and the Sports. Our results demonstrate that the proposed method outperforms the baseline approaches with more than 2% accuracy, 3% accuracy, 3% accuracy and 5% mAP on UCF101, HMDB51, Sports and Willow 7 Actions datasets, respectively.

The existing hashing methods mainly handle either the feature based nearest-neighbor search or the category-level image retrieval, whereas a few efforts are devoted to instance retrieval problem. In this paper, we propose a binary multi-view fusion framework for directly recovering a latent Hamming subspace from the multi-view features for instance retrieval. More specifically, the multi-view subspace reconstruction and the binary quantization are integrated in a unified framework so as to minimize the discrepancy between the original multi-view high-dimensional Euclidean space and the resulting compact Hamming subspace. Besides, our method is essentially an unsupervised learning scheme without any labeled data involved, and thus can be used in the cases when the supervised information is unavailable or insufficient. Experiments on public benchmark and large-scale datasets reveal that our method achieves competitive retrieval performance comparable to the state-of-the-arts and has excellent scalability in large-scale scenario.

Although deep learning makes major breakthroughs in object detection, object detection still faces several limitations listed as follows: (1) Many works underplay the feature selection, leading to the resulting key features are not prominent enough and prone to noise; (2) Many works pass back features in a layer-by-layer manner to achieve multi-scale features. However, as the distance of layers from each other increases, the semantics are diluted, and the transfer of information between layers becomes difficult. To overcome these problems, we propose a new Interconnected Feature Pyramid Networks (IFPN) for feature enhancement. It can simultaneously select attentive features through the attention mechanism and realize the free flow of information. On the basis of the improvements, we design a new IFPN Detector. Experiments on COCO dataset and Smart UVM dataset show that our method can bring a significant improvement.

The semantic segmentation task is highly related to detection and apparently can provide complementary information for detection. In this paper, we propose integrating deep semantic segmentation feature maps into the original pedestrian detection framework which combines feature channels with AdaBoost classifiers. Firstly, we develop shallow-deep channels by concatenating shallow hand-crafted and deep segmentation channels to capture appearance clues as well as semantic attributes. Then a set of manually designed filters are utilized on the new channels to generate more response feature maps. Finally a cascade AdaBoost classifier is learned for hard negatives selection and pedestrian detection. With abundant feature information, our proposed detector achieves superior results on Caltech USA 10x and ETH dataset.

Different from traditional human activity recognition, human activity prediction aims to recognize an unfinished activity, typically in absence of explicit temporal progress status. In this paper, we propose a new human activity prediction approach by extending the recently proposed generalized time warping (GTW) 20, which allows an efficient and flexible alignment of two or more multi-dimensional time series. More specifically, for each activity video, either complete or incomplete, we first decompose it into a sequence of short video segments. Then, we represent each segment by the local spatial-temporal statistics using the classical bag-of-visual-words model. In this way, the comparison between a query sequence (i.e., containing an incomplete activity) and a reference sequence (i.e., containing a full activity) boils down to the problem of aligning their corresponding segment sequences. While GTW treats different portions of a sequence as equally important, our task is in favor of early portions since an incomplete activity video always aligns from the beginning of a complete one. Thus motivated, we develop a temporally-weighted GTW (TGTW) algorithm for the activity prediction problem by encouraging alignment in the early portion of an activity sequence. Finally, the similarity derived from TGTW is combined with the k-nearest neighbors algorithm for predicting the activity class of an input sequence. The proposed approach is evaluated on several publicly available datasets in comparison with state-of-the-art approaches. The experimental results and analysis clearly demonstrate the effectiveness of the proposed approach.

Zero-shot learning (ZSL), a type of structured multioutput learning, has attracted much attention due to its requirement of no training data for target classes. Conventional ZSL methods usually project visual features into semantic space and assign labels by finding their nearest prototypes. However, this type of nearest neighbor search (NNS)-based method often suffers from great performance degradation because of the nonuniform variances between different categories. In this article, we propose a probabilistic framework by taking covariance into account to deal with the above-mentioned problem. In this framework, we define a new latent space, which has two characteristics. The first is that the features in this space should gather within the classes and scatter between the classes, which is implemented by triplet learning; the second is that the prototypes of unseen classes are synthesized with nonnegative coefficients, which are generated by nonnegative matrix factorization (NMF) of relations between the seen classes and the unseen classes in attribute space. During training, the learned parameters are the projection model for triplet network and the nonnegative coefficients between the unseen classes and the seen classes. In the testing phase, visual features are projected into latent space and assigned with the labels that have the maximum probability among unseen classes for classic ZSL or within all classes for generalized ZSL. Extensive experiments are conducted on four popular data sets, and the results show that the proposed method can outperform the state-of-the-art methods in most circumstances.

Of late, there are many studies on the robust discriminant analysis, which adopt L1-norm as the distance metric, but their results are not robust enough to gain universal acceptance. To overcome this problem, the authors of this article present a nonpeaked discriminant analysis (NPDA) technique, in which cutting L1-norm is adopted as the distance metric. As this kind of norm can better eliminate heavy outliers in learning models, the proposed algorithm is expected to be stronger in performing feature extraction tasks for data representation than the existing robust discriminant analysis techniques, which are based on the L1-norm distance metric. The authors also present a comprehensive analysis to show that cutting L1-norm distance can be computed equally well, using the difference between two special convex functions. Against this background, an efficient iterative algorithm is designed for the optimization of the proposed objective. Theoretical proofs on the convergence of the algorithm are also presented. Theoretical insights and effectiveness of the proposed method are validated by experimental tests on several real data sets.

In recent years, 2D vehicle detection has experienced impressive progress. Despite of the outstanding performance achieved by the state-of-the-art methods, it is still very challenging to detect vehicles in real scene due to the existence of small targets in images. In this work, we propose a Cascade Multi-Scale Region-based Convolutional Neural Network method (CMS R-CNN) for accurate small vehicles detection. Our method combines two key insights: (1) we propose a efficient multi-scale feature fusion network, which combines spatial information of feature maps with all scales, to enhance the ability to detect small vehicles; (2) we introduce a novel cascade framework which consists of a sequence of detectors trained with the increasing IoU thresholds strategy to further achieve better detection performance. Experiments show that our method achieves a high accuracy on the benchmark of KITTI dataset (77.0% in difficult data of KITTI) and outperforms previous work by a remarkable margin (6.2% higher than Faster R-CNN in difficult data of KITTI).

Databases are of great significance to researchers to achieve a satisfactory model. The lack of data is always a bottleneck to facial landmark localization, especially for the dense facial landmark detection. In this article, we provide a new dataset, called Dense Landmark Localization (DLL) database, which contains 39,198 images and is annotated in high quality. Annotating dense landmarks is a very tedious work due to two challenges. (a) Not every facial point has clear definition. Some of them distribute uniformly along the contour. Their labelled positions are determined by subjective judgement of the annotators, so that the quality of the annotation is poor. (b) Adjusting facial points one by one is time-consuming. The workload will increase dramatically when there are more points. To overcome the aforementioned problems, we propose a semiautomatic annotation tool to annotate dense points with much less clicks.

In this work, we propose a 3D-like bounding box to detect vehicles in RGB image based on the 2D bounding box detection. Taking advantage of key-points detection by detecting eight corners of the 3D box surrounding the vehicle, our 3D-like bounding box is capable of estimating the vehicle position more accurately than traditional object detection methods, while provides the information of the front and back of the vehicle or even the orientation that is of great help in advanced driver assistance system. We present two improved strategies for data annotation to evaluate the proposed method. The experiments on KITTI object detection database show that our presented two strategies exhibit promising performance.

Learning long-term dependences (LTDs) with recurrent neural networks (RNNs) is challenging due to their limited internal memories. In this paper, we propose a new external memory architecture for RNNs called an external addressable long-term and working memory (EALWM)-augmented RNN. This architecture has two distinct advantages over existing neural external memory architectures, namely the division of the external memory into two parts—long-term memory and working memory—with both addressable and the capability to learn LTDs without suffering from vanishing gradients with necessary assumptions. The experimental results on algorithm learning, language modeling, and question answering demonstrate that the proposed neural memory architecture is promising for practical applications.

Recently, multi-view features have significantly pro- moted the performance of image re-ranking by providing complementary image descriptions. Without loss of generality, in multi-view re-ranking, multiple heterogeneous visual features of high dimensionality are projected onto a low-dimensional subspace, and thus the resulting latent representation can be used for the subsequent similarity-based ranking. Albeit effective, this standard mechanism underplays the intrinsic structure underlying the latent subspace and does not take into account the substantial noise in the original multi-view feature spaces. In this paper, we propose a robust multi-view feature learning strategy for accurate image re-ranking. Due to the dramatic variability in visual appearance for different target images, it is necessary to uncover the shared components underlying those query-related instances that are visually unlike for improving the re-ranking accuracy. Consequently, it is reasonable to assume the latent subspace enjoys the low-rank property and thus the subspace recovery can be achieved via the low-rank modeling accordingly. In addition, the real-world data are usually partially contaminated and it is required to appropriately model the sample-dependent data noise. Towards this end, we employ l2,1- norm based sparsity constraint to model the sample-specific mapping noise for enhancing the model robustness. In order to produce discriminative representations, we encode a similarity preserving term in our multi-view embedding framework. As a result, the sample separability is maximally maintained in the latent subspace with sufficient discriminative power. The extensive experimental evaluations on public landmark benchmarks reveal that our approach achieves impressive performance superior to the state-of-the-art, which thus demonstrates the efficacy of the proposed method.

Fabric defects have an important influence on the quality of the fabric product. Automatic fabric defect detection is a crucial part for quality control in the textile industry. The primary challenge of fabric defects identification is not only to find the existing defects, but also to classify them into different types. In this paper, we propose a novel fabric defect detection and classification method consists of three main steps. Firstly, the fabric image is cropped into a set of image patches and each patch is labeled with specified defect type. Secondly, the visual saliency map is generated from the patch to localize defects with specified visual attention. Then, the combination of visual salience map with raw image input into a convolutional neural network for robust feature representation, and finally output its predicted defect type. During the testing section, defect inspection runs in a sliding window schemes using the trained model, and both the type and position of each defect are obtained simultaneously. Our method tries to investigate the combination of visual saliency and one-stage object detector with feature pyramid, which fully makes use of information from multi-resolution guided with visual attention. Besides, soft-cutoff loss is employed to further improve the performance of the method, and our network can be learnt in an end-to-end manner. Experiments based on our fabric defect image datasets, the proposed method can achieve a 98.52% accuracy of classification. This method is comparable to the usual two-stage detector with more compact model parameters, makes it valuable in the industrial application.

Facial landmark detection is one of the most important tasks in face image and video analysis. Existing algorithms based on deep convolutional neural networks have achieved good performance in public benchmarks and practical applications such as face verification, expression analysis, beauty applications and so on. However, the performance of a facial landmark detector degrades significantly when dealing with challenging facial images in the presence of extreme appearance variations such as pose, expression, occlusion, etc. To mitigate these difficulties, we propose a robust facial landmark detection algorithm based on coordinates regression in an end-to-end training fashion. By using the soft-argmax function, the network weights can be optimised with a mixed loss function. The online pose-based data augmentation technology is used to effectively solve the data imbalance problem and improve the robustness of the proposed method. Experiments conducted on the 300-W and AFLW datasets demonstrate that the performance of the proposed algorithm is competitive to the state-of-the-art heatmap regression algorithms, in terms of accuracy. Besides, our method achieves real-time speed on 300-W with 68 landmarks, which runs at 85 FPS on a Tesla v100 GPU.