Search

Multiview learning (MVL), which enhances the learners' performance by coordinating complementarity and consistency among different views, has attracted much attention. The multiview generalized eigenvalue proximal support vector machine (MvGSVM) is a recently proposed effective binary classification method, which introduces the concept of MVL into the classical generalized eigenvalue proximal support vector machine (GEPSVM). However, this approach cannot guarantee good classification performance and robustness yet. In this article, we develop multiview robust double-sided twin SVM (MvRDTSVM) with SVM-type problems, which introduces a set of double-sided constraints into the proposed model to promote classification performance. To improve the robustness of MvRDTSVM against outliers, we take L1-norm as the distance metric. Also, a fast version of MvRDTSVM (called MvFRDTSVM) is further presented. The reformulated problems are complex, and solving them are very challenging. As one of the main contributions of this article, we design two effective iterative algorithms to optimize the proposed nonconvex problems and then conduct theoretical analysis on the algorithms. The experimental results verify the effectiveness of our proposed methods.

Traditional methods for unsupervised domain adaptation often leverage a projection matrix or a neural network as the feature extractor or classifier, where the feature extractor shared by the source and target domains enables the sample distributions to be aligned in the feature space, and simultaneously makes the source domain features separability enough for the classifier. However, only the alignment of both domains is not enough because the inter-class distance of some categories in the target domain may be too small, i.e., the feature separability is poor, which often leads to the bad condition that some samples are projected to the classification boundaries and thus misclassified. To solve this problem, we propose a pluggable generic auxiliary distribution (GAD) module for target domain in this paper. The proposed GAD module can iteratively refine the prediction of the target domain samples to increase the separability of the learned features, thereby increasing the distance between features of different categories. This operation can finally reduce the possibility of the target domain samples falling near the classification boundary, and leads to the improvement of classification accuracy for the target domain. Extensive experiments on several popular datasets are conducted, and the results demonstrate the effectiveness of the proposed method.

Face super-resolution (FSR) is an effective way to deal with low-resolution (LR) face images, which can infer the latent high-resolution (HR) face images from the LR inputs. In contrast with traditional FSR methods such as interpolation, learning-based methods generate more realistic HR images of LR faces by exploiting the relationship between HR and LR images . In this paper, we propose a novel FSR learning approach based on orthonormalized partial least squares referred to as OPLS-SR. It first learns a latent coherent feature space of low-dimensional HR and LR face embeddings via a recursive optimization, and then super-resolves the LR face images through global face reconstruction and facial detail compensation. Experimental results on the CAS-PEAL-R1 and FERET face databases have demonstrated the effectiveness of the proposed OPLS-SR method in terms of quantitative and qualitative evaluations.

Although deep features have achieved the state-of-the-art performance in action recognition recently, the hand-crafted shallow features still play a critical role in characterizing human actions for taking advantage of visual contents in an intuitive way such as edge features. Therefore, the shallow features can serve as auxiliary visual cues supplementary to deep representations. In this paper, we propose a discriminative subspace learning model (DSLM) to explore the complementary properties between the hand-crafted shallow feature representations and the deep features. As for the RGB action recognition, this is the first work attempting to mine multi-level feature complementaries by the multi-view subspace learning scheme. To sufficiently capture the complementary information among heterogeneous features, we construct the DSLM by integrating the multi-view reconstruction error and classification error into an unified objective function. To be specific, we first use Fisher Vector to encode improved dense trajectories (iDT+FV) for shallow representations and two-stream convolutional neural network models (T-CNN) for generating deep features. Moreover, the presented DSLM algorithm projects multi-level features onto a shared discriminative subspace with the complementary information and discriminating capacity simultaneously incorporated. Finally, the action types of test samples are identified by the margins from the learned compact representations to the decision boundary. The experimental results on three datasets demonstrate the effectiveness of the proposed method.

In recent years, visual question answering (VQA) has become topical. The premise of VQA's significance as a benchmark in AI, is that both the image and textual question need to be well understood and mutually grounded in order to infer the correct answer. However, current VQA models perhaps `understand' less than initially hoped, and instead master the easier task of exploiting cues given away in the question and biases in the answer distribution. In this paper we propose the inverse problem of VQA (iVQA). The iVQA task is to generate a question that corresponds to a given image and answer pair. We propose a variational iVQA model that can generate diverse, grammatically correct and content correlated questions that match the given answer. Based on this model, we show that iVQA is an interesting benchmark for visuo-linguistic understanding, and a more challenging alternative to VQA because an iVQA model needs to understand the image better to be successful. As a second contribution, we show how to use iVQA in a novel reinforcement learning framework to diagnose any existing VQA model by way of exposing its belief set: the set of question-answer pairs that the VQA model would predict true for a given image. This provides a completely new window into what VQA models `believe' about images. We show that existing VQA models have more erroneous beliefs than previously thought, revealing their intrinsic weaknesses. Suggestions are then made on how to address these weaknesses going forward., arXiv admin note: text overlap with arXiv:1710.03370

Conventional multi-view re-ranking methods usually perform asymmetrical matching between the region of interest (ROI) in the query image and the whole target image for similarity computation. Due to the inconsistency in the visual appearance, this practice tends to degrade the retrieval accuracy particularly when the image ROI, which is usually interpreted as the image objectness, accounts for a smaller region in the image. Since Privileged Information (PI), which can be viewed as the image prior, enables well characterizing the image objectness, we are aiming at leveraging PI for further improving the performance of the multi-view re-ranking accuracy in this paper. Towards this end, we propose a discriminative multi-view re-ranking approach in which both the original global image visual contents and the local auxiliary PI features are simultaneously integrated into a unified training framework for generating the latent subspaces with sufficient discriminating power. For the on-the-fly re-ranking, since the multi-view PI features are unavailable, we only project the original multi-view image representations onto the latent subspace, and thus the re-ranking can be achieved by computing and sorting the distances from the multi-view embeddings to the separating hyperplane. Extensive experimental evaluations on the two public benchmarks Oxford5k and Paris6k reveal our approach provides further performance boost for accurate image re-ranking, whilst the comparative study demonstrates the advantage of our method against other multi-view re-ranking methods.

The Visual Object Tracking challenge VOT2021 is the ninth annual tracker benchmarking activity organized by the VOT initiative. Results of 71 trackers are presented; many are state-of-the-art trackers published at major computer vision conferences or in journals in recent years. The VOT2021 challenge was composed of four sub-challenges focusing on different tracking domains: (i) VOT-ST2021 challenge focused on short-term tracking in RGB, (ii) VOT-RT2021 challenge focused on "real-time" short-term tracking in RGB, (iii) VOT-LT2021 focused on long-term tracking, namely coping with target disappearance and reappearance and (iv) VOT-RGBD2021 challenge focused on long-term tracking in RGB and depth imagery. The VOT-ST2021 dataset was refreshed, while VOT-RGBD2021 introduces a training dataset and sequestered dataset for winner identification. The source code for most of the trackers, the datasets, the evaluation kit and the results along with the source code for most trackers are publicly available at the challenge website 1 .

We present a Siamese-like Dual-branch network based on solely Transformers for tracking. Given a template and a search image, we divide them into non-overlapping patches and extract a feature vector for each patch based on its matching results with others within an attention window. For each token, we estimate whether it contains the target object and the corresponding size. The advantage of the approach is that the features are learned from matching, and ultimately, for matching. So the features are aligned with the object tracking task. The method achieves better or comparable results as the best-performing methods which first use CNN to extract features and then use Transformer to fuse them. It outperforms the state-of-the-art methods on the GOT-10k and VOT2020 benchmarks. In addition, the method achieves real-time inference speed (about $40$ fps) on one GPU. The code and models will be released., Comment: ICCV21 Workshops

Existing visual object tracking usually learns a bounding-box based template to match the targets across frames, which cannot accurately learn a pixel-wise representation, thereby being limited in handling severe appearance variations. To address these issues, much effort has been made on segmentation-based tracking, which learns a pixel-wise object-aware template and can achieve higher accuracy than bounding-box template based tracking. However, existing segmentation-based trackers are ineffective in learning the spatio-temporal correspondence across frames due to no use of the rich temporal information. To overcome this issue, this paper presents a novel segmentation-based tracking architecture, which is equipped with a spatio-appearance memory network to learn accurate spatio-temporal correspondence. Among it, an appearance memory network explores spatio-temporal non-local similarity to learn the dense correspondence between the segmentation mask and the current frame. Meanwhile, a spatial memory network is modeled as discriminative correlation filter to learn the mapping between feature map and spatial map. The appearance memory network helps to filter out the noisy samples in the spatial memory network while the latter provides the former with more accurate target geometrical center. This mutual promotion greatly boosts the tracking performance. Without bells and whistles, our simple-yet-effective tracking architecture sets new state-of-the-arts on the VOT2016, VOT2018, VOT2019, GOT-10K, TrackingNet, and VOT2020 benchmarks, respectively. Besides, our tracker outperforms the leading segmentation-based trackers SiamMask and D3S on two video object segmentation benchmarks DAVIS16 and DAVIS17 by a large margin. The source codes can be found at https://github.com/phiphiphi31/DMB., 12 pages

Fisher’s criterion is one of the most popular discriminant criteria for feature extraction. It is defined as the generalized Rayleigh quotient of the between-class scatter distance to the within-class scatter distance. Consequently, Fisher’s criterion does not take advantage of the discriminant information in the class covariance differences, and hence, its discriminant ability largely depends on the class mean differences. If the class mean distances are relatively large compared with the within-class scatter distance, Fisher’s criterion-based discriminant analysis methods may achieve a good discriminant performance. Otherwise, it may not deliver good results. Moreover, we observe that the between-class distance of Fisher’s criterion is based on the $\ell _{2}$ -norm, which would be disadvantageous to separate the classes with smaller class mean distances. To overcome the drawback of Fisher’s criterion, in this paper, we first derive a new discriminant criterion, expressed as a mixture of absolute generalized Rayleigh quotients , based on a Bayes error upper bound estimation, where mixture of Gaussians is adopted to approximate the real distribution of data samples. Then, the criterion is further modified by replacing $\ell _{2}$ -norm with $\ell _{1}$ one to better describe the between-class scatter distance, such that it would be more effective to separate the different classes. Moreover, we propose a novel $\ell _{1}$ -norm heteroscedastic discriminant analysis method based on the new discriminant analysis (L1-HDA/GM) for heteroscedastic feature extraction, in which the optimization problem of L1-HDA/GM can be efficiently solved by using the eigenvalue decomposition approach. Finally, we conduct extensive experiments on four real data sets and demonstrate that the proposed method achieves much competitive results compared with the state-of-the-art methods.

To mitigate the problems of visual ambiguity and domain shift in conventional zero-shot learning (ZSL), in this paper, we propose a novel method, namely, dual-verification network (DVN), which accepts features and attributes in a pairwise manner as input and verifies the result in both the attribute and feature spaces. First, the DVN projects a feature onto an orthogonal space, where the projected feature has maximum correlation with its corresponding attribute and is orthogonal to all the other attributes. Second, we adopt the concept of semantic feature representation, which computes the relationship between the semantic feature and class labels. Based on this concept, we project the attributes onto the feature space by extending the attributes and labels from the class level to instance level. In addition, we employ a deep architecture and utilize the cross entropy loss to train an end-to-end network for dual verification. Extensive experiments in ZSL and generalized ZSL are performed on four well-known datasets, and the results show that the proposed DVN exhibits a competitive performance relative to the state-of-the-art methods.

Eyeglasses removal is challenging in removing different kinds of eyeglasses, e.g., rimless glasses, full-rim glasses, and sunglasses, and recovering appropriate eyes. Due to the significant visual variants, the conventional methods lack scalability. Most existing works focus on the frontal face images in the controlled environment, such as the laboratory, and need to design specific systems for different eyeglass types. To address the limitation, we propose a unified eyeglass removal model called the eyeglasses removal generative adversarial network (ERGAN), which could handle different types of glasses in the wild. The proposed method does not depend on the dense annotation of eyeglasses location but benefits from the large-scale face images with weak annotations. Specifically, we study the two relevant tasks simultaneously, that is, removing eyeglasses and wearing eyeglasses. Given two face images with and without eyeglasses, the proposed model learns to swap the eye area in two faces. The generation mechanism focuses on the eye area and invades the difficulty of generating a new face. In the experiment, we show the proposed method achieves a competitive removal quality in terms of realism and diversity. Furthermore, we evaluate ERGAN on several subsequent tasks, such as face verification and facial expression recognition. The experiment shows that our method could serve as a preprocessing method for these tasks.

Recently, there are many works on discriminant analysis, which promote the robustness of models against outliers by using L₁- or L2,1-norm as the distance metric. However, both of their robustness and discriminant power are limited. In this article, we present a new robust discriminant subspace (RDS) learning method for feature extraction, with an objective function formulated in a different form. To guarantee the subspace to be robust and discriminative, we measure the within-class distances based on L2,s-norm and use L2,p-norm to measure the between-class distances. This also makes our method include rotational invariance. Since the proposed model involves both L2,p-norm maximization and L2,s-norm minimization, it is very challenging to solve. To address this problem, we present an efficient nongreedy iterative algorithm. Besides, motivated by trace ratio criterion, a mechanism of automatically balancing the contributions of different terms in our objective is found. RDS is very flexible, as it can be extended to other existing feature extraction techniques. An in-depth theoretical analysis of the algorithm's convergence is presented in this article. Experiments are conducted on several typical databases for image classification, and the promising results indicate the effectiveness of RDS.

Crowd counting is an important task in computer vision, which has many applications in video surveillance. Although the regression-based framework has achieved great improvements for crowd counting, how to improve the discriminative power of image representation is still an open problem. Conventional holistic features used in crowd counting often fail to capture semantic attributes and spatial cues of the image. In this paper, we propose integrating semantic information into learning locality-aware feature (LAF) sets for accurate crowd counting. First, with the help of a convolutional neural network, the original pixel space is mapped onto a dense attribute feature map, where each dimension of the pixelwise feature indicates the probabilistic strength of a certain semantic class. Then, LAF built on the idea of spatial pyramids on neighboring patches is proposed to explore more spatial context and local information. Finally, the traditional vector of locally aggregated descriptor (VLAD) encoding method is extended to a more generalized form weighted-VLAD (W-VLAD) in which diverse coefficient weights are taken into consideration. Experimental results validate the effectiveness of our presented method.

© 2013 IEEE. In the traditional graph embedding framework, the graph is usually built by k-NN or r-ball. Since it is difficult to manually set the parameters k and r in the high-dimensional space, sparse representation-based methods are usually introduced to automatically build the graphs. In recent years, nuclear norm-based matrix regression (NMR) has been proposed for face recognition using the low rank structural information (i.e., the image matrix-based error model). Inspired by NMR, we give a NMR-based projections (NMRP) method for feature extraction and recognition. The experiments on FERET and extended Yale B face databases show that NMR can be used to build the graph while NMRP is an effective feature extraction method.

Graph-based methods have been widely adopted to detect salient objects in images. However, there are two limitations of these methods. First, only one kind of query is employed for saliency propagation on the graph. Second, these methods only represent pairwise relations between vertices and thus give an incomplete representation of the relationships between image regions. In this paper, we propose a foreground- and background-queries-based hypergraph optimization framework for salient region detection. In this framework, both foreground queries and background queries are explicitly exploited to uniformly highlight the salient foreground and suppress the non-salient background. Furthermore, to include both the pairwise and the higher-order relations among two or more vertices, a probabilistic hypergraph is constructed based on local spatial correlation, global spatial correlation, and color correlation to represent the relations among image regions from different views. Extensive experimental results demonstrate the effectiveness of the proposed framework.

Zero-shot learning (ZSL), a type of structured multioutput learning, has attracted much attention due to its requirement of no training data for target classes. Conventional ZSL methods usually project visual features into semantic space and assign labels by finding their nearest prototypes. However, this type of nearest neighbor search (NNS)-based method often suffers from great performance degradation because of the nonuniform variances between different categories. In this article, we propose a probabilistic framework by taking covariance into account to deal with the above-mentioned problem. In this framework, we define a new latent space, which has two characteristics. The first is that the features in this space should gather within the classes and scatter between the classes, which is implemented by triplet learning; the second is that the prototypes of unseen classes are synthesized with nonnegative coefficients, which are generated by nonnegative matrix factorization (NMF) of relations between the seen classes and the unseen classes in attribute space. During training, the learned parameters are the projection model for triplet network and the nonnegative coefficients between the unseen classes and the seen classes. In the testing phase, visual features are projected into latent space and assigned with the labels that have the maximum probability among unseen classes for classic ZSL or within all classes for generalized ZSL. Extensive experiments are conducted on four popular data sets, and the results show that the proposed method can outperform the state-of-the-art methods in most circumstances.

Of late, there are many studies on the robust discriminant analysis, which adopt L1-norm as the distance metric, but their results are not robust enough to gain universal acceptance. To overcome this problem, the authors of this article present a nonpeaked discriminant analysis (NPDA) technique, in which cutting L1-norm is adopted as the distance metric. As this kind of norm can better eliminate heavy outliers in learning models, the proposed algorithm is expected to be stronger in performing feature extraction tasks for data representation than the existing robust discriminant analysis techniques, which are based on the L1-norm distance metric. The authors also present a comprehensive analysis to show that cutting L1-norm distance can be computed equally well, using the difference between two special convex functions. Against this background, an efficient iterative algorithm is designed for the optimization of the proposed objective. Theoretical proofs on the convergence of the algorithm are also presented. Theoretical insights and effectiveness of the proposed method are validated by experimental tests on several real data sets.

Learning long-term dependences (LTDs) with recurrent neural networks (RNNs) is challenging due to their limited internal memories. In this paper, we propose a new external memory architecture for RNNs called an external addressable long-term and working memory (EALWM)-augmented RNN. This architecture has two distinct advantages over existing neural external memory architectures, namely the division of the external memory into two parts—long-term memory and working memory—with both addressable and the capability to learn LTDs without suffering from vanishing gradients with necessary assumptions. The experimental results on algorithm learning, language modeling, and question answering demonstrate that the proposed neural memory architecture is promising for practical applications.

Remote sensing is a classical area of research that has been involved in many crucial applications, including urban development agriculture, scene interpretation, defense, weather, and other non-Earth observations. In the last decade, the analysis of hyperspectral images (HSIs) acquired by remote sensors has gained substantial attention and is increasingly becoming an active research discipline. However, there are some main challenges in hyperspectral data classification, such as ultra-high dimensionality of data, a limited number of labeled instances, and large spatial variability of spectral signature. These challenges degrade the ability to differentiate the pairwise distance between points and make it difficult to discriminate the most relevant features, causing the classification performance to give wrong or inaccurate results. Therefore, in processing hyperspectral images, the classification approaches have been proposed jointly by dimensionality reduction. Several feature extraction based HSI have been developed to solve the classification problem in hyperspectral images. These methods aim to reduce the dimensionality of the data while preserving the discriminative information of both spectral and spatial features.

Recently, multi-view features have significantly pro- moted the performance of image re-ranking by providing complementary image descriptions. Without loss of generality, in multi-view re-ranking, multiple heterogeneous visual features of high dimensionality are projected onto a low-dimensional subspace, and thus the resulting latent representation can be used for the subsequent similarity-based ranking. Albeit effective, this standard mechanism underplays the intrinsic structure underlying the latent subspace and does not take into account the substantial noise in the original multi-view feature spaces. In this paper, we propose a robust multi-view feature learning strategy for accurate image re-ranking. Due to the dramatic variability in visual appearance for different target images, it is necessary to uncover the shared components underlying those query-related instances that are visually unlike for improving the re-ranking accuracy. Consequently, it is reasonable to assume the latent subspace enjoys the low-rank property and thus the subspace recovery can be achieved via the low-rank modeling accordingly. In addition, the real-world data are usually partially contaminated and it is required to appropriately model the sample-dependent data noise. Towards this end, we employ l2,1- norm based sparsity constraint to model the sample-specific mapping noise for enhancing the model robustness. In order to produce discriminative representations, we encode a similarity preserving term in our multi-view embedding framework. As a result, the sample separability is maximally maintained in the latent subspace with sufficient discriminative power. The extensive experimental evaluations on public landmark benchmarks reveal that our approach achieves impressive performance superior to the state-of-the-art, which thus demonstrates the efficacy of the proposed method.

Vehicle detection is an important part of robot environmental perception. In this paper, a 3D vehicle detection method using a single image is proposed to generate the 3D space coordinate information of the object using monocular vision for autonomous driving. The proposed method works under the multi-task framework and integrates 2D object detection, 3D object detection, orientation estimation and key point detection into one unified deep convolution neural network (DCNN) which could be trained by end-to-end learning. Besides, our proposed method is built by modifying Fast R-CNN using multi-task learning, and thus our proposed method is named multi-task Faster R-CNN (MT-Faster R-CNN). The experiments on KITTI dataset are conducted to evaluate our proposed method and the other 3D vehicle detection methods. The experimental results demonstrate that our proposed method is competitive and could significantly assist autonomous driving.

International audience; The principal component analysis network (PCANet), which is one of the recently proposed deep learning architectures, achieves the state-of-the-art classification accuracy in various databases. However, the visualization or explanation of the PCANet is lacked. In this paper, we try to explain why PCANet works well from energy perspective point of view based on a set of experiments. The paper shows that the error rate of PCANet is qualitatively correlated with the inverse of the logarithm of BlockEnergy, which is the energy after the block sliding process of PCANet, and also this relation is quantified by using curve fitting method. The proposed energy explanation approach can also be used as a testing method for checking if every step of the constructed networks is necessary.

© 2018 International Joint Conferences on Artificial Intelligence. All right reserved. The performance of data-driven learning approaches is often unsatisfactory when the training data is inadequate either in quantity or quality. Manually labeled privileged information (PI), e.g., attributes, tags or properties, is usually incorporated to improve classifier learning. However, the process of manually labeling is time-consuming and labor-intensive. To address this issue, we propose to enhance classifier learning by extracting PI from untagged corpora, which can effectively eliminate the dependency on manually labeled data. In detail, we treat each selected PI as a subcategory and learn one classifier for per subcategory independently. The classifiers for all subcategories are then integrated together to form a more powerful category classifier. Particularly, we propose a new instance-level multi-instance learning (MIL) model to simultaneously select a subset of training images from each subcategory and learn the optimal classifiers based on the selected images. Extensive experiments demonstrate the superiority of our approach.

We propose the inverse problem of Visual question answering (iVQA), and explore its suitability as a benchmark for visuo-linguistic understanding. The iVQA task is to generate a question that corresponds to a given image and answer pair. Since the answers are less informative than the questions, and the questions have less learnable bias, an iVQA model needs to better understand the image to be successful than a VQA model. We pose question generation as a multi-modal dynamic inference process and propose an iVQA model that can gradually adjust its focus of attention guided by both a partially generated question and the answer. For evaluation, apart from existing linguistic metrics, we propose a new ranking metric. This metric compares the ground truth question's rank among a list of distractors, which allows the drawbacks of different algorithms and sources of error to be studied. Experimental results show that our model can generate diverse, grammatically correct and content correlated questions that match the given answer., CVPR18 Spotlight

Steerable properties dominate the design of traditional filters, e.g., Gabor filters, and endow features the capability of dealing with spatial transformations. However, such excellent properties have not been well explored in the popular deep convolutional neural networks (DCNNs). In this paper, we propose a new deep model, termed Gabor Convolutional Networks (GCNs or Gabor CNNs), which incorporates Gabor filters into DCNNs to enhance the resistance of deep learned features to the orientation and scale changes. By only manipulating the basic element of DCNNs based on Gabor filters, i.e., the convolution operator, GCNs can be easily implemented and are compatible with any popular deep learning architecture. Experimental results demonstrate the super capability of our algorithm in recognizing objects, where the scale and rotation changes occur frequently. The proposed GCNs have much fewer learnable network parameters, and thus is easier to train with an endtoend pipeline. The source code will be here 1.

Multihop range-free localization methods could obtain relatively reasonable location estimation in the isotropic network; however, during the practical application, it is often affected by various anisotropic factors such as the radio irregularity and barriers, which can significantly reduce its performance. In this paper, we propose a new approach for multihop localization in wireless sensor network based on nonlinear mapping and learning algorithm. The proposed method is simple, efficient, higher accuracy and no need to set complex parameter in that only hop-counts information and position information of the beacons are used for the localization. The proposed approach is composed of two steps: firstly, this algorithm uses kernel function to define the connectivity information (hop-counts) between nodes, then, learning method is used to guide and build the inter-node localization model; secondly, the hop-counts between the unknown nodes and beacons are used to estimate the coordinate of unknown nodes. We evaluate our algorithm under various anisotropic network and real environment, and analyze its performance. We also compare our approach with several available advanced approaches, and demonstrate the superior performance of our proposed algorithm in terms of location estimation adaptability.

This paper presents a simple and robust feature descriptor method, namely local dominant orientation feature histogram (LDOFH). In this method, the discriminant histogram contains dominant orientation, and the corresponding relative energy value is obtained by calculating the direction and the amplitude of the gradient of each pixel over a local patch. The LM-NNDA (Yang et al. in Pattern Recognit 44:1387–1402, 2011) method based on the principal component analysis (PCA) method is finally employed to reduce the redundancy information and get the low-dimensional and discriminative features. We apply this descriptor on AR, IMM face image databases. Experimental results demonstrate the effectiveness of the proposed LDOFH method.

Semantic gap, which refers to the limitation that low-level hand-crafted visual features insufficiently encode high-level semantic concepts contained in the images, has been a challenging issue in image retrieval and significantly impairs the performance of real-world retrieval systems. Despite massive efforts that have been devoted to developing effective image signatures, e.g., Bag-of-Visual-Words (BOVW), the Fisher Vector (FV) and the Vector of Locally Aggregated Descriptors (VLAD), these mid-level image features still fail to handle the problem of semantic gap and thus lead to suboptimal results. Towards this end, a large body of work focuses on introducing attribute learning into a variety of vision applications. As inherent nature that describes the intrinsic properties of objects, such as color, shape and rigidity, learned attributes serve as intermediate representations that bridge the semantic gap. However, conventional attribute embedding methods are generally developed for image global representation while ignoring local spatial cues, which prevents them from achieving desirable performance. In this paper, we attempt to encode weak spatial information into attribute embedding for effective image retrieval. Specifically, we partition the image into regular grids and extract Classemes attribute vector from each patch, which results in a large pool of Classemes descriptors followed by VLAD aggregation for generating holistic representation. In order to produce a compact and discriminative code, we employ a piecewise Fisher Discriminant Analysis (FDA) for dimension reduction and concatenate all the compressed Classemes into a single vector coined Spatially Pooled Attributes (SPA). Thorough experimental evaluation and comparative study on three public benchmarks demonstrate the superiority of the proposed approach.

© 2016 IEEE. Given an unreliable visual patterns and insufficient query information, content-based image retrieval is often suboptimal and requires image re-ranking using auxiliary information. In this paper, we propose a discriminative multiview interactive image re-ranking (DMINTIR), which integrates user relevance feedback capturing users' intentions and multiple features that sufficiently describe the images. In DMINTIR, heterogeneous property features are incorporated in the multi-view learning scheme to exploit their complementarities. In addition, a discriminatively learned weight vector is obtained to reassign updated scores and target images for re-ranking. Compared with other multi-view learning techniques, our scheme not only generates a compact representation in the latent space from the redundant multi-view features but also maximally preserves the discriminative information in feature encoding by the large-margin principle. Furthermore, the generalization error bound of the proposed algorithm is theoretically analyzed and shown to be improved by the interactions between the latent space and discriminant function learning. Experimental results on two benchmark data sets demonstrate that our approach boosts baseline retrieval quality and is competitive with the other state-of-the-art re-ranking strategies.

In this paper, we propose using high-level action units to represent human actions in videos and, based on such units, a novel sparse model is developed for human action recognition. There are three interconnected components in our approach. First, we propose a new context-aware spatial-temporal descriptor, named locally weighted word context, to improve the discriminability of the traditionally used local spatial-temporal descriptors. Second, from the statistics of the context-aware descriptors, we learn action units using the graph regularized nonnegative matrix factorization, which leads to a part-based representation and encodes the geometrical information. These units effectively bridge the semantic gap in action recognition. Third, we propose a sparse model based on a joint l2,1-norm to preserve the representative items and suppress noise in the action units. Intuitively, when learning the dictionary for action representation, the sparse model captures the fact that actions from the same class share similar units. The proposed approach is evaluated on several publicly available data sets. The experimental results and analysis clearly demonstrate the effectiveness of the proposed approach.

The “CNN-RNN” design pattern is increasingly widely applied in a variety of image annotation tasks including multi-label classification and captioning. Existing models use the weakly semantic CNN hidden layer or its transformas the image embedding that provides the interface between the CNN and RNN. This leaves the RNN overstretched with two jobs: predicting the visual concepts and modelling their correlations for generating structured annotation output.Importantly this makes the end-to-end training of the CNN and RNN slow and ineffective due to the difficulty of back propagating gradients through the RNN to train the CNN. We propose a simple modification to the design pattern thatmakes learning more effective and efficient. Specifically, we propose to use a semantically regularised embedding layer as the interface between the CNN and RNN. Regularising the interface can partially or completely decouple the learningproblems, allowing each to be more effectively trained and jointly training much more efficient. Extensive experiments show that state-of-the art performance is achieved on multi-label classification as well as image captioning.

In this paper, we propose a Multi-Manifold Discriminant Analysis (MMDA) method for an image feature extraction and pattern recognition based on graph embedded learning and under the Fisher discriminant analysis framework. In an MMDA, the within-class graph and between-class graph are, respectively, designed to characterize the within-class compactness and the between-class separability, seeking for the discriminant matrix to simultaneously maximize the between-class scatter and minimize the within-class scatter. In addition, in an MMDA, the within-class graph can represent the sub-manifold information, while the between-class graph can represent the multi-manifold information. The proposed MMDA is extensively examined by using the FERET, AR and ORL face databases, and the PolyU finger-knuckle-print databases. The experimental results demonstrate that an MMDA is effective in feature extraction, leading to promising image recognition performance.