Search

With the advantages of low power consumption, flexibility, and high readability against bright ambiance, electrophoretic displays (EPDs) have wide application prospects in the fields of education, smart supermarkets, Internet of Things, smart homes, wearable devices, etc. EPDs suffer from a severe history dependence during grayscale switching, which results in annoying ghost images. However, currently, it is difficult to distinguish diverse types of ghost images automatically; thus, lookup-tables (LUTs) for multi-grayscale waveform design cannot be effectively generated but require cumbersome manual adjustment. In this article, we proposed to adopt a convolutional neural network (CNN) to automatically recognize ghost images, based on which, LUTs that could suppress ghost images and achieve accurate grayscales were automatically generated for waveform design. Moreover, the workforce for manual adjustment was significantly saved. The results suggest that the CNN is a powerful tool for EPDs to achieve better image quality, as well as less manual cost. [ABSTRACT FROM AUTHOR]

Digitally unwrapping images of paper sheets is crucial for accurate document scanning and text recognition. This paper presents a method for automatically rectifying curved or folded paper sheets from a few images captured from multiple viewpoints. Prior methods either need expensive 3D scanners or model deformable surfaces using over-simplified parametric representations. In contrast, our method uses regular images and is based on general developable surface models that can represent a wide variety of paper deformations. Our main contribution is a new robust rectification method based on ridge-aware 3D reconstruction of a paper sheet and unwrapping the reconstructed surface using properties of developable surfaces via $\ell _1$ conformal mapping. We present results on several examples including book pages, folded letters and shopping receipts. [ABSTRACT FROM AUTHOR]

This paper proposes Value-Aware solid-state drive (SSD) with fast access speed and low power consumption by eliminating error-correcting code (ECC). Value-Aware SSD utilizes the error tolerance of image recognition application using a deep neural network (DNN) to enhance reliability. In a previous paper, which proposes Value-Aware SSD, fast ECC decoder is implemented and SSD is evaluated with the 32-bit floating-point data format. On the other hand, in this paper, the proposed Value-Aware SSD is analyzed with 32-bit and 8-bit fixed-point data format and achieves the higher reliability even without ECC by newly proposed two techniques, Critical Bit Error Reduction (CBER) and Middle & Lower Page Error Reduction (M&L-PER). CBER and M&L-PER are proposed for 32-bit and 8-bit data format of application, respectively. These techniques modulate the threshold voltage ($V_{\mathrm {TH}}$) distribution of memory cells by recognizing the importance of each stored bit. By proposed CBER, as much as 15% bit error rate (BER) of NAND flash is allowed while the application provides high image recognition accuracy. Even if bit precision is truncated to 8 bit, 3.9% BER is accepted by M&L-PER. The fast read access and low power consumption are realized because ECC is not required. Finally, this paper analyzes the Value-Aware techniques with 3-D multi-level cell (MLC) NAND flash to compare the effects for 3D-MLC and triple-level cell (TLC) NAND flash. [ABSTRACT FROM AUTHOR]

Deep learning models have been shown to be vulnerable to adversarial attacks. Adversarial attacks are imperceptible perturbations added to an image such that the deep learning model misclassifies the image with a high confidence. Existing adversarial defenses validate their performance using only the classification accuracy. However, classification accuracy by itself is not a reliable metric to determine if the resulting image is “adversarial-free”. This is a foundational problem for online image recognition applications where the ground-truth of the incoming image is not known and hence we cannot compute the accuracy of the classifier or validate if the image is “adversarial-free” or not. This paper proposes a novel privacy preserving framework for defending Black box classifiers from adversarial attacks using an ensemble of iterative adversarial image purifiers whose performance is continuously validated in a loop using Bayesian uncertainties. The proposed approach can convert a single-step black box adversarial defense into an iterative defense and proposes three novel privacy preserving Knowledge Distillation (KD) approaches that use prior meta-information from various datasets to mimic the performance of the Black box classifier. Additionally, this paper proves the existence of an optimal distribution for the purified images that can reach a theoretical lower bound, beyond which the image can no longer be purified. Experimental results on six public benchmark datasets namely: 1) Fashion-MNIST, 2) CIFAR-10, 3) GTSRB, 4) MIO-TCD, 5) Tiny-ImageNet, and 6) MS-Celeb show that the proposed approach can consistently detect adversarial examples and purify or reject them against a variety of adversarial attacks. [ABSTRACT FROM AUTHOR]

Laser trimming promises an increasing yield in microfabrication for the high performance Micro Electro-Mechanical Systems (MEMS) resonant devices by overcoming manufacturing tolerances and tuning sensors or actuators for certain operating conditions. How to achieve a high precision and efficiency trimming at low cost is the critical issue that the technical communities are most concerned. In this paper, we proposed an automatic laser trimming system for silicon MEMS devices based on a picosecond laser source and image recognition technique (IRT). Through the design of laser focusing optical path and coaxial imaging optical path, the observation of laser machining process is realized. A set of picosecond laser trimming parameters is determined by experiments, which enables a 4 $\mu {\mathrm{ m}}$ width trench on the silicon resonator. Compared with the femtosecond laser trimming systems, the cost of the pico-laser system is reduced by more than 60%. The shape matching algorithm based on OpenCV and the chord-to-point distance accumulation (CPDA) algorithm for MEMS resonators image measurement solve the problem of inefficiency of manual trimming for complex resonators. A dual-mass tuning fork resonator is used to demonstrate the feasibility of the trimming approach. It is shown that the system realizes a fine-tuning (< 0.3Hz) of the resonant frequencies. The trimming time of a single device using the automatic trimming method is reduced from more than 1 hour manual trimming to less than 1 minute. Finally, the structural stiffness imbalance of tuning fork resonator caused by the manufacturing error is eliminated, which verifies the feasibility of the system. [ABSTRACT FROM AUTHOR]

This paper proposes a momentum algorithm based on Borges difference and an adaptive momentum (Adam) algorithm based on Borges difference to update parameters, which can adjust the momentum information more flexibly. The Borges difference is proposed from the definition of Borges derivative to be combined with the gradient algorithm in convolutional neural networks. The proposed momentum algorithm based on Borges difference and Adam algorithm based on Borges difference can be adjusted more flexibly in order to speed up the convergence. The parameter optimization algorithm with the Borges difference presents a better performance compared with the integer-order momentum algorithm and integer-order Adam algorithm, with the proposed nonlinear adjustment method for the parameter tuning of convolutional neural networks. By analyzing experimental results of Fashion-MNIST dataset and CIFAR-10 dataset, the optimization algorithms based on Borges difference proposed in this paper gain better effects on the optimization model compared with the corresponding ones based on the integer-order difference, and can speed up the convergence speed and recognition accuracy of the image recognition. [ABSTRACT FROM AUTHOR]

Few-shot action recognition classifies new actions with only few training samples, of which the mainstream methods adopt class means to obtain prototypes as the representations of each category. However, affected by sample capacity and extreme samples, mean-of-class prototypes can’t well represent the average level of samples. In this paper, we enhance the prototypes from multiple dimensions for better classification. We firstly propose a novel similarity optimization mechanism where Prototype Aggregation Adaptive Loss (PAAL) is designed to deeply mine the similarity between samples and prototypes for enhancing the ability of inter-class differential detail identification. Secondly, for mitigating the impact of the samples on class prototypes, we refactor the prototype calculation formula with Cross-Enhanced Prototype (CEP) to narrow intra-class differences in which Reweighted Similarity Attention (RSA) is designed to update prototypes. Finally, Dynamic Temporal Transformation (DTT) is proposed to alleviate inconsistent distribution of temporal information for obtaining better video-level descriptors. Extensive experiments on standard benchmark datasets demonstrate that our proposed method achieves the state-of-the-art results. [ABSTRACT FROM AUTHOR]

Dynamic vision sensors (event cameras) have recently been introduced to solve a number of different vision tasks such as object recognition, activities recognition, tracking, etc. Compared with the traditional RGB sensors, the event cameras have many unique advantages such as ultra low resources consumption, high temporal resolution and much larger dynamic range. However, these cameras only produce noisy and asynchronous events of intensity changes, i.e., event-streams rather than frames, where conventional computer vision algorithms can't be directly applied. In our opinion the key challenge for improving the performance of event cameras in vision tasks is finding the appropriate representations of the event-streams so that cutting-edge learning approaches can be applied to fully uncover the spatio-temporal information contained in the event-streams. In this paper, we focus on the event-based human gait identification task and investigate the possible representations of the event-streams when deep neural networks are applied as the classifier. We propose new event-based gait recognition approaches basing on two different representations of the event-stream, i.e., graph and image-like representations, and use graph-based convolutional network (GCN) and convolutional neural networks (CNN) respectively to recognize gait from the event-streams. The two approaches are termed as EV-Gait-3DGraph and EV-Gait-IMG. To evaluate the performance of the proposed approaches, we collect two event-based gait datasets, one from real-world experiments and the other by converting the publicly available RGB gait recognition benchmark CASIA-B. Extensive experiments show that EV-Gait-3DGraph achieves significantly higher recognition accuracy than other competing methods when sufficient training samples are available. However, EV-Gait-IMG converges more quickly than graph-based approaches while training and shows good accuracy with only few number of training samples (less than ten). So image-like presentation is preferable when the amount of training data is limited. [ABSTRACT FROM AUTHOR]

Domain adaptation leverages rich knowledge from a related source domain so that it can be used to perform tasks in a target domain. For more knowledge to be obtained under relaxed conditions, domain adaptation methods have been widely used in pattern recognition and image classification. However, most of the existing domain adaptation methods only consider how to minimize different distributions of the source and target domains, which neglects what should be transferred for a specific task and suffers negative transfer by distribution outliers. To address these problems, in this paper, we propose a novel domain adaptation method called weighted correlation embedding learning (WCEL) for image classification. In the WCEL approach, we seamlessly integrated correlation learning, graph embedding, and sample reweighting into a unified learning model. Specifically, we extracted the maximum correlated features from the source and target domains for image classification tasks. In addition, two graphs were designed to preserve the discriminant information from interclass samples and neighborhood relations in intraclass samples. Furthermore, to prevent the negative transfer problem, we developed an efficient sample reweighting strategy to predict the target with different confidence levels. To verify the performance of the proposed method in image classification, extensive experiments were conducted with several benchmark databases, verifying the superiority of the WCEL method over other state-of-the-art domain adaptation algorithms. [ABSTRACT FROM AUTHOR]

In this paper, a Multi-scale Contrastive Graph Convolutional Network (MC-GCN) method is proposed for unconstrained face recognition with image sets, which takes a set of media (orderless images and videos) as a face subject instead of single media (an image or video). Due to factors such as illumination, posture, media source, etc., there are huge intra-set variances in a face set, and the importance of different face prototypes varies considerably. How to model the attention mechanism according to the relationship between prototypes or images in a set is the main content of this paper. In this work, we formulate a framework based on graph convolutional network (GCN), which considers face prototypes as nodes to build relations. Specifically, we first present a multi-scale graph module to learn the relationship between prototypes at multiple scales. Moreover, a Contrastive Graph Convolutional (CGC) block is introduced to build attention control model, which focuses on those frames with similar prototypes (contrastive information) between pair of sets instead of simply evaluating the frame quality. The experiments on IJB-A, YouTube Face, and an animal face dataset clearly demonstrate that our proposed MC-GCN outperforms the state-of-the-art methods significantly. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNN) and Graph Neural Networks (GNN), such as Graph Attention Networks (GAT), are two classic neural network models, which are applied to the processing of grid data and graph data respectively. They have achieved outstanding performance in hyperspectral images (HSIs) classification field, which have attracted great interest. However, CNN has been facing the problem of small samples and GNN has to pay a huge computational cost, which restrict the performance of the two models. In this paper, we propose Weighted Feature Fusion of Convolutional Neural Network and Graph Attention Network (WFCG) for HSI classification, by using the characteristics of superpixel-based GAT and pixel-based CNN, which proved to be complementary. We first establish GAT with the help of superpixel-based encoder and decoder modules. Then we combined the attention mechanism to construct CNN. Finally, the features are weighted fusion with the characteristics of two neural network models. Rigorous experiments on three real-world HSI data sets show WFCG can fully explore the high-dimensional feature of HSI, and obtain competitive results compared to other state-of-the art methods. [ABSTRACT FROM AUTHOR]

Accurate alignment of the watch hand is a critical procedure in the industrial production of watches. However, compared with the traditional ideal laboratorial case, the watch-hand alignment suffers various external disturbances from the aged equipment and industrial operating conditions, which seriously influence the alignment precision. To achieve accurate watch-hand alignment in the complex industrial environment, this paper develops a robotic micromanipulation system and proposes a corresponding robust control strategy. First, a micromanipulation system with five degrees of freedom is set up and integrated with an optical microscope. Then, a set of proper machine vision methods is adopted to obtain accurate position information in disregard of the interference terms. Third, the dynamic model of the controlled watch hand is built with consideration of the external disturbances from the production environment. Since the velocity information cannot be observed accurately through the machine vision process, we design a state observer to acquire the position and velocity values for the following control strategy. After that, a sliding mode controller with the radical basis function neural network is proposed to achieve high-precision alignment while resisting the external disturbance. Finally, simulation and experimental results prove that the proposed method can realize the alignment of the watch hand within 6 s with the accuracy of $2\times 10^{-3}$ rad, which is enhanced at least two times compared with the traditional manual method. Note to Practitioners—This paper was motivated by the challenging problem of the watch-hand alignment in the industrial production. Unlike the traditional manipulation, this alignment is subjected to various disturbances from the aged equipment and complex operating conditions, including complex forces at the air–liquid operation interface and the confusing scratches, undesired flaking particles, as well as the reflection of light in the image. Here, we propose a microrobotic system and a robust control strategy to precisely align the watch hand and eliminate the influence from the above disturbances. In the machine vision process for recognizing the object, filtering, contour/corner detection, and template matching methods are adopted to acquire accurate position information despite the interference in image processing. To avoid the disturbance from the complex forces during the alignment, we develop the robust sliding mode controller to guarantee the alignment precision. The experiment results indicate that the proposed system can align the watch hand with the accuracy enhanced at least two times as previous manual method. This paper provides a general solution for micromanipulation under disturbance in industrial production environment. [ABSTRACT FROM AUTHOR]

Due to their good rotational invariance and stability, image continuous orthogonal moments are intensively applied in rotationally invariant recognition and image processing. However, most moments produce numerical instability, which impacts the image reconstruction and recognition performance. In this paper, a new set of invariant continuous orthogonal moments, polar harmonic Fourier moments (PHFMs), free of numerical instability is designed. The radial basis functions (RBFs) of the PHFMs are much simpler than those of the Chebyshev-Fourier moments (CHFMs), orthogonal Fourier-Mellin moments (OFMMs), Zernike moments (ZMs), and pseudo-Zernike moments (PZMs). For the same degree, the RBFs of the PHFMs have more zeros and are more evenly distributed than those of the ZMs and PZMs. Therefore, PHFMs do not suffer from information suppression problem; hence, the image description ability of the PHFMs is superior to that of the ZMs and PZMs. Moreover, the RBFs of the PHFMs are always less than or equal to 1.0 near the unit disk center, whereas those of the OFMMs, PZMs, CHFMs, and radial harmonic Fourier moments (RHFMs) are infinite (implying numerical instability). This indicates that PHFMs can outperform these moments in image reconstruction tasks. We theoretically and experimentally demonstrate that PHFMs outperform the above moments in reconstructing images and recognizing rotationally invariant objects considering noise and various attacks. This paper also details the significance of the PHFM phase in image reconstruction, angle estimation using PHFMs, and the accurate moment selection of the PHFMs. [ABSTRACT FROM AUTHOR]

Recently, food recognition has received more and more attention in image processing and computer vision for its great potential applications in human health. Most of the existing methods directly extracted deep visual features via convolutional neural networks (CNNs) for food recognition. Such methods ignore the characteristics of food images and are, thus, hard to achieve optimal recognition performance. In contrast to general object recognition, food images typically do not exhibit distinctive spatial arrangement and common semantic patterns. In this paper, we propose a multi-scale multi-view feature aggregation (MSMVFA) scheme for food recognition. MSMVFA can aggregate high-level semantic features, mid-level attribute features, and deep visual features into a unified representation. These three types of features describe the food image from different granularity. Therefore, the aggregated features can capture the semantics of food images with the greatest probability. For that solution, we utilize additional ingredient knowledge to obtain mid-level attribute representation via ingredient-supervised CNNs. High-level semantic features and deep visual features are extracted from class-supervised CNNs. Considering food images do not exhibit distinctive spatial layout in many cases, MSMVFA fuses multi-scale CNN activations for each type of features to make aggregated features more discriminative and invariable to geometrical deformation. Finally, the aggregated features are more robust, comprehensive, and discriminative via two-level fusion, namely multi-scale fusion for each type of features and multi-view aggregation for different types of features. In addition, MSMVFA is general and different deep networks can be easily applied into this scheme. Extensive experiments and evaluations demonstrate that our method achieves state-of-the-art recognition performance on three popular large-scale food benchmark datasets in Top-1 recognition accuracy. Furthermore, we expect this paper will further the agenda of food recognition in the community of image processing and computer vision. [ABSTRACT FROM AUTHOR]

This paper addresses the problem of matching common node correspondences among multiple graphs referring to an identical or related structure. This multi-graph matching problem involves two correlated components: i) the local pairwise matching affinity across pairs of graphs; ii) the global matching consistency that measures the uniqueness of the pairwise matchings by different composition orders. Previous studies typically either enforce the matching consistency constraints in the beginning of an iterative optimization, which may propagate matching error both over iterations and across graph pairs; or separate affinity optimization and consistency enforcement into two steps. This paper is motivated by the observation that matching consistency can serve as a regularizer in the affinity objective function especially when the function is biased due to noises or inappropriate modeling. We propose composition-based multi-graph matching methods to incorporate the two aspects by optimizing the affinity score, meanwhile gradually infusing the consistency. We also propose two mechanisms to elicit the common inliers against outliers. Compelling results on synthetic and real images show the competency of our algorithms. [ABSTRACT FROM PUBLISHER]

Recent studies have shown that a Deep Convolutional Neural Network (DCNN) trained on a large image dataset can be used as a universal image descriptor and that doing so leads to impressive performance for a variety of image recognition tasks. Most of these studies adopt activations from a single DCNN layer, usually a fully-connected layer, as the image representation. In this paper, we proposed a novel way to extract image representations from two consecutive convolutional layers: one layer is used for local feature extraction and the other serves as guidance to pool the extracted features. By taking different viewpoints of convolutional layers, we further develop two schemes to realize this idea. The first directly uses convolutional layers from a DCNN. The second applies the pre-trained CNN on densely sampled image regions and treats the fully-connected activations of each image region as a convolutional layer's feature activations. We then train another convolutional layer on top of that as the pooling-guidance convolutional layer. By applying our method to three popular visual classification tasks, we find that our first scheme tends to perform better on applications which demand strong discrimination on lower-level visual patterns while the latter excels in cases that require discrimination on category-level patterns. Overall, the proposed method achieves superior performance over existing approaches for extracting image representations from a DCNN. In addition, we apply cross-layer pooling to the problem of image retrieval and propose schemes to reduce the computational cost. Experimental results suggest that the proposed method achieves promising results for the image retrieval task. [ABSTRACT FROM AUTHOR]

The task of feature selection is to find the most representative features from the original high-dimensional data. Because of the absence of the information of class labels, selecting the appropriate features in unsupervised learning scenarios is much harder than that in supervised scenarios. In this paper, we investigate the potential of locally linear embedding (LLE), which is a popular manifold learning method, in feature selection task. It is straightforward to apply the idea of LLE to the graph-preserving feature selection framework. However, we find that this straightforward application suffers from some problems. For example, it fails when the elements in the feature are all equal; it does not enjoy the property of scaling invariance and cannot capture the change of the graph efficiently. To solve these problems, we propose a new filter-based feature selection method based on LLE in this paper, which is named as LLE score. The proposed criterion measures the difference between the local structure of each feature and that of the original data. Our experiments of classification task on two face image data sets, an object image data set, and a handwriting digits data set show that LLE score outperforms state-of-the-art methods, including data variance, Laplacian score, and sparsity score. [ABSTRACT FROM PUBLISHER]

Automotive radars have faced to various jamming signals with different types and parameters. One counter measure is to identify the fine-grained modulation types of these jamming signals, which can provide supplementary information for intercepting. Most existing signal modulation recognition methods attempt to establish a machine-learning mechanism by training with a large number of annotated samples, which is hardly applicable in real-world electronic reconnaissance scenarios where only a few samples can be intercepted in advance. Few-Shot Learning aims to learn from base classes with many samples and infuse this knowledge to support classes with only a few samples, thus realizing model generalization. In this paper, inspired by the fact that the energy distribution of the signal of interest is always concentrated in time-frequency images, a novel few-shot learning framework based on foreground segmentation is proposed, which benefits from its powerful filtering to remove the noise and clutter from background. The experimental results show that the proposed method can achieve excellent performance for fine-grained signal modulation recognition even with only one support sample and is robust under different signal-to-noise-ratio conditions. [ABSTRACT FROM AUTHOR]

Mobile image recognition services, which provide people with image recognition services through the cameras of mobile devices, are revolutionizing our lives. However, most existing cloud/edge-based approaches suffer from two major limitations, (i) Low recognition accuracy and high network bandwidth pressure, and (ii) Not easy to extract features based on currently available resources of mobile devices. In this paper, we propose a resource-aware feature extraction framework for mobile image recognition services. The proposed framework consists of discriminative feature extraction (DFE) and NestDFE algorithms. The DFE algorithm can generate an extractor ${{\mathbf E}}$ E to extract discriminative features from the image data set on the edge server and images on mobile devices. Thus, the proposed framework can achieve higher recognition accuracy and require mobile devices to upload less feature data to the edge server. The NestDFE algorithm generates a single multi-capacity extractor that acts as a series of sub-extractors and enables mobile devices to dynamically select sub-extractors. Experimental results show that the proposed framework improves recognition accuracy by about 23 percent and reduces network traffic by about 76 percent compared with existing approaches. [ABSTRACT FROM AUTHOR]

This article is a comment on the recent TPAMI paper (Gopalan et al., 2012) that introduced a blur-invariant distance measure between two images. We point out two mistakes of the theory presented in (Gopalan et al., 2012) and propose a correction. We also compare the original and corrected methods experimentally. [ABSTRACT FROM AUTHOR]

These instructions give guidelines for preparing papers for this publication. Presents information for authors publishing in this journal. [ABSTRACT FROM PUBLISHER]

This paper focuses on the thorax disease classification problem in chest X-ray (CXR) images. Different from the generic image classification task, a robust and stable CXR image analysis system should consider the unique characteristics of CXR images. Particularly, it should be able to: 1) automatically focus on the disease-critical regions, which usually are of small sizes; 2) adaptively capture the intrinsic relationships among different disease features and utilize them to boost the multi-label disease recognition rates jointly. In this paper, we propose to learn discriminative features with a two-branch architecture, named ConsultNet, to achieve those two purposes simultaneously. ConsultNet consists of two components. First, an information bottleneck constrained feature selector extracts critical disease-specific features according to the feature importance. Second, a spatial-and-channel encoding based feature integrator enhances the latent semantic dependencies in the feature space. ConsultNet fuses these discriminative features to improve the performance of thorax disease classification in CXRs. Experiments conducted on the ChestX-ray14 and CheXpert dataset demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

In this paper, we aim to build a comprehensive face detection system which provides a one-stop solution to various practical challenges for face detection in realistic scenarios, e.g., detecting faces from multiple-views, faces with occlusions, exaggerated expressions or blurred faces. Moreover, we introduce an automatic data harvest algorithm to effectively improve the generalization performance of the system even when collecting training faces containing various challenging patterns is difficult. In particular, we introduce three critical components to build the system, i.e., a recently widely used deep convolutional neural network (CNN), a novel blur-aware bi-channel network architecture, and a new self-learning mechanism capable of exploiting video contexts continuously. The aforementioned challenges except for detecting blurred faces can potentially be addressed by the CNN component owing its robustness to local deformation of target faces. The more challenging problem of detecting blurred faces is addressed by the bi-channel architecture component which processes blurred and clear faces adaptively. In addition, to address the difficulties in improving the generalization performance of the learning-based face detection system, we introduce a video-context-based self-learning mechanism into the system, which enables the system to continuously enhance its performance by harvesting faces with challenging training patterns automatically. To exploit video context, the detector is applied to massive unlabeled videos, and challenging faces are captured based on temporal inference. These recaptured faces, generally corresponding to one or multiple challenges mentioned above, are fed into the detection system to further improve its performance. Extensive experiments with the proposed detection system provide new state-of-the-art performance on FDDB data set, PASCAL face data set, AFW data set, and WIDER Face data set. [ABSTRACT FROM AUTHOR]

In this paper, we devise and optimize schemes for the resistive random-access memory (RRAM)-based hardware implementation of convolutional neural networks (CNNs). The key achievements are as follows: 1) a specific circuit for CNN and corresponding operation methods is presented; 2) quantization schemes for utilizing binary RRAM or RRAM with multilevel resistances as synapses are proposed, and simulations show that our CNN system percentage accuracy for the MNIST data set using multilevel RRAM and 97% accuracy using binary RRAM as synapses; 3) the influence of factors such as the number and size of kernels, as well as the device conductance variation, on the final recognition accuracy is discussed. Three ways to reduce the hardware cost are also analyzed; and 4) we implement the online learning function using the developed CNN system with a binary spike-time-dependent plasticity protocol and achieve up to 94% accuracy for the online learning tasks. [ABSTRACT FROM AUTHOR]

Fine-grained image analysis (FGIA) is a longstanding and fundamental problem in computer vision and pattern recognition, and underpins a diverse set of real-world applications. The task of FGIA targets analyzing visual objects from subordinate categories, e.g., species of birds or models of cars. The small inter-class and large intra-class variation inherent to fine-grained image analysis makes it a challenging problem. Capitalizing on advances in deep learning, in recent years we have witnessed remarkable progress in deep learning powered FGIA. In this paper we present a systematic survey of these advances, where we attempt to re-define and broaden the field of FGIA by consolidating two fundamental fine-grained research areas – fine-grained image recognition and fine-grained image retrieval. In addition, we also review other key issues of FGIA, such as publicly available benchmark datasets and related domain-specific applications. We conclude by highlighting several research directions and open problems which need further exploration from the community. [ABSTRACT FROM AUTHOR]

Commercial clouds now heavily use CPUs in DL (deep learning) because there are large numbers of CPUs which would otherwise sit idle during off-peak periods. Following the trend, CPU vendors have not only released high-performance many-core CPUs but also developed efficient math kernel libraries. However, current DL platforms cannot scale well to a large number of CPU cores, making many-core CPUs inefficient in DL computation. We analyze the memory access patterns of various layers and identify the root cause of the low scalability, i.e., the per-layer barriers that are implicitly imposed by current platforms which assign one single instance (i.e., one batch of input data) to a CPU. The barriers cause severe memory bandwidth contention and CPU starvation in the access-intensive layers (like activation and BN). This paper presents a novel approach called ParaX, which boosts the performance of DL on multi-NUMA (non-uniform memory access) many-core CPUs by effectively alleviating bandwidth contention and CPU starvation. Our key idea is to assign one instance to each CPU core instead of to the entire CPU, so as to remove the per-layer barriers on the executions of the many cores. ParaX designs an ultralight scheduling policy which sufficiently overlaps the access-intensive layers with the compute-intensive ones to avoid contention, and proposes a NUMA-aware gradient server mechanism for training which leverages shared memory to substantially reduce the overhead of per-iteration parameter synchronization. We have implemented ParaX on MXNet. Extensive evaluation on a two-NUMA Intel 8280 CPU shows that ParaX significantly improves the training/inference throughput for all tested models (for image recognition and natural language processing) by $1.73\times \sim 2.93{\times}$ 1. 73 × ∼ 2. 93 × . [ABSTRACT FROM AUTHOR]

Person attribute recognition aims to identify the attribute labels from the pedestrian images. Extracting contextual relation from the images and attributes, including the spatial-semantic relations, the spatial context and the semantic correlation, is beneficial to enhance the discrimination of the features for recognizing the attributes. Thus, this work proposes a sequence contextual relation learning (SCRL) method to capture these relations. It first embeds the images and attributes into sequences in two branches. Then SCRL flexibly learns the contextual relation from the sequences with the parallel attention model structure, which integrates the inter-attention and intra-attention models. The inter-attention module is utilized to extract the spatial-semantic relations, while the intra-attention is designed to gain the spatial context and the semantic correlation. Both attention modules are comprised of several parallel attention units and each unit can obtain the pairwise relations in one subspace. Therefore, they obtain the relations in multiple subspaces, which can improve the comprehensiveness of the relation learning. Additionally, for the sake of better extraction of spatial-semantic relations, this paper employs connectionist temporal classification (CTC) loss which is capable of driving the network to enforce monotonic alignment between the image and attribute. It can also accelerate the convergence of the network by the algorithm in it. Extensive experiments on five public datasets, i.e., Market-1501 attribute, Duke attribute, PETA, RAP and PA-100K datasets, demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

One of the major causes of poor results in license plate recognition is low quality of images affected by multiple factors, such as severe illumination condition, complex background, different weather conditions, night light, and perspective distortions. In this paper, we propose a new mathematical model based on Riesz fractional operator for enhancing details of edge information in license plate images to improve the performances of text detection and recognition methods. The proposed model performs convolution operation of the Riesz fractional derivative over each input image by enhancing the edge strength in it. To test the performance of the proposed model, we conduct experiments on benchmark license plate image databases, namely, UCSD and ICDAR 2015-SR competition text image databases. Experimental results on enhancement show that the proposed model outperforms the existing baseline enhancement techniques in terms of quality measures. Furthermore, experimental results on text detection and recognition show that text detection and recognition rates are improved significantly after enhancement compared with before enhancement. [ABSTRACT FROM AUTHOR]

Multiple-view triangulation by $\ell _\infty$ minimisation has become established in computer vision. State-of-the-art $\ell _\infty$ triangulation algorithms exploit the quasiconvexity of the cost function to derive iterative update rules that deliver the global minimum. Such algorithms, however, can be computationally costly for large problem instances that contain many image measurements, e.g., from web-based photo sharing sites or long-term video recordings. In this paper, we prove that $\ell _\infty$ triangulation admits a coreset approximation scheme, which seeks small representative subsets of the input data called coresets. A coreset possesses the special property that the error of the $\ell _\infty$ solution on the coreset is within known bounds from the global minimum. We establish the necessary mathematical underpinnings of the coreset algorithm, specifically, by enacting the stopping criterion of the algorithm and proving that the resulting coreset gives the desired approximation accuracy. On large-scale triangulation problems, our method provides theoretically sound approximate solutions. Iterated until convergence, our coreset algorithm is also guaranteed to reach the true optimum. On practical datasets, we show that our technique can in fact attain the global minimiser much faster than current methods. [ABSTRACT FROM AUTHOR]

Deep ConvNets have shown their good performance in image classification tasks. However, there still remains problems in deep video representations for action recognition. On one hand, current video ConvNets are relatively shallow compared with image ConvNets, which limits their capability of capturing the complex video action information; on the other hand, temporal information of videos is not properly utilized to pool and encode the video sequences. Toward these issues, in this paper we utilize two state-of-the-art ConvNets, i.e., the very deep spatial net (VGGNet ) and the temporal net from Two-Stream ConvNets , for action representation. The convolutional layers and the proposed new layer, called frame-diff layer, are extracted and pooled with two temporal pooling strategies: Trajectory pooling and Line pooling. The pooled local descriptors are then encoded with vector of locally aggregated descriptors (VLAD) to form the video representations. In order to verify the effectiveness of the proposed framework, we conduct experiments on UCF101 and HMDB51 data sets. It achieves accuracy of 92.08% on UCF101, which is the state-of-the-art, and the accuracy of 65.62% on HMDB51, which is comparable to the state-of-the-art. In addition, we propose the new Line pooling strategy, which can speed up the extraction of feature and achieve the comparable performance of the Trajectory pooling. [ABSTRACT FROM AUTHOR]

Person re-identification (Re-Id) across nonoverlapping camera views is one of challenging problems in surveillance video analysis. The difficulties in person Re-Id mainly come from the large appearance variations caused by camera view angle, human pose, illumination, and occlusion. Recently, extensive efforts have been cast into addressing this problem by developing invariant features or discriminative distance metrics. However, there is still a lack of systematic evaluations on the pipeline for feature extraction and combination. In this paper, we propose a spatial pyramid-based statistical feature extraction framework as a unified pipeline of feature extraction and combination for person Re-Id, and systematically evaluate the configuration details in feature extraction and the fusion strategies in feature combination. Extensive experiments on benchmark datasets demonstrate the critical components in feature extraction. Moreover, by combining multiple features, our proposed approach can yield state-of-the-art performance. It should be mentioned that our approach achieves rank 1 matching rate of 45.8% on dataset VIPeR and 61.5% on dataset CUHK01, respectively. [ABSTRACT FROM AUTHOR]

Machine-learning algorithms have shown outstanding image recognition/classification performance for computer vision applications. However, the compute and energy requirement for implementing such classifier models for large-scale problems is quite high. In this paper, we propose feature driven selective classification (FALCON) inspired by the biological visual attention mechanism in the brain to optimize the energy-efficiency of machine-learning classifiers. We use the consensus in the characteristic features (color/texture) across images in a dataset to decompose the original classification problem and construct a tree of classifiers (nodes) with a generic-to-specific transition in the classification hierarchy. The initial nodes of the tree separate the instances based on feature information and selectively enable the latter nodes to perform object specific classification. The proposed methodology allows selective activation of only those branches and nodes of the classification tree that are relevant to the input while keeping the remaining nodes idle. Additionally, we propose a programmable and scalable neuromorphic engine (NeuE) that utilizes arrays of specialized neural computational elements to execute the FALCON-based classifier models for diverse datasets. The structure of FALCON facilitates the reuse of nodes while scaling up from small classification problems to larger ones thus allowing us to construct classifier implementations that are significantly more efficient. We evaluate our approach for a 12-object classification task on the Caltech101 dataset and ten-object task on CIFAR-10 dataset by constructing FALCON models on the NeuE platform in 45-nm technology. Our results demonstrate up to $3.66\boldsymbol \times $ improvement in energy-efficiency for no loss in output quality, and even higher improvements of up to $5.91\boldsymbol \times $ with 3.9% accuracy loss compared to an optimized baseline network. In addition, FALCON shows an improvement in training time of up to $1.96\boldsymbol \times $ as compared to the traditional classification approach. [ABSTRACT FROM AUTHOR]

Wide models such as generalized linear models and factorization-based models have been extensively used in various predictive applications, e.g., recommendation, CTR prediction, and image recognition. Due to the memory bounded property of the models, the performance improvement on CPU is reaching the limitation. GPU is known to have many computation units and high memory bandwidth, and becomes a promising platform for training machine learning models. However, the GPU training for the wide models is far from optimal due to the sparsity and irregularity in wide models. The existing GPU-based wide models are even slower than the ones using CPU. The classical training schema of the wide models does not optimized for the GPU architecture, which suffers from large amount of random memory accesses and redundant read/write of intermediate values. In this paper, we propose an efficient GPU-training framework for the large-scale wide models, named cuWide. To fully benefit from the memory hierarchy of GPU, cuWide applies a new flow-based schema for training, which leverages the spatial and temporal locality of wide models to drastically reduce the amount of communication with GPU global memory. To do so, we adopt a bigraph computation model to efficiently realize the flow-based schema and exploit three flexible interfaces for programming. Further, we use the 2D partition of mini-batch (in sample and feature dimensions) with proposed graph abstraction to optimize GPU memory access for sparse data, and apply several spatial-temporal caching mechanisms (importance-based model caching and cross-stage accumulation caching mechanisms) to achieve a high performance kernel. To efficiently implement cuWide, we also propose several GPU-oriented optimizations, including feature-oriented data layout to enhance the data locality, replication mechanism to reduce update conflicts in shared memory, and multi-stream scheduling to overlap data transferring and kernel computing. We show that cuWide can be up to more than 20× faster than the state-of-the-art GPU solutions and multi-core CPU solutions. [ABSTRACT FROM AUTHOR]

Human adaptability relies crucially on learning and merging knowledge from both supervised and unsupervised tasks: the parents point out few important concepts, but then the children fill in the gaps on their own. This is particularly effective, because supervised learning can never be exhaustive and thus learning autonomously allows to discover invariances and regularities that help to generalize. In this paper we propose to apply a similar approach to the problem of object recognition across domains: our model learns the semantic labels in a supervised fashion, and broadens its understanding of the data by learning from self-supervised signals on the same images. This secondary task helps the network to focus on object shapes, learning concepts like spatial orientation and part correlation, while acting as a regularizer for the classification task over multiple visual domains. Extensive experiments confirm our intuition and show that our multi-task method, combining supervised and self-supervised knowledge, provides competitive results with respect to more complex domain generalization and adaptation solutions. It also proves its potential in the novel and challenging predictive and partial domain adaptation scenarios. [ABSTRACT FROM AUTHOR]

The deep learning-based automatic recognition of the scanning or exposing region in medical imaging automation is a promising new technique, which can decrease the heavy workload of the radiographers, optimize imaging workflow and improve image quality. However, there is little related research and practice in X-ray imaging. In this paper, we focus on two key problems in X-ray imaging automation: automatic recognition of the exposure moment and the exposure region. Consequently, we propose an automatic video analysis framework based on the hybrid model, approaching real-time performance. The framework consists of three interdependent components: Body Structure Detection, Motion State Tracing, and Body Modeling. Body Structure Detection disassembles the patient to obtain the corresponding body keypoints and body Bboxes. Combining and analyzing the two different types of body structure representations is to obtain rich spatial location information about the patient body structure. Motion State Tracing focuses on the motion state analysis of the exposure region to recognize the appropriate exposure moment. The exposure region is calculated by Body Modeling when the exposure moment appears. A large-scale dataset for X-ray examination scene is built to validate the performance of the proposed method. Extensive experiments demonstrate the superiority of the proposed method in automatically recognizing the exposure moment and exposure region. This paradigm provides the first method that can enable automatically and accurately recognize the exposure region in X-ray imaging without the help of the radiographer. [ABSTRACT FROM AUTHOR]

Quantization is a promising technique to reduce the computation and storage costs of DNNs. Low-bit ($\leq8$ bits) precision training remains an open problem due to the difficulty of gradient quantization. In this paper, we find two long-standing misunderstandings of the bias of gradient quantization noise. First, the large bias of gradient quantization noise, instead of the variance, is the key factor of training accuracy loss. Second, the widely used stochastic rounding cannot solve the training crash problem caused by the gradient quantization bias in practice. Moreover, we find that the asymmetric distribution of gradients causes a large bias of gradient quantization noise. Based on our findings, we propose a novel adaptive piecewise quantization method to effectively limit the bias of gradient quantization noise. Accordingly, we propose a new data format, Piecewise Fixed Point (PWF), to present data after quantization. We apply our method to different applications including image classification, machine translation, optical character recognition, and text classification. We achieve approximately $1.9\sim 3.5\times $ speedup compared with full precision training with an accuracy loss of less than 0.5%. To the best of our knowledge, this is the first work to quantize gradients of all layers to 8 bits in both large-scale CNN and RNN training with negligible accuracy loss. [ABSTRACT FROM AUTHOR]

In leveraging manifold learning in domain adaptation (DA), graph embedding-based DA methods have shown their effectiveness in preserving data manifold through the Laplace graph. However, current graph embedding DA methods suffer from two issues: 1). they are only concerned with preservation of the underlying data structures in the embedding and ignore sub-domain adaptation, which requires taking into account intra-class similarity and inter-class dissimilarity, thereby leading to negative transfer; 2). manifold learning is proposed across different feature/label spaces separately, thereby hindering unified comprehensive manifold learning. In this paper, starting from our previous DGA-DA, we propose a novel DA method, namely ${A}$ ttention ${R}$ egularized Laplace ${G}$ raph-based ${D}$ omain ${A}$ daptation (ARG-DA), to remedy the aforementioned issues. Specifically, by weighting the importance across different sub-domain adaptation tasks, we propose the ${A}$ ttention ${R}$ egularized Laplace Graph for class aware DA, thereby generating the attention regularized DA. Furthermore, using a specifically designed FEEL strategy, our approach dynamically unifies alignment of the manifold structures across different feature/label spaces, thus leading to comprehensive manifold learning. Comprehensive experiments are carried out to verify the effectiveness of the proposed DA method, which consistently outperforms the state of the art DA methods on 7 standard DA benchmarks, i.e., 37 cross-domain image classification tasks including object, face, and digit images. An in-depth analysis of the proposed DA method is also discussed, including sensitivity, convergence, and robustness. [ABSTRACT FROM AUTHOR]

Contrastive self-supervised learning (CSSL) has achieved promising results in extracting visual features from unlabeled data. Most of the current CSSL methods are used to learn global image features with low-resolution that are not suitable or efficient for pixel-level tasks. In this paper, we propose a coarse-to-fine CSSL framework based on a novel contrasting strategy to address this problem. It consists of two stages, one for encoder pre-training to learn global features and the other for decoder pre-training to derive local features. Firstly, the novel contrasting strategy takes advantage of the spatial structure and semantic meaning of different regions and provides more cues to learn than that relying only on data augmentation. Specifically, a positive pair is built from two nearby patches sampled along the direction of the texture if they fall into the same cluster. A negative pair is generated from different clusters. When the novel contrasting strategy is applied to the coarse-to-fine CSSL framework, global and local features are learned successively by forcing the positive pair close to each other and the negative pair apart in an embedding space. Secondly, a discriminant constraint is incorporated into the per-pixel classification model to maximize the inter-class distance. It makes the classification model more competent at distinguishing between different categories that have similar appearance. Finally, the proposed method is validated on four SAR images for land-cover classification with limited labeled data and substantially improves the experimental results. The effectiveness of the proposed method is demonstrated in pixel-level tasks after comparison with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In the field of image set classification, most existing works focus on exploiting effective latent discriminative features. However, it remains a research gap to efficiently handle this problem. In this paper, benefiting from the superiority of hashing in terms of its computational complexity and memory costs, we present a novel Discrete Metric Learning (DML) approach based on the Riemannian manifold for fast image set classification. The proposed DML jointly learns a metric in the induced space and a compact Hamming space, where efficient classification is carried out. Specifically, each image set is modeled as a point on Riemannian manifold after which the proposed DML minimizes the Hamming distance between similar Riemannian pairs and maximizes the Hamming distance between dissimilar ones by introducing a discriminative Mahalanobis-like matrix. To overcome the shortcoming of DML that relies on the vectorization of Riemannian representations, we further develop Bilinear Discrete Metric Learning (BDML) to directly manipulate the original Riemannian representations and explore the natural matrix structure for high-dimensional data. Different from conventional Riemannian metric learning methods, which require complicated Riemannian optimizations (e.g., Riemannian conjugate gradient), both DML and BDML can be efficiently optimized by computing the geodesic mean between the similarity matrix and inverse of the dissimilarity matrix. Extensive experiments conducted on different visual recognition tasks (face recognition, object recognition, and action recognition) demonstrate that the proposed methods achieve competitive performance in terms of accuracy and efficiency. [ABSTRACT FROM AUTHOR]

Reading arbitrary-shaped text in an end-to-end fashion has received particularly growing interested in computer vision. In this paper, we study the problem of scene text spotting, which aims to detect and recognize text from cluttered images simultaneously and propose an end-to-end trainable neural network named Boundary TextSpotter. Different from existing methods that describe the shape of text instance with bounding box or shape mask, Boundary TextSpotter formulates it as a set of boundary points. Besides, the representation of such boundary points provides the order of reading text. Benefiting from the representation on both detection and recognition, Boundary TextSpotter can easily deal with the text of arbitrary shapes. Further, to efficiently detect the boundary points of the text, a single-stage text detector is proposed, which can almost perform at a real-time speed. Experiments on three challenging datasets, including ICDAR2015, Total-Text and CTW1500 demonstrate that the proposed method achieves state-of-the-art or competitive results, meanwhile significantly improving the inference speed. [ABSTRACT FROM AUTHOR]

Recently, embedding and metric-based few-shot learning (FSL) has been introduced into hyperspectral image classification (HSIC) and achieved impressive progress. To further enhance the performance with few labeled samples, we in this paper propose a novel FSL framework for HSIC with a class-covariance metric (CMFSL). Overall, the CMFSL learns global class representations for each training episode by interactively using training samples from the base and novel classes, and a synthesis strategy is employed on the novel classes to avoid overfitting. During the meta-training and meta-testing, the class labels are determined directly using the Mahalanobis distance measurement rather than an extra classifier. Benefiting from the task-adapted class-covariance estimations, the CMFSL can construct more flexible decision boundaries than the commonly used Euclidean metric. Additionally, a lightweight cross-scale convolutional network (LXConvNet) consisting of 3D and 2D convolutions is designed to thoroughly exploit the spectral-spatial information in the high-frequency and low-frequency scales with low computational complexity. Furthermore, we devise a spectral-prior-based refinement module (SPRM) in the initial stage of feature extraction, which cannot only force the network to emphasize the most informative bands while suppressing the useless ones, but also alleviate the effects of the domain shift between the base and novel categories to learn a collaborative embedding mapping. Extensive experiment results on four benchmark data sets demonstrate that the proposed CMFSL can outperform the state-of-the-art methods with few-shot annotated samples. [ABSTRACT FROM AUTHOR]

Metric-based methods achieve promising performance on few-shot classification by learning clusters on support samples and generating shared decision boundaries for query samples. However, existing methods ignore the inaccurate class center approximation introduced by the limited number of support samples, which consequently leads to biased inference. Therefore, in this paper, we propose to reduce the approximation error by class center calibration. Specifically, we introduce the so-called Pair-wise Similarity Module (PSM) to generate calibrated class centers adapted to the query sample by capturing the semantic correlations between the support and the query samples, as well as enhancing the discriminative regions on support representation. It is worth noting that the proposed PSM is a simple plug-and-play module and can be inserted into most metric-based few-shot learning models. Through extensive experiments in metric-based models, we demonstrate that the module significantly improves the performance of conventional few-shot classification methods on four few-shot image classification benchmark datasets. Codes are available at: https://github.com/PRIS-CV/Pair-wise-Similarity-module. [ABSTRACT FROM AUTHOR]

Unsupervised domain adaptation (UDA) enables a learning machine to adapt from a labeled source domain to an unlabeled target domain under the distribution shift. Thanks to the strong representation ability of deep neural networks, recent remarkable achievements in UDA resort to learning domain-invariant features. Intuitively, the goal is that a good feature representation and the hypothesis learned from the source domain can generalize well to the target domain. However, the learning processes of domain-invariant features and source hypotheses inevitably involve domain-specific information that would degrade the generalizability of UDA models on the target domain. The lottery ticket hypothesis proves that only partial parameters are essential for generalization. Motivated by it, we find in this paper that only partial parameters are essential for learning domain-invariant information. Such parameters are termed transferable parameters that can generalize well in UDA. In contrast, the rest parameters tend to fit domain-specific details and often cause the failure of generalization, which are termed untransferable parameters. Driven by this insight, we propose Transferable Parameter Learning (TransPar) to reduce the side effect of domain-specific information in the learning process and thus enhance the memorization of domain-invariant information. Specifically, according to the distribution discrepancy degree, we divide all parameters into transferable and untransferable ones in each training iteration. We then perform separate update rules for the two types of parameters. Extensive experiments on image classification and regression tasks (keypoint detection) show that TransPar outperforms prior arts by non-trivial margins. Moreover, experiments demonstrate that TransPar can be integrated into the most popular deep UDA networks and be easily extended to handle any data distribution shift scenarios. [ABSTRACT FROM AUTHOR]

Adversarial training (AT) has been demonstrated to be effective in improving model robustness by leveraging adversarial examples for training. However, most AT methods are in face of expensive time and computational cost for calculating gradients at multiple steps in generating adversarial examples. To boost training efficiency, fast gradient sign method (FGSM) is adopted in fast AT methods by calculating gradient only once. Unfortunately, the robustness is far from satisfactory. One reason may arise from the initialization fashion. Existing fast AT generally uses a random sample-agnostic initialization, which facilitates the efficiency yet hinders a further robustness improvement. Up to now, the initialization in fast AT is still not extensively explored. In this paper, focusing on image classification, we boost fast AT with a sample-dependent adversarial initialization, i.e., an output from a generative network conditioned on a benign image and its gradient information from the target network. As the generative network and the target network are optimized jointly in the training phase, the former can adaptively generate an effective initialization with respect to the latter, which motivates gradually improved robustness. Experimental evaluations on four benchmark databases demonstrate the superiority of our proposed method over state-of-the-art fast AT methods, as well as comparable robustness to advanced multi-step AT methods. The code is released at https://github.com//jiaxiaojunQAQ//FGSM-SDI. [ABSTRACT FROM AUTHOR]

Zero-shot recognition has been a hot topic in recent years. Since no direct supervision is available, researchers use semantic information as the bridge instead. However, most zero-shot recognition methods jointly model images on the class level without considering the distinctive character of each image. To solve this problem, in this paper, we propose a novel exemplar-based, semantic guided zero-shot recognition method (EBSG). Both visual and semantic information of each image is used. We train visual sub-model to separate each image from the other images of different classes. We also train semantic sub-model to separate this image from the other images described with different semantics. We concatenate the outputs of visual and semantic sub-models to represent images. Image classification model is then learned by measuring visual similarity and semantic consistency of both source and target images. We conduct zero-shot recognition experiments on four widely used datasets. Experimental results show the effectiveness of the proposed EBSG method. [ABSTRACT FROM AUTHOR]

Model fine-tuning is a widely used transfer learning approach in person Re-identification (ReID) applications, which fine-tuning a pre-trained feature extraction model into the target scenario instead of training a model from scratch. It is challenging due to the significant variations inside the target scenario, e.g., different camera viewpoint, illumination changes, and occlusion. These variations result in a gap between each mini-batch’s distribution and the whole dataset’s distribution when using mini-batch training. In this paper, we study model fine-tuning from the perspective of the aggregation and utilization of the dataset’s global information when using mini-batch training. Specifically, we introduce a novel network structure called Batch-related Convolutional Cell (BConv-Cell), which progressively collects the dataset’s global information into a latent state and uses it to rectify the extracted feature. Based on BConv-Cells, we further proposed the Progressive Transfer Learning (PTL) method to facilitate the model fine-tuning process by jointly optimizing BConv-Cells and the pre-trained ReID model. Empirical experiments show that our proposal can greatly improve the ReID model’s performance on MSMT17, Market-1501, CUHK03, and DukeMTMC-reID datasets. Moreover, we extend our proposal to the general image classification task. The experiments in several image classification benchmark datasets demonstrate that our proposal can significantly improve baseline models’ performance. The code has been released at https://github.com/ZJULearning/PTL [ABSTRACT FROM AUTHOR]

Video quality assessment (VQA) task is an ongoing small sample learning problem due to the costly effort required for manual annotation. Since existing VQA datasets are of limited scale, prior research tries to leverage models pre-trained on ImageNet to mitigate this kind of shortage. Nonetheless, these well-trained models targeting on image classification task can be sub-optimal when applied on VQA data from a significantly different domain. In this paper, we make the first attempt to perform self-supervised pre-training for VQA task built upon contrastive learning method, targeting at exploiting the plentiful unlabeled video data to learn feature representation in a simple-yet-effective way. Specifically, we implement this idea by first generating distorted video samples with diverse distortion characteristics and visual contents based on the proposed distortion augmentation strategy. Afterwards, we conduct contrastive learning to capture quality-aware information by maximizing the agreement on feature representations of future frames and their corresponding predictions in the embedding space. In addition, we further introduce distortion prediction task as an additional learning objective to push the model towards discriminating different distortion categories of the input video. Solving these prediction tasks jointly with the contrastive learning not only provides stronger surrogate supervision signals, but also learns the shared knowledge among the prediction tasks. Extensive experiments demonstrate that our approach sets a new state-of-the-art in self-supervised learning for VQA task. Our results also underscore that the learned pre-trained model can significantly benefit the existing learning based VQA models. Source code is available at https://github.com/cpf0079/CSPT. [ABSTRACT FROM AUTHOR]