Search

In recent years, object re-identification (ReID) performance based on deep convolutional networks has reached a very high level and has seen outstanding progress. The existing methods merely focus on the robustness of features and classification accuracy but ignore the relationship among different features (i.e., the relationship between gallery–gallery pairs or probe–gallery pairs). In particular, a probe located at the decision boundary is the key to suppressing object ReID performance. We consider this probe as a hard sample. Recent studies have shown that Graph Convolutional Networks (GCN) significantly improve the relationship among features. However, applying the GCN to object ReID is still an open question. This paper proposes two learnable GCN modules: the Feature Aggregation Graph Convolutional Network (FA-GCN) and the Evaluation Connectivity Graph Convolutional Network (EC-GCN). Specifically, the pre-work selects an arbitrary feature extraction network to extract features in the object ReID dataset. Given a probe, FA-GCN aggregates neighboring nodes through the affinity graph of the gallery set. Afterward, EC-GCN uses a random probability gallery sampler to construct subgraphs for evaluating the connectivity of probe–gallery pairs. Finally, we jointly aggregate the node features and connectivity ratios as a new distance matrix. Experimental results on two person ReID datasets (Market-1501 and DukeMTMC-ReID) and one vehicle ReID dataset (VeRi-776) show that the proposed method achieves state-of-the-art performance. • We propose a novel framework containing two learnable GCN components. • Our framework can be inserted into arbitrary deep neural networks. • We formulate a new sampling strategy RPGS. • We improve the state-of-the-art methods on both person and vehicle ReID datasets. [ABSTRACT FROM AUTHOR]

Video captioning aims to briefly describe the content of a video in accurate and fluent natural language, which is a hot research topic in multimedia processing. As a bridge between video and natural language, video captioning is a challenging task that requires a deep understanding of video content and effective utilization of diverse video multimodal information. Existing video captioning methods usually ignore the relative importance between different frames when aggregating frame-level video features and neglect the global semantic correlations between videos and texts in learning visual representations, resulting in the learned representations less effective. To address these problems, we propose a novel framework, namely Global Semantic Enhancement Network (GSEN) to generate high-quality captions for videos. Specifically, a feature aggregation module based on a lightweight attention mechanism is designed to aggregate frame-level video features, which highlights features of informative frames in video representations. In addition, a global semantic enhancement module is proposed to enhance semantic correlations for video and language representations in order to generate semantically more accurate captions. Extensive qualitative and quantitative experiments on two public benchmark datasets MSVD and MSR-VTT demonstrate that the proposed GSEN can achieve superior performance than state-of-the-art methods. • A video captioning framework called global semantic enhancement network is proposed. • It highlights features of informative frames in aggregated video representations. • It enhances semantic correlations between video and language representations. • Superiority of the method over state-of-the-art was shown on two benchmarks. [ABSTRACT FROM AUTHOR]

• Propose a multi-scale feature learning network for hand gesture recognition. • Propose a ESA module to fuse more semantics for hand gesture feature learning. • Use the mask learning branch to guide the network to focus on hand-specific regions. • The proposed method achieves promising performance on challenging HGR databases. Hand gesture recognition from images is a longstanding computer vision task that can be used to build a potential bridge for human-computer interaction and sign language translation. For number of methods proposed for hand gesture recognition (HGR); however, difficult scenarios such as different scales of hand gestures and complex backgrounds exist, making them less effective. In this paper, we propose an end-to-end multiscale feature learning network for HGR, which consists of a CNN-based backbone network, a feature aggregation pyramid network (FAPN) embedded with a two-stage expansion-squeeze-aggregation (ESA) module, and three task-specific prediction branches. First, the backbone network extracts multiscale features from the original hand gesture images. Furthermore, the FAPN embedded with two-stage ESA extensively exploits multiscale feature information and learns hand gesture-specific features at different scales. Then, the mask loss guides the network to locate hand-specific regions during the training stage, and finally, the classification and regression branches output the category and location of a hand gesture during the model training and prediction. The experimental results on two publicly available datasets show that the proposed method outperforms most state-of-the-art HGR methods. [ABSTRACT FROM AUTHOR]

• We propose a human identification method based on dental impression images. First, we extract a geometric feature called the calibrated offset distance to describe the dental arch. The dental arch is a distinctive feature of individual teeth. To address the challenges in describing the dental arch, this paper introduces a novel calibrated offset distance that captures the characteristics of the arc, changes in radian, and distances between arc breakpoints simultaneously. • We propose an adaptive threshold strategy for the tooth print detection model based on the YOLOv8 network. We design a gradient training strategy to achieve a robust detection model. Building upon its high positioning accuracy, we propose an adaptive threshold strategy to further enhance the detection accuracy. • We propose a multi-scale and multi-dimensional tooth print aggregation feature. In order to enhance the ability to recognize individuals with similar teeth structures, we further integrate SIFT features to provide a detailed description based on geometric features. This aggregation feature is composed to improve the performance of human identification. In large-scale natural disasters and special criminal cases, surface features of bodies, such as faces and fingerprints, are easily destroyed. Teeth possess strong high-temperature resistance, corrosion resistance, and high hardness, which can compensate for the shortcomings of the aforementioned situations. This paper proposes an identification method based on the aggregated features of multi-scale dental impression images. Firstly, a method exploiting the adaptive object detection method based on YOLOv8 is proposed to segment toothprints. Next, a novel geometric feature named calibrated offset distance is extracted, combined with the SIFT feature method, to extract multi-scale and multi-dimensional features from the global toothprint, local toothprints, and single-tooth prints. Finally, all features are aggregated to enhance the descriptive ability and robustness. Experimental results indicate that the method proposed in this paper demonstrates good identification performance. [ABSTRACT FROM AUTHOR]

Highlights • We formulate the convolutional feature aggregation as an SPD matrix non-linear generation and transformation problem on the Riemannian manifold. • Our model can be implemented under an end-to-end learning framework. • We design three novel layers to implement the convolutional feature aggregation. • We exploit the faster matrix operation to speed up the computation, and the component decomposition and retraction of the orthogonal Stiefel manifold to carry out the backpropagation. • Our approach notably outperforms the state-of-the-art methods. Abstract Recent studies have shown that aggregating convolutional features of a Convolutional Neural Network (CNN) can obtain impressive performance for a variety of computer vision tasks. The Symmetric Positive Definite (SPD) matrix becomes a powerful tool due to its remarkable ability to learn an appropriate statistic representation to characterize the underlying structure of visual features. In this paper, we propose a method of aggregating deep convolutional features into a robust representation through the SPD generation and the SPD transformation under an end-to-end deep network. To this end, several new layers are introduced in our method, including a nonlinear kernel generation layer, a matrix transformation layer, and a vector transformation layer. The nonlinear kernel generation layer is employed to aggregate convolutional features into a kernel matrix which is guaranteed to be an SPD matrix. The matrix transformation layer is designed to project the original SPD representation to a more compact and discriminative SPD manifold. The vectorization and normalization operations are performed in the vector transformation layer to take the upper triangle elements of the SPD representation and carry out the power normalization and l 2 normalization to reduce the redundancy and accelerate the convergence. The SPD matrix in our network can be considered as a mid-level representation bridging convolutional features and high-level semantic features. Results of extensive experiments show that our method notably outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

• We found the local high correlation mapping relationship between hazeclear image pairs. • A lightweight and fast network is proposed for real-time image dehazing. • A novel multi-scale feature extraction subnetwork is embedded into the dehazing model. • Our method surpasses previous state-of-the-art lightweight dehazing methods. • The size of our proposed network model is only 53.70 KB. • The average runtime for a 620 × 460 image on the CPU is as low as 0.12 seconds. [Display omitted] Haze causes visual degradation and obscures image information, which gravely affects the reliability of computer vision tasks in real-time systems. Leveraging an enormous number of learning parameters as the restoration costs, learning-based methods have gained significant success, but they are runtime intensive or memory inefficient. In this paper, we propose a local multi-scale feature aggregation network, called LMFA-Net, which has a lightweight model structure and can be used for real-time dehazing. By learning the local mapping relationship between the clean value of a haze image at a certain point and its surrounding local region, LMFA-Net can directly restore the final haze-free image. In particular, we adopt a novel multi-scale feature extraction sub-network (M-Net) to extract features from different scales. As a lightweight network, LMFA-Net can achieve fast and efficient dehazing. Extensive experiments demonstrate that our proposed LMFA-Net surpasses previous state-of-the-art lightweight dehazing methods in both quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

• A new yet effective lightweight shared multiple sub-encoders design for feature sharing and feature reusing across the encoder. • A novel Cascading Context Mining (CCM) module which segments the regions with different receptive fields at different spatial dimensions of the input. • A new Hybrid Path Attention Semantic Aggregation (HPA-SA) module for identifying and localizing all the objects in a complex scene accurately. • The proposed model produced the state-of-the-art result on four datasets among the existing semantic models having 2 million parameters or less. Recent studies show a significant growth in semantic segmentation. However, many semantic segmentation models still have a large number of parameters, making them unsuitable for resource-constrained embedded devices. To address this issue, we propose an efficient Shared Feature Reuse Segmentation (SFRSeg) model containing several novelties: a new yet effective shared-branch multiple sub-encoders design, a context mining module and a semantic aggregating module for better context granularity. In particular, our shared-branch approach improves the entire feature hierarchy by sharing the spatial and context knowledge in both shallow and deep branches. After every shared point in each sub-encoder, a proposed cascading context mining (CCM) module is deployed to filter out the noisy spatial details from the feature maps and provides a diverse size of receptive fields for capturing the latent context between multi-scale geometric shapes in the scene. To overcome the gradient vanishing issue at the early stage, we reduce the number of layers in the first sub-encoder and employ a unique multiple sub-encoders design which reprocesses the rich global feature maps through multiple sub-encoders for better feature refinement. Later, the rich semantic features generated by the efficient sub-encoders at different levels are fused by the proposed Hybrid Path Attention Semantic Aggregation (HPA-SA) module that effectively reduces the semantic gap between feature maps at different levels and alleviate the well-known boundary degeneration effect. To make it computationally efficient for resource-constrained embedded devices, a series of lightweight methods such as a lightweight encoder, a squeeze-and-excitation design, separable convolution filters, channel reduction (CR) are carefully exploited. With an exceptional performance on Cityscapes (70.6% test mIoU) and CamVid (74.7% test mIoU) data sets, the proposed model is shown to be superior over existing light real-time semantic segmentation models whilst having only 1.6 million parameters. [ABSTRACT FROM AUTHOR]

• A relation extraction module that models implicit relationships by expectation-maximization. • A relation-based convolutional descriptor aggregation method for global descriptors. • A novel pipeline for weakly supervised fine-grained image retrieval. • Extensive experiments show that the proposed method achieves new state-of-the-art. Fine-Grained Image Retrieval (FGIR) is a fundamental yet challenging task that has recently received considerable attention. However, two critical issues remain unresolved. On the one hand, convolutional neural networks (CNNs) trained with image-level labels tend to focus on the most discriminative image patches but overlook the implicit relation among them. On the other hand, existing large models developed for FGIR are computationally expensive and difficult to learn discriminative features. To address these issues without additional object-level annotations or localization sub-networks, we propose a novel unified framework for fine-grained image retrieval. Specifically, a novel Relation-based Convolutional Descriptor Aggregation (RCDA) method for extracting subtle yet discriminative features from fine-grained images is introduced. The RCDA method consists of a local feature generation network and a relation extraction (RE) module that models both explicit information and implicit relations. The explicit information is modeled by computing feature similarities, while the implicit relation is mined via an expectation-maximization algorithm. Moreover, we further leverage the knowledge distillation technique to optimize the parameters of the feature generation network and speed up the fine-tuning procedure by transferring knowledge from a large model to a smaller model. Experimental results on three benchmark datasets (CUB-200-2011, Stanford-Car and FGVC-Aircraft) demonstrate that the proposed method not only achieves a significant improvement over baseline models but also outperforms state-of-the-art methods by a large margin (6.4%, 1.3%, 23.2%, respectively). [ABSTRACT FROM AUTHOR]

• We are among the first ones to investigate the drawbacks of maxpooling in deep point cloud networks. And based on our insightful analysis, a kernel-based feature aggregation framework is innovatively proposed for orderless point analysis via deep networks. The framework enables modeling nonlinear feature relationships which are complementary to max-pooling in a flexible manner. • The order-invariance property of max-pooling can be naturally kept by simply choosing an order-invariant kernel function, e.g., the commonly used RBF kernel or Polynomial kernel. At the same time, the high efficiency of max-pooling is also maintained and only very minor additional computational cost is introduced, which will be demonstrated later through theoretical analysis and experimental study. • The superiority and generability of the proposed method have been extensively verified in both supervised and unsupervised point cloud analysis tasks with various representative backbone networks. Specifically, the supervised tasks include 3D object classification, part segmentation, scene segmentation while the unsupervised task refers to place retrieval. The backbone networks involved in the evaluation include PointNet, DGCNN and PCT. Various effective deep networks have been developed for analysis of 3D point clouds. One key step in these networks is to aggregate the features of orderless points into a compact representation for the cloud. As a typical order-invariant aggregation method, max-pooling has been widely applied. However, while enjoying simplicity and high efficiency, max-pooling does not fully exploit the feature information since it not only ignores the non-maximum elements in each feature dimension but also neglects the interactions between different dimensions. These drawbacks of max-pooling motivate us to explore advanced feature aggregation methods for 3D point cloud analysis. The desired advanced method should be capable of modeling richer information between the point features than max-pooling, and, at the same time, it can readily replace max-pooling without the need to modify other parts of the existing network architecture. To this end, this paper proposes a novel kernel-based feature aggregation framework for 3D point cloud analysis for the first time. The proposed method effectively considers all the elements in each dimension and models the nonlinear interactions between feature dimensions as complementary information to max-pooling. In addition, it is a plug-in module that can be integrated to many common networks as a replacement of max-pooling. Comprehensive experiments are conducted to demonstrate the consistently superior performance and high generality of the proposed method over max-pooling. Specifically, the proposed kernel-based feature aggregation framework consistently improves the max-pooling with three representative backbones of PointNet, DGCNN and PCT across four 3D point cloud based analysis tasks, including supervised 3D object classification, 3D part segmentation, indoor semantic segmentation and one additional unsupervised place retrieval task. Especially, it shows remarkable performance improvement over max-pooling in the unsupervised retrieval task, demonstrating its advantage in forming 3D point cloud representation. [ABSTRACT FROM AUTHOR]

The presence of Antinuclear Autoantibodies (ANA) in human serum is connected with several autoimmune diseases. Indirect Immunofluorescence (IIF) imaging of human epithelial type-2 cells (HEp-2) is the dominant protocol used for the identification of ANA. However, due to limitations in the processes, several attempts have been made to automate the procedure of HEp-2 cell classification. In this work, we focus on the task of HEp-2 cell classification and we propose a novel method for local feature encoding that allows us to generalize the concept of residual encoding in sparse vectors. More specifically, our method hierarchically aggregates the residual of the feature vectors’ sparse representation leading to a Vector of Hierarchically Aggregated Residuals (VHAR). Using SIFT descriptors computed on a dense grid and multiple scales, as well as considering spatial information our method achieves 78.0% classification and 82.3% mean class rate (MCR) on ICPR2012 and ICPR2014 HEp-2 cell contest datasets respectively. [ABSTRACT FROM AUTHOR]

• Infinite-dimensional features are directly aggregated without approximation error. • Our descriptors contain infinite order statistics of input features. • The sigmoid kernel is introduced to construct infinite-dimensional features. • Our method outperforms the state-of-the-art finite-dimensional and infinite-dimensional feature aggregation methods. Aggregating infinite-dimensional features has demonstrated superiority compared with their finite-dimensional counterparts. However, most existing methods approximate infinite-dimensional features with finite-dimensional representations, which inevitably results in approximation error and inferior performance. In this paper, we propose a non-approximate aggregation method that directly aggregates infinite-dimensional features rather than relying on approximation strategies. Specifically, since infinite-dimensional features are infeasible to store, represent and compute explicitly, we introduce a factorized bilinear model to capture pairwise second-order statistics of infinite-dimensional features as a global descriptor. It enables the resulting aggregation formulation to only involve the inner product in an infinite-dimensional space. The factorized bilinear model is calculated by a Sigmoid kernel to generate informative features containing infinite order statistics. Experiments on four visual tasks including the fine-grained, indoor scene, texture, and material classification, demonstrate that our method consistently achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we introduce a new algorithm for clustering and aggregating relational data (CARD). We assume that data is available in a relational form, where we only have information about the degrees to which pairs of objects in the data set are related. Moreover, we assume that the relational information is represented by multiple dissimilarity matrices. These matrices could have been generated using different sensors, features, or mappings. CARD is designed to aggregate pairwise distances from multiple relational matrices, partition the data into clusters, and learn a relevance weight for each matrix in each cluster simultaneously. The cluster dependent relevance weights offer two advantages. First, they guide the clustering process to partition the data set into more meaningful clusters. Second, they can be used in subsequent steps of a learning system to improve its learning behavior. The performance of the proposed algorithm is illustrated by using it to categorize a collection of 500 color images. We represent the pairwise image dissimilarities by six different relational matrices that encode color, texture, and structure information. [Copyright &y& Elsevier]