Search

Abstract Person Re‐identification (Re‐ID) is a task of matching target pedestrians under cross‐camera surveillance. Learning discriminative feature representations is the main issue for person Re‐ID. A few recent methods introduce text descriptions as auxiliary information to enhance feature representations, as it offers richer semantic information and perspective consistency. However, these works usually process text and images separately, which leads to the absence of cross‐modal interactions. In this article, a Dual‐modal Graph Attention Interaction Network (Dual‐GAIN) is proposed to integrate visual features and textual features into a heterogeneous graph to model the relationship between them, simultaneously. The proposed Dual‐GAIN mainly consists of two components: a dual‐stream feature extractor and a Graph Attention Interaction Network (GAIN). Specifically, the two‐stream feature extractor is utilised to extract visual features and textual features respectively. Then, visual local features and textual features are treated as nodes to construct a multi‐modal graph. Cosine similarity constrained attention weights are introduced in GAIN, which is designed for cross‐modal interaction and feature fusion on this heterogeneous multi‐modal graph. Experiments on public large‐scale datasets, that is, Market‐1501, CUHK03 labelled, and CUHK03 detected, demonstrate our method achieves the state‐of‐the‐art performance.

Abstract Although most state‐of‐the‐art action recognition models have adopted a two‐stream 3D convolutional structure as a backbone network, few works have studied the impact of loss functions on action recognition models. In addition, sparsity is used as a key prior knowledge in many fields. However, as far as is known, no one has studied the influence of the sparsity of network output on the output of deep learning‐based action recognition models. Therefore, this paper proposes a novel two‐stream inflated 3D ConvNet based on the sparse regularization (SRI3D) model for action recognition. In order to allow the network to learn the sparsity of output, the ℓ1 norm is embedded in the loss function in regularization form in a plug‐and‐play manner. It can make the classification result after the fusion of the two‐stream network only be the category with the highest confidence in one of the streams and not the other cases. The proposed loss function based on sparse regularization makes the output vector of the neural network as sparse as possible so that the classification results will not be ambiguous. Experimental results show that compared with other state‐of‐the‐art models, this SRI3D has a competitive advantage on Kinetics‐400, Something‐Something V2, UCF‐101 and HMDB‐51.

Abstract In this article, an effective approach is proposed to recognise the 2D+3D facial expression automatically based on orthogonal Tucker decomposition using factor priors (OTDFPFER). As a powerful technique, Tucker decomposition on the basis of the low rank approximation is often used to extract the useful information from the constructed 4D tensor composed of 3D face scans and 2D images aiming to maintain correlations and their structural information. Finding a set of projected factor matrices is our ultimate goal. During the 4D tensor modelling process, high similarities among samples will emerge because of the information missed partially. Based on the tensor orthogonal Tucker decomposition, the involved core tensor with the structured sparsity, and a graph regularisation term via the graph Laplacian matrix together with the fourth factor matrix are employed for better characterisation of the generated similarities and for keeping the consistency of low dimensional space. To recover the missing information, a framework for tensor completion (TC) will be embedded naturally. Finally, an alternating direction method coupled with the majorisation‐minimisation scheme is designed to solve the resulting tensor completion problem. The numerical experiments are conducted on the Bosphorus and the BU‐3DFE databases with promising recognition accuracies.

Abstract Part misalignment of the human body caused by complex variations in viewpoint and pose poses a fundamental challenge to person re‐identification. This letter examines Res2Net as the backbone network to extract multi‐scale appearance features. At the same time, it uses the human parsing model to extract part features, which can be used as an attention stream to guide part features re‐calibration from the spatial dimension. Additionally, in order to ensure the diversity of features, SAG‐PAN effectively integrates the global appearance features of person image with part fine‐grained features. The experimental results on the Market‐1501, DukeMTMC‐reID and CUHK03 datasets show that the proposed SAG‐PAN achieved superior performance against the existing state‐of‐the‐art methods.

Scene graph generation is to recognise objects and their semantic relationships in an image and can help computers understand visual scene. To improve relationship prediction, geometry information is essential and usually incorporated into relationship features. Existing methods use coordinates of objects to encode their spatial layout. However, in this way, they neglect the context of objects. In this study, to take full use of spatial knowledge efficiently, the authors propose a novel subgraph and object context‐masked network (SOCNet) consisting of spatial mask relation inference (SMRI) and hierarchical message passing (HMP) modules to address the scene graph generation task. In particular, to take advantage of spatial knowledge, SMRI masks partial context of object features depending on their spatial layout of objects and corresponding subgraph to facilitate their relationship recognition. To refine the features of objects and subgraphs, they also propose HMP that passes highly correlated messages from both microcosmic and macroscopic aspects through a triple‐path structure including subgraph–subgraph, object–object, and subgraph–object paths. Finally, statistical co‐occurrence probability is used to regularise relationship prediction. SOCNet integrates HMP and SMRI into a unified network, and comprehensive experiments on visual relationship detection and visual genome datasets indicate that SOCNet outperforms several state‐of‐the‐art methods on two common tasks.

Capsule neural network is a new and popular technique in deep learning. However, the traditional capsule neural network does not extract features sufficiently before the dynamic routing between capsules. In this study, one double enhanced capsule neural network (E2‐Capsnet) that uses AU‐aware attention for facial expression recognition (FER) is proposed. The E2‐Capsnet takes advantage of dynamic routing between capsules and has two enhancement modules which are beneficial to FER. The first enhancement module is the convolutional neural network with AU‐aware attention, which can focus on the active areas of the expression. The second enhancement module is the capsule neural network with multiple convolutional layers, which enhances the ability of the feature representation. Finally, the squashing function is used to classify the facial expression. The authors demonstrate the effectiveness of E2‐Capsnet on the two public benchmark datasets, RAF‐DB and EmotioNet. The experimental results show that their E2‐Capsnet is superior to the state‐of‐the‐art methods. The code is available at https://github.com/ShanCao18/E2‐Capsnet.

The modeling, computational complexity, and accuracy of spatio-temporal models are the three major foci in the field of video action recognition. The traditional 2D convolution has low computational complexity, but it cannot capture the temporal relationships. Although the 3D convolution can obtain good performance, it is with both high computational complexity and a large number of parameters. In this paper, we propose a plug-and-play Spatio-Temporal Shift Module (STSM), which is a both effective and high-performance module. STSM can be easily inserted into other networks to increase or enhance the ability of the network to learn spatio-temporal features, effectively improving performance without increasing the number of parameters and computational complexity. In particular, when 2D CNNs and STSM are integrated, the new network may learn spatio-temporal features and outperform networks based on 3D convolutions. We revisit the shift operation from the perspective of matrix algebra, i.e., the spatio-temporal shift operation is a convolution operation with a sparse convolution kernel. Furthermore, we extensively evaluate the proposed module on Kinetics-400 and Something-Something V2 datasets. The experimental results show the effectiveness of the proposed STSM, and the proposed action recognition networks may also achieve state-of-the-art results on the two action recognition benchmarks.

Spaced repetition, key for long-term memory retention through optimized review schedules based on predicted memory retention, is increasingly vital for effective language learning. Traditional methods, however, often fail to account for individual cognitive variations and material difficulty, resulting in a lack of high adaptability and effectiveness. To address this, our study introduces the Multidimensional Cognition Regression (MCR) model. MCR incorporates the Difficulty Engineering (DE) module, which integrates both objective and subjective factors to evaluate the intricacy of the content. Moreover, MCR further leverages a variety of user memory and cognitive characteristics, combined with psychological insights and machine learning techniques, to predict the memory ``half-life" of material. This approach transcends methods like Half-Life Regression proven effective on Duolingo, reducing prediction errors demonstrated by lower Mean Absolute Error. Based on the predictive modeling of memory's halflife and corresponding biological memory patterns, we opt to schedule reviews at the juncture when the memory decays to its halflife point. Empirical validation in real-world settings showed enhanced retention efficiency.