Search

After having introduced a comprehensive general solution framework for few-shot learning (FSL) classification problems, we provide details of the data augmentation schemes and the learning techniques used, which are critical in minimizing the risk of overfitting, due to the fact that only a few labeled samples are available to fine-tune a robust encoder. Once such a general solution framework is in place, we obtain a convolutional neural network (CNN)-based encoder, which is evolved along a number of architectural brick axes from a baseline ResNet-50 model, while in each step of this evolution process, the adoption or not of the temporary incremental model will be assessed through this general solution framework. This is called a ConvNeXt encoder. We then insert an efficient channel attention (ECA) module to more effectively process the features extracted in the depthwise convolutional layer in the ConvNeXt encoder, with a minimal increase in the number of parameters used, and obtain a novel encoder: ConvNeXt-ECA. We apply the ConvNeXt-ECA encoder in the general solution framework on 5 popular FSL benchmark datasets and found that our ConvNeXt-ECA encoder achieves the best average few-shot classification accuracy when compared with other state-of-the-art results.

For solving the facial expression recognition (FER) problem, we introduce a novel feature extractor called the coordinate-based neighborhood attention mechanism (CNAM), which uses the coordinate attention (CA) method to capture the directional relationships in separate horizontal and vertical directions, the input features from a preprocessing unit, and then passes this to two residual blocks, one consisting of the neighborhood attention (NA) mechanism, which captures the local interaction of features within the neighborhood of a feature vector, while the other one contains a channel attention implemented by a multilayer perceptron (MLP). We apply the feature extractor, the CNAM module, to four FER benchmark datasets, namely, RAF-DB, AffectNet(7cls), AffectNet(8cls), and CK+, and through qualitative and quantitative analysis techniques, we conclude that the insertion of the CNAM module could decrease the intra-cluster distances and increase the inter-cluster distances among the high-dimensional feature vectors. The CNAM compares well with other state-of-the-art (SOTA) methods, being the best-performing method for the AffectNet(7cls) and CK+ datasets, while for the RAF-DB and AffectNet(8cls) datasets, its performance is among the top-performing SOTA methods.

In studying the joint object detection and classification problem for facial expression recognition (FER) deploying the YOLOX framework, we introduce a novel feature extractor, called neighborhood coordinate attention Mamba (NCAMamba) to substitute for the original feature extractor in the Feature Pyramid Network (FPN). NCAMamba combines the background information reduction capabilities of Mamba, the local neighborhood relationship understanding of neighborhood attention, and the directional relationship understanding of coordinate attention. The resulting FER-YOLO-NCAMamba model, when applied to two unaligned FER benchmark datasets, RAF-DB and SFEW, obtains significantly improved mean average precision (mAP) scores when compared with those obtained by other state-of-the-art methods. Moreover, in ablation studies, it is found that the NCA module is relatively more important than the Visual State Space (VSS), a version of using Mamba for image processing, and in visualization studies using the grad-CAM method, it reveals that regions around the nose tip are critical to recognizing the expression; if it is too large, it may lead to erroneous prediction, while a small focused region would lead to correct recognition; this may explain why FER of unaligned faces is such a challenging problem.

Segmentation of human actions is a major research problem in video understanding. A number of existing approaches demonstrate that performing action segmentation before action recognition results in better recognition performance. In this paper, we address the problem of action segmentation in an online manner. We first extend the clustering-based image segmentation approach into a temporal one, where hierarchical supervoxel levels for action segmentation are generated accordingly. We then propose a streaming approach to flatten the hierarchical levels into one based on uniform entropy slice, in order to preserve important information in the video. The flattened level contains the silhouette of a human with the structure of body parts labeled in different labels. We then combine the human structure information and the original video frames to “strengthen” the action in a video, which paves the way for accurate action recognition. The experimental results show that our online approach achieves satisfactory performance regarding action segmentation or recognition on various publicly available data sets, including the DAVIS data set, the UCF Sports data set, and the KTH data set.