Search

In industrial manufacturing, how to accurately classify defective products and locate the location of defects has always been a concern. Previous studies mainly measured similarity based on extracting single‐scale features of samples. However, only using the features of a single scale is hard to represent different sizes and types of anomalies. Therefore, the authors propose a set of memory banks of multi‐scale features (MBMF) to enrich feature representation and detect and locate various anomalies. To extract features of different scales, different aggregation functions are designed to produce the feature maps at different granularity. Based on the multi‐scale features of normal samples, the MBMF are constructed. Meanwhile, to better adapt to the feature distribution of the training samples, the authors proposed a new iterative updating method for the memory banks. Testing on the widely used and challenging dataset of MVTec AD, the proposed MBMF achieves competitive image‐level anomaly detection performance (Image‐level Area Under the Receiver Operator Curve (AUROC)) and pixel‐level anomaly segmentation performance (Pixel‐level AUROC). To further evaluate the generalisation of the proposed method, we also implement anomaly detection on the BeanTech AD dataset, a commonly used dataset in the field of anomaly detection, and the Fashion‐MNIST dataset, a widely used dataset in the field of image classification. The experimental results also verify the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Due to the huge cost of manual annotations, the labelled data may not be sufficient to train a dynamic facial expression (DFR) recogniser with good performance. To address this, the authors propose a multi‐modal pre‐training method with a pseudo‐label guidance mechanism to make full use of unlabelled video data for learning informative representations of facial expressions. First, the authors build a pre‐training dataset of videos with aligned vision and audio modals. Second, the vision and audio feature encoders are trained through an instance discrimination strategy and a cross‐modal alignment strategy on the pre‐training data. Third, the vision feature encoder is extended as a dynamic expression recogniser and is fine‐tuned on the labelled training data. Fourth, the fine‐tuned expression recogniser is adopted to predict pseudo‐labels for the pre‐training data, and then start a new pre‐training phase with the guidance of pseudo‐labels to alleviate the long‐tail distribution problem and the instance‐class confliction. Fifth, since the representations learnt with the guidance of pseudo‐labels are more informative, a new fine‐tuning phase is added to further boost the generalisation performance on the DFR recognition task. Experimental results on the Dynamic Facial Expression in the Wild dataset demonstrate the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

Compared with natural images, underwater images are usually degraded with blur, scale variation, colour shift and texture distortion, which bring much challenge for computer vision tasks like object detection. In this case, generic object detection methods usually fail to achieve satisfactory performance. The main reason is considered that the current methods lack sufficient discriminativeness of feature representation for the degraded underwater images. A a novel multi‐scale feature representation and interaction network for underwater object detection is proposed, in which two core modules are elaborately designed to enhance the discriminativeness of feature representation for underwater images. The first is the Context Integration Module, which extracts rich context information from high‐level features and is integrated with the feature pyramid network to enhance the feature representation in a multi‐scale way. The second is the Dual‐refined Attention Interaction Module, which further enhances the feature representation by sufficient interactions between different levels of features both in channel and spatial domains based on attention mechanism. The proposed model is evaluated on four public underwater datasets. The experimental results compared with state‐of‐the‐art object detection methods show that the proposed model has leading performance, which verifies that it is effective for underwater object detection. In addition, object detection experiments on a foggy dataset of Real‐world Task‐driven Testing Set (RTTS) and the natural image dataset of pattern analysis statistical modelling and computational learning, visual object classes (PASCAL VOC) are conducted. The results show that the proposed model can be applied on the degraded dataset of RTTS but fails on PASCAL VOC. [ABSTRACT FROM AUTHOR]

Hand gesture recognition is a challenging topic in the field of computer vision. Multimodal hand gesture recognition based on RGB-D is with higher accuracy than that of only RGB or depth. It is not difficult to conclude that the gain originates from the complementary information existing in the two modalities. However, in reality, multimodal data are not always easy to acquire simultaneously, while unimodal RGB or depth hand gesture data are more general. Therefore, one hand gesture system is expected, in which only unimordal RGB or Depth data is supported for testing, while multimodal RGB-D data is available for training so as to attain the complementary information. Fortunately, a kind of method via multimodal training and unimodal testing has been proposed. However, unimodal feature representation and cross-modality transfer still need to be further improved. To this end, this paper proposes a new 3D-Ghost and Spatial Attention Inflated 3D ConvNet (3DGSAI) to extract high-quality features for each modality. The baseline of 3DGSAI network is Inflated 3D ConvNet (I3D), and two main improvements are proposed. One is 3D-Ghost module, and the other is the spatial attention mechanism. The 3D-Ghost module can extract richer features for hand gesture representation, and the spatial attention mechanism makes the network pay more attention to hand region. This paper also proposes an adaptive parameter for positive knowledge transfer, which ensures that the transfer always occurs from the strong modality network to the weak one. Extensive experiments on SKIG, VIVA, and NVGesture datasets demonstrate that our method is competitive with the state of the art. Especially, the performance of our method reaches 97.87% on the SKIG dataset using only RGB, which is the current best result. [ABSTRACT FROM AUTHOR]

Point cloud semantic segmentation, a crucial research area in the 3D computer vision, lies at the core of many vision and robotics applications. Due to the irregular and disordered of the point cloud, however, the application of convolution on point clouds is challenging. In this article, we propose the "coordinate convolution," which can effectively extract local structural information of the point cloud, to solve the inapplicability of conventional convolution neural network (CNN) structures on the 3D point cloud. The "coordinate convolution" is a projection operation of three planes based on the local coordinate system of each point. Specifically, we project the point cloud on three planes in the local coordinate system with a joint 2D convolution operation to extract its features. Additionally, we leverage a self‐encoding network based on image semantic segmentation U‐Net structure as the overall architecture of the point cloud semantic segmentation algorithm. The results demonstrate that the proposed method exhibited excellent performances for point cloud data sets corresponding to various scenes. [ABSTRACT FROM AUTHOR]

Subspace data representation has recently become a common practice in many computer vision tasks. Low-rank representation (LRR) is one of the most successful models for clustering vectorial data according to their subspace structures. This paper explores the possibility of extending LRR for subspace data on Grassmann manifold. Rather than directly embedding the Grassmann manifold into the symmetric matrix space, an extrinsic view is taken to build the self-representation in the local area of the tangent space at each Grassmannian point, resulting in a localized LRR method on Grassmann manifold. A novel algorithm for solving the proposed model is investigated and implemented. The performance of the new clustering algorithm is assessed through experiments on several real-world data sets including MNIST handwritten digits, ballet video clips, SKIG action clips, and DynTex++ data set and highway traffic video clips. The experimental results show that the new method outperforms a number of state-of-the-art clustering methods. [ABSTRACT FROM AUTHOR]

Different from image-based 3D pose estimation, video-based 3D pose estimation gains performance improvement with temporal information. However, these methods still face the challenge of insufficient generalization ability, including human motion speed, body shape, and camera distance. To address the above problems, we propose a novel approach, referred to as joint Spatial–temporal Multi-scale Transformers and Pose Transformation Equivalence Constraints (SMT-PTEC) for 3D human pose estimation from videos. We design a more general spatial–temporal multi-scale feature extraction strategy, and introduce optimization constraints that adapt to the diversity of data to improve the accuracy of pose estimation. Specifically, we first introduce a spatial multi-scale transformer to extract multi-scale features of pose and establish a cross-scale information transfer mechanism, which effectively explores the underlying knowledge of human motion. Then, we present a temporal multi-scale transformer to explore multi-scale dependencies between frames, enhance the adaptability of the network to human motion speed, and improve the estimation accuracy through a context aware fusion of multi-scale predictions. Moreover, we add pose transformation equivalence constraints by changing the training samples with horizontal flipping, scaling, and body shape transformation to effectively overcome the influence of camera distance and body shape for the prediction accuracy. Extensive experimental results demonstrate that our approach achieves superior performance with less computational complexity than previous state-of-the-art methods. Code is available at https://github.com/JNGao123/SMT-PTEC. [ABSTRACT FROM AUTHOR]

3D object reconstruction from arbitrary view intensity images is a challenging but meaningful research topic in computer vision. The main limitations of existing approaches are that they lack complete and efficient prior information and might not be able to deal with serious occlusion or partial observation of 3D objects, which may produce incomplete and unreliable reconstructions. To reconstruct structure and recover missing or unseen parts of objects, category prior and intrinsic geometry relation are particularly useful and necessary during the 3D reconstruction process. In this paper, we propose Category-and-Intrinsic-Geometry Guided Network (CIGNet) for 3D coarse-to-fine reconstruction from arbitrary view intensity images by leveraging category prior and intrinsic geometry relation. CIGNet combines a category prior guided reconstruction module with an intrinsic geometry relation guided refinement module. In the first reconstruction module, we leverage semantic class context by adding a supervision term over object categories to output coarse reconstructed results. In the second refinement module, we model the coarse 3D volumetric data as 2D slices and consider intrinsic geometry relations between them to design graph structures of coarse 3D volumes to finish the graph-based refinement. CIGNet can accomplish high-quality 3D reconstruction tasks by exploring the intra-category characteristics of objects as well as the intrinsic geometry relations of each object, both of which serve as useful complements to the visual information of images, in a coarse-to-fine fashion. Extensive quantitative and qualitative experiments on a synthetic dataset ShapeNet and real-world datasets Pix3D , Statue Model Repository , and BlendedMVS indicate that CIGNet outperforms several state-of-the-art methods in terms of accuracy and detail recovery. [ABSTRACT FROM AUTHOR]

Superpixel segmentation, which amounts to partitioning an image into a number of superpixels each of which is a set of pixels sharing common visual meanings, requires specific needs for different computer vision tasks. Graph based methods, as a kind of popular superpixel segmentation method, regard an image as a weighted graph whose nodes correspond to pixels of the image, and partition all pixels into superpixels according to the similarity between pixels over various feature spaces. Despite their improvement of the performance of segmentation, these methods ignore high-order relationship between them incurred from either locally neighboring pixels or structured layout of the image. Moreover, they measure the similarity of pairwise pixels using Gaussian kernel where a robust radius parameter is difficult to find for pixels which exhibit multiple features (e.g., texture, color, brightness). In this paper, we propose an adaptive hypergraph superpixel segmentation (AHS) of intensity images for solving both issues. AHS constructs a hypergraph by building the hyperedges with an adaptive neighborhood scheme, which explores an intrinsic relationship of pixels. Afterwards, AHS encodes the relationship between pairwise pixels using characteristics of current two pixels as well as their neighboring pixels defined by hyperedges. Essentially, AHS models the relationship of pairwise pixels in a high-order group fashion while graph based methods evaluate it in a one-vs-one fashion. Experiments on four datasets demonstrate that AHS achieves higher or comparable performance compared with state-of-the-art methods. • We propose a superpixel segmentation which consists of hypergraph based representation of images and Canny based boundary fine-tuning. • We model the high-order relationship of pixels by building the hyperedges with an adaptive neighborhood scheme. [ABSTRACT FROM AUTHOR]

• An SCA-Net that explicitly explores and integrates global and local contextual information. • An LPCM embedded in the endcoding network to grant coarse-scale feature maps better discrimination ability. • An SPCM in feature decoding to help integrate multi-scale feature maps and compensate saliency predictions with fine detailed cues. Salient Object Detection (SOD) is a fundamental problem in the field of computer vision. This paper presents a novel Spatial Context-Aware Network (SCA-Net) for SOD in images. Compared with other recent deep learning based SOD algorithms, SCA-Net can more effectively aggregate multi-level deep features. A Long-Path Context Module (LPCM) is employed to grant better discrimination ability to feature maps that incorporate coarse global information. Consequently, a more accurate initial saliency map can be obtained to facilitate subsequent predictions. SCA-Net also adopts a Short-Path Context Module (SPCM) to progressively enforce the interaction between local contextual cues and global features. Extensive experiments on five large-scale benchmarks demonstrate that SCA-Net achieves favorable performance against very recent state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]