Search

This document is a guest editorial from the journal IET Computer Vision, discussing the topic of advanced image restoration and enhancement. The editorial highlights the challenges faced in this field, such as the complexity of degradation models for real-world low-quality images and the difficulty of acquiring paired data. It also introduces a special issue of the journal that includes five accepted papers, which focus on video reconstruction and image super-resolution. The editorial concludes by providing brief summaries of each accepted paper. The guest editors of the special issue are also mentioned, along with their research interests and affiliations. [Extracted from the article]

The authors present a comprehensive overview of image processing and analysis work done to support research into the model flowering plant Arabidopsis thaliana. Beside the plant's importance in biological research, using image analysis to obtain experimental measurements of it is an interesting vision problem in its own right, involving the segmentation and analysis of sequences of images of objects whose shape varies between individual specimens and also changes over time. While useful measurements can be obtained by segmenting a whole plant from the background, they suggest that the increased range and precision of measurements made available by leaf-level segmentation makes this a problem well worth solving. A variety of approaches have been tried by biologists as well as computer vision researchers. This is an interdisciplinary area and the computer vision community has an important contribution to make. They suggest that there is a need for publicly available datasets with ground truth annotations to enable the evaluation of new approaches and to support the building of training data for modern data-driven computer vision approaches, which are those most likely to result in the kind of fully automated systems that will be of use to biologists. [ABSTRACT FROM AUTHOR]

An automatic computer vision algorithm that detects individual paper fibres from an image, assesses the possibility of grasping the detected fibres with microgrippers and detects the suitable grasping points is presented. The goal of the algorithm is to enable automatic fibre manipulation for mechanical characterisation, which has traditionally been slow manual work. The algorithm classifies the objects in images based on their morphology, and detects the proper grasp points from the individual fibres by applying given geometrical constraints. The authors test the ability of the algorithm to detect the individual fibres with 35 images containing more than 500 fibres in total, and also compare the graspability analysis and the calculated grasp points with the results of an experienced human operator with 15 images that contain a total of almost 200 fibres. The detection results are outstanding, with fewer than 1% of fibres missed. The graspability analysis gives sensitivity of 0.83 and specificity of 0.92, and the average distance between the grasp points of the human and the algorithm is 220 µm. Also, the choices made by the algorithm are much more consistent than the human choices. [ABSTRACT FROM AUTHOR]

The object tracking algorithm based on Siamese network often extracts the deep feature of the target to be tracked from the first frame of the video sequence as a template, and uses the template for the whole tracking process. Because the manually annotated target in the first frame of video sequence is more accurate, these algorithms often have stable performance. However, it is difficult to adapt to the changing target features only using the target template extracted from the first frame. Inspired by the feature fusion network based on a transformer, this paper proposes a template update module called multi‐template temporary information fusion module (MTFM), which can be trained offline. By fusing multiple target template features on time series, the template can always adapt to the changes of target appearance in the tracking process. In order to train the MTFM, this paper proposes a training method using time series data and Mean Square Error (MSE) as the loss function. This paper uses the MTFM on SiamFC++ tracker, and obtains good experimental results in three challenging datasets, including VOT2016, OTB100 and GOT‐10k. The running speed of the algorithm on graphics processing unit (GPU) is maintained at about 200fps, which exhibits good real‐time performance. [ABSTRACT FROM AUTHOR]

A dual-balanced network for long-tail distribution object detection Gong et al. present a method for object detection with a long-tail distribution that includes a dual-balanced network and balanced classification loss. Few-shot logo detection Hou et al. present a cutting-edge few-shot detection method for logo images. The past decade has witnessed remarkable achievements in computer vision, owing to the fast development of deep learning. [Extracted from the article]

The text in the natural scene can express rich semantic information, which helps people understand and analyse daily things. This paper focuses on the problems of discrete text spatial distribution and variable text geometric size in natural scenes with complex backgrounds and proposes an end‐to‐end natural scene text detection method based on DBNet. The authors first use IResNet as the backbone network, which does not increase network parameters while retaining more text features. Furthermore, a module with Transformer is introduced in the feature extraction stage to strengthen the correlation between high‐level feature pixels. Then, the authors add a spatial pyramid pooling structure in the end of feature extraction, which realises the combination of local and global features, enriches the expressive ability of feature maps, and alleviates the detection limitations caused by the geometric size of features. Finally, to better integrate the features of each level, a dual attention module is embedded after multi‐scale feature fusion. Extensive experiments on the MSRA‐TD500, CTW1500, ICDAR2015, and MLT2017 data set are conducted. The results showed that IDBNet can improve the average precision, recall, and F‐measure of a text compared with the state of art text detection methods and has higher predictive ability and practicability. [ABSTRACT FROM AUTHOR]

With the rapid development of 3D imaging sensors, such as depth cameras and laser scanning systems, 3D data has become increasingly accessible. TOPIC B - POSE ESTIMATION Gao et al., in their paper 'Efficient 6D Object Pose Estimation based on Attentive Multi-Scale Contextual Information', proposed an end-to-end 6D pose estimation network to utilise multi-scale contextual features learnt from two heterogeneous data. Experimental results conducted on the multi-vehicle stereo event camera dataset demonstrate the effectiveness of EAGAN on both the depth and optical flow estimation tasks. [Extracted from the article]

Perception systems in autonomous vehicles need to accurately detect and classify objects within their surrounding environments. Numerous types of sensors are deployed on these vehicles, and the combination of such multimodal data streams can significantly boost performance. The authors introduce a novel sensor fusion framework using deep convolutional neural networks. The framework employs both camera and LiDAR sensors in a multimodal, multiview configuration. The authors leverage both data types by introducing two new innovative fusion mechanisms: element‐wise multiplication and multimodal factorised bilinear pooling. The methods improve the bird's eye view moderate average precision score by +4.97% and +8.35% on the KITTI dataset when compared to traditional fusion operators like element‐wise addition and feature map concatenation. An in‐depth analysis of key design choices impacting performance, such as data augmentation, multi‐task learning, and convolutional architecture design is offered. The study aims to pave the way for the development of more robust multimodal machine vision systems. The authors conclude the paper with qualitative results, discussing both successful and problematic cases, along with potential ways to mitigate the latter. [ABSTRACT FROM AUTHOR]

Due to the robustness of skeleton data to human scale, illumination changes, dynamic camera views, and complex backgrounds, great progress has been made in skeleton‐based video anomaly detection in recent years. The spatio‐temporal graph convolutional network has been proven to be effective in modelling the spatio‐temporal dependencies of non‐Euclidean data such as human skeleton graphs, and the autoencoder based on this basic unit is widely used to model sequence features. However, due to the limitations of the convolution kernel, the model cannot capture the correlation between non‐adjacent joints, and it is difficult to deal with long‐term sequences, resulting in an insufficient understanding of behaviour. To address this issue, this paper applies the Transformer to the human skeleton and proposes the Spatio‐Temporal Enhanced Graph‐Transformer AutoEncoder (STEGT‐AE) to improve the capability of modelling. In addition, the multi‐memory model with skip connections is employed to provide different levels of coding features, thereby enhancing the ability of the model to distinguish similar heterogeneous behaviours. Furthermore, the STEGT‐AE has a single encoder‐double decoder architecture, which can improve the detection performance by the combining reconstruction and prediction error. The experimental results show that performances of STEGT‐AE is significantly better than other advanced algorithms on four baseline datasets. [ABSTRACT FROM AUTHOR]

Grasping detection, which involves identifying and assessing the grasp ability of objects by robotic systems, has garnered significant attention in recent years due to its pivotal role in the development of robotic systems and automated assembly processes. Despite notable advancements in this field, current methods often grapple with both practical and theoretical challenges that hinder their real‐world applicability. These challenges encompass low detection accuracy, the burden of oversized model parameters, and the inherent complexity of real‐world scenarios. In response to these multifaceted challenges, a novel lightweight grasping detection model that not only addresses the technical aspects but also delves into the underlying theoretical complexities is introduced. The proposed model incorporates attention mechanisms and residual modules to tackle the theoretical challenges posed by varying object shapes, sizes, materials, and environmental conditions. To enhance its performance in the face of these theoretical complexities, the proposed model employs a Convolutional Block Attention Module (CBAM) to extract features from RGB and depth channels, recognising the multifaceted nature of object properties. Subsequently, a feature fusion module effectively combines these diverse features, providing a solution to the theoretical challenge of information integration. The model then processes the fused features through five residual blocks, followed by another CBAM attention module, culminating in the generation of three distinct images representing capture quality, grasping angle, and grasping width. These images collectively yield the final grasp detection results, addressing the theoretical complexities inherent in this task. The proposed model's rigorous training and evaluation on the Cornell Grasp dataset demonstrate remarkable detection accuracy rates of 98.44% on the Image‐wise split and 96.88% on the Object‐wise split. The experimental results strongly corroborate the exceptional performance of the proposed model, underscoring its ability to overcome the theoretical challenges associated with grasping detection. The integration of the residual module ensures rapid training, while the attention module facilitates precise feature extraction, ultimately striking an effective balance between detection time and accuracy. [ABSTRACT FROM AUTHOR]

Many mid-level and high-level computer vision tasks, such as object segmentation, detection and recognition, image retrieval and classification, and video tracking and understanding, still have not leveraged the advantages offered by spectral information. The increasing accessibility and affordability of spectral imaging technology have revolutionised computer vision, allowing for data capture across various wavelengths beyond the visual spectrum. [Extracted from the article]

With the goal of solving the problem of feature extractors lacking strong supervision training and insufficient time information concerning single‐sequence model learning, a hierarchical sequence memory network with a multi‐level iterative optimisation strategy is proposed for continuous sign language recognition. This method uses the spatial‐temporal fusion convolution network (STFC‐Net) to extract the spatial‐temporal information of RGB and Optical flow video frames to obtain the multi‐modal visual features of a sign language video. Then, in order to enhance the temporal relationships of visual feature maps, the hierarchical memory sequence network is used to capture local utterance features and global context dependencies across time dimensions to obtain sequence features. Finally, the decoder decodes the final sentence sequence. In order to enhance the feature extractor, the authors adopted a multi‐level iterative optimisation strategy to fine‐tune STFC‐Net and the utterance feature extractor. The experimental results on the RWTH‐Phoenix‐Weather multi‐signer 2014 dataset and the Chinese sign language dataset show the effectiveness and superiority of this method. [ABSTRACT FROM AUTHOR]

Although low‐rank representation (LRR)‐based subspace learning has been widely applied for feature extraction in computer vision, how to enhance the discriminability of the low‐dimensional features extracted by LRR based subspace learning methods is still a problem that needs to be further investigated. Therefore, this paper proposes a novel low‐rank preserving embedding regression (LRPER) method by integrating LRR, linear regression, and projection learning into a unified framework. In LRPER, LRR can reveal the underlying structure information to strengthen the robustness of projection learning. The robust metric L2,1‐norm is employed to measure the low‐rank reconstruction error and regression loss for moulding the noise and occlusions. An embedding regression is proposed to make full use of the prior information for improving the discriminability of the learned projection. In addition, an alternative iteration algorithm is designed to optimise the proposed model, and the computational complexity of the optimisation algorithm is briefly analysed. The convergence of the optimisation algorithm is theoretically and numerically studied. At last, extensive experiments on four types of image datasets are carried out to demonstrate the effectiveness of LRPER, and the experimental results demonstrate that LRPER performs better than some state‐of‐the‐art feature extraction methods. [ABSTRACT FROM AUTHOR]

Accurate target prediction, especially bounding box estimation, is a key problem in visual tracking. Many recently proposed trackers adopt the refinement module called IoU predictor by designing a high‐level modulation vector to achieve bounding box estimation. However, due to the lack of spatial information that is important for precise box estimation, this simple one‐dimensional modulation vector has limited refinement representation capability. In this study, a novel IoU predictor (IoUNet++) is designed to achieve more accurate bounding box estimation by investigating spatial matching with a spatial cross‐layer interaction model. Rather than using a one‐dimensional modulation vector to generate representations of the candidate bounding box for overlap prediction, this paper first extracts and fuses multi‐level features of the target to generate template kernel with spatial description capability. Then, when aggregating the features of the template and the search region, the depthwise separable convolution correlation is adopted to preserve the spatial matching between the target feature and candidate feature, which makes their IoUNet++ network have better template representation and better fusion than the original network. The proposed IoUNet++ method with a plug‐and‐play style is applied to a series of strengthened trackers including DiMP++, SuperDiMP++ and SuperDIMP_AR++, which achieve consistent performance gain. Finally, experiments conducted on six popular tracking benchmarks show that their trackers outperformed the state‐of‐the‐art trackers with significantly fewer training epochs. [ABSTRACT FROM AUTHOR]

When describing an image, people can rapidly extract the topic from the image and find the main object, generating sentences that match the main idea of the image. However, most of the scene graph generation methods do not emphasise the importance of the topic of the image. Consequently, the captions generated by the scene graph‐based image captioning models cannot reflect the topic in the image then expressing the central idea of the image. In this paper, we propose a method for image captioning based on topic scene graphs (TSG). Firstly, we propose the structure of topic scene graphs that express images' topics and the relationships between objects. Then, combined with the topic scene graph, we utilise the salient object detection to generate the topic scene graph highlighting the salient objects of the image. Note that our framework is agnostic to any scene graph‐based image captioning model and thus can be widely applied in the community which seeks salient object predictions. We compare the performance of our topic scene graph with the state‐of‐the‐art scene graph generation models and mainstream image captioning models on MSCOCO and Visual Genome datasets, both achieving better performance. [ABSTRACT FROM AUTHOR]

To address the problem that the features extracted by CNN‐based facial expression recognition (FER) do not consider structural information, a region adaptive correlation deep network (RACN) is proposed. The network consists of two branches. In one branch, the features obtained by applying CNN to facial sub‐blocks are used as the input of the proposed second‐order region correlation network (SRCN), which obtains structural features by adaptively learning the correlation of facial regions. Furthermore, they are fused with the parallel branch‐extracted global features to obtain a comprehensive high‐semantic feature representation. Finally, weights are assigned to the two features through the channel attention mechanism for more accurate expression classification. Experimental results show that our method can effectively extract expression features in an end‐to‐end manner, improve the accuracy of FER, and achieve competitive performance without relying on any a priori knowledge. And the region‐adaptive correlation feature extraction branch RACN can be applied to other deep learning networks to extract discriminative structural‐adaptive features. To the best of our knowledge, our work is the first to enrich the feature representation for end‐to‐end static FER by adaptively obtaining more discriminative regional adaptive correlation feature vectors via the autocorrelation matrix combined with CNN compared to the existing literature. [ABSTRACT FROM AUTHOR]

Semantic action recognition aims to classify actions based on the associated semantics, which can be applied in video captioning and human‐machine interaction. In this paper the problem is addressed by jointly learning multiple pose lexicons based on multiple body parts. Specifically, multiple visual pose models are learnt, and one visual pose model is associated with one body part, which characterises the likelihood of an observed video frame being generated from hidden visual poses. Moreover, multiple pose lexicon models are simultaneously learnt along with visual pose models. One pose lexicon model is associated with one body part that establishes a probabilistic mapping between the hidden visual poses and semantic poses parsed from textual instructions. To capture the temporal relations among body parts, a transition model is also learnt to measure the probability of the alignment transitioned from one position to another position. The body part‐based pose lexicon learning provides a novel method of cross‐modality semantic correlation, which can be applied in other spatial and temporal data. Action classification is finally formulated as the problem of finding the maximum posterior probability that a given multiple sequences of visual frames follow multiple sequences of semantic poses, subject to the most likely visual pose sequences and alignment sequences. Experiments were conducted on five action datasets to validate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

The existing monocular depth estimation methods based on deep learning have difficulty in estimating the depth near the edges of the objects in an image when the depth distance between these objects changes abruptly and decline in accuracy when an image has more noises. Furthermore, these methods consume more hardware resources because they have huge network parameters. To solve these problems, this paper proposes a depth estimation method based on weighted fusion and point‐wise convolution. The authors design a maximum‐average adaptive pooling weighted fusion module (MAWF) that fuses global features and local features and a continuous point‐wise convolution module for processing the fused features derived from the (MAWF) module. The two modules work closely together for three times to perform weighted fusion and point‐wise convolution of features of multi‐scale from the encoder output, which can better decode the depth information of a scene. Experimental results show that our method achieves state‐of‐the‐art performance on the KITTI dataset with δ1 up to 0.996 and the root mean square error metric down to 8% and has demonstrated the strong generalisation and robustness. [ABSTRACT FROM AUTHOR]

Room layout estimation aims to predict the location and range of layout planes of interior spaces. Previous works treat each layout plane as an independent individual without considering the ordinal relation between walls, resulting the loss of the wall planes and the lack of integrity. This paper proposes a novel two‐branch neural networks model to estimate 3D layouts of cuboid and non‐cuboid room types. The model embeds the ordinal relation between layout planes into the layout segmentation branch through an proposed ordinal classification loss function, and outputs both pixel‐level layout segmentation maps and layout plane parameter maps. Then, the instance‐level plane parameters of each layout plane are determined by using an instance‐aware pooling layer. Finally, the sharpness of layout edges of the 2D layout semantic segmentation map is optimized by using an improved depth map intersection algorithm. Furthermore, we annotate a large‐scale 3D room layout estimation dataset, InteriorNet‐Layout, to obtain a steady model. Experiments on synthesized real‐world datasets show that the proposed method achieves faster calculation while maintaining high accuracy. Code is available at https://github.com/Hui‐Yao/3D‐ordinal‐layout‐estimation. [ABSTRACT FROM AUTHOR]

Makeup transfer (MT) aims to transfer the makeup style from a given reference makeup face image to a source image while preserving face identity and background information. In recent years, MT has attracted the attention of many scholars, and it has a wide range of application prospects and research value. Since then, many methods have been proposed to accomplish MT, most of which are based on Generative Adversarial Network methods. A taxonomy of existing algorithms in the field of MT is first proposed. Then, evaluation methods are proposed, existing methods are analysed, and existing datasets are introduced. This paper finally discusses the current problems in the field of MT and the trend of future research. [ABSTRACT FROM AUTHOR]

Stereo matching algorithms have been developed for many years but basically focus only on the implementation of existing datasets and are rarely applied to real scenarios, such as industrial robot scenarios. Traditional stereo matching algorithms have a high error rate, and deep learning algorithms are difficult to obtain good results in real scenarios because of their weak generalisation ability and difficult access to training data. In order to use stereo matching algorithms for industrial robot guidance, it is better to design a new traditional algorithm with low time complexity for the characteristics of industrial robot scenarios dominated by planar facets. This paper proposes a new matching method based on subrows of pixels, instead of individual pixels, in order to improve robustness of matching and reduce running time. First, the pixel strings from the same row of the left and right images are divided into several colour‐identical or colour‐gradient segments. Then, the colour and length of the two left and right pixel segment are used as clues to determine a matching relation and obtain the matching type. Then, all match types can be determined according to non‐crossing mapping. Each match type can reason backward to the corresponding spatial state of the stimulus source so that the disparity of pixels in pixel segments representing the spatial state can be calculated. This new matching method makes full use of the stimulus homology constraints and projective geometric constraints of row‐aligned images. The method can obtain good results in industrial robot scenarios and be applied for industrial robot guidance. [ABSTRACT FROM AUTHOR]

Convolutional neural network (CNN), as a classic deep learning algorithm, has been applied to various computer vision tasks. However, most classic CNN models focus on the extraction and utilisation of spatial domain features, while ignoring the potential ability of frequency domain feature extraction. In this study, the mechanism in the backbone design is explored. Firstly, the traditional DCT formula is converted into a convolution form through mathematical derivation. On the basis of a a new type of convolution, namely the DCT Convolution is designed. It is more applicable to deep learning network architectures. Secondly, based on the DCT Convolution, a new cross‐domain fusion network named CDF‐Net is designed. The frequency domain and spatial domain features of the input sample are extracted and fused by the network. CDF‐Net is a general network framework which can be applied to most existing prevalent networks. Finally, various experiments are conducted. On image classification task, for Imagenet2012 dataset, the method proposed was applied to ResNet50, and the accuracy of Top1 was increased by 3.684%. On object detection task, for COCO2017 dataset, the method proposed in this study was applied to ResNet50 and ResNeXt50, mAP were improved by 0.5% and 1.2% respectively. [ABSTRACT FROM AUTHOR]

Deep learning‐based single view 3D reconstruction is a hot topic in computer vision. However, predicting a more realistic 3D point cloud from a single image is an ill‐posed problem. In recent years, most of the 3D point cloud prediction methods based on single view are straight‐through structure, which will cause the loss of part of feature information and the loss of part of detail information of the resulting point clouds, which will lead to the unsatisfactory visual effect of reconstructed point clouds. In this paper, a Feature‐Enhanced 3D point clouds generation Network (3D‐FENet) from a single image is proposed. In order to enhance the feature information of RGB image, edge extraction module is adopted. In the process of point cloud generation, a point cloud pyramid is designed, which combines low resolution point cloud with high resolution point cloud to enhance the local details of the generated point clouds. In the fine‐tuning stage, the differential projection module is used to fine‐tune the whole network by 2D projection of reconstructed point clouds. Experimental results show that the performance of the authors' proposed method is better than the state‐of‐the‐art studies. [ABSTRACT FROM AUTHOR]

Cross‐modality person re‐identification is a challenging task due to the large visual appearance difference between RGB and infrared images. Existing studies mainly focus on learning local features and ignore the correlation between local features. In this paper, the Integration Graph Attention Network is proposed to learn the completed correlation between local features via the graph structure. To this end, the authors learn the coarse‐fine attention weights to aggregate the local features by considering local detail and global information. Furthermore, the Multi‐Centre Constrained Loss is proposed to optimise the feature similarity by constraining the centres of modality and identity. It simultaneously utilises three kinds of centre constraints, that is intra‐identity centre constraint, modality centre constraint, and inter‐identity centre constraint, in order to reduce the influence of modality information explicitly. The proposed method is evaluated on two standard benchmark datasets, that is SYSU‐MM01 and RegDB, and the results demonstrate that the authors' method achieves better performance than the state‐of‐the‐art methods, for example, surpassing NFS by 4.8% and 6.0% mAP on the single‐shot setting in All‐search and Indoor‐search modes, respectively. [ABSTRACT FROM AUTHOR]

Traditional unsupervised domain adaptation (UDA) usually assumes that the source domain has labels and the target domain has no labels. In a real environment, labelled source domain data usually comes from multiple different distributions. To handle this problem, multi‐source unsupervised domain adaptation (MUDA) is proposed. Multi‐source unsupervised domain adaptation aims to adapt the model trained on multi‐labelled source domains to the unlabelled target domain. In this paper, a novel MUDA method by domain‐specific feature recalibration and alignment (FRA) is proposed. Specifically, to achieve feature recalibration, the authors leverage channel attention to pick out significant channels and spatial attention to focus on important features in different channels. Such integration of channel and spatial attention can lead to effective domain‐specific feature recalibration that may be of great importance to MUDA. In addition, to achieve better MUDA, the authors propose domain‐specific feature alignment which consists of Maximum Mean Discrepancy and JS‐divergence loss. Maximum Mean Discrepancy can reduce the difference between the source domain and target domain. Meanwhile, JS‐divergence loss may ensure the prediction consistency of different classifiers in the source domains. Four experiments have proved that FRA can achieve significantly better results in popular benchmarks for MUDA. [ABSTRACT FROM AUTHOR]

As a basic component in the field of computer vision, the pedestrian detection plays an essential role in several real‐world applications such as video surveillance. The promising performance has been achieved in pedestrian detection relying on deep learning, but large‐scale variance and small‐scale pedestrian detection remain inherently hard as before. In order to deal with the aforementioned problems, this paper proposes a multi‐scale pedestrian detection method with global–local attention and multi‐scale receptive field context (MRFC). To make the network focus on small‐scale pedestrians, we add a high‐resolution detection branch on the original detector. To better integrate the incongruous semantic feature, the global–local attention module is embedded to highlight the feature representation of pedestrians so as to implement the feature fusion effectively. In order to adapt the receptive field of the network to achieve scale‐variance detection, the MRFC is applied. Based on integrating the above structures, the proposed method achieves competitive results on Caltech and CityPersons datasets. The source code is released in https://github.com/xiaopan999/yolov5‐pedestrian_detection. [ABSTRACT FROM AUTHOR]

Diabetic Retinopathy (DR) grading into different stages of severity continues to remain a challenging issue due to the complexities of the disease. Diabetic Retinopathy grading classifies retinal images to five levels of severity ranging from 0 to 5, which represents No DR, Mild non‐proliferative diabetic retinopathy (NPDR), Moderate NPDR, Severe NPDR, and proliferative diabetic retinopathy. With the advancement of Deep Learning, studies on the application of the Convolutional Neural Network (CNN) in DR grading have been on the rise. High accuracy and sensitivity are the desired outcome of these studies. This paper reviewed recently published studies that employed CNN for DR grading to 5 levels of severity. Various approaches are applied in classifying retinal images which are, (i) by training CNN models to learn the features for each grade and (ii) by detecting and segmenting lesions using information about their location such as microaneurysms, exudates, and haemorrhages. Public and private datasets have been utilised by researchers in classifying retinal images for DR. The performance of the CNN models was measured by accuracy, specificity, sensitivity, and area under the curve. The CNN models and their performance varies for every study. More research into the CNN model is necessary for future work to improve model performance in DR grading. The Inception model can be used as a starting point for subsequent research. It will also be necessary to investigate the attributes that the model uses for grading. [ABSTRACT FROM AUTHOR]

Modern convolutional neural networks (CNNs)‐based face detectors have achieved tremendous strides due to large annotated datasets. However, misaligned results with high detection confidence but low localization accuracy restrict the further improvement of detection performance. In this paper, the authors first predict high confidence detection results on the training set itself. Surprisingly, a considerable part of them exist in the same misalignment problem. Then, the authors carefully examine these cases and point out that annotation misalignment is the main reason. Later, a comprehensive discussion is given for the replacement rationality between predicted and annotated bounding‐boxes. Finally, the authors propose a novel Bounding‐Box Deep Calibration (BDC) method to reasonably replace misaligned annotations with model predicted bounding‐boxes and offer calibrated annotations for the training set. Extensive experiments on multiple detectors and two popular benchmark datasets show the effectiveness of BDC on improving models' precision and recall rate, without adding extra inference time and memory consumption. Our simple and effective method provides a general strategy for improving face detection, especially for light‐weight detectors in real‐time situations. [ABSTRACT FROM AUTHOR]

Research on tooth surface design based on deep neural networks has recently achieved progress in terms of both accuracy and execution efficiency. However, unrealistic outputs are still a challenging issue, partially because of (1) the lack of semantic guidance, (2) the inability to discover and rectify false results, and (3) the lack of exploration of structural coherence in intermediate layers. In this paper, we present an approach to predict depth images for designed teeth based on a conditional generative adversarial network (CGAN) by incorporating semantic guidance. Moreover, the uncertainty of semantic inference is employed to improve the model outputs, and a structural coherence loss is proposed for adversarial learning to enhance the discrimination capability of the network in intermediate layers. We evaluate the performance of our approach with the Shining3D tooth dataset. The experimental results show that our method produces better results than the other available approaches in terms of accuracy. [ABSTRACT FROM AUTHOR]

Skeleton‐based human action recognition has emerged recently thanks to its compactness and robustness to appearance variations. Although impressive results have been obtained in recent years, the performance of skeleton‐based action recognition methods has to be improved to be deployed in real‐time applications. Recently, a lightweight network structure named Double‐feature Double‐motion Network (DD‐Net) has been proposed for the skeleton‐based human action recognition. With high speed, the DD‐Net achieves state‐of‐the‐art performance on hand and body actions. The DD‐Net could not distinguish actions if they have a weak connection with the global trajectories. However, the DD‐Net is suitable for human action recognition where actions strongly correlate to the global trajectories. In this paper, the authors propose TD‐Net, an improved version of the DD‐Net in which a new branch is added. The new branch takes the normalised coordinates of joints (NCJ) to enrich the spatial information. On five datasets for skeleton‐based human activity recognition that are MSR‐Action3D, CMDFall, JHMDB, FPHAB, and NTU RGB + D, the TD‐Net consistently obtains superior performance compared with the baseline model DD‐Net. The proposed method outperforms different state‐of‐the‐art methods, including both hand‐designed and deep learning‐based methods on four datasets (MSR‐Action3D, CMDFall, JHMDB, and FPHAB). Furthermore, the generalisation of the proposed method is confirmed through cross‐dataset evaluation. To illustrate the potential use of the model for real‐time human action recognition, the authors have deployed an application on an edge device. The experimental result shows that the application can process up to 40 fps for pose estimation using MediaPipe. It takes only 0.04 ms to recognise an action from skeleton sequences. [ABSTRACT FROM AUTHOR]

Most deep‐learning‐based multi‐view stereo series studies are concerned with improving the depth prediction accuracy of noise‐free images. However, it is difficult to obtain off‐the‐set clean images in practice and 3D convolutional neural networks require a lot of computing resources. To make full use of its computing power, different types of information can be processed simultaneously in the network. For these two issues, this paper proposes a novel multi‐stage network architecture to address depth inference and denoising simultaneously. Specifically, 2D feature maps are first converted into 3D cost volumes containing pixel information and depth information through differentiable homography and Gaussian probability mapping. Then, the cost volume is input into the regularisation module in each network stage to obtain the predicted probability volumes. Furthermore, simple static weights lead to training failure, and it is necessary to dynamically adjust the loss function by gradient normalisation. The proposed method can dispose of pixel information and depth information simultaneously and both reach an excellent level. Extensive experimental results show that the authors' work surpasses the state‐of‐the‐art denoising on the DTU dataset (adding Gaussian–Poisson noise) and is more robust to noise images in depth inference. [ABSTRACT FROM AUTHOR]

The colour of the point cloud provides abundant information for perception tasks such as autonomous driving and virtual reality. However, few prior work studied the automatic colourisation of colourless point clouds. In this paper, the authors propose a novel method named as Point cloud Colorization Network based on Relevance Embedding (PCCN‐RE) relying on three structures: a relevance embedding structure that efficiently captures local information through the calculation of a covariance matrix within nearby points; a weighted pooling structure designed to facilitate the fusion of features; an enhanced spatial transform network structure that keeps the invariance of input point clouds. On the ShapeNetCore dataset, our PCCN‐RE generates more authentic colour than state‐of‐the‐art methods for colourless point clouds and achieves the highest results by obtaining a Peak Signal to Noise Ratio of 9.40 and a Structural Similarity Index of 0.62. [ABSTRACT FROM AUTHOR]

Point cloud matching is an important procedure in a variety of computer vision tasks. Traditional point cloud matching methods have made great progress, while neural network‐based approaches are becoming a trend, powered by their strong capabilities of feature extraction. Existing point matching neural networks, however, mainly focus on the rigid transformation. More complex transformations should also be considered in many scenarios. In this regard, the authors extend the rigid registration to non‐rigid cases and propose a network called the Scale Robust Point Matching (SRPM)‐Net for scale point matching. This robust structure‐preserving network is implemented by incorporating Lie group parametrisation. It is conducted by Lie group linearisation representation with the constraints of parameters under the corresponding basis of Lie algebra. SRPM‐Net preserves the structure of the solution and avoids degeneration. The contributions of this paper lie in two aspects: Most importantly, SRPM‐Net provides an extendable framework for handling complicated transformations. Secondly, it introduces a new feature learning module, which better preserves the shape structure by aggregating the high‐dimensional feature and calculating the normal vector of point cloud surface automatically. Experimental results show that SRPM‐Net is more robust and accurate than existing traditional and recent deep learning methods under various situations. [ABSTRACT FROM AUTHOR]

The multi‐camera calibration is an essential step for many spatially aware applications, such as robotic navigation, augmented reality, and 3D human pose estimation. Traditional calibration methods use off‐the‐shelf checkerboards or triangles as the known world coordinate system and their corresponding corners are set as control points, which heavily depends on specific calibration patterns and is not suitable for calibration pattern‐denied environments. In this paper, an automatic calibration method is proposed to calibrate the multi‐camera system without the aid of a known calibration pattern. The key idea of the proposed method is that the authors consider the human body, which is always available, as the counterpart of the calibration pattern. The authors' approach starts with binocular camera calibration, in which the extrinsic and intrinsic parameters are calculated in order and followed by a joint optimisation. With the results of each pair of binocular camera calibration, the multi‐camera system calibration is carried out in three steps: (i) parameters initialisation, (ii) extrinsic parameters optimisation, and (iii) jointly optimising intrinsic and extrinsic parameters. Since the authors' approach does not require additional calibration patterns except for one visible person, it is flexible and easy to be implemented. Real experiments are conducted in different scenes, camera angles, and camera settings. Human pose estimation with the multi‐camera system is additionally performed for exhaustive experiments. The experimental results demonstrate that the authors' method shows superior performance than the traditional method with the aid of a specific calibration pattern. [ABSTRACT FROM AUTHOR]

6D pose estimation has been pervasively applied to various robotic applications, such as service robots, collaborative robots, and unmanned warehouses. However, accurate 6D pose estimation is still a challenge problem due to the complexity of application scenarios caused by illumination changes, occlusion and even truncation between objects, and additional refinement is required for accurate 6D object pose estimation in prior work. Aiming at the efficiency and accuracy of 6D object pose estimation in these complex scenes, this paper presents a novel end‐to‐end network, which effectively utilises the contextual information within a neighbourhood region of each pixel to estimate the 6D object pose from RGB‐D images. Specifically, our network first applies the attention mechanism to extract effective pixel‐wise dense multimodal features, which are then expanded to multi‐scale dense features by integrating pixel‐wise features at different scales for pose estimation. The proposed method is evaluated extensively on the LineMOD and YCB‐Video datasets, and the experimental results show that the proposed method is superior to several state‐of‐the‐art baselines in terms of average point distance and average closest point distance. [ABSTRACT FROM AUTHOR]

Despite their success, existing human pose estimation approaches mostly have complex architectures, high cost, and lack of lightweight modules. To address this problem, this paper proposes a Ghost Shuffle Lightweight Pose Network (GSLPN) with a more lightweight and efficient network architecture than the popular Lightweight Pose Network. First, in order to condense the scale of the network while maintaining its performance, we stack two lightweight modules, depthwise convolution and the Ghost module, to build our initial prototype bottleneck. Then, we impose a channel shuffle operation on the prototype bottleneck to shuffle the sequence of the feature maps for constructing Ghost Shuffle Bottleneck (Ghost Shuffle Bottleneck) with effective feature representation so as to develop a GSLPN. Second, a lightweight, efficient parallel attention mechanism, Lightweight Pose Parallel Attention, is proposed to improve keypoint locating accuracy. An experiment validating the proposed method showed that GSLPN achieved competitive performance with a smaller model size and less computational complexity than state‐of‐the‐art methods, indicating that the GSLPN is a superior approach for human pose estimation. [ABSTRACT FROM AUTHOR]

View graph construction aims to effectively organise disordered image dataset through image retrieval technique before structure from motion (SfM). Existing view graph construction methods usually fail to handle scenes with duplicate structure, because these methods solely treat the construction of view graph as a process of image‐pair‐wise matching and lack in exploiting images' topological details in dataset. In this paper, we handle this problem from a novel perspective to construct view graph in a global paradigm by introducing an end‐to‐end graph convolutional network (GCN). First, a location‐aware embedding module is introduced to encode images into a feature space that takes into account the feature's location by using Vision Transformer architecture, improving the distinction between features of duplicate structure. Second, graph convolutional network that consists of topological relationship preserving module and feature metric learning module is proposed. Topological relationship preserving network is proposed to help nodes maintain their connected neighbourhood features. By merging the topological connected information into images' embedding, our method can process image matching in a global mode, thus improving the disambiguation ability for images with duplicate scenes. Then a feature metric learning network is embedded into GCN to dynamically compute the linkage prediction among nodes based on their features. Finally, our method combines these three parts to jointly optimise nodes' features and linkage prediction in an end‐to‐end paradigm. We make qualitative and quantitative comparisons based on three public benchmark datasets and demonstrate that our proposed method performs favourably against other state‐of‐the‐art methods. [ABSTRACT FROM AUTHOR]

The work on grayscale image colourisation has been significantly improved. Currently, learning‐based methods have achieved some great colourisation effects, but existing colour edge bleeding, especially when colourful cartoon characters. In this paper, we focus on the colourisation of cartoon characters from a series in an adversarial environment with a line art network, whose name is LineGAN. LineGAN learns the corresponding colour mapping from datasets, improving the accuracy of image colourisation. Our methods limit the colour boundary overflow by adding a line art frame in the generator. Extensive experiment results on cartoon image colourisation tasks demonstrate that the proposed method can achieve effective results. [ABSTRACT FROM AUTHOR]

The transformer architecture has been the dominant framework for today's image captioning tasks because of its superior performance. However, existing methods based on transformer often lack the integrated use of multi‐level semantic information and are weak in maintaining the relevance of captions to the image. In this paper, a semantic‐meshed and content‐guided transformer network is introduced for image captioning to solve these problems. The semantic‐meshed mechanism allows the model to generate words by selecting semantic information of multiple interaction levels adaptively through attention‐based reconstruction. And the content‐guided module guides the words generation by using attribute features that represent the image content, which aims to keep the generated caption consistent with the main content of the image. Experiments on dataset on the MSCOCO captioning dataset are conducted to validate the authors' model and achieve superior results compared to other state‐of‐the‐art method approaches. [ABSTRACT FROM AUTHOR]

Image recognition based on machine learning has been widely utilised in the computer vision field. In the image recognition process, quite a few positive and negative instances are needed for effective machine learning. However, some invalid instances selected from the instance candidates, particularly for negative instances, will result in reduced image recognition accuracy and wasted resources. When making the instance selections for machine learning, if a large number of negative instances are selected that exhibit a semantic distance too close to the positive instances, the recognition accuracy will be considerably reduced. The selection of valid negative instances from the instance candidates has become a new challenge in the field of computer vision. In this paper a new method containing several different algorithms is presented, that more effectively selects valid negative instances. In this innovative process, a Wordnet subtree, semantic distance based on feature descriptors and an improved K-nearest neighbour model were all utilised. Experiments were implemented using a large database to determine the effectiveness of this method. The results have demonstrated that this method can significantly improve the recognition results. [ABSTRACT FROM AUTHOR]

Adversarial examples (AEs) against deep neural networks (DNNs) raise wide concerns about the robustness of DNNs. Existing detection mechanisms are often limited to a given attack algorithm. Therefore, it is highly desirable to develop a robust detection approach that remains effective for a large group of attack algorithms. In addition, most of the existing defences only perform well for small images (e.g. MNIST and Canadian institute for advanced research (CIFAR)) rather than large images (e.g. ImageNet). In this paper, the authors propose a robust and effective defence method for analysing the sensitivity of various AEs, especially in a much harder case (large images). Their method first creates a feature map from the input space to the new feature space, by utilising 19 different feature mapping methods. Then, a detector is learned with the machine-learning algorithm to recognise the unique distribution of AEs. Their extensive evaluations on their proposed detector show that their detector can achieve: (i) low false-positive rate (<1%), (ii) high true-positive rate (higher than 98%), (iii) low overhead (<0.1 s per input), and (iv) good robustness (work well across different learning models, attack algorithms, and parameters), which demonstrate the efficacy of the proposed detector in practise. [ABSTRACT FROM AUTHOR]

Given the importance of privacy protection in databases and other institutions, federated learning (FL) is used to benefit training machine learning models based on these decentralised and private data so as to address the growing vision tasks. However, for federated learning, statistical heterogeneity continues to be a major problem. Recently, plenty of personalised federated learning methods have been explored to solve the problem of statistical heterogeneity. The usage of trained base layers and the effect of feature extraction in personalised layers, however, are hardly considered in those methods that employ the learning personalised models approach. To address the problem of the statistical heterogeneity in image analysis, PCCFED, a personalised federated learning method utilizing the strategy of parameter decoupling is proposed. It should be emphasised that the authors' personalised federated learning method decouples the personalised (P) layers into a connecting (C) layer and classifier (C) layer in order to enhance the effectiveness of feature learning for personalised layers. Further, an approach is proposed to fully use the base layers to adapt a personalised model based on the newly admitted institution's dataset through meta‐transfer. The performance of the proposed PCCFED on three datasets is evaluated under the practical non‐independent and identically distributed (non‐IID) setting. Extensive experiments demonstrate that compared with baseline methods, the proposed framework achieves the best performance in federated learning and fine‐tuning. Through FL, the investigation reveals a method to reduce statistical heterogeneity while protecting the institutions' privacy. [ABSTRACT FROM AUTHOR]

Clean energy is a major trend. The importance of photovoltaic power generation is also growing. Photovoltaic power generation is mainly affected by the weather. It is full of uncertainties. Previous work has relied chiefly on historical photovoltaics data for time series forecasts. However, unforeseen weather conditions can sometimes skew. Consequently, a spatial‐temporal‐meteorological‐long short‐term memory prediction model (STM‐LSTM) is proposed to compensate for the shortage of photovoltaic prediction models for uncertainties. This model can simultaneously process satellite image data, historical meteorological data, and historical power generation data. In this way, historical patterns and meteorological change information are extracted to improve the accuracy of photovoltaic prediction. STM‐LSTM processes raw satellite data to obtain cloud image data. It can extract cloud motion information using the dense optical flow method. First, the cloud images are processed to extract cloud position information. By adaptive attentive learning of images in different bands, a better representation for subsequent tasks can be obtained. Second, it is important to process historical meteorological data to learn meteorological change patterns. Last but not least, the historical photovoltaic power generation sequences are combined to obtain the final photovoltaic prediction results. After a series of experimental validation, the performance of the proposed STM‐LSTM model has a good improvement compared with the baseline model. [ABSTRACT FROM AUTHOR]

In this paper, a novel tensor dimensionality reduction (TDR) approach is proposed, which maintains the local geometric structure of tensor data by tensor local linear embedding and explores the global feature by optimising global subspace projection. Firstly, we analyse the local linear feature of tensor data for learning the linear separable embedding of the tenor data. Furthermore, a global subspace projection distance minimisation strategy is introduced to extract the global characteristic of the tensor data. The aim of this strategy is to find an optimal low‐dimensional subspace for TDR. In particular, two novel TDR algorithms are developed by the ensemble of tensor local feature preservation and global subspace projection distance minimisation, which express the subspace projection optimisation as an iteration optimisation problem and a Rayleigh quotient problem, respectively. The extensive experimental results on tensor data classification and clustering have demonstrated the proposed algorithms performed well. [ABSTRACT FROM AUTHOR]

Automated segmentation of tumors in breast ultrasound (US) images is challenging due to poor image quality, presence of speckle noise, shadowing effects and acoustic enhancement. This paper improves the multi-scale Gaussian kernel induced fuzzy C-means clustering method with spatial bias correction (MsGKFCM_S). Furthermore, it presents a hybrid segmentation method, using both the features of the MsGKFCM_S clustering and active contour driven by a region-scalable fitting energy function. The result obtained from the MsGKFCM_S method is utilised to initialise the contour that spreads to identify the estimated regions. It also helps to estimate the several controlling parameters of the curve evolution process. The proposed approach is evaluated on a database of 127 breast US images consisting of 75 malignant and 52 solid benign cases. The performance of proposed approach is compared with other related techniques, using performance measures such as Jaccard Index, dice similarity, shape similarity, Hausdroff difference, area difference, accuracy and F-measure. Results indicate that the proposed approach can successfully detect lesions in breast US images, with high accuracy of 97.889 and 97.513%. Moreover, the proposed approach has the capability of handling shadowing effects, acoustic enhancement and multiple lesions. [ABSTRACT FROM AUTHOR]

In India, cervical cancer is the second most common type of cancer in females. Pap smear is a simple cytology test for the detection of cancer in its early stages. To obtain the best results from the Pap smear, expert pathologist are required. Availability of pathologist in India is far below the required numbers, especially in rural parts. In this paper, multiple texturebased features are introduced for the extraction of relevant and informative features from single-cell images. First-order histogram, GLCM, LBP, Laws, and DWT are used for texture feature extraction. These methods help to recognise the contour of the nucleus and cytoplasm. ANN and SVM are used to classify the single-cell images either normal or cancerous based on the trained features. ANN and SVM are used on every single feature as well as on the combination of all features. Best results are obtained with a combination of all features. The system is evaluated on generated dataset MNITJ, containing 330 single cervical cell images and also on publicly available benchmark Herlev data set. Experimental results show that the proposed texturebased features give significantly better results in cervical cancer detection when compared with state of the art shape-based features regarding accuracy. [ABSTRACT FROM AUTHOR]

This paper addresses the 3D face reconstruction and semantic annotation from a single‐view noisy depth image. A deep neural network‐based coarse‐to‐fine framework is presented to take advantage of 3D morphable model (3DMM) regression and per‐vertex geometry refinement. The low‐dimensional subspace coefficients of the 3DMM initialize the global facial geometry, being prone to be over‐smooth because of the low‐pass characteristics of the shape subspace. The proposed geometry refinement subnetwork predicts per‐vertex displacements to enrich local details, which is learned from unlabelled noisy depth images based on the registration‐like loss. In order to guarantee the semantic correspondence between the resultant 3D face and the depth image, a semantic consistency constraint is introduced to adapt an annotation model learned from the synthetic data to real noisy depth images. The resultant depth annotations are required to be consistent with the label propagation from the coarse and refined parametric 3D faces. The proposed coarse‐to‐fine reconstruction scheme and the semantic consistency constraint are evaluated on the depth‐based 3D face reconstruction and semantic annotation. The series of experiments demonstrate that the proposed approach achieves the performance improvements over compared methods regarding 3D face reconstruction and depth image annotation. [ABSTRACT FROM AUTHOR]