Search

Small moving objects at far distance always occupy only one or a few pixels in image and exhibit extremely limited visual features, which bring great challenges to motion detection. Highly evolved visual systems endow flying insects with remarkable ability to pursue tiny mates and prey, providing a good template to develop image processing method for small target motion detection. The insects’ excellent sensitivity to small moving objects is believed to come from a class of specific neurons called small target motion detectors (STMDs). However, existing STMD-based methods often experience performance degradation when coping with complex natural scenes. In this paper, we propose a bio-inspired visual system with spatio-temporal feedback mechanism (called Spatio-Temporal Feedback STMD) to suppress false positive background movement while enhancing system responses to small targets. Specifically, the proposed visual system is composed of two complementary subnetworks and a feedback loop. The first subnetwork is designed to extract spatial and temporal movement patterns of cluttered background by neuronal ensemble coding. The second subnetwork is developed to capture small target motion information where its output and signal from the first subnetwork are integrated together via the feedback loop to filter out background false positives in a recurrent manner. Experimental results demonstrate that the proposed spatio-temporal feedback visual system is more competitive than existing methods in discriminating small moving targets from complex natural environments. [ABSTRACT FROM AUTHOR]

We present our automated real-time socket inspection system capable of detecting an assortment of defects including metallic and liquid staining, loose capacitors and pins, and other debris and foreign material (FM). Our test tools pick and place manufactured units into sockets for electrical testing. Any debris accumulated inside the test sockets will likely damage subsequent units until the defective socket is replaced. To quickly capture in-situ defects and mitigate further damage, we equipped each pick-and-place arm with a new vision system designed to fit within the existing tool. The tight footprint constraints required a highly compact imaging system which resulted in a variety of image artifacts, creating several unique challenges. Our inspection algorithm utilizes a variety of advanced computer vision and machine learning techniques to normalize images, extract and match features, register the images, suppress unwanted artifacts, and detect defects. The detected changes are then sent to a deep learning classifier to further filter between true defects and natural socket deterioration. The flagged socket images can be manually dispositioned by the user and the socket can be sent for repair or cleaning as needed. [ABSTRACT FROM AUTHOR]

Intelligent manufacturing raises higher requirements for tool condition monitoring (TCM) in terms of accuracy, robustness, and adaptability. At present, direct methods based on image processing and deep learning have made breakthroughs in TCM. However, some issues, such as image quality, model parameters, and dataset scale in the abovementioned methods, restrict industrial applications of TCM. Regarding the abovementioned issue, the purpose of this article is to propose a lightweight network model based on multiple activation functions to promote the intelligent industrial application of TCM. First, the image quality mechanism caused by complex working conditions is analyzed in industrial environments. Correspondingly, data augmentation is adopted to solve the problem of data scale under the premise of ensuring data quality and richness. Then, the adaptive activation function and the hard version of swish are introduced at the front and second half of the network to avoid information loss and reduce the activation function cost. Finally, a lightweight network based on cloud-edge collaboration for TCM is constructed. The model is iteratively optimized in the cloud and inferenced on the edge embedded device. The accuracy and adaptability of the proposed network are verified by accelerating milling cutter life under multiple working conditions. [ABSTRACT FROM AUTHOR]

Classification of 3D objects – the selection of a category in which each object belongs – is of great interest in the field of machine learning. Numerous researchers use deep neural networks to address this problem, altering the network architecture and representation of the 3D shape used as an input. To investigate the effectiveness of their approaches, we conduct an extensive survey of existing methods and identify common ideas by which we categorize them into a taxonomy. Second, we evaluate 11 selected classification networks on two 3D object datasets, extending the evaluation to a larger dataset on which most of the selected approaches have not been tested yet. For this, we provide a framework for converting shapes from common 3D mesh formats into formats native to each network, and for training and evaluating different classification approaches on this data. Despite being partially unable to reach the accuracies reported in the original papers, we compare the relative performance of the approaches as well as their performance when changing datasets as the only variable to provide valuable insights into performance on different kinds of data. We make our code available to simplify running training experiments with multiple neural networks with different prerequisites. [ABSTRACT FROM AUTHOR]

Defect pattern detection and classification are challenging for thin-film-transistor liquid-crystal display (TFT-LCD) manufacturing. Limitations of the existing solutions for automatic optical inspection can be traced in part to the lack of a framework within which different existing and new defect patterns can be analyzed, while integrating domain knowledge and effective technologies. This study aims to develop a framework for image-based defect classification that employs the convolution neural networks without using complex and time-consuming image-processing processes in advance. An empirical study was conducted in a leading TFT-LCD manufacturing in Taiwan for validation. The results have shown that the defect patterns can be effectively classified by the proposed convolutional neural networks that outperform the existing approaches such as Support Vector Machine and Random Forest. The developed solution is implemented to effectively support the engineers. [ABSTRACT FROM AUTHOR]

Image quality assessment (IQA) is an important task of image processing and has diverse applications, such as image super-resolution reconstruction, image transmission and monitoring systems. This paper proposes a perceptual hashing algorithm with complementary color wavelet transform (CCWT) and compressed sensing (CS) for reduced-reference (RR) IQA. The CCWT is exploited to decompose input color image into different sub-bands. Since the calculation of CCWT uses all color channels without discarding any information, the distortions introduced by digital operations on color channels are preserved in the CCWT sub-bands. The block-based CS is used to extract features from the CCWT sub-bands. As the Euclidean distance between the block-based CS features is slightly influenced by content-preserving operations, perceptual features constructed by Euclidean distances are robust, discriminative and compact. Hash sequence is finally determined by quantifying the perceptual features. Effectiveness of the proposed hashing is verified by various experiments on four open image databases. Experimental results demonstrate that the proposed hashing is superior to some state-of-the-art algorithms in terms of classification and RR IQA application. [ABSTRACT FROM AUTHOR]

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA. [ABSTRACT FROM AUTHOR]

Deep learning (DL) methods usually need to collect a large amount of labeled data to extract deep features. However, due to the difficulty of obtaining numerous labeled data from synthetic aperture radar (SAR) images, unsupervised feature learning has been focused on SAR image processing. In this article, we propose a 3-D sparse model (3-DSM) to extract deep sparse features from SAR images in an unsupervised way. Concretely, 3-DSM learns the convolution kernels by minimizing the error between the features and the constructed sparse maps, without labeled samples. Thus, the discriminative features can be extracted in an unsupervised way by the learned convolution kernels and are able to capture the main structural information of SAR images. Furthermore, to the best of our knowledge, 3-DSM first specifies the sparsity of convolution kernels, with each convolution kernel exhibiting its independence from the others and the redundancy of convolution kernels being diminishing. It means that each convolution kernel extracts its unique structural features of SAR images. Consequently, in the feature extraction, 3-D sparsities have been specified, including width, height, and depth, with the acquisition of discriminative less-redundant features. The effectiveness of 3-DSM is demonstrated by the feature extraction and segmentation of the simulated and real SAR images. [ABSTRACT FROM AUTHOR]

Saliency detection by human refers to the ability to identify pertinent information using our perceptive and cognitive capabilities. While human perception is attracted by visual stimuli, our cognitive capability is derived from the inspiration of constructing concepts of reasoning. Saliency detection has gained intensive interest with the aim of resembling human ‘perceptual’ system. However, saliency related to human ‘cognition’, particularly the analysis of complex salient regions (‘cogitating’ process), is yet to be fully exploited. We propose to resemble human cognition, coupled with human perception, to improve saliency detection. We recognize saliency in three phases (‘Seeing’ - ‘Perceiving’ - ‘Cogitating), mimicking human's perceptive and cognitive thinking of an image. In our method, ‘Seeing’ phase is related to human perception, and we formulate the ‘Perceiving’ and ‘Cogitating’ phases related to the human cognition systems via deep neural networks (DNNs) to construct a new module (Cognitive Gate) that enhances the DNN features for saliency detection. To the best of our knowledge, this is the first work that established DNNs to resemble human cognition for saliency detection. In our experiments, our approach outperformed 17 benchmarking DNN methods on six well-recognized datasets, demonstrating that resembling human cognition improves saliency detection. [ABSTRACT FROM AUTHOR]

The automatic extraction of lake water is one of the research hotspots in the field of remote sensing image processing. Due to the small interclass variance between lakes and other ground objects, and the complex texture characteristics of lake boundaries, existing methods often have problems such as over-segmentation and inaccurate boundary segmentation when segmenting lake water bodies. To alleviate these problems, this article designs an end-to-end semantic segmentation network [noise-canceling transformer network (NT-Net)] for the automatic extraction of lake water bodies from remote sensing images. Aiming at the problem of over-segmentation caused by nonlake objects, an interference attenuation module is designed in the network. This module can model the key features that are distinguishable and suitable for segmenting lake water by analyzing the difference in feature representation between lakes and other ground objects, thus suppressing the feature representation of nonlake objects. To more accurately segment the lake boundary, a multilevel transformer module is designed. This module can capture the context association of boundary information and enhance the feature representation of boundary information by using the self-attention mechanism. The comparative experimental results show that, compared with the current mainstream semantic segmentation networks, the method in this article has advantages in extracting lake water bodies comprehensively and coherently. [ABSTRACT FROM AUTHOR]

Change detection for remote sensing images involves detecting regional surface changes of interest between two images taken of the same geographical area but at different times. In image processing, the spatial domain uses grayscale values to describe an image. The frequency is directly related to the spatial change rate, so the frequency domain can be intuitively associated with patterns of intensity variations in the image. These two domains provide different perspectives for image interpretation. Most existing deep-learning-based methods formulate change detection as a pixel-wise binary classification problem and utilize various strategies to extract information in the spatial domain. However, they rarely pay attention to the rich information in the frequency domain. To address this problem, we propose an end-to-end joint frequency-spatial domain network (JFSDNet) to implement remote sensing optical image change detection. Specifically, we introduce frequency information into the change detection to supplement the loss of image details caused by downsampling. In addition, we employ a frequency selection module to adaptively discriminate and choose frequency clues by reducing the complexity of the frequency features. The JFSDNet is applied to two publicly available datasets: the change-detection dataset (CDD) dataset and the LEarning VIsion and Remote sensing Change Detection (LEVIR-CD) dataset. Compared with other methods, both visual interpretation and quantitative assessment confirmed that our proposed method achieved a favorable performance. [ABSTRACT FROM AUTHOR]

Deep learning models have shown great potential in remote sensing (RS) image processing and analysis. Nevertheless, there are insufficient labeled samples to train deep networks, which seriously affects the performance of these models. To resolve this contradiction, we propose a generative self-supervised feature learning (S2FL) architecture for multimodal RS image land cover classification. Specifically, multiple complementary observed views are constructed from multimodal RS images, which are employed for following generative self-supervised learning (SSL). The proposed S2FL architecture is capable of extracting high-level meaningful feature representations from multiview data, and this process does not require any labeled information, providing a feasible solution to relieve the urgent need for annotated samples. The learned features are normalized and merged with corresponding spectral information to further improve the discriminative capability of feature representations, and we utilize these fused features for land cover classification. Compared with the existing supervised, semi-supervised, and self-supervised approaches, the proposed generative self-supervised model achieves superior performance in terms of feature learning and land cover classification, especially in the small sample classification case. [ABSTRACT FROM AUTHOR]

Missing data reconstruction is a classical, yet challenging problem in remote sensing (RS) image processing due to the complex atmospheric environment and variability of satellite sensors. Most of the contemporary reconstruction methods either handle only one specific task or require supplementary data, while the single input for multitask reconstruction has not been explored yet. In this article, we propose a novel generative adversarial network-based unified framework for missing RS image reconstruction, which is capable of various reconstruction tasks given only single-source data as input. Specifically, we first propose a mask extraction network (MEN) to obtain a united soft mask, which represents the intrinsic prior under various scenarios and indicates not only location, but also context information. The versatility of mask extraction enables the multitask reconstruction of RS images. Besides, we propose a unified inpainting network (UIN) to repair diverse degraded images. Being specifically tailored for RS images, dilated pyramidal convolutions (DPCs) and an attention fusion mechanism (AFM) are introduced to further improve the feature extraction ability and thus exhaustly leveraging the single-input information. Extensive experiments demonstrate the uncompromising performance of the proposed method against state-of-the-art multiinput methods on diverse missing restoration. Moreover, further exploration shows the potential of the proposed method to utilize joint spatial–spectral–temporal information, which is evaluated to outperform existing competitors on remote sense images. [ABSTRACT FROM AUTHOR]

Arbitrary style transfer aims at migrating the style of a reference style painting to a target content image. Existing methods find it challenging to achieve good content fidelity and style migration at the same time. Moreover, they all rely on manually defined content and style, which is of limited universality and robustness. In this paper, we propose to introduce contrastive learning into style transfer, instructing the network to automatically learn to model the structural content and artistic style based on natural contrastive relationships in style transfer. Compared with existing methods, our learned modeling of content and style is more robust and universal. In addition, we further propose instance-wise contrastive style losses and a patch-wise contrastive content loss to guide style transfer. Combining the proposed contrastive losses and two self-reconstruction strategies, we develop a new style transfer framework, which is pluggable and can be flexibly applied to various style transfer modules. Experimental results demonstrate that our method has strong flexibility and synthesizes stylized images with higher quality. [ABSTRACT FROM AUTHOR]

Feature matching, which refers to identifying and then corresponding the same or similar visual pattern from two or more images, is a key technique in any image processing task that requires establishing good correspondences between images. Given potential correspondences (matches) in two scenes, a novel whole-part deep learning framework, termed as Context Structure Representation Network (CSR-Net), is designed to infer the probabilities of arbitrary correspondences being inliers. Traditional approaches commonly build the local relation between correspondences by manually engineered criteria. Different from existing attempts, the main idea of our work is to learn explicitly neighborhood structure of each correspondence, allowing us to formulate the matching problem into a dynamic local structure consensus evaluation in an end-to-end fashion. For this purpose, we propose a permutation-invariant STructure Representation (STR) learning module, which can easily merge different types of networks into a unified architecture to deal with sparse matches directly. By the collaborative use of STR, we introduce a Context-Aware Attention (CAA) mechanism to adaptively re-calibrate structure features via a rotation-invariant context aware encoding and simple feature gating, thus arising the ability of fine-grained patterns recognition. Moreover, to further weaken the cost of establishing reliable correspondences, the CSR-Net is formulated as whole-part consensus learning, where the aim of whole level is compensating rigid transformations. In order to demonstrate our CSR-Net can effectively boost the baselines, we intensively experiment on image matching and other visual tasks. The results of the experiment confirm that the matching performances of CSR-Net have significantly improved over nine state-of-the-art competitors. [ABSTRACT FROM AUTHOR]

3D volumetric image processing has attracted increasing attention in the last decades, in which one major research area is to develop efficient lossless volumetric image compression techniques to better store and transmit such images with massive amount of information. In this work, we propose the first end-to-end optimized learning framework for losslessly compressing 3D volumetric data. Our approach builds upon a hierarchical compression scheme by additionally introducing the intra-slice auxiliary features and estimating the entropy model based on both intra-slice and inter-slice latent priors. Specifically, we first extract the hierarchical intra-slice auxiliary features through multi-scale feature extraction modules. Then, an Intra-slice and Inter-slice Conditional Entropy Coding module is proposed to fuse the intra-slice and inter-slice information from different scales as the context information. Based on such context information, we can predict the distributions for both intra-slice auxiliary features and the slice images. To further improve the lossless compression performance, we also introduce two new gating mechanisms called Intra-Gate and Inter-Gate to generate the optimal feature representations for better information fusion. Eventually, we can produce the bitstream for losslessly compressing volumetric images based on the estimated entropy model. Different from the existing lossless volumetric image codecs, our end-to-end optimized framework jointly learns both intra-slice auxiliary features at different scales for each slice and inter-slice latent features from previously encoded slices for better entropy estimation. The extensive experimental results indicate that our framework outperforms the state-of-the-art hand-crafted lossless volumetric image codecs (e.g., JP3D) and the learning-based lossless image compression method on four volumetric image benchmarks for losslessly compressing both 3D Medical Images and Hyper-Spectral Images. [ABSTRACT FROM AUTHOR]

Cartoonization as a special type of artistic style transfer is a difficult image processing task. The current existing artistic style transfer methods cannot generate satisfactory cartoon-style images due to that artistic style images often have delicate strokes and rich hierarchical color changes while cartoon-style images have smooth surfaces without obvious color changes, and sharp edges. To this end, we propose a cartoon loss based generative adversarial network (CartoonLossGAN) for cartoonization. Particularly, we first reuse the encoder part of the discriminator to build a compact generative adversarial network (GAN) based cartoonization architecture. Then we propose a novel cartoon loss function for the architecture. It can imitate the process of sketching to learn the smooth surface of the cartoon image, and imitate the coloring process to learn the coloring of the cartoon image. Furthermore, we also propose an initialization strategy, which is used in the scenario of reusing the discriminator to make our model training easier and more stable. Extensive experimental results demonstrate that our proposed CartoonLossGAN can generate fantastic cartoon-style images, and outperforms four representative methods. [ABSTRACT FROM AUTHOR]

As two practical and important image processing tasks, color demosaicking (CDM) and super-resolution (SR) have been studied for decades. However, most literature studies these two tasks independently, ignoring the potential benefits of a joint solution. In this paper, aiming at efficient and effective joint demosaicking and super-resolution (JDSR), a well-designed two-stage convolutional neural network (CNN) architecture is proposed. For the first stage, by making use of the sampling-pattern information, a pattern-aware feature extraction (PFE) module extracts features directly from the Bayer-sampled low-resolution (LR) image, while keeping the resolution of the extracted features the same as the input. For the second stage, a dual-branch feature refinement (DFR) module effectively decomposes the features into two components with different spatial frequencies, on which different learning strategies are applied. On each branch of the DFR module, the feature refinement unit, namely, densely-connected dual-path enhancement blocks (DDEB), establishes a sophisticated nonlinear mapping from the LR space to the high-resolution (HR) space. To achieve strong representational power, two paths of transformations and the channel attention mechanism are adopted in DDEB. Extensive experiments demonstrate that the proposed method is superior to the sequential combination of state-of-the-art (SOTA) CDM and SR methods. Moreover, with much smaller model size, our approach also surpasses other SOTA JDSR methods. [ABSTRACT FROM AUTHOR]

Joint photographic experts group (JPEG) compression is widely used in image processing and computer vision. Detecting double compressed JPEG images is a common problem in forensics and detecting compressed images with the same quantization matrix remains a challenging task. However, most existing methods were designed for detection in grayscale images and cannot fully use the unique characteristics of color images (such as the relationship between channels and color information). In addition, the performance of existing methods is unsatisfactory for low JPEG quality factors and in cross detection experiments. To solve these problems, we analyze the stability of a color image to obtain the convergence error and transposition error. According to the convergence characteristics of color JPEG images, the continuous compression by the same quantization matrix can make the JPEG image tend to be stable. The final stable state and the convergence process are determined by the number of compressions of the original image. Thus, continuously compressed JPEG images can be regarded as a continuous frame to obtain the convergence error. As the color image converges, its ability to resist interference decreases. To reflect the changes in anti-interference ability, the transposition operation is used to disturb the color JPEG image to obtain the transposition error. In addition, quaternion mapping is used to retain the relationship between continuously compressed JPEG images and enlarge the influence caused by transposition operation. In our experiments on several image databases, the proposed method outperforms existing methods in different settings. [ABSTRACT FROM AUTHOR]

Cloud and cloud shadow segmentation is one of the most important issues in remote sensing image processing. Most of the remote sensing images are very complicated. In this work, a dual-branch model composed of transformer and convolution network is proposed to extract semantic and spatial detail information of the image, respectively, to solve the problems of false detection and missed detection. To improve the model’s feature extraction, a mutual guidance module (MGM) is introduced, so that the transformer branch and the convolution branch can guide each other for feature mining. Finally, in view of the problem of rough segmentation boundary, this work uses different features extracted by the transformer branch and the convolution branch for decoding and repairs the rough segmentation boundary in the decoding part to make the segmentation boundary clearer. Experimental results on the Landsat-8, Sentinel-2 data, the public dataset high-resolution cloud cover validation dataset created by researchers at Wuhan University (HRC_WHU), and the public dataset Spatial Procedures for Automated Removal of Cloud and Shadow (SPARCS) demonstrate the effectiveness of our method and its superiority to the existing state-of-the-art cloud and cloud shadow segmentation approaches. [ABSTRACT FROM AUTHOR]

The wire bonding process is one of the most critical processes in semiconductor packaging. The electrical performance and reliability of IC chips must be evaluated by probe testing before wire bonding. In the probe test, the probe usually leaves traces on the surface of the pad. Large probe marks tend to cause a decrease in bond adhesion and so increase the possibility of ball bond lifting from the pads. Shear force is an important measure of the adhesion between the ball and pad. The prediction of the shear force helps to understand the bonding quality in advance. In this study, the six features of the probe marks were extracted by the automatic image recognition method. Using three machine learning techniques (logistic regression, support vector machine, and random forest) based on principal component analysis (PCA), the shear force is estimated based on six features before wire bonding. The results indicate that the proposed PCA-based random forest could identify bad chips with 97.92% accuracy before wire bonding. Unnecessary chips can be found early, and unnecessary wire bonding can be avoided, thereby saving process time and cost. The proposed method would improve the efficiency and quality yield of the semiconductor packaging process. [ABSTRACT FROM AUTHOR]

Spatial failure patterns in wafer defect maps are very useful for root cause analysis, which is essential for yield optimization in the manufacturing process. Especially, the scratch defect type is the most challenging pattern to recognize because the position, shape, size and curvature vary widely from one scratch to another. Discontinuity points within scratches also contribute to the low recognition rate, and such points are often hidden defective dies that become reliability threat. Previous studies on defect pattern identification show that the recognition rate for scratches is among the lowest in all patterns even if the overall accuracy is high. In this paper, we propose a novel scratch pattern recognition method. In this paper, image processing techniques are applied to recognize scratch patterns in wafer maps. The Hough transform is first employed to identify line segments, from which scratches can be reconstructed. In contrast to previous machine-learning based methods, there is no need to train a complicated prediction model so that the computation time is small. The method is validated by using wafer maps from six real products and public data. Experimental results show that the proposed method achieves high recall, precision and accuracy in all cases. [ABSTRACT FROM AUTHOR]

This work proposes an interpretable radiomics approach to differentiate between malignant and benign focal liver lesions (FLLs) on contrast-enhanced ultrasound (CEUS). Although CEUS has shown promise for differential FLLs diagnosis, current clinical assessment is performed only by qualitative analysis of the contrast enhancement patterns. Quantitative analysis is often hampered by the unavoidable presence of motion artifacts and by the complex, spatiotemporal nature of liver contrast enhancement, consisting of multiple, overlapping vascular phases. To fully exploit the wealth of information in CEUS, while coping with these challenges, here we propose combining features extracted by the temporal and spatiotemporal analysis in the arterial phase enhancement with spatial features extracted by texture analysis at different time points. Using the extracted features as input, several machine learning classifiers are optimized to achieve semiautomatic FLLs characterization, for which there is no need for motion compensation and the only manual input required is the location of a suspicious lesion. Clinical validation on 87 FLLs from 72 patients at risk for hepatocellular carcinoma (HCC) showed promising performance, achieving a balanced accuracy of 0.84 in the distinction between benign and malignant lesions. Analysis of feature relevance demonstrates that a combination of spatiotemporal and texture features is needed to achieve the best performance. Interpretation of the most relevant features suggests that aspects related to microvascular perfusion and the microvascular architecture, together with the spatial enhancement characteristics at wash-in and peak enhancement, are important to aid the accurate characterization of FLLs. [ABSTRACT FROM AUTHOR]

Remote sensing images, especially hyperspectral images (HSIs), are extremely vulnerable to random noise and stripe noise. As a key aspect of HSI data quality improvement, stripe noise removal has always been a pervasive issue in remote sensing image processing. Convolutional neural networks have been applied for HSI data destriping. However, the existing methods lose the stripe-free component of the original image to a certain extent. These models also ignore the global spatial context of images and the correlation between spatial information and spectral information. Therefore, we propose a novel destriping convolutional network to overcome the problems with the existing methods. Octave convolution is used to extract cross-frequency features, and separate and compress the low-frequency information of the images, while dilation convolution (Dila-Conv) is used to reduce the amount of required calculation and also preserve the key image information. In addition, Dila-Conv can expand the receptive field to obtain multiscale features. Finally, a cross-channel enhanced spatial–spectral feature fusion module is used to acquire and integrate spatial context information and interchannel dependencies on a global scale as auxiliary information so that the network model can learn and pay attention to key feature information, specifically, “what to look for” and “where to look at,” which can facilitate the distinction between stripe and stripe-free components. Experimental results obtained using multiple datasets demonstrated that the proposed method can outperform the existing comparable methods and can produce satisfactory results in terms of visual effects and quantitative evaluation. [ABSTRACT FROM AUTHOR]

The profound impact of deep learning and particularly of convolutional neural networks (CNNs) in automatic image processing has been decisive for the progress and evolution of remote sensing (RS) hyperspectral imaging (HSI) processing. Indeed, CNNs have stated themselves as the current state of the art, reaching unparalleled results in HSI classification. However, most CNNs were designed for RGB images, and their direct application to HSI data analysis could lead to nonoptimal solutions. Moreover, CNNs perform classification based on the identification of specific features, neglecting the spatial relationships between different features (i.e., their arrangement) due to pooling techniques. The capsule network (CapsNet) architecture is an attempt to overcome this drawback by nesting several neural layers within a capsule, connected by dynamic routing, both to identify not only the presence of a feature but also its instantiation parameters and to learn the relationships between different features. Although this mechanism improves the data representations, enhancing the classification of HSI data, it still acts as a black box, without control of the most relevant features for classification purposes. Indeed, important features could be discriminated against. In this article, a new multiple attention-guided CapsNet is proposed to improve feature processing for RS-HSIs’ classification, both to improve computational efficiency (in terms of parameters) and increase accuracy. Hence, the most representative visual parts of the images are identified using a detailed feature extractor coupled with attention mechanisms. Extensive experimental results have been obtained on five real datasets, demonstrating the great potential of the proposed method compared to other state-of-the-art classifiers. [ABSTRACT FROM AUTHOR]

Feature matching, which aims to seek good correspondences from an image pair of the same or similar scene, is one of the important studies on digital remote sensing (RS) image processing. However, alongside the common degradation problems, such as geometric distortion, RS images also often face nonlinear radiation distortions, thereby posing more complex matching patterns. To reduce the cost of establishing reliable correspondences, this article proposes a simple, but effective end-to-end hierarchical learning framework, termed interactive generative structure learning network (IGS-Net). The key thinking of our approach is to offer a structure self-generate learning mechanism, called interactive generative structure learning (IGSL) block, for modeling the local context information of potential correspondences. Specifically, IGSL contains two novel operations: adaptive structure-aware representation (ASR) and physical constraint embedding. Besides, we introduce a coarse-to-fine geometry estimation pipeline aligning two sets of feature points to weaken the degree of randomness in matching patterns, thus improving the generalization ability of representation learning. Overall, this differentiable representation learning architecture can be inserted into existing classification models easily for robust outlier detection and removal. In order to demonstrate that our IGS-Net can boost the baselines, we intensively experiment on both single modality and multimodal RS image datasets. The large amounts of experiment results reveal that the matching performances of IGS-Net are significantly improved over eight state-of-the-art competitors. [ABSTRACT FROM AUTHOR]

Retrieving 3D shapes based on 2D images is a challenging research topic, due to the significant gap between different domains. Recently, various approaches have been proposed to handle this problem. However, the majority of methods target the cross-domain retrieval task as a pure domain adaptation problem, which focuses on the alignment but ignores the visual relevance between the 2D images and their corresponding 3D shapes. To fundamentally decrease the divergence between different domains, we propose a novel cross-domain learning network (CLN) for 2D image-based 3D shape retrieval task. First, we estimate the pose information from the 2D image to guide the view rendering of 3D shapes, which increases the visual correlations of the cross-domain data to eliminate the divergence between them. Second, we introduce a novel joint learning network, considering both the domain-specific characteristics and the cross-domain interactions for data alignment, which further compensates for the gap between different domains by controlling the distance of intra- and inter-classes. After the metric learning process, discriminative descriptors of images and shapes are generated for the cross-domain retrieval task. To prove the effectiveness and robustness of the proposed method, we conduct extensive experiments on the MI3DOR, SHREC’13, and SHREC’14 datasets. The experimental results demonstrate the superiority of our proposed method, and significant improvements have been achieved compared with state-of-the-art methods. [ABSTRACT FROM AUTHOR]

An angular clustering method based on image processing is proposed in this article. It is used to identify clusters in 2-D representations of propagation channels. The approach uses operations such as watershed segmentation and is particularly well suited for clustering directional channels obtained by beam steering at millimeter wave. This situation occurs for instance with electronic beam steering using analog antenna arrays during beam-training process or channel modeling measurements using either electronic or mechanical beam steering. In particular, the proposed technique is used here to cluster 2-D power angular spectrum maps. The proposed clustering is unsupervised and is well suited to preserve the shape of clusters by considering the angular connection between neighbor samples, which is useful to obtain more accurate descriptions of channel angular properties. The approach is found to outperform approaches based on K-power-means in terms of accuracy as well as computational resources. The technique is assessed in simulation using the IEEE 802.11ad channel model and in measurement using experiments conducted at 60 GHz in an indoor environment. [ABSTRACT FROM AUTHOR]

Object detection from remote sensing images (RSIs) is a basic topic in the area of aviation and satellite image processing, which has great effects on geological disaster detection, agricultural planning, and land utilization. However, it is always faced with several severe difficulties. For instance, the scale of the target spans over a very wide range, and the difference between the target size and the image size is huge, as some targets only account for a dozen pixels compared with the RSI of the megapixel level. In this work, an innovative object detection network (GLNet) is proposed for remote sensing imagery. Our approach integrates global context clues extracted by the multiscale perception (MSP) module and local spatial contextual correlations encoded by Clip long short-term memory (Clip-LSTM). These rich semantic features are further exploited to design a self-adapted anchor (SA) module to alleviate the scale variations in RSIs. Extensive experiments are executed on several public easily accessed benchmarks, including DOTA, NWPU VHR-10, and DIOR. Experimental results have demonstrated that our GLNet outperforms numerous latest methods. [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (CNNs) have emerged as a new paradigm for Mammogram diagnosis. Contemporary CNN-based computer-aided-diagnosis systems (CADs) for breast cancer directly extract latent features from input mammogram image and ignore the importance of morphological features. In this paper, we introduce a novel end-to-end deep learning framework for mammogram image processing, which computes mass segmentation and simultaneously predicts diagnosis results. Specifically, our method is constructed in a dual-path architecture that solves the mapping in a dual-problem manner, with an additional consideration of important shape and boundary knowledge. One path, called the Locality Preserving Learner (LPL), is devoted to hierarchically extracting and exploiting intrinsic features of the input. Whereas the other path, called the Conditional Graph Learner (CGL), focuses on generating geometrical features via modeling pixel-wise image to mask correlations. By integrating the two learners, both the cancer semantics and cancer representations are well learned, and the component learning paths in return complement each other, contributing an improvement to the mass segmentation and cancer classification problem at the same time. In addition, by integrating an automatic detection set-up, the DualCoreNet achieves fully automatic breast cancer diagnosis practically. Experimental results show that in benchmark DDSM dataset, DualCoreNet has outperformed other related works in both segmentation and classification tasks, achieving 92.27% DI coefficient and 0.85 AUC score. In another benchmark INbreast dataset, DualCoreNet achieves the best mammography segmentation (93.69% DI coefficient) and competitive classification performance (0.93 AUC score). [ABSTRACT FROM AUTHOR]

The development of deep learning has brought new prospects into the field of hyperspectral classification, and the classification ability of this method for the classification of hyperspectral images (HSIs) has been continuously improved. However, there are still some problems that must be solved; for example, the spectral–spatial features of HSIs are not effectively extracted, the labeled samples in the data set are limited, and the number of samples in different categories is imbalanced. To facilitate the progress of hyperspectral classification, a spectral–spatial fractal residual convolutional neural network with data balance augmentation is proposed here. In this network, a data balance augmentation approach that can solve the problems of limited labeled data and imbalanced categories is proposed. In addition, the spectral–spatial residual module is proposed to learn the spectral–spatial information and alleviate the problem of model degradation effectively. In addition, the spectral–spatial focal structure, which can guarantee the integrity of the information, is introduced. Moreover, the spectral–spatial dimensional transformation module, which can reduce the size and number of hyperspectral feature maps without losing the fine features, is presented. In particular, the proposed network has a strong ability to classify the categories that have a small number of samples and reaches the state-of-the-art level for three benchmark data sets. [ABSTRACT FROM AUTHOR]

In the new wave of artificial intelligence, deep learning is impacting various industries. As a closely related area, optimization algorithms greatly contribute to the development of deep learning. But the reverse applications are still insufficient. Is there any efficient way to solve certain optimization problem through deep learning? The key is to convert the optimization to a representation suitable for deep learning. In this article, a traveling salesman problem (TSP) is studied. Considering that deep learning is good at image processing, an image representation method is proposed to transfer a TSP to an image. Based on samples of a ten city TSP, a fully convolutional network (FCN) is used to learn the mapping from a feasible region to an optimal solution. The training process is analyzed and interpreted through stages. A visualization method is presented to show how an FCN can understand the training task of a TSP. Once the training is completed, no significant effort is required to solve a new TSP and the prediction is obtained on the scale of milliseconds. The results show good performance in finding the global optimal solution. Moreover, the developed FCN model has been demonstrated on TSP’s with different city numbers, proving excellent generalization performance. [ABSTRACT FROM AUTHOR]

Feature discretization is one of the most relevant techniques for data preprocessing in remote sensing research area. Its main goal is to transform the continuous features of images into discrete ones to improve the efficiency of intelligent image processing algorithms, thus helping experts to more easily understand and use the acquired remote sensing data. In this article, we focus on feature discretization for classification of high-resolution remote sensing images in coastal areas. In these images, interactions among multiple bands exist, noises interfere, and maritime domain-specific prior knowledge is difficult to get. To address these challenges, we propose a hybrid metric method, based on information entropy and chi-square test, to calculate the stability of the discrete interval and the similarity of adjacent intervals. In addition, we use the degree of dependence among knowledge from the rough set theory as the evaluation criterion for discretization schemes and then scan each band in turn with the strategy of first splitting then merging, to obtain the optimal set of discrete features. Our method has been compared with the best state-of-the-art discretization algorithms on the GF-2 and Landsat 8 satellite datasets. Experiments show that the proposed method achieves better classification accuracy for high-resolution remote sensing images in coastal areas. It can not only effectively mine the correlation between features but also filter the outliers in bands, thus producing as few discrete intervals as possible while ensuring data consistency. [ABSTRACT FROM AUTHOR]

Multi-view learning has become a significant research topic in image processing, data mining and machine learning due to the proliferation of multi-view data. Considering the difficulty in obtaining labeled data in many real applications, we focus on the multi-view unsupervised feature selection problem. Most existing multi-view feature selection introduce an identical similarity matrix among different views, which cannot preserve the specific correlation between each single view. Also, some of these methods just consider either global or local structures. In this paper, we propose an embedding method, Adaptive Similarity Embedding for Unsupervised Multi-View Feature Selection (ASE-UMFS). This method reduces the high-dimensional data to the low dimensions and unifies different views to a combination weight matrix. We also use parameters to constraint the similarity matrix for the local structure, where the regularization term is used to add a prior of uniform distribution; taking into account of the independence in projection matrix among different views, optimization of the similarity matrix is further improved. To confirm the effectiveness of ASE-UMFS, comparisons are made with benchmark algorithm on real-world data sets. The experimental results demonstrate that the proposed algorithm outperforms several state-of-the-art methods in multi-view learning. [ABSTRACT FROM AUTHOR]

With the development of video and image processing technology, the field of video tampering forensics is facing enormous challenges. Specifically, as the fundamental basis of judicial forensics, passive forensics for object removal video forgery is particularly essential. To extract tampering traces in video more sufficiently, the author proposed a spatiotemporal trident network based on the spatial rich model (SRM) and 3D convolution (C3D), which provides three branches and can theoretically improve the detection and localization accuracy of tampered regions. Based on the spatiotemporal trident network, a temporal detector and a spatial locator were designed to detect and locate the tampered regions in the temporal and spatial domains of videos. For the temporal detector, 3D CNNs were employed in three branches as the encoders and a bidirectional long short-term memory (BiLSTM) as the decoder. For the spatial locator, a backbone network named C3D-ResNet12 was designed as the encoder of the three branches, and the region proposal networks (RPNs) were employed as the decoders in three branches. In addition, we optimized the loss functions of the above two algorithms based on focal loss and GIoU loss. The experimental results revealed the effectiveness of spatiotemporal detection and localization algorithms: for temporal forgery detection, the accuracy of the frame classification increased to 99+%; for spatial forgery localization, the successful localization rate of the tampered regions in forged frames reached 96+%, and the mean intersection over union of the located tampered regions and the real tampered regions reached 62+%. [ABSTRACT FROM AUTHOR]

This article develops a deep learning object detector to provide accurate six-degree-of-freedom (DoF) information of the drogue relative to a monocular camera onboard a flying unmanned aerial vehicle. An object detector helps to provide the needed information for an autonomous vehicle to dock and refuel without the need for human intervention. This object detector can detect eight different beacons by training on 8746 images of a mock drogue. Once these beacons were detected, a nonlinear least squares algorithm that uses the collinearity equations as a system model takes the beacon's location on the captured image to provide an accurate six-DoF navigation solution. These navigation solutions from the object detector were evaluated on multiple metrics and then compared to navigation solutions provided by a VICON motion tracking system. Finally, Monte Carlo analysis, using the collinearity equations as a system model, evaluates an object detector's performance with various noise levels. [ABSTRACT FROM AUTHOR]

In recent years, Salient Object Detection (SOD) has shown great success with the achievements of large-scale benchmarks and deep learning techniques. However, existing SOD methods mainly focus on natural images with low-resolutions, e.g., $400\times 400$ or less. This drawback hinders them for advanced practical applications, which need high-resolution, detail-aware results. Besides, lacking of the boundary detail and semantic context of salient objects is also a key concern for accurate SOD. To address these issues, in this work we focus on the High-Resolution Salient Object Detection (HRSOD) task. Technically, we propose the first end-to-end learnable framework, named Dual ReFinement Network (DRFNet), for fully automatic HRSOD. More specifically, the proposed DRFNet consists of a shared feature extractor and two effective refinement heads. By decoupling the detail and context information, one refinement head adopts a global-aware feature pyramid. Without increasing too much computational burden, it can boost the spatial detail information, which narrows the gap between high-level semantics and low-level details. In parallel, the other refinement head adopts hybrid dilated convolutional blocks and group-wise upsamplings, which are very efficient in extracting contextual information. Based on the dual refinements, our approach can enlarge receptive fields and obtain more discriminative features from high-resolution images. Experimental results on high-resolution benchmarks (the public DUT-HRSOD and the proposed DAVIS-SOD) demonstrate that our method is not only efficient but also performs more accurate than other state-of-the-arts. Besides, our method generalizes well on typical low-resolution benchmarks. [ABSTRACT FROM AUTHOR]

Plant disease diagnosis is very critical for agriculture due to its importance for increasing crop production. Recent advances in image processing offer us a new way to solve this issue via visual plant disease analysis. However, there are few works in this area, not to mention systematic researches. In this paper, we systematically investigate the problem of visual plant disease recognition for plant disease diagnosis. Compared with other types of images, plant disease images generally exhibit randomly distributed lesions, diverse symptoms and complex backgrounds, and thus are hard to capture discriminative information. To facilitate the plant disease recognition research, we construct a new large-scale plant disease dataset with 271 plant disease categories and 220,592 images. Based on this dataset, we tackle plant disease recognition via reweighting both visual regions and loss to emphasize diseased parts. We first compute the weights of all the divided patches from each image based on the cluster distribution of these patches to indicate the discriminative level of each patch. Then we allocate the weight to each loss for each patch-label pair during weakly-supervised training to enable discriminative disease part learning. We finally extract patch features from the network trained with loss reweighting, and utilize the LSTM network to encode the weighed patch feature sequence into a comprehensive feature representation. Extensive evaluations on this dataset and another public dataset demonstrate the advantage of the proposed method. We expect this research will further the agenda of plant disease recognition in the community of image processing. [ABSTRACT FROM AUTHOR]

With the continuous development of rail transit fully automatic operation, the urgent need to improve train operation safety makes obstacle detection become the research focus. In this work, a flexible and efficient multiscale one-stage object detector FE-YOLO was proposed for image obstacle detection. The feature extraction network is composed of attention module, downsampling module, residual block, spatial pyramid pooling (SPP) module, and so on. A repeatable bidirectional cross-scale path aggregation module was designed as the core of the feature fusion network. The dataset RT2021 of rail transit obstacles was constructed based on the real scene. The mean of average precision (mAP), detection time, iteration time, parameters, and anti-interference ability were used to compare FE-YOLO with other classic object detectors. The results showed that FE-YOLO has the best comprehensive performance. The mAP can reach 92.57%, and the single-frame detection time on the onboard embedded device is up to 0.0989 s. The ablation experiment verified the effectiveness of each module of FE-YOLO and has the best generalization performance on the PASCAL VOC dataset. Finally, a rail transit obstacle detection system was developed, and multiple sensors were used to improve the detection accuracy. Experiments showed that the detection system can work normally in different environments. [ABSTRACT FROM AUTHOR]

The micro/nanohardness profile has been extensively used in manufacturing to understand material properties from external surfaces to internal material. The microhardness profile requires the indentation dimension and depth measurements. Dimension measurement in most cases is interfered by noise, texture, or defects, while depth identification asks for accurate stitching of local micrographs and recognition of tilted machined surfaces. However, few commercial microhardness testers can address the issues, while the manual measurement is limited by low robustness and efficiency. To fill this gap, this paper proposes an automatic and accurate measurement method of microhardness profile based on image processing. The method can stitch local micrographs, recognize indentation dimensions and depths, generate the microhardness profile, and analyze the material property with only one click. Experiments proved the method enjoys high accuracy, automation, and robustness. [ABSTRACT FROM AUTHOR]

The dropper is one of the core components of the high speed railway catenary and dropper failure will lead to serious transportation accidents. It is very important to carry out dropper failure and defect detection. There are a large number of droppers installed in the catenary. The dropper images are collected by the high-definition camera installed on the top of the moving catenary inspection vehicles. The image quality and image consistency are poor, and it is very difficult to identify dropper defects automatically. The railway department and companies lack efficient and intelligent detection methods. This article innovatively proposes a multialgorithm fusion image processing technology, and builds a dropper recognition and failure–defect detection model based on a deep learning algorithm and subpixel level dropper defect detection model, and achieves high accuracy dropper failure and defect detection. The detection model based on the Faster R-CNN algorithm is studied to realize the positioning and recognition of the dropper and the failure detection of bending slack and broken dropper. The subpixel level dropper defect detection algorithm is based on the fusion of the image preprocessing, dropper fine positioning algorithm, edge fitting and bending zoom algorithms, the Hough transform algorithm, and so on. These can be used to realize detection of defects, such as microdeformation, dropper-strands loosened, dropper-strands broken, and foreign body adhesions. The test is verified by the catenary images taken from a practical high speed railway. The detection accuracy, real-time performance, and stability of the algorithm meet the needs of inspection and maintenance for a high speed railway. [ABSTRACT FROM AUTHOR]