Search

Complex-valued convolutional neural networks (CV-CNN) have recently gained recognition in feature representation learning. It implements the repeated application of the operations in convolution, local average pooling, and the absolute value of the resulting vectors. However, it is only conducted in the complex spatial domain, and lacks effective representation of directionality, singularity, and regularity in the complex spectral domain for anomaly detection of images. This is the key to feature learning representation of high-order singularity. To solve this problem, a complex-valued contourlet neural network (C$^{2}$N$^{2}$) is proposed in this article. It is novel in this sense that, different from the CV-CNN in the spatial domain, the spectral stream of C$^{2}$N$^{2}$ can enhance the multiresolution sparse representation of nonsubsampled contourlet (NSCT) with multiscales and multidirections for images. Furthermore, the spectral feature integration module is proposed to capture the statistical properties of the NSCT coefficients. It is shown that the proposed network can improve the distinguishability of feature learning and classification ability in theoretical analysis and experiments on three benchmark datasets (Flevoland, Xi'an, and Germany) compared with developed methods. Polarimetric synthetic aperture radar image classification is widely used in the fields of agriculture, forestry, and military. It must be emphasized that there is potential in effective feature learning representation and the generalization capability of C$^{2}$N$^{2}$ in deep learning, recognition, and interpretation.

The foundation model (FM) has garnered significant attention for its remarkable transfer performance in downstream tasks. Typically, it undergoes task-agnostic pretraining on a large dataset and can be efficiently adapted to various downstream applications through fine-tuning. While FMs have been extensively explored in language and other domains, their potential in remote sensing has also begun to attract scholarly interest. However, comprehensive investigations and performance comparisons of these models on remote sensing tasks are currently lacking. In this survey, we provide essential background knowledge by introducing key technologies and recent developments in FMs. Subsequently, we explore essential downstream applications in remote sensing, covering classification, localization, and understanding. Our analysis encompasses over 30 FMs in both natural and remote sensing fields, and we conduct extensive experiments on more than 10 datasets, evaluating global feature representation, local feature representation, and target localization. Through quantitative assessments, we highlight the distinctions among various FMs and confirm that pretrained large-scale natural FMs can also deliver outstanding performance in remote sensing tasks. After that, we systematically presented a brain-inspired framework for remote sensing foundation models (RSFMs). We delve into the brain-inspired characteristics in this framework, including structure, perception, learning, and cognition. To conclude, we summarize 12 open problems in RSFMs, providing potential research directions. Our survey offers valuable insights into the burgeoning field of RSFMs and aims to foster further advancements in this exciting area.

Transformer has shown excellent performance in remote sensing field with long-range modeling capabilities. Remote sensing video (RSV) moving object detection and tracking play indispensable roles in military activities as well as urban monitoring. However, transformers in these fields are still at the exploratory stage. In this survey, we comprehensively summarize the research prospects of transformers in RSV moving object detection and tracking. The core designs of remote sensing transformers and advanced transformers are first analyzed. It mainly includes the attention mechanism evolution for specific tasks, the fitting ability design of input mapping, diverse feature representation, model optimization, etc. The architectural characteristics of RSV detection and tracking are then described across two aspects. One is moving object detection for motion-based traditional background subtractions and appearance-based deep learning models. The other is object tracking for single and multiple targets. The research difficulties mainly include the blurred foreground in RSV data, the irregular object movement in traditional background subtraction, and the severe object occlusion in object tracking. Following that, the potential significance of transformers is discussed according to some thorny problems in RSV. Finally, we summarize ten open challenges of transformers in RSV, which may be used as a reference for promoting future research.

Brain-inspired algorithms have become a new trend in next-generation artificial intelligence. Through research on brain science, the intelligence of remote sensing algorithms can be effectively improved. This article summarizes and analyzes the essential properties of brain cognize learning and the recent advance of remote sensing interpretation. First, this article introduces the structural composition and the properties of the brain. Then, five represent brain-inspired algorithms are studied, including multiscale geometry analysis, compressed sensing, attention mechanism, reinforcement learning, and transfer learning. Next, this article summarizes the data types of remote sensing, the development of typical applications of remote sensing interpretation, and the implementations of remote sensing, including datasets, software, and hardware. Finally, the top ten open problems and the future direction of brain-inspired remote sensing interpretation are discussed. This work aims to comprehensively review the brain mechanisms and the development of remote sensing and to motivate future research on brain-inspired remote sensing interpretation.

This paper presents a pansharpening technique based on the non-subsampled contourlet transform (NSCT) and convolutional autoencoder (CAE). NSCT is exceptionally proficient at presenting orientation information and capturing the internal geometry of objects. First, it’s used to decompose the multispectral (MS) and panchromatic (PAN) images into high-frequency and low-frequency components using the same number of decomposition levels. Second, a CAE network is trained to generate original low-frequency PAN images from their spatially degraded versions. Low-resolution multispectral images are then fed into the trained convolutional autoencoder network to generate estimated high-resolution multispectral images. Third, another CAE network is trained to generate original high-frequency PAN images from their spatially degraded versions. The result of low-pass CAE is fed to the trained high-pass CAE to generate estimated high-resolution multispectral images. The final pan-sharpened image is accomplished by injecting the detailed map of the spectral bands into the corresponding estimated high-resolution multispectral bands. The proposed method is tested on QuickBird datasets and compared with some existing pan-sharpening techniques. Objective and subjective results demonstrate the efficiency of the proposed method.

Pixelwise classification plays an important role for image interpretation. The remote sensing images, especially polarimetric synthetic aperture radar (PolSAR), have provided wide applications for both military and civilian users regardless of weather or lighting conditions. However, the classification of heterogeneous imagery is still challenging. In this article, we propose a novel deep fuzzy graph convolutional network (DFGCN) for pixelwise classification of PolSAR imagery. Inspired by the fuzzy logic, the imagery (i.e., a reflection of the backscattering of the ground) is represented by a fuzzy graph, whose node is associated with a feature vector, and fuzzy weights denoting the similarity degrees between nodes are defined by using both feature distance and spatial relation. We also construct a new graph residual module stacking multiple residual layers to extend the depth of our fuzzy graph network. Moreover, we present a graph shrinking strategy to reduce the graph size for training and predicting the labels of large-scale PolSAR imagery, which can cut down the computational cost. The experimental results on various real-world PolSAR datasets indicate that DFGCN dramatically enhances classification accuracies on both heterogeneous and homogeneous regions with limited labeled pixels compared with the state-of-the-art methods.

The recent years has witnessed a rapid development of Deep Learning (DL) based Automation Modulation Classification (AMC) methods, which has proved to outperform traditional classification approaches. In order to disturb the deep neural networks for AMC, in this paper, we propose an adversarial attack method to generate fake signals for fooling DL-based classifiers. Firstly, some constraints on visual difference and recoverability of fake signals are defined. Next, a Few Data Point Attacker (FDPA) is proposed to generate fake signals with few perturbed data points via differential evolution algorithm. Some experiments are taken on a public dataset, RML 2016.10a, and the results show that fake signals generated by the FDPA can remarkably reduce the accuracies of three types of DL-based AMC classifiers, a Convolutional Neural Network (CNN) based classifier, a Long Short-Term Memory Recurrent Neural Network (LSTM-RNN) based classifier, and a classifier combined with CNN and LSTM-RNN. The code will be available.

In this paper a new Sparse Tensor Auto-Encoder (STAE) model is proposed to learn a latent and discriminative feature representations for saliency detection. By formulating the background patches as holistic high-dimensional tensors and learning multi-dimensional dictionary to code image patches, the coding error can precisely reveal the difference between salient object and background. Then a saliency map can be derived by a subsequent refinement of the representation errors of patches via image segmentation. Several benchmark datasets are used to verify the effectiveness of the proposed method. The results show that the proposed STAE can accurately locate saliency region and outperform its counterparts.

Handcraft features are commonly used for signal classification, which is a time-consuming feature engineering. In order to develop a general and robust feature learning method for radio signals, a novel One-dimensional Deep Attention Convolution Network (ODACN) is proposed to automatically extract discriminative features and classify various kinds of signals. First, one-dimensional (1-D) sparse filters are designed to learn hierarchical features of raw signals. Second, an attention layer is constructed to weight and assemble feature maps, to derive more context-relevant representation. By using simple 1-D filtering, ODACN is characteristic of less parameters and lower computation complexity than traditional Convolutional Neural Networks (CNNs). Moreover, feature attention can mimic a succession of partial glimpses of humans and focus on context parts of signals, thus helps in recognizing signals even at low Signal-to-Noise Ratio (SNR). Some experiments are taken to classify 31 kinds of signals with different modulation and channel coding types, and the results show that ODACN can achieve accurate classification of very similar signals, without any prior knowledge and manual operation.

Compared with natural image super-resolution, hyperspectral image super-resolution (HSR) is more complex because the redundancy in spectral bands and spatial information. To overcome the difficulties exist in HSR, in this paper, we propose a tensor spatial-spectral joint correlation based HSR method. Start with the tensor representation, we construct a series of fourth-order tensors to preserve the intrinsic structure of hyperspectral images, and then explore the spatial-spectral joint correlation based on meaningful interpretations of tensor canonical matrices. To further constrain the spectral characteristics, we analyze the sparsity of the spectral gradients and model it with Laplacian prior. Then, the two regularizations are combined with the reconstruction model to develop a new HSR method. Finally, an iterative optimization algorithm based on alternating direction method of multiplier (ADMM) and augmented Lagrangian multiplier method is proposed to reconstruct the high-resolution hyperspectral images. Experimental results on several data sets illustrate the effectiveness of our proposed method both in visual and numerical comparisons.

The past decade has witnessed a prosperity of sparsity-inspired face hallucination methods that use sparse prior and instances to generate High-Resolution (HR) faces. However, they need numerous Low-Resolution (LR) and HR instance pairs and adopt approximate sparse coding, which will bring bias to the recovery and suffer from high computational burden. In this paper we advance a Single Face Image Hallucination (SFIH) method from a new perspective of Non-linear Learning Compressive Sensing (NLCS), which can recover HR faces from a surprisingly small number of HR faces. The nonlinear sparse coding of facial images is explored, and a Deep AutoEncoder (DAE) network is constructed for learning a kernel function from a single HR instance set. SFIH is then reduced to an analytic compressive recovery problem by reformulating linear sparse coding as a nonlinear DAE model. By exploring the nonlinear sparsity in the feature space, NLCS can accurately and rapidly recover HR facial images with large magnification factor and exhibit robustness to LR-HR instance pairs mapping. Some experiments are taken on realizing 3X, 6X, 9X amplification of face images, and the results prove its efficiency and superiority to its counterparts.

Deep Neural Networks (DNNs) have achieved remarkable accuracy improvements for automatic modulation classification. However, the employed networks often have millions of parameters and need very high computation, which makes it difficult to deploy these models on portable devices with limited resources. We propose a cross model deep learning scheme to build a lightweight deep network for accurate modulation classification. Firstly, a large Hybrid DNN (HDNN) that is composed of convolutional and recurrent layers is constructed and trained for automatic and accurate classification of signals. Then we build a smaller Layered Resnet Network (LRN) with shallow layers and few nodes. The HDNN and LRN are taken as a Teacher Model (TM) and a Student Model (SM) respectively. Finally, a knowledge distillation method is proposed to guide the learning of the SM, by formulating a teaching loss from the prediction of the TM to train the SM. The performances of the proposed HDNN and LRN are investigated on the public RadioML2016.10a and RadioML2016.10b data sets. The experimental results show that the trained HDNN presents state-of-the-art classification results and the LRN trained in this scheme takes only about a sixth of the HDNN's inference time and consumes only 472.3KB for storage, with a slight accuracy decrease compared with the large HDNN.

Desired by sparse subset learning, in this paper, a hyperspectral band selection method via pairwise band agreement with spatial-spectral graph regularier, referred as Regularized Band Selection via Learned Pairwise Agreement (RBS-LPA), was proposed. The process was formulated as a graph-regularized row-sparse constrained optimization problem, which select a few representative bands to code the all bands based on the learned pairwise band agreement. In RBS-LPA, a spatial-spectral informative graph, constructed by spatial-spectral neighbor relationship, is incorporated to encode both the spatial and spectral geometrical structure. By combining the learning procedure with graph regularizer jointly, the graph regularizer can adaptively change with the sparse representative bands, which ensures that the selected bands can well preserve the local manifold structure. Experimental results on three real urban hyperspectral data demonstrate the efficiency of the proposed RBS-LPA and achieve convincing performance.

Remote Sensing Images (RSIs) often have extremely wide width and abundant terrain. In order to achieve rapid object detection in large RSIs, in this paper, a Deep Hash Assisted Network (DHAN) is constructed by introducing a hashing encoding of images in a two-stage deep neural network. Different with the available detection networks, DHAN first locates candidate object regions and then transfers the learned features to another Region Proposal Network (RPN) for detection. On the one hand, it can avoid the calculations on the background irrelevant to objects. On the other hand, the built hash encoding layer in DHAN can accelerate the detection via binary hash features. Moreover, a self attention layer is designed and combined with the convolution layer, to distinguish relatively small objects regions from a very large scene. The proposed method is tested on several public data sets, and the comparison results show that DHAN can remarkably improve the detection efficiency on large RSIs and simultaneously achieve high detection accuracy.

Tensor representation is the most natural and effective way to preserve the structural information of hyperspectral image (HSI), and thus is very beneficial to HSI processing. This paper represents the spectral features of each testing pixel and its spatial neighbors as a Spatial Neighbor Tensor (SNT), whose spectral vectors can be simultaneously sparse coded by the spectral vectors of a few common training pixels. The obtained sparse coding coefficients could be regarded as a Slice Sparse Coding Tensor (SSCT), which can be adaptively learnt and utilized to predict the labels of HSI pixels. Furthermore, to improve the efficiency, the Compressive Dimensionality Reduction (CDR) is introduced into tensor slice sparse coding for optimizing SSCT and thus an SSCT based Classifier with CDR (SSCTC-CDR) is proposed for hyperspectral image classification (HIC). The performance of SSCTC-CDR is evaluated on three real HSI data, and the results show that it can obtain high-accurate classification result with relatively low computation lost.

In this paper, a new Separable Attention Capsule Network (SACN) is proposed for signal classification. SACN is a light-weight network composed of multi-channel separable convolution layer, attention module and classification layer. First, depth-wise convolution is employed to extract features of signals in a low-complexity manner, and the multi-channel network structure is designed to increase the network width to improve the diversity of features of signals. Then a channel attention module is followed by a capsule network whose element contains a group of neurons. This attention module can explore the interdependence among channels to use global information to selectively strengthen some important channels, thus achieving the improvement of generalization ability of SACN. Some experiments are taken on several datasets with communication and radar signals, and the comparison results prove the efficiency of SACN and the superiority to its counterparts.

Nowadays, with the development of remote sensing technology, very-high-resolution (VHR) remote sensing image object detection technology attracts more and more attention. However, various challenges still exist in remote sensing image object detection field, such as the complex and varied appearances, the expensive manual annotation, and difficult in fast detecting the large scene image. In this paper, we propose a multi-scale image block-level fully convolutional neural network (MIF-CNN) for remote sensing image object detection, which can solve the above problems to a certain degree. First, the training data sets which only require class labels and do not need the bounding box label can reduce the spend of manual annotation during training. Second, the MIF-CNN is designed to extract the multi-scale-based high-level feature which can better represent the various types of objects. The image block-level fully convolutional network contributes to improving computing efficiency and can directly detect any size of the input image, including a large scene remote sensing image. In the testing phase, the large scene image is directly input to MIF-CNN model, and the detection results are generated from the MIF-CNN output maps which are improved by the proposed bounding boxes modification strategy and local re-recognized strategy. The experiments on NWPU VHR-10 [1] and two Airports data sets demonstrate the effectiveness of the proposed method.

Synthetic aperture radar (SAR) image classification aims at labeling pixels with different categories and this is both, a fundamental step for automatic target recognition (ATR) and a prerequisite for further interpretation. In the past decades, various methods have been proposed for the classification of SAR targets and among them are discriminative dictionary learning (DDL) methods. These DDL methods have recently gained attention from researchers' community due to the fact that they are very powerful on both, representation and discrimination during the classification process of SAR images. However, most of the existing DDL methods adopt l0-norm or l1-norm to ensure the sparsity, but in general, these DDL methods suffer from a high computational burden. Furthermore, it is important to minimize the execution time in the phase of online testing for the scenario of onboard real-time or near real-time SAR automatic target recognition such as modern unmanned aerial vehicle SAR platforms. That said, on reducing execution time, we are confronted with the problem of enhancing recognition efficiency while maintaining its accuracy. In order to solve this problem, our paper proposes a fast DDL method (named as FaDDL) based on a nonlinear analysis co-sparse model by adopting an l1,∞-norm ball as a constraint to replace l0-norm or l1-norm on the coding coefficient matrix. The experimental results show that our proposed method significantly reduces execution time, without losing the classification accuracy.

The past few years have witnessed the prosperity of establishing deep architectures for modeling complex structured data. In this paper, motivated by the hierarchical, multi-scale and sparse characteristics of Human Visual System (HVS), we advance a new Deep and Sparse Representative (DSR) saliency detection approach for natural images. Under the assumption that salient regions are considered as sparse representatives of an image, we gradually select the salient regions via a deep and hierarchical sparse coding model. Moreover, in order to well preserve image structures and achieve more efficient coding, we formulate image patches as tensors and propose a Sparse Tensor Coding based DSR (STC-DSR) approach to find representatives. A generalized Laplacian regularizer is further cast on the coding to refine the coefficients. Finally, the saliency map is formulated by a spatial aggregation and multi-scale fusion of multilayer representatives. The proposed STC-DSR is investigated on the synthetic and public datasets and the experimental results prove its efficiency and superiority to its counterparts.

With the rapid development of earth observation satellites, on-orbit data processing is becoming more and more desirable. In this paper, a new on-orbit change detection method for Synthetic Aperture Radar (SAR) images, is proposed via an Extreme Self-paced Learning Machine (ESLM). First, a reflectivity-spatial affinity is defined to measure the similarity between two segmented super-pixels, to identify the initial three groups of pixels: strictly changed, strictly unchanged and fuzzy pixels. Then a new extreme self-paced learning machine is developed, by gradually selecting the most confident changed pixels and predicting the changed pixels in an incremental pattern. Moreover, both the labeled and unlabeled samples are explored to realize semi-supervised classification. Different with the available methods, ESLM works in a self-paced learning pattern and achieves accurate detection, for it can automatically choose the training samples and explore unlabeled samples to enhance the online prediction of changes. Therefore, ESLM has the characteristics of accurate and robust detection, parameter free, low-complexity and rapid implementation, which is very suitable for on-orbit processing. Some experiments are taken on five real benchmark datasets, and the results verify the effectiveness of ESLM.

Extracting matched details of the PANchromatic (PAN) image and injecting them into the MultiSpectral (MS) images, is very crucial in pansharpening. In this paper, a new pansharpening method based on Joint Local Low Rank Decomposition (JLLRD) and Hierarchical Geometric Filtering (HGF) is proposed. First, a cascaded geometric filtering is performed on the PAN and MS images, to extract their multi-scale directional details. Then a joint local low rank decomposition is developed to deduce low-rank and sparse components for injection. Finally, an adaptive injection rule based on spectral correlation coefficient, is designed to further reduce spectral distortion of the fused images. Several experiments are taken to investigate the performance of the proposed JLLRD-HGF method, and the results show that it can extract more accurate injection details and produce less spectral and spatial distortions than its counterparts.

Inspired by human’s learning characteristic that knowledge is gradually learned little by little, a Spatial-Polarimetric Reinforcement Learning (SPRL) approach is proposed for Polarimetric Synthetic Aperture Radar (PolSAR) data classification, from a new perspective of reinforcement learning. In our method, each pixel has its own “state” and “action”, and can modify its “action” based on interactions with the “environment”. A spatial-polarimetric “reward” function, is designed from a local neighborhood region to explore both the spatial and polarimetric information for more accurate classification. Thus a self-evolution and model-free classifier can be obtained, which has simple principle and robustness to speckle noises existed in the data. By an interaction with the environment, SPRL can obtain high classification accuracy when only very few labeled pixels are available. Several real PolSAR datasets are used to investigate the effectiveness of the proposed method, and the results show that SPRL is superior to its counterparts.

Object detection is one of the most important and challenging branches of computer vision, which has been widely applied in people’s life, such as monitoring security, autonomous driving and so on, with the purpose of locating instances of semantic objects of a certain class. With the rapid development of deep learning algorithms for detection tasks, the performance of object detectors has been greatly improved. In order to understand the main development status of object detection pipeline thoroughly and deeply, in this survey, we analyze the methods of existing typical detection models and describe the benchmark datasets at first. Afterwards and primarily, we provide a comprehensive overview of a variety of object detection methods in a systematic manner, covering the one-stage and two-stage detectors. Moreover, we list the traditional and new applications. Some representative branches of object detection are analyzed as well. Finally, we discuss the architecture of exploiting these object detection methods to build an effective and efficient system and point out a set of development trends to better follow the state-of-the-art algorithms and further research.

Panchromatic (PAN) and multispectral (MS) imagery classification is one of the hottest topics in the field of remote sensing. In recent years, deep learning techniques have been widely applied in many areas of image processing. In this paper, an end-to-end learning framework based on deep multiple instance learning (DMIL) is proposed for MS and PAN images' classification using the joint spectral and spatial information based on feature fusion. There are two instances in the proposed framework: one instance is used to capture the spatial information of PAN and the other is used to describe the spectral information of MS. The features obtained by the two instances are concatenated directly, which can be treated as simple fusion features. To fully fuse the spatial-spectral information for further classification, the simple fusion features are fed into a fusion network with three fully connected layers to learn the high-level fusion features. Classification experiments carried out on four different airborne MS and PAN images indicate that the classifier provides feasible and efficient solution. It demonstrates that DMIL performs better than using a convolutional neural network and a stacked autoencoder network separately. In addition, this paper shows that the DMIL model can learn and fuse spectral and spatial information effectively, and has huge potential for MS and PAN imagery classification. [ABSTRACT FROM AUTHOR]