Search

This paper proposes a complex-valued convolutional neural network for land form classification and discovery in interferometric synthetic aperture radar (InSAR). Since the amount of satellite-borne SAR data has been increasing drastically, it is necessary to structurize the local features contained in observation data prior to utilization in the so-called big data framework for higher usability. Convolutional neural networks have such potential in general. However, there exists no network that can deal with complex amplitude data obtained in InSAR consistently. In this paper, we propose a complex-valued convolutional neural network to deal with InSAR. We demonstrate that the network classifies slopes and plains adaptively and, moreover, indicates small volcanos similar to a sample volcano (Omuroyama) included in the InSAR data. We also find their characteristic features emerging in the kernels in the convolution layers. These results reveal that the proposed complex-valued convolutional neural network is capable of successfully discovering unidentified lands similar to a prepared sample, which is highly useful for the InSAR data structurization. [ABSTRACT FROM AUTHOR]

Compression of hyperspectral images onboard of spacecrafts is a tradeoff between the limited computational resources and the ever-growing spatial and spectral resolution of the optical instruments. As such, it requires low-complexity algorithms with good rate-distortion performance and high throughput. In recent years, the Consultative Committee for Space Data Systems (CCSDS) has focused on lossless and near-lossless compression approaches based on predictive coding, resulting in the recently published CCSDS 123.0-B-2 recommended standard. While the in-loop reconstruction of quantized prediction residuals provides excellent rate-distortion performance for the near-lossless operating mode, it significantly constrains the achievable throughput due to data dependencies. In this paper, we study the performance of a faster method based on the prequantization of the image followed by a lossless predictive compressor. While this is well known to be suboptimal, one can exploit powerful signal models to reconstruct the image at the ground segment, recovering part of the suboptimality. In particular, we show that convolutional neural networks can be used for this task and that they can recover the whole SNR drop incurred at a bit rate of 2 bits per pixel. [ABSTRACT FROM AUTHOR]

Change detection is one of the central problems in earth observation and was extensively investigated over recent decades. In this paper, we propose a novel recurrent convolutional neural network (ReCNN) architecture, which is trained to learn a joint spectral–spatial–temporal feature representation in a unified framework for change detection in multispectral images. To this end, we bring together a convolutional neural network and a recurrent neural network into one end-to-end network. The former is able to generate rich spectral-spatial feature representations, while the latter effectively analyzes temporal dependence in bitemporal images. In comparison with previous approaches to change detection, the proposed network architecture possesses three distinctive properties: 1) it is end-to-end trainable, in contrast to most existing methods whose components are separately trained or computed; 2) it naturally harnesses spatial information that has been proven to be beneficial to change detection task; and 3) it is capable of adaptively learning the temporal dependence between multitemporal images, unlike most of the algorithms that use fairly simple operation like image differencing or stacking. As far as we know, this is the first time that a recurrent convolutional network architecture has been proposed for multitemporal remote sensing image analysis. The proposed network is validated on real multispectral data sets. Both visual and quantitative analyses of the experimental results demonstrate competitive performance in the proposed mode. [ABSTRACT FROM AUTHOR]

Frequency filtering is widely used in routine processing of seismic data to improve the signal-to-noise ratio (SNR) of recorded signals and by doing so to improve subsequent analyses. In this paper, we develop a new denoising/decomposition method, DeepDenoiser, based on a deep neural network. This network is able to simultaneously learn a sparse representation of data in the time–frequency domain and a non-linear function that maps this representation into masks that decompose input data into a signal of interest and noise (defined as any non-seismic signal). We show that DeepDenoiser achieves impressive denoising of seismic signals even when the signal and noise share a common frequency band. Because the noise statistics are automatically learned from data and require no assumptions, our method properly handles white noise, a variety of colored noise, and non-earthquake signals. DeepDenoiser can significantly improve the SNR with minimal changes in the waveform shape of interest, even in the presence of high noise levels. We demonstrate the effect of our method on improving earthquake detection. There are clear applications of DeepDenoiser to seismic imaging, micro-seismic monitoring, and preprocessing of ambient noise data. We also note that the potential applications of our approach are not limited to these applications or even to earthquake data and that our approach can be adapted to diverse signals and applications in other settings. [ABSTRACT FROM AUTHOR]

The nonstationary and non-Gaussian distribution of the high-resolution (HR) synthetic aperture radar (SAR) image provides much valuable information. However, the current methods, especially deep learning models, directly learn spatial features from HR SAR data while ignoring global statistical information. Combining the local spatial features and global statistical properties of HR SAR images is urgently needed to capture complete HR SAR characteristics. In this paper, a feature fusion network (Fusion-Net) using both deep spatial context encoder and nonstationary joint statistical model (NS-JSM) is proposed for the first time. Fusion-Net realizes the fusion description of local spatial and global statistical features in an end-to-end supervised classification framework. First, a deep spatial context encoder network (DSCEN) is designed based on multiscale group convolution (MSGC) module and channel attention (CA) module. The DSCEN expands the scope of context information extraction with few parameters and increases the interaction between high- level feature channels. Then, the NS-JSM is adopted to capture the unique SAR statistical information. Specifically, the SAR image is transformed into the Gabor wavelet domain. The produced sub-band magnitudes and phases are modeled by the log-normal and uniform distribution. The covariance matrix (CM) is calculated for mapped sub-band data to capture the interscale and intrascale nonstationary correlation. Finally, the group compression and smooth normalization units are introduced into Fusion-Net to fuse the statistical features and spatial features, which not only exploits the complementary information between different features but also optimizes the fusion feature representation. Experiments on four real HR SAR images validate the superiority of the proposed method over other related algorithms. [ABSTRACT FROM AUTHOR]

Recently, hyperspectral image (HSI) classification approaches based on deep learning (DL) models have been proposed and shown promising performance. However, because of very limited available training samples and massive model parameters, DL methods may suffer from overfitting. In this paper, we propose an end-to-end 3-D lightweight convolutional neural network (CNN) (abbreviated as 3-D-LWNet) for limited samples-based HSI classification. Compared with conventional 3-D-CNN models, the proposed 3-D-LWNet has a deeper network structure, less parameters, and lower computation cost, resulting in better classification performance. To further alleviate the small sample problem, we also propose two transfer learning strategies: 1) cross-sensor strategy, in which we pretrain a 3-D model in the source HSI data sets containing a greater number of labeled samples and then transfer it to the target HSI data sets and 2) cross-modal strategy, in which we pretrain a 3-D model in the 2-D RGB image data sets containing a large number of samples and then transfer it to the target HSI data sets. In contrast to previous approaches, we do not impose restrictions over the source data sets, in which they do not have to be collected by the same sensors as the target data sets. Experiments on three public HSI data sets captured by different sensors demonstrate that our model achieves competitive performance for HSI classification compared to several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Object detection is a basic issue of very high-resolution remote sensing images (RSIs) for automatically labeling objects. At present, deep learning has gradually gained the competitive advantage for remote sensing object detection, especially based on convolutional neural networks (CNNs). Most of the existing methods use the global information in the fully connected feature vector and ignore the local information in the convolutional feature cubes. However, the local information can provide spatial information, which is helpful for accurate localization. In addition, there are variable factors, such as rotation and scaling, which affect the object detection accuracy in RSIs. In order to solve these problems, this paper presents a hierarchical robust CNN. First, multiscale convolutional features are extracted to represent the hierarchical spatial semantic information. Second, multiple fully connected layer features are stacked together so as to improve the rotation and scaling robustness. Experiments on two data sets have shown the effectiveness of our method. In addition, a large-scale high-resolution remote sensing object detection data set is established to make up for the current situation that the existing data set is insufficient or too small. The data set is available at https://github.com/CrazyStoneonRoad/TGRS-HRRSD-Dataset. [ABSTRACT FROM AUTHOR]

In multiclass classification, convolutional neural network (CNN) is generally coupled with the cross-entropy (CE) loss, which only penalizes the predicted probability corresponding to a ground truth class and ignores the interclass relationship. We argue that CNN can be improved by using a better loss function. On the other hand, the Wasserstein distance (WD) is a well-known metric used to measure the distance between two distributions. Directly solving the WD problem requires a prohibitively large amount of computation time, whereas the cheaper iterative algorithms have a variety of shortcomings such as computational instability and difficulty in selecting parameters. In this paper, we address these issues by giving an analytical solution to the WD problem—for the first time, we find that for two distributions in hierarchically organized data space, WD has a closed-form solution, which we call “hierarchical WD (HWD).” We use this theory to construct novel loss functions that overcome the shortcomings of CE loss. To this end, multi-CNN information fusion that provides the basis for building category hierarchies is carried out first. Then, the semantic relationship among classes is modeled as a binary tree. Then, CNN coupled with an HWD-based loss, i.e., hierarchical Wasserstein CNN (HW-CNN), is trained to learn deep features. In this way, prior knowledge about the interclass relationship is embedded into HW-CNN, and information from several CNNs provides guidance in the process of training individual HW-CNNs. We conducted extensive experiments over two publicly available remote sensing data sets and achieved a state-of-the-art performance in scene classification tasks. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) denoising is a crucial preprocessing procedure to improve the performance of the subsequent HSI interpretation and applications. In this paper, a novel deep learning-based method for this task is proposed, by learning a nonlinear end-to-end mapping between the noisy and clean HSIs with a combined spatial–spectral deep convolutional neural network (HSID-CNN). Both the spatial and spectral information are simultaneously assigned to the proposed network. In addition, multiscale feature extraction and multilevel feature representation are, respectively, employed to capture both the multiscale spatial–spectral feature and fuse different feature representations for the final restoration. The simulated and real-data experiments demonstrate that the proposed HSID-CNN outperforms many of the mainstream methods in both the quantitative evaluation indexes, visual effects, and HSI classification accuracy. [ABSTRACT FROM AUTHOR]

Due to the urgent demand for remote sensing big data analysis, large-scale remote sensing image retrieval (LSRSIR) attracts increasing attention from researchers. Generally, LSRSIR can be divided into two categories as follows: uni-source LSRSIR (US-LSRSIR) and cross-source LSRSIR (CS-LSRSIR). More specifically, US-LSRSIR means the inquiry remote sensing image and images in the searching data set come from the same remote sensing data source, whereas CS-LSRSIR is designed to retrieve remote sensing images with a similar content to the inquiry remote sensing image that are from a different remote sensing data source. In the literature, US-LSRSIR has been widely exploited, but CS-LSRSIR is rarely discussed. In practical situations, remote sensing images from different kinds of remote sensing data sources are continually increasing, so there is a great motivation to exploit CS-LSRSIR. Therefore, this paper focuses on CS-LSRSIR. To cope with CS-LSRSIR, this paper proposes source-invariant deep hashing convolutional neural networks (SIDHCNNs), which can be optimized in an end-to-end manner using a series of well-designed optimization constraints. To quantitatively evaluate the proposed SIDHCNNs, we construct a dual-source remote sensing image data set that contains eight typical land-cover categories and 10 000 dual samples in each category. Extensive experiments show that the proposed SIDHCNNs can yield substantial improvements over several baselines involving the most recent techniques. [ABSTRACT FROM AUTHOR]

Most convolutional neural network (CNN)-based cloud detection methods are built upon the supervised learning framework that requires a large number of pixel-level labels. However, it is expensive and time-consuming to manually annotate pixelwise labels for massive remote sensing images. To reduce the labeling cost, we propose an unsupervised domain adaptation (UDA) approach to generalize the model trained on labeled images of source satellite to unlabeled images of the target satellite. To effectively address the domain shift problem on cross-satellite images, we develop a novel UDA method based on grouped features alignment (GFA) and entropy minimization (EM) to extract domain-invariant representations to improve the cloud detection accuracy of cross-satellite images. The proposed UDA method is evaluated on “Landsat- $8~\rightarrow $ ZY-3” and “GF- $1\rightarrow $ ZY-3” domain adaptation tasks. Experimental results demonstrate the effectiveness of our method against existing state-of-the-art UDA approaches. The code of this paper has been made available online (https://github.com/nkszjx/grouped-features-alignment). [ABSTRACT FROM AUTHOR]

Classification of remotely sensed images into land cover or land use is highly dependent on geographical information at least at two levels. First, land cover classes are observed in a spatially smooth domain separated by sharp region boundaries. Second, land classes and observation scale are also tightly intertwined: they tend to be consistent within areas of homogeneous appearance, or regions, in the sense that all pixels within a roof should be classified as roof, independently on the spatial support used for the classification. In this paper, we follow these two observations and encode them as priors in an energy minimization framework based on conditional random fields (CRFs), where classification results obtained at pixel and region levels are probabilistically fused. The aim is to enforce the final maps to be consistent not only in their own spatial supports (pixel and region) but also across supports, i.e., by getting the predictions on the pixel lattice and on the set of regions to agree. To this end, we define an energy function with three terms: 1) a data term for the individual elements in each support (support-specific nodes); 2) spatial regularization terms in a neighborhood for each of the supports (support-specific edges); and 3) a regularization term between individual pixels and the region containing each of them (intersupports edges). We utilize these priors in a unified energy minimization problem that can be optimized by standard solvers. The proposed 2L↯CRF model consists of a CRF defined over a bipartite graph, i.e., two interconnected layers within a single graph accounting for interlattice connections. 2L↯CRF is tested on two very high-resolution data sets involving submetric satellite and subdecimeter aerial data. In all cases, 2L↯CRF improves the result obtained by the independent base model (either random forests or convolutional neural networks) and by standard CRF models enforcing smoothness in the spatial domain. [ABSTRACT FROM AUTHOR]

It is a feasible and promising way to utilize deep neural networks to learn and extract valuable features from synthetic aperture radar (SAR) images for SAR automatic target recognition (ATR). However, it is too difficult to effectively train the deep neural networks with limited raw SAR images. In this paper, we propose a new approach to do SAR ATR, in which a multiview deep learning framework was employed. Based on the multiview SAR ATR pattern, we first present a flexible mean to generate adequate multiview SAR data, which can guarantee a large amount of inputs for network training without needing many raw SAR images. Then, a unique deep convolutional neural network containing a parallel network topology with multiple inputs is adopted. The features of input SAR images from different views will be learned by the proposed network layer by layer; meanwhile, the learned features from the distinct views are fused in different layers progressively. Therefore, the proposed framework is able to achieve a superior recognition performance, and requires only a small number of raw SAR images for network training samples generation. Experimental results have shown the superiority of the proposed framework based on the Moving and Stationary Target Acquisition and Recognition data set. [ABSTRACT FROM AUTHOR]

As the acquisition of very high resolution (VHR) satellite images becomes easier owing to technological advancements, ever more stringent requirements are being imposed on automatic image interpretation. Moreover, per-pixel classification has become the focus of research interests in this regard. However, the efficient and effective processing and the interpretation of VHR satellite images remain a critical task. Convolutional neural networks (CNNs) have recently been applied to VHR satellite images with considerable success. However, the prevalent CNN models accept input data of fixed sizes and train the classifier using features extracted directly from the convolutional stages or the fully connected layers, which cannot yield pixel-to-pixel classifications. Moreover, training a CNN model requires large amounts of labeled reference data. These are challenging to obtain because per-pixel labeled VHR satellite images are not open access. In this paper, we propose a framework called the unsupervised-restricted deconvolutional neural network (URDNN). It can solve these problems by learning an end-to-end and pixel-to-pixel classification and handling a VHR classification using a fully convolutional network and a small number of labeled pixels. In URDNN, supervised learning is always under the restriction of unsupervised learning, which serves to constrain and aid supervised training in learning more generalized and abstract feature. To some degree, it will try to reduce the problems of overfitting and undertraining, which arise from the scarcity of labeled training data, and to gain better classification results using fewer training samples. It improves the generality of the classification model. We tested the proposed URDNN on images from the Geoeye and Quickbird sensors and obtained satisfactory results with the highest overall accuracy (OA) achieved as 0.977 and 0.989, respectively. Experiments showed that the combined effects of additional kernels and stages may have produced better results, and two-stage URDNN consistently produced a more stable result. We compared URDNN with four methods and found that with a small ratio of selected labeled data items, it yielded the highest and most stable results, whereas the accuracy values of the other methods quickly decreased. For some categories with fewer training pixels, accuracy for categories from other methods was considerably worse than that in URDNN, with the largest difference reaching almost 10%. Hence, the proposed URDNN can successfully handle the VHR image classification using a small number of labeled pixels. Furthermore, it is more effective than state-of-the-art methods. [ABSTRACT FROM PUBLISHER]

Remote sensing (RS) scene classification is a challenging task due to various land covers contained in RS scenes. Recent RS classification methods demonstrate that aggregating the multilayer convolutional features, which are extracted from different hierarchical layers of a convolutional neural network, can effectively improve classification accuracy. However, these methods treat the multilayer convolutional features as equally important and ignore the hierarchical structure of multilayer convolutional features. Multilayer convolutional features not only provide complementary information for classification but also bring some interference information (e.g., redundancy and mutual exclusion). In this paper, a gated bidirectional network is proposed to integrate the hierarchical feature aggregation and the interference information elimination into an end-to-end network. First, the performance of each convolutional feature is quantitatively analyzed and a superior combination of convolutional features is selected. Then, a bidirectional connection is proposed to hierarchically aggregate multilayer convolutional features. Both the top–down direction and the bottom–up direction are considered to aggregate multilayer convolutional features into the semantic-assist feature and appearance-assist feature, respectively, and a gated function is utilized to eliminate interference information in the bidirectional connection. Finally, the semantic-assist feature and appearance-assist feature are merged for classification. The proposed method can compete with the state-of-the-art methods on four RS scene classification data sets (AID, UC-Merced, WHU-RS19, and OPTIMAL-31). [ABSTRACT FROM AUTHOR]

This paper addresses the problem of radar-based human gait recognition based on the dual-channel deep convolutional neural network (DC-DCNN). To enrich the limited radar data set of human gaits and provide a benchmark for classifier training, evaluation, and comparison, it proposes an effective method for radar echo generation from the infrared, publicly accessible motion capture (MOCAP) data set. According to the different nonstationary characteristics of micro-Doppler (m-D) for the torso and limbs, it enhances their distinguishable joint time–frequency (JTF) features by applying the short-time Fourier transforms (SFTFs) with varying sliding window length and then designs the DC-DCNN structure to achieve refined human gait recognition by separate feature extraction and fusion. Experiments have shown that compared with the traditional single-channel deep convolutional neural network (SC-DCNN), the proposed method achieves higher recognition accuracy in refined human gait classification without incurring additional radar resources and could be readily extended to refined recognition of other human activities. [ABSTRACT FROM AUTHOR]

The shortage of training samples remains one of the main obstacles in applying the neural networks to the hyperspectral images classification. To fuse the spatial and spectral information, pixel patches are often utilized to train a model, which may further aggregate this problem. In the existing works, an artificial neural network (ANN) model supervised by centerloss (ANNC) was introduced. Training merely with spectral information, the ANNC yields discriminative spectral features suitable for the subsequent classification tasks. In this paper, we propose a novel convolutional neural network (CNN)-based spatial feature fusion (CSFF) algorithm, which allows a smart integration of spatial information to the spectral features extracted by ANNC. As a critical part of CSFF, a CNN-based discriminant model is introduced to estimate whether two pixels belong to the same class. At the testing stage, by applying the discriminant model to the pixel pairs generated by a test pixel and each of its neighbors, the local structure is estimated and represented as a customized convolutional kernel. The spectral–spatial feature is generated by a convolutional operation between the estimated kernel and the corresponding spectral features within a local region. The final label is determined by classifying the resulting spectral–spatial feature. Without increasing the number of training samples or involving pixel patches at the training stage, the CSFF framework achieves the state of the art by declining 20%–50% classification failures in experiments on three well-known hyperspectral images. [ABSTRACT FROM AUTHOR]

In this paper, we describe Hydra, an ensemble of convolutional neural networks (CNNs) for geospatial land classification. The idea behind Hydra is to create an initial CNN that is coarsely optimized but provides a good starting pointing for further optimization, which will serve as the Hydra’s body. Then, the obtained weights are fine-tuned multiple times with different augmentation techniques, crop styles, and classes weights to form an ensemble of CNNs that represent the Hydra’s heads. By doing so, we prompt convergence to different endpoints, which is a desirable aspect for ensembles. With this framework, we were able to reduce the training time while maintaining the classification performance of the ensemble. We created ensembles for our experiments using two state-of-the-art CNN architectures, residual network (ResNet), and dense convolutional networks (DenseNet). We have demonstrated the application of our Hydra framework in two data sets, functional map of world (FMOW) and NWPU-RESISC45, achieving results comparable to the state-of-the-art for the former and the best-reported performance so far for the latter. Code and CNN models are available at https://github.com/maups/hydra-fmow. [ABSTRACT FROM AUTHOR]

With the large amount of high-spatial resolution images now available, scene classification aimed at obtaining high-level semantic concepts has drawn great attention. The convolutional neural networks (CNNs), which are typical deep learning methods, have widely been studied to automatically learn features for the images for scene classification. However, scene classification based on CNNs is still difficult due to the scale variation of the objects in remote sensing imagery. In this paper, a multiscale CNN (MCNN) framework is proposed to solve the problem. In MCNN, a network structure containing dual branches of a fixed-scale net (F-net) and a varied-scale net (V-net) is constructed and the parameters are shared by the F-net and V-net. The images and their rescaled images are fed into the F-net and V-net, respectively, allowing us to simultaneously train the shared network weights on multiscale images. Furthermore, to ensure that the features extracted from MCNN are scale invariant, a similarity measure layer is added to MCNN, which forces the two feature vectors extracted from the image and its corresponding rescaled image to be as close as possible in the training phase. To demonstrate the effectiveness of the proposed method, we compared the results obtained using three widely used remote sensing data sets: the UC Merced data set, the aerial image data set, and the google data set of SIRI-WHU. The results confirm that the proposed method performs significantly better than the other state-of-the-art scene classification methods. [ABSTRACT FROM AUTHOR]

Following the great success of deep convolutional neural networks (CNNs) in computer vision, this paper proposes a complex-valued CNN (CV-CNN) specifically for synthetic aperture radar (SAR) image interpretation. It utilizes both amplitude and phase information of complex SAR imagery. All elements of CNN including input-output layer, convolution layer, activation function, and pooling layer are extended to the complex domain. Moreover, a complex backpropagation algorithm based on stochastic gradient descent is derived for CV-CNN training. The proposed CV-CNN is then tested on the typical polarimetric SAR image classification task which classifies each pixel into known terrain types via supervised training. Experiments with the benchmark data sets of Flevoland and Oberpfaffenhofen show that the classification error can be further reduced if employing CV-CNN instead of conventional real-valued CNN with the same degrees of freedom. The performance of CV-CNN is comparable to that of existing state-of-the-art methods in terms of overall classification accuracy. [ABSTRACT FROM PUBLISHER]

Deep convolutional neural networks have been dominating in the field of hyperspectral image (HSI) classification. However, single convolutional kernel can limit the receptive field and fail to capture the sequential properties of data. The self-attention-based Transformer can build global sequence information, among which the Swin Transformer (SwinT) integrates sequence modeling capability and prior information of the visual signals (e.g., locality and translation invariance). Based on SwinT, we propose a 3-D SwinT (3DSwinT) to accommodate the 3-D properties of HSI and capture the rich spatial–spectral information of HSI. Currently, supervised learning is still the most commonly used method for remote sensing image interpretation. However, pixel-by-pixel HSI classification demands a large number of high-quality labeled samples that are time-consuming and costly to collect. As unsupervised learning, self-supervised learning (SSL), especially contrastive learning, can learn semantic representations from unlabeled data and, hence, is becoming a potential alternative to supervised learning. On the other hand, current contrastive learning methods are all single level or single scale, which do not consider complex and variable multiscale features of objects. Therefore, this article proposes a novel 3DSwinT-based hierarchical contrastive learning (3DSwinT-HCL) method, which can fully exploit multiscale semantic representations of images. Besides, we propose a multiscale local contrastive learning (MS-LCL) module to mine the pixel-level representations in order to adapt to downstream dense prediction tasks. A series of experiments verify the great potential and superiority of 3DSwinT-HCL. [ABSTRACT FROM AUTHOR]

Aerial scene classification remains challenging as: 1) the size of key objects in determining the scene scheme varies greatly and 2) many objects irrelevant to the scene scheme are often flooded in the image. Hence, how to effectively perceive the region of interests (RoIs) from a variety of sizes and build more discriminative representation from such complicated object distribution is vital to understand an aerial scene. In this article, we propose a novel all grains, one scheme (AGOS) framework to tackle these challenges. To the best of our knowledge, it is the first work to extend the classic multiple instance learning (MIL) into multigrain formulation. Specifically, it consists of a multigrain perception (MGP) module, a multibranch multi-instance representation (MBMIR) module, and a self-aligned semantic fusion (SSF) module. First, our MGP module preserves the differential dilated convolutional features from the backbone, which magnifies the discriminative information from multigrains. Then, our MBMIR module highlights the key instances in the multigrain representation under the MIL formulation. Finally, our SSF module allows our framework to learn the same scene scheme from multigrain instance representations and fuses them, so that the entire framework is optimized as a whole. Notably, our AGOS is flexible and can be easily adapted to existing convolutional neural networks (CNNs) in a plug-and-play manner. Extensive experiments on UCM, aerial image dataset (AID), and Northwestern Polytechnical University (NWPU) benchmarks demonstrate that our AGOS achieves a comparable performance against the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The current regional-scale process-based photosynthesis models use biome-specified values of maximum carboxylation rate at 25 °C ($V_{m25}$) in simulating ecosystem gross primary productivity (GPP). These models ignore the variations in $V_{m25}$ over time and space, resulting in substantial errors in regional estimates of cropland GPP. Thus, to resolve this problem, we used the ensemble Kalman filter (EnKF) to assimilate tower-based GPP from five maize flux sites into a process-based mode to obtain the “apparent” value of $V_{m25}$ and then modeled this parameter using machine learning (ML) algorithms. The results showed that $V_{m25}$ increased during the early growing season and then decreased after reaching a peak value in the middle of the growing season. The coefficient of determination ($R^{2}$) root mean square error (RMSE) for satellite-driven coupled photosynthesis and evapotranspiration simulator (SCOPES)-Crop with EnKF-derived varied $V_{m25}$ in simulating daily GPP across all site-days increased (decreased) by 0.17 (5.63 $\mu \text {mol}\,\text {m}^{-2}\,\text {s}^{-1}$) on average compared to that for the model with fixed $V_{m25}$. We used four ML algorithms, namely artificial neural network, random forest, extreme gradient enhancement, and convolutional neural network (CNN), to model the $V_{m25}$ of maize. The CNN algorithm yielded the best results. The average of the $R^{2}$ (RMSE) values of simulated GPP using CNN-based $V_{m25}$ over the three flux sites is 0.93 (1.95 $\mu \text {mol}\,\text {m}^{-2}\,\text {s}^{-1}$), higher (smaller) than that using fixed $V_{m25}$. This study implies that representing the seasonal variations in $V_{m25}$ can facilitate improved estimates of GPP and the ML methods are useful tools for modeling the variation in $V_{m25}$. [ABSTRACT FROM AUTHOR]

Hyperspectral images (HSIs) contain abundant information in the spatial and spectral domains, allowing for a precise characterization of categories of materials. Convolutional neural networks (CNNs) have achieved great success in HSI classification, owing to their excellent ability in local contextual modeling. However, CNNs suffer from fixed filter weights and deep convolutional layers, which lead to a limited receptive field and high computational burden. The recent vision transformer (ViT) models long-range dependencies with a self-attention mechanism and has been an alternative backbone to CNNs traditionally used in HSI classification. However, such transformer-based architectures designate all the input pixels of the receptive field as feature tokens in terms of feature embedding and self-attention, which inevitably limits the ability for learning multiscale features and increases the computational cost. To overcome this issue, we propose a local semantic feature aggregation-based transformer (LSFAT) architecture which allows transformers to represent long-range dependencies of multiscale features more efficiently. We introduce the concept of the homogeneous region into the transformer by considering a pixel aggregation strategy and further propose neighborhood-aggregation-based embedding (NAE) and attention (NAA) modules, which are able to adaptively form multiscale features and capture locally spatial semantics among them in a hierarchical transformer architecture. A reusable classification token is included together with the feature tokens in the attention calculation. In the last stage, a fully connected layer is used to perform classification on the reusable token after transformer encoding. We verify the effectiveness of the NAE and NAA modules compared with the traditional ViT through extensive experiments. Our results demonstrate the excellent classification performance of the proposed method in comparison to other state-of-the-art approaches on several public HSIs. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs) have demonstrated impressive capabilities in the representation and classification of hyperspectral remote sensing images. Traditional CNNs require massive data to sufficiently train the network. To tackle this problem, graph convolutional network (GCN) has been introduced for hyperspectral image classification. GCN methods usually construct the graph from either spectral or spatial domain, which has not adequately explored the information in the joint spectral–spatial domain. In this article, we propose a superpixel spectral–spatial feature fusion graph convolution network for hyperspectral image classification (S3FGCN). S3FGCN can comprehensively use information in spectral, spatial, and spectral–spatial domains with limited data. Moreover, to enhance the performance, we explore a shared weights’ GCN in the spectral–spatial domain. To further improve the efficiency, superpixels are used to construct the adjacency matrix. Finally, dynamic sampling is adopted to make the model focus more on difficult samples. In the experiments on four datasets, S3FGCN demonstrates better accuracy compared with the state-of-the-art hyperspectral image classification methods. [ABSTRACT FROM AUTHOR]

We investigate the problem of efficiently localizing sketch depicted scenes in a remote sensing image dataset. We pose the problem as that of remote sensing image retrieval with sketch queries and explore the use of hashing techniques to achieve efficient retrieval. Given two training datasets of sketches and remote sensing images that have a common set of class labels, we develop a hashing strategy that coconstructs two hash code books for the sketches and the remote sensing images separately. The hash code book coconstruction strategy encourages hash codes for the sketches and remote sensing images from different classes to be far away from one another and those from the same class to be close. This property is maintained by two cohesion intensive cues: 1) an interclass pairwise disperse cue (InterPDC) and 2) an intraclass pairwise balance cue (IntraPBC). We use the two coconstructed hash code books for training two linear mapping models that generate hash codes for sketches and remote sensing images separately. Sorting the Hamming distance between the sketch hash codes and the remote sensing image hash codes renders efficient remote sensing image retrieval with sketch queries. This enables localizing the sketch depicted scenes in the remote sensing image dataset. In addition, our method can also be used for fast localizing sketch depicted scenes in a remote sensing image of large size. Extensive experiments on public datasets validate the effectiveness and efficiency of our method. [ABSTRACT FROM AUTHOR]

Satellite-based synthetic aperture radar (SAR) can provide low-cost, frequent environment monitoring for dynamic intertidal zones. The critical problem is to realize pixel-level classification of SAR images of the intertidal zones with excellent and robust performance. Recently, deep learning, in particular deep convolutional neural networks, has provided us with promising solutions to this problem. Based on a sophisticated deep learning-based pixel-level classification model $\text{U}^{2}$ -Net, we propose an MB- $\text{U}^{2}$ -ACNet model suitable for intertidal zone land cover classification using dual-polarimetric SAR data integrated with environmental information, such as wind speed and tide level information. The MB- $\text{U}^{2}$ -ACNet model has a multibranch nested U-shaped encoding–decoding structure. We extract and fuse features from multiple data sources, including satellite remote sensing and environmental information, by establishing the multibranch structure. Furthermore, we propose an asymmetric convolution residual U-block for each encoding–decoding stage to improve the model’s feature extraction ability. Moreover, the model with attention mechanisms better distinguishes the importance of features from the channel’s perspectives and spatial dimensions. We construct a dataset with 106 Sentinel-1 SAR images from 2016 to 2020 for environmental monitoring in the intertidal zone of Shanghai Nanhui. On the dataset, the proposed model reaches the overall classification accuracy of 96.40% and the mean intersection over union score of 0.8307. The experiments show the advantages of the proposed model compared with the benchmarking models due to better feature extraction and multisource information fusion. In addition, the contributions of every added substructure are analyzed systematically. [ABSTRACT FROM AUTHOR]

Change detection (CD) is among the most important applications in remote sensing (RS) that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end users, and they omit important information such as relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article, we propose to describe the changes over bitemporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take bitemporal images as input and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bitemporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in the absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images, and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bitemporal images. The datasets are available at the following link: https://disi.unitn.it/~melgani/datasets.html. [ABSTRACT FROM AUTHOR]

As a typical application of remote sensing technology, change detection can find the ground information changes by acquiring the images of the same region at different times. The change detection using the synthetic aperture radar (SAR) with the advantages of all day and all-weather usually monitors the significant surface change, such as flood disasters and earthquake deformation. However, when it comes to detecting subtle changes such as vehicle traces, the traditional methods ignoring the phase coherence between image pairs cannot intensify these faint changes in the difference image. The SAR coherent change detection (CCD) based on repeat-pass repeat-geometry complex images utilizing both the intensity and phase fraction could exhibit the subtle vehicle trace in the difference image. However, the complicated background and decorrelation factors significantly affect the quality of difference images, further causing great trouble for automatic trace detection. This article proposes the spatial feature enhanced Unet and adaptive data augmentation to realize vehicle trace detection. More specifically, the pseudocolor image is first synthesized based on a two-stage coherence estimation method. Then, considering the long-continuity and parallel distribution of vehicle trace samples, the enhanced Unet is constructed by fusing spatial convolutional neural network and spatial attention mechanism. After that, the adaptation data augmentation strategy is presented by introducing manual registration errors and multiple estimation windows. Finally, the experimental results on the Sandia CCD data and our measured data demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

We propose a novel deep learning method using an attention-based fully convolutional dense network (FCDNet) for automatic earthquake detection. The FCDNet consists of encoder–decoder parts with skip connections, where each encoder–decoder block contains a block of densely connected layers to enhance the feature learning capability. The spatial attention mechanism is added within the FCDNet to assign greater attention to useful features and hence improve the accuracy of earthquake detection. The time–frequency representations of three-component seismograms produced by the Stockwell transform are used for better extracting the hidden data features. The attention-based FCDNet extracts the time–frequency features needed for distinguishing the seismic signal from the background noise. We evaluate the performance of the proposed method using a Mediterranean dataset. The attention-based FCDNet is trained using 90% of the Mediterranean dataset and tested using the remaining 10%. Accordingly, the training and testing accuracies are 97.71% and 97.02%, respectively. The intersection over union (IoU), precision, recall, and F1 score of the attention-based FCDNet are 93.80%, 99.72%, 99.55%, and 99.64%, respectively. Moreover, to evaluate the generalization ability of the trained model, we utilize 100000 seismic waveforms recorded in different seismic regions from the global STanford EArthquake Dataset (STEAD) dataset for testing, which shows robust performance. We also apply the attention-based FCDNet to the Japanese seismic data and compare the performance to the CRED and SCALODEEP methods. The attention-based FCDNet outperforms the benchmark methods and achieves a higher detection accuracy of 99.46%. The attention-based FCDNet is additionally evaluated using one-day continuous seismic data recording a seismic swarm that occurred in the Helike region. As a result, the attention-based FCDNet recognizes 135 earthquakes and raises 15 false alarms with a detection accuracy of 90.06%. [ABSTRACT FROM AUTHOR]

Accurate classification of remote sensing (RS) images is a perennial topic of interest in the RS community. Recently, transfer learning, especially for fine-tuning pretrained convolutional neural networks (CNNs), has been proposed as a feasible strategy for RS scene classification. However, because the target domain (i.e., the RS images) and the source domain (e.g., ImageNet) are quite different, simply using the model pretrained on an ImageNet dataset presents some difficulties. The RS images and the pretrained models need to be properly adjusted to build a better classification system. In this study, an adaptive learning strategy for transferring a CNN-based model is proposed. First, an adaptive transform is used to adjust the original size of the RS image to a certain size, which is tailored to the input of the subsequent pretrained model. Then, an adaptive transferring model is proposed to automatically learn what knowledge from the pretrained model should be transferred to the RS scene classification model. Finally, in combination with a label smoothing approach, an adaptive label is presented to generate soft labels based on the statistics of the classification model predictions for each category, which is beneficial for learning the relationships between the target and nontarget categories of scenes. In general, the proposed methods adaptively manage the input, model, and label simultaneously, which leads to better classification performance for RS scene classification. The proposed methods are tested on three widely used datasets, and the obtained results show that the proposed methods provide competitive classification accuracy compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

China is the largest coal consumer in the world. The massive exploitation and utilization of coal resources have resulted in serious problems of heavy metal pollution and environmental contamination, such as soil degradation, water pollution, crop damage, and even threatening human lives. Therefore, monitoring soil heavy metal pollution quickly and in real time is an urgent task at present. This research not only formulated a new preprocessing method enlightened by few-shot learning for soil hyperspectral data but also combined it with other soil-related auxiliary information to extract effective information from the soil hyperspectrum, at the end of which different regression methods were adopted to predict soil heavy metal contamination. This test used 168 actual soil samples from the Eastern Junggar coalfield in Xinjiang for verification. Since copper in the soil is a trace element and the corresponding spectral characteristics are affected by other impurities, improper use of hyperspectral preprocessing methods may introduce interference information or may delete useful information, which makes the model effect unsatisfied. To effectively address the above-mentioned problems, the preprocessing method of this experiment combined the second-order differential derivation, and the data enhancement (DA) method together with the addition of auxiliary information to allow more effective features to be entered into the model. Next, the attentive interpretable tabular learning (TabNet) model was improved in three different ways using the original TabNet model and three improved TabNet models to create regression models. One of the improved TabNet models had the best effect, with a list of the top 30 features according to the degree of importance. Meanwhile, the regression prediction of Cu content using four different convolutional neural networks (CNNs) revealed that the model with the residual block was the strongest and slightly outperformed the improved TabNet model, but lacked interpretation of the input data. Besides, this experiment also employed different preprocessing methods for regression prediction on various models and found that the traditional preprocessing methods performed best in traditional regression models [e.g., partial least square regression (PLSR)] and underperformed in deep learning models. The selected optimal model was compared with PLSR and CNN models. The results indicated that both the improved TabNet model and the improved CNN model had better performance using the new preprocessing approach proposed in this article, with improved TabNet yielding a coefficient of determination ($\text{R}^{2}$), root-mean-square error (RMSE), and the ratio of performance to interquartile range (RPIQ) of 0.94, 1.341, and 4.474, respectively. The improved CNN model had a coefficient of determination of 0.942, an RMSE of 1.324, and an interquartile range of 4.531 in the test dataset. [ABSTRACT FROM AUTHOR]

Standard hyperspectral (HS) pansharpening relies on fusion to enhance low-resolution HS (LRHS) images to the resolution of their matching panchromatic (PAN) images, whose practical implementation is normally under a stipulation of scale invariance of the model across the training phase and the pansharpening phase. By contrast, arbitrary resolution HS (ARHS) pansharpening seeks to pansharpen LRHS images to any user-customized resolutions. For such a new HS pansharpening task, it is not feasible to train and store convolution neural network (CNN) models for all possible candidate scales, which implies that the single model acquired from the training phase should be capable of being generalized to yield HS images with any resolutions in the pansharpening phase. To address the challenge, a novel variable subpixel convolution (VSPC)-based CNN (VSPC-CNN) method following our arbitrary upsampling CNN (AU-CNN) framework is developed for ARHS pansharpening. The VSPC-CNN method comprises a two-stage elevating thread. The first stage is to improve the spatial resolution of the input HS image to that of the PAN image through a prepansharpening module, and then, a VSPC-encapsulated arbitrary scale attention upsampling (ASAU) module is cascaded for arbitrary resolution adjustment. After training with given scales, it can be generalized to pansharpen HS image to arbitrary scales under the spatial patterns invariance across the training and pansharpening phases. Experimental results from several specific VSPC-CNNs on both simulated and real HS datasets show the superiority of the proposed method. [ABSTRACT FROM AUTHOR]

While deep learning algorithms have achieved good results in hyperspectral image (HSI) classification, several supervised classification algorithms rely on a large number of labeled samples to get adequate performance. Collecting a large number of labeled samples is expensive in many real applications. To address this issue, a novel semisupervised HSI classification framework called deep representative feature learning (DRFL) with virtual adversarial training (DRFL-VAT) is developed in this article. By embedding the local manifold learning (LML) into the fully connected layers of a convolutional neural network (CNN), our newly developed DRFL can learn representative features. The VAT regularization is adopted to exploit the prediction label distribution of training samples and addresses the overfitting problem. Finally, the objective function of DRFL-VAT is solved by a customized algorithm. We test our method on three widely public HSI datasets and our results show that our method is competitive when compared to other state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have been widely used in hyperspectral image classification (HSIC). Although the current CNN-based methods have achieved good performance, they still face a series of challenges. For example, the receptive field is limited, information is lost in down-sampling layer, and a lot of computing resources are consumed for deep networks. To overcome these problems, we proposed a multilevel spectral–spatial transformer network (MSTNet) for HSIC. The structure of MSTNet is an image-based classification framework, which is efficient and straightforward. Based on this framework, we designed a self-attentive encoder. First, HSIs are processed into sequences. Meanwhile, a learned positional embedding (PE) is added to integrate spatial information. Then, a pure transformer encoder (TE) is employed to learn feature representations. Finally, the multilevel features are processed by decoders to generate the classification results in the original image size. The experimental results based on three real hyperspectral datasets demonstrate the efficiency of the proposed method in comparison with the other related CNN-based methods. [ABSTRACT FROM AUTHOR]

Recently, deep-learning-based denoising methods for hyperspectral images (HSIs) have been comprehensively studied and achieved impressive performance because they can effectively extract complex and nonlinear image features. Compared with deep-learning-based methods, the nonlocal similarity-based denoising methods are more suitable for images containing edges or regular textures. We propose a powerful HSI denoising method, termed non-local 3-D convolutional neural network (NL-3DCNN), combining traditional machine learning and deep learning techniques. NL-3DCNN exploits the high spectral correlation of an HSI using subspace representation, and the corresponding representation coefficients are termed eigenimages. The high spatial correlation in eigenimages is exploited by grouping nonlocal similar patches, which are denoised by a 3-D convolutional neural network. The numerical and graphical denoising results of the simulated and real data show that the proposed method is superior to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The detection of early signs of volcanic unrest preceding an eruption in the form of ground deformation in interferometric synthetic aperture radar (InSAR) data is critical for assessing volcanic hazard. In this work, we treat this as a binary classification problem of InSAR images and propose a novel deep learning methodology that exploits a rich source of synthetically generated interferograms to train quality classifiers that perform equally well in real interferograms. The imbalanced nature of the problem, with orders of magnitude fewer positive samples, coupled with the lack of a curated database with labeled InSAR data, sets a challenging task for conventional deep learning architectures. We propose a new framework for domain adaptation, in which we learn class prototypes from synthetic data with vision transformers. We report detection accuracy that amounts to the highest reported accuracy on a large test set for volcanic unrest detection. Moreover, we built upon this knowledge by learning a new, nonlinear, projection between the learned representations and prototype space, using pseudo-labels produced by our model from an unlabeled real InSAR dataset. This leads to the new state-of-the-art with 97.1% accuracy on our test set. We demonstrate the robustness of our approach by training a simple ResNet-18 convolutional neural network on the unlabeled real InSAR dataset with pseudo-labels generated from our top transformer prototype model. Our methodology provides a significant improvement in performance without the need of manually labeling any sample, opening the road for further exploitation of synthetic InSAR data in various remote sensing applications. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) classification has been a hot topic for decides, as HSIs have rich spatial and spectral information, and provide a strong basis for distinguishing different land-cover objects. Benefiting from the development of deep learning technologies, deep learning-based HSI classification methods have achieved promising performance. Recently, several neural architecture search (NAS) algorithms have been proposed for HSI classification, which further improves the accuracy of HSI classification to a new level. In this article, NAS and transformer are combined for handling the HSI classification task for the first time. Compared with the previous work, the proposed method has two main differences. First, we revisit the search spaces designed in previous HSI classification NAS methods and propose a novel hybrid search space, consisting of the space-dominated cell and the spectrum-dominated cell. Compared with search spaces proposed in previous works, the proposed hybrid search space is more aligned with the characteristic of HSI data, that is, HSIs have a relatively low spatial resolution and an extremely high spectral resolution. Second, to further improve the classification accuracy, we attempt to graft the emerging transformer module on the automatically designed convolutional neural network (CNN) to add global information to local region focused features learned by CNN. Experimental results on three public HSI datasets show that the proposed method achieves much better performance than comparison approaches, including manually designed networks and NAS-based HSI classification methods. Especially on the most recently captured dataset Houston University, overall accuracy is improved by nearly 6 percentage points. Code is available at https://github.com/Cecilia-xue/HyT-NAS. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs) have achieved state-of-the-art performance in infrared small target detection. However, the limited number of public training data restricts the performance improvement of CNN-based methods. To handle the scarcity of training data, we propose a method that can generate synthetic training data for infrared small target detection. We adopt the generative adversarial network framework where synthetic background images and infrared small targets are generated in two independent processes. In the first stage, we synthesize infrared images by transforming visible images into infrared ones. In the second stage, target masks are implanted on the transformed images. Then, the proposed intensity modulation network synthesizes realistic target objects that can be diversely generated from further image processing. Experimental results on the recent public dataset show that, when we train various detection networks using the dataset composed of both real and synthetic images, detection networks yield better performance than using real data only. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have been extensively utilized for hyperspectral image (HSI) and light detection and ranging (LiDAR) data classification. However, CNNs have not been much explored for joint HSI and LiDAR image classification. Therefore, this article proposes a joint feature learning (HSI and LiDAR) and fusion mechanism using CNN and spatial morphological blocks, which generates highly accurate land-cover maps. The CNN model comprises three Conv3D layers and is directly applied to the HSIs for extracting discriminative spectral–spatial feature representation. On the contrary, the spatial morphological block is able to capture the information relevant to the height or shape of the different land-cover regions from LiDAR data. The LiDAR features are extracted using morphological dilation and erosion layers that increase the robustness of the proposed model by considering elevation information as an additional feature. Finally, both the obtained features from CNNs and spatial morphological blocks are combined using an additive operation prior to the classification. Extensive experiments are shown with widely used HSIs and LiDAR datasets, i.e., University of Houston (UH), Trento, and MUUFL Gulfport scene. The reported results show that the proposed model significantly outperforms traditional methods and other state-of-the-art deep learning models. The source code for the proposed model will be made available publicly at https://github.com/AnkurDeria/HSI+LiDAR. [ABSTRACT FROM AUTHOR]

Seismic full-waveform inversion (FWI) is able to build high-resolution velocity model based on the full information carried by seismic wave. However, FWI requires an accurate enough initial model to ensure convergence. In this article, we propose a new nonlinear FWI method to mitigate the initial model dependence problem. Specifically, we first propose a nonlinear operator within the hybrid model- and data-driven framework based on the frequency controllable envelope operator (FCEO) and a deep learning (DL) architecture U-Net. FCEO is used to obtain the envelope of a band-limited data and U-Net realizes the mapping from this envelope to that corresponding to a lower frequency band. The U-Net is trained in a self-supervised manner that avoids the reliance on labeled data and benefits the generalization ability. Based on the nonlinear operator, a nonlinear FWI method is proposed by defining a new misfit function. In addition, the calculation of gradient is derived using the adjoint state method. Using numerical examples, we investigate the performance of the proposed nonlinear operator and the new nonlinear FWI method. The results clearly demonstrate that the proposed nonlinear operator is effective in obtaining low-frequency envelope data, and the new nonlinear FWI method has advantages over common method in mitigating cycle-skipping and building an initial model for conventional FWI. [ABSTRACT FROM AUTHOR]

Accurate and reliable building footprint maps are vital to urban planning and monitoring, and most existing approaches fall back on convolutional neural networks (CNNs) for building footprint generation. However, one limitation of these methods is that they require strong supervisory information from massive annotated samples for network learning. State-of-the-art semi-supervised semantic segmentation networks with consistency training can help deal with this issue by leveraging a large amount of unlabeled data, which encourages the consistency of model output on data perturbation. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples, enabling to impose additional constraints. Prior semi-supervised semantic segmentation networks have established cluster assumption, in which the decision boundary should lie in the vicinity of low sample density. In this work, we observe that for building footprint generation, low-density regions are more apparent at the intermediate feature representations within the encoder than the encoder’s input or output. Therefore, we propose an instruction to assign the perturbation to the intermediate feature representations within the encoder, which considers the spatial resolution of input remote sensing imagery and the mean size of individual buildings in the study area. The proposed method is evaluated on three datasets with different resolutions: Planet dataset (3 m/pixel), Massachusetts dataset (1 m/pixel), and Inria dataset (0.3 m/pixel). Experimental results show that the proposed approach can well extract more complete building structures and alleviate omission errors. [ABSTRACT FROM AUTHOR]

Improving the spatial resolution of hyperspectral images (HSIs) has traditionally been an important topic in the field of remote sensing. Many approaches have been proposed based on various theories, including component substitution, multiresolution analysis, spectral unmixing, Bayesian probability, and tensor representation. However, these methods have some common disadvantages such that their performance degrades dramatically as the up-scale ratio increases, and they have little concern for the per-pixel radiometric accuracy of the sharpened image. Moreover, many learning-based methods have been proposed through decades of innovations, but most of them require a large set of training pairs, which is unpractical for many real problems. To solve these problems, we propose a stable hyperspectral sharpening method based on the Laplacian pyramid and the generative convolutional neural network (CNN), which achieves superior radiometric accuracy of the sharpened data in different up-scale ratios based on one single input pair. First, with a low-resolution HSI (LR-HSI) and high-resolution multispectral image (HR-MSI) pair, the preliminary high-resolution HSI (HR-HSI) is calculated via linear regression. Then, the high-frequency details of the preliminary HR-HSI are estimated via the subtraction between it and the CNN-generated-blurry version. By injecting the details to the output of the generative CNN with the LR-HSI as input, the final HR-HSI is obtained. Nine different state-of-the-art sharpening methods are chosen as our baselines, and three different datasets with different scene content are tested. Furthermore, the target detection method, the adaptive coherence estimator (ACE), is conducted on the reconstructed HR-HSI to evaluate the per-pixel radiometric accuracy. The results demonstrate that the proposed method has the best and the most stable performance in terms of spectral and spatial accuracies. [ABSTRACT FROM AUTHOR]

Change detection (CD) of remote sensing (RS) images has enjoyed remarkable success by virtue of convolutional neural networks (CNNs) with promising discriminative capabilities. However, CNNs lack the capability of modeling long-range dependencies in bitemporal image pairs, resulting in inferior identifiability against the same semantic targets yet with varying features. The recently thriving Transformer, on the contrary, is warranted, for practice, with global receptive fields. To jointly harvest the local-global features and circumvent the misalignment issues caused by step-by-step downsampling operations in traditional backbone networks, we propose an intra-scale cross-interaction and inter-scale feature fusion network (ICIF-Net), explicitly tapping the potential of integrating CNN and Transformer. In particular, the local features and global features, respectively, extracted by CNN and Transformer, are interactively communicated at the same spatial resolution using a linearized Conv Attention module, which motivates the counterpart to glimpse the representation of another branch while preserving its own features. In addition, with the introduction of two attention-based inter-scale fusion schemes, including mask-based aggregation and spatial alignment (SA), information integration is enforced at different resolutions. Finally, the integrated features are fed into a conventional change prediction head to generate the output. Extensive experiments conducted on four CD datasets of bitemporal (RS) images demonstrate that our ICIF-Net surpasses the other state-of-the-art (SOTA) approaches. [ABSTRACT FROM AUTHOR]

In this article, we propose an efficient and generalizable framework based on a deep convolutional neural network (CNN) for multisource remote sensing (RS) data joint classification. While recent methods are mostly based on multistream architectures, we use group convolution (GConv) to construct equivalent network architectures efficiently within a single-stream network. Based on a recent technique called dynamic grouping convolution (DGConv), we further propose a network module named separable DGConv (SepDGConv), to make GConv hyperparameters, and, thus, the overall network architecture, learnable during network training. In the experiments, the proposed method is applied to residual network (ResNet) and UNet, and the adjusted networks are verified on three very diverse benchmark datasets (i.e., Houston2018 data, Berlin data, and MUUFL Gulfport Hyperspectral and LiDAR Airborne Data Set (MUUFL) data). Experimental results demonstrate the effectiveness of the proposed single-stream CNNs, and in particular, SepG-ResNet18 improves the state-of-the-art classification overall accuracy (OA) on hyperspectral–synthetic aperture radar (HS–SAR) Berlin dataset from 62.23% to 68.21%. In the experiments we have two interesting findings. First, using DGConv generally reduces test OA variance. Second, multistream is harmful to model performance if imposed to the first few layers, but becomes beneficial if applied to deeper layers. Altogether, the findings imply that the multistream architecture, instead of being a strictly necessary component in deep learning models for multisource RS data, essentially plays the role of model regularizer. Our code is publicly available at https://github.com/yyyyangyi/CNNs-for-Multi-Source-Remote-Sensing-Data-Fusion. We hope our work can inspire novel research in the future. [ABSTRACT FROM AUTHOR]

The lack of labels is one of the major challenges in hyperspectral image (HSI) classification. Widely used Deep Learning (DL) models such as convolutional neural networks (CNNs) experience serious performance degradation when training samples are limited. In contrast, graph convolutional networks (GCNs) can simultaneously exploit the insufficient labeled data and massive unlabeled data of HSI in a semisupervised learning fashion. However, in order to reduce computational cost and mitigate noise, existing GCN-based classification methods usually perform superpixel segmentation as a preprocessing step and implement feature extraction as well as node classification on the predefined superpixel graph, where one superpixel might incorporate pixels with different labels. Moreover, the local spectral–spatial information within superpixels is generally ignored. To alleviate these two issues, we propose an end-to-end mixhop superpixel-based GCN (EMS-GCN) framework for HSI classification. Specifically, we first introduce the differentiable superpixel segmentation algorithm to map the pixel representations into a superpixel feature space, which allows refining the superpixel boundary with the training of the network. After that, a superpixel graph is constructed and fed into a novel mixhop superpixel-based GCN, where both the local information within superpixels and long-range information among superpixels are extracted, while the structure of the superpixel graph is updated at the same time. Finally, the enhanced superpixel representations are mapped back into a pixel feature space to conduct pixel-wise classification. Extensive experiments demonstrate the effectiveness of the proposed EMS-GCN method compared with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Infrared small target detection plays an important role in target warning, ground monitoring, and flight guidance. Existing methods typically utilize local-contrast information of each pixel to detect infrared small targets, neglecting the interior relation between target pixels or background pixels. The mere use of the local information of one pixel, however, is not sufficient for accurate detection, which may lead to missing detection and false alarms. As a harmonious whole, information between pixels are necessary to determine if a pixel belongs to the target or the background. Motivated by the fact that pixels from targets or backgrounds are correlated with each other, we propose a coarse-to-fine interior attention-aware network (IAANet) for infrared small target detection. Specifically, a region proposal network (RPN) is first applied to obtain coarse target regions and filter out backgrounds. Then, we leverage a transformer encoder to model the attention between pixels in coarse target regions, outputting attention-aware features. Finally, predictions are obtained by feeding attention-aware features to a classification head. Extensive experiments show that our approach is capable of detecting targets precisely, of suppressing a variety of false alarm sources, and works effectively in various background environments and target appearances. We show that our IAANet outperforms the state-of-the-art methods by a large margin. Code will be made available at: https://github.com/kwwcv/iaanet. [ABSTRACT FROM AUTHOR]

Recently, several recurrent neural network (RNN)-based models have been proposed to delineate the outlines of buildings from very high resolution (VHR) remote sensing images. These models first use convolutional neural networks (CNNs) to recognize the boundary fragments by learning probability maps of both edges and corners and then feed them into RNN to find and link a set of sequent corners into external boundaries of buildings. However, caused by the category imbalance of edges and corners, the local ambiguity of edge detection is very serious, which significantly affects the accuracy of predicted outline corners. To tackle this challenge, this article introduces a convolutional RNN embedded with line segment information (LSI-RNN), a novel network that aims to directly detect line segment instead of edges. To achieve this, LSI-RNN utilizes an additional cotraining branch to generate an attraction field map (AFM) by neural discriminative dimensionality reduction (NDDR) layer. Consequently, the conventional classification problem of edges is converted to a regression problem of line segments, thus solving the aforementioned issues. Experimental results over three remote sensing datasets with different spatial resolutions show that the proposed method consistently outperforms other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The scene classification of high spatial resolution (HSR) imagery involves labeling an HSR image with a specific high-level semantic class according to the composition of the semantic objects and their spatial relationships. As such, scene classification has attracted increased attention in recent years, and many different algorithms have now been proposed for the cross-scene classification task. However, the recently proposed scene classification methods based on deep convolutional neural networks (CNNs) still suffer from domain shift problems, because of the training data and validation data not following the assumption of independent and identical distributions. The employment of generative adversarial networks has been found to be an effective way to bridge the domain shift/gap. However, the existing cross-scene classification methods do not use the classification information in the target domain, and the domain classifier is task-independent for different scene classification tasks. In this article, to solve this problem, domain adaptation via a task-specific classifier (DATSNET) framework is proposed for HSR image scene classification. Task-specific classifiers and minimizing and maximizing, “ i.e., minimaxing,” of the classifier discrepancy are integrated in the DATSNET framework. The task-specific classifiers are proposed to align the distributions of the source domain features and target domain features by utilizing task-specific decision boundaries in the target domain. In order to align the two task-specific classifiers’ feature distributions, minimaxing the defined discrepancy between the different classifiers in an adversarial manner is proposed to obtain better task-specific classifier boundaries in the target domain and a better-aligned feature distribution in both domains. The experimental results obtained with different remote sensing cross-scene classification tasks demonstrate that the proposed method achieves a significantly improved performance compared with the other state-of-the-art remote sensing cross-scene classification algorithms. [ABSTRACT FROM AUTHOR]

With the increasing availability and resolution of satellite sensor data, multispectral (MS) and panchromatic (PAN) images are the most popular data that are used in remote sensing among applications. This article proposes a novel cross-resolution hidden layer feature fusion (CRHFF) approach for joint classification of multiresolution MS and PAN images. In particular, shallow spectral and spatial features at a global scale are first extracted from an MS image. Then, deep cross-resolution hidden layer features extracted from MS and PAN are fused from patches at a local scale according to an autoencoder (AE)-like deep network. Finally, the selected multiresolution hidden layer features are classified in a supervised manner. By taking advantage of integrated shallow-to-deep and global-to-local features from the high-resolution MS and PAN images, the cross-resolution latent information can be extracted and fused in order to better model imaged objects from the multimodal representation and finally increase the classification accuracy. Experimental results obtained on three real multiresolution datasets covering complex urban scenarios confirm the effectiveness of the proposed approach in terms of higher accuracy and robustness with respect to literature methods. [ABSTRACT FROM AUTHOR]