Search

As an important yet challenging task in Earth observation, change detection (CD) is undergoing a technological revolution, given the broadening application of deep learning. Nevertheless, existing deep learning-based CD methods still suffer from two salient issues: 1) incomplete temporal modeling, and 2) space-time coupling. In view of these issues, we propose a more explicit and sophisticated modeling of time and accordingly establish a pair-to-video change detection (P2V-CD) framework. First, a pseudo transition video that carries rich temporal information is constructed from the input image pair, interpreting CD as a problem of video understanding. Then, two decoupled encoders are utilized to spatially and temporally recognize the type of transition, and the encoders are laterally connected for mutual promotion. Furthermore, the deep supervision technique is applied to accelerate the model training. We illustrate experimentally that the P2V-CD method compares favorably to other state-of-the-art CD approaches in terms of both the visual effect and the evaluation metrics, with a moderate model size and relatively lower computational overhead. Extensive feature map visualization experiments demonstrate how our method works beyond making contrasts between bi-temporal images. Source code is available at https://github.com/Bobholamovic/CDLab. [ABSTRACT FROM AUTHOR]

In he past decade, object detection has achieved significant progress in natural images but not in aerial images, due to the massive variations in the scale and orientation of objects caused by the bird’s-eye view of aerial images. More importantly, the lack of large-scale benchmarks has become a major obstacle to the development of object detection in aerial images (ODAI). In this paper, we present a large-scale Dataset of Object deTection in Aerial images (DOTA) and comprehensive baselines for ODAI. The proposed DOTA dataset contains 1,793,658 object instances of 18 categories of oriented-bounding-box annotations collected from 11,268 aerial images. Based on this large-scale and well-annotated dataset, we build baselines covering 10 state-of-the-art algorithms with over 70 configurations, where the speed and accuracy performances of each model have been evaluated. Furthermore, we provide a code library for ODAI and build a website for evaluating different algorithms. Previous challenges run on DOTA have attracted more than 1300 teams worldwide. We believe that the expanded large-scale DOTA dataset, the extensive baselines, the code library and the challenges can facilitate the designs of robust algorithms and reproducible research on the problem of object detection in aerial images. [ABSTRACT FROM AUTHOR]

A voltage drop normally occurs when a power source runs for a load system through long-distance cables. To compensate for such voltage drops, it is necessary to provide remote detecting and controlling of the load-side voltage and boost to the required voltage. In this article, a novel impedance detection method providing more reliable and accurate resistance and inductance measured results for remote cables is presented. Meanwhile, a digital–analog hybrid control strategy based on the impedance detection method is presented to compensate for the voltage drop and delayed response caused by cable intrinsic impedance. Compared with traditional methods, the proposed strategy performs detection and compensation at the source of the cable, and can be integrated into power source devices without imposing extra burden to the load system, which is more attractive and promising in engineering application. Finally, a prototype is built for experimental verification, and a systematic and intuitive comparison is conducted with existing voltage-drop compensation methods. The experimental and comparative results indicate that the problems of voltage drop and delayed response caused by long-distance cables are solved, and the accurate and rapid regulation of the remote voltage is achieved. [ABSTRACT FROM AUTHOR]

The neural network-based remote sensing image change detection method faces a large amount of imaging interference and severe class imbalance problems under high-resolution conditions, which bring new challenges to the accuracy of the detection network. In this work, to address the imaging interference caused by different imaging angles and times, the siamese strategy and multi-head self-attention mechanism are used to reduce the imaging differences between the dual-temporal images and fully exploit the inter-temporal information. Secondly, a learnable multi-part feature learning module is used to adaptively exploit features from different scales to obtain more comprehensive features. Finally, a mixed loss function strategy is used to ensure that the network converges effectively and excludes the adverse interference of a large number of negative samples to the network. Extensive experiments show that our method outperforms numerous methods on LEVIR-CD, WHU, and DSIFN datasets. [ABSTRACT FROM AUTHOR]

With the increasing of on-orbit satellites, target dynamic monitoring becomes more and more important in space situation awareness applications. Currently, there are some exploratory methods monitoring attitude-stabilized targets using various high-resolution remote sensing technologies. However, it is still a big challenge to achieve dynamic estimation of spin satellite with single sensor. Inspired by the existing matching based works, this article presents a dynamic estimation method of spin satellite using single-station inverse synthetic aperture radar (ISAR) images. When the angle accumulation is set to be a constant in each ISAR imaging period, the projection change of target structures in the long-term observation sequence is described with the hidden Markov model to build the mathematic expression of target on-orbit state. In this condition, target sequential attitude can be solved with Viterbi decoding algorithm even the azimuth scaling of each image is difficulty due to target spin motion. Then, combined with the radar observation geometry, target sequential attitude parameters are substituted into a spin parameter optimization, which is solved by particle swarm optimization algorithm. In the end, target instantaneous attitude vectors and spin speed are used to express its dynamic during the observation period. Simulation experiments of a typical spin spacecraft, Tiangong-I, confirm the feasibility of the proposed algorithm, and its superiority is also investigated by comparison with the existing work. [ABSTRACT FROM AUTHOR]

Reducing the feature distribution shift caused by the factor of visual-environment changes, named visual-environment changes (VE-changes), is a hot issue in domain adaptation learning. However, in the semantic segmentation task of remote sensing imageries, besides VE-changes, the change of semantic-scene changes (SS-changes) is another factor raising the domain gap, which brings the label distribution shift. For example, although urban and rural share the same landcover label, there is still a gap in label distribution. If there is little relation that can be found in neither feature nor label space, forcibly adapting to a new domain could have a high risk of negative transfer. Hence, we propose a new Transitive Domain Adaptation method for Remote Sensing (TDARS) images. First, we introduce an intermediate domain to enlarge the relation between the given source and target domains. Second, we learn from primary and nonprimary confident classes to increase the likelihood of transferring valuable information. As a result, TDARS enables the given source and target domains to be connected through the selected intermediate domain and performs effective knowledge transfer among all domains. The proposed method is evaluated on three domain adaptation datasets of remote sensing images. Extensive experiments show that the approach can effectively handle the domain shift problem from remote sensing images compared to other state-of-the-art domain adaptation methods. [ABSTRACT FROM AUTHOR]

Few-shot remote sensing scene classification (FSRSSC) is an important task, which aims to recognize novel scene classes with few examples. Recently, several studies attempt to address the FSRSSC problem by following few-shot natural image classification methods. These existing methods have made promising progress and achieved superior performance. However, they all overlook two unique characteristics of remote sensing images: 1) object cooccurrence that multiple objects tend to appear together in a scene image and 2) object spatial correlation that these cooccurrence objects are distributed in the scene image following some spatial structure patterns. Such unique characteristics are very beneficial for FSRSSC, which can effectively alleviate the scarcity issue of labeled remote sensing images since they can provide more refined descriptions for each scene class. To fully exploit these characteristics, we propose a novel scene-graph-matching-based meta-learning framework for FSRSSC, called SGMNet. In this framework, a scene graph construction module is carefully designed to represent each test remote sensing image or each scene class as a scene graph, where the nodes reflect these cooccurrence objects; meanwhile, the edges capture the spatial correlations between these cooccurrence objects. Then, a scene graph matching module is further developed to evaluate the similarity score between each test remote sensing image and each scene class. Finally, based on the similarity scores, we perform the scene class prediction via a nearest neighbor classifier. We conduct extensive experiments on the UCMercedLandUse, WHU19, AID, and NWPU-RESISC45 datasets. The experimental results show that our method obtains superior performance over the previous state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Recently, the burgeoning demands for captioning-related applications have inspired great endeavors in the remote sensing community. However, current benchmark datasets are deficient in data volume, category variety, and description richness, which hinders the advancement of new remote sensing image captioning approaches, especially those based on deep learning. To overcome this limitation, we present a larger and more challenging benchmark dataset termed NWPU-Captions is available at https://github.com/HaiyanHuang98/NWPU-Captions. NWPU-Captions contains 157 500 sentences, with all 31 500 images annotated manually by seven experienced volunteers. The superiority of NWPU-Captions over current publicly available benchmark datasets not only lies in its much larger scale but also in its wider coverage of complex scenes and the richness and variety of describing vocabularies. Furthermore, a novel encoder–decoder architecture, multilevel and contextual attention network (MLCA-Net), is proposed. MLCA-Net employs a multilevel attention module to adaptively aggregate image features of specific spatial regions and scales and introduces a contextual attention module to explore the latent context hidden in remote sensing images. MLCA-Net improves the flexibility and diversity of the generated captions while keeping their accuracy and conciseness by exploring the properties of scale variations and semantic ambiguity. Finally, the effectiveness, robustness, and generalization of MLCA-Net are proved through extensive experiments on existing datasets and NWPU-Captions. [ABSTRACT FROM AUTHOR]

The Hydrometeorological Center of Russia receives agrometeorological information from about 950 stations one time per ten days and the remote sensing Advanced Scatterometer (ASCAT) data from three Meteorological Operational (MetOp) satellites. We suggest a combined objective analysis (OA) of the available water content based on the available water content measurements at agrometeorological stations and on remote sensing data. The new version of OA is constructed using two neural networks and the backpropagation of error to learn it simultaneously. The first neural network is used to convert the ASCAT data into the available water content values, and the second network is used to estimate the inhomogeneities of soil moisture fields. We use the optimal interpolation (OI) method for assimilation of the ground-based data. In the new version, we evaluate the correlation functions (CFs) of inhomogeneous non-Gaussian fields, not from sample statistics but from machine learning methods. The method takes into account the combining of various datasets: ASCAT data, Food and Agriculture Organization (FAO) soil types, European Space Agency (ESA) GlobCover, and National Center for Atmospheric Research (NCAR) climate data. [ABSTRACT FROM AUTHOR]

Aerial scene classification remains challenging as: 1) the size of key objects in determining the scene scheme varies greatly and 2) many objects irrelevant to the scene scheme are often flooded in the image. Hence, how to effectively perceive the region of interests (RoIs) from a variety of sizes and build more discriminative representation from such complicated object distribution is vital to understand an aerial scene. In this article, we propose a novel all grains, one scheme (AGOS) framework to tackle these challenges. To the best of our knowledge, it is the first work to extend the classic multiple instance learning (MIL) into multigrain formulation. Specifically, it consists of a multigrain perception (MGP) module, a multibranch multi-instance representation (MBMIR) module, and a self-aligned semantic fusion (SSF) module. First, our MGP module preserves the differential dilated convolutional features from the backbone, which magnifies the discriminative information from multigrains. Then, our MBMIR module highlights the key instances in the multigrain representation under the MIL formulation. Finally, our SSF module allows our framework to learn the same scene scheme from multigrain instance representations and fuses them, so that the entire framework is optimized as a whole. Notably, our AGOS is flexible and can be easily adapted to existing convolutional neural networks (CNNs) in a plug-and-play manner. Extensive experiments on UCM, aerial image dataset (AID), and Northwestern Polytechnical University (NWPU) benchmarks demonstrate that our AGOS achieves a comparable performance against the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Low-resolution ship detection from optical satellite image sequences is critical in high-orbit remote sensing satellite applications. However, it is still a difficult problem due to the following challenges: 1) the size of the ship is tiny in the low-resolution image; 2) the ship target is dim and the contrast with the background is low; and 3) the interference of cloud and fog covering is complex and changeable. For these reasons, the targets are easily lost during the detection. In fact, the Clearer the Objects against to the background, the more Confident the Observers can detect it. In light of these considerations, we propose a COCO-Net to detect the small dynamic objects on low-resolution images in this article. First, the multiframe images are associated by introducing motion information as an effective compensation for small object features. Second, an integrated dual-supervised network that processes single-level tasks hierarchically is presented to adaptively enhance the input data quality of object detection without being limited by diverse scene disturbances. Third, a unified region of interest (ROI)-loss scheme that modulates the loss function of the first component by introducing ROI-masks from the second component is utilized to make the first component also work for object detection. In addition, we construct a new dataset for the small dynamic object detection based on the GaoFen-4 satellite imagery. Comprehensive experiments on a self-assembled dataset from the GaoFen-4 satellite show the superior performance of the proposed method compared to state-of-the-art object detectors. [ABSTRACT FROM AUTHOR]

Due to the all-weather and all-time imaging capability of synthetic aperture radar (SAR), SAR remote sensing analysis has attracted much attention recently. However, compared with the optical images, SAR images are more difficult to be interpreted. If an SAR image could be translated into its corresponding optical image, then the generated optical image would be helpful for assisting the interpretation. Addressing this issue, we investigate how to translate SAR images into optical ones in this work and propose a parallel generative adversarial model for SAR-to-optical image translation, called parallel generative adversarial network (Parallel-GAN), consisting of a backbone image translation subnetwork and an adjoint optical image reconstruction subnetwork. Under the proposed model, the backbone image translation subnetwork is designed to translate SAR images into optical ones, and simultaneously some of its intermediate layers are required to output similar latent features to those from the corresponding layers of the adjoint image reconstruction subnetwork. Thanks to the imposed hierarchical latent optical features, the proposed Parallel-GAN could achieve the SAR-to-optical image translation effectively. Extensive experimental results on three public datasets demonstrate that the proposed method outperforms ten state-of-the-art methods for SAR-to-optical image translation. [ABSTRACT FROM AUTHOR]

Although ship detection methods based on deep learning have achieved remarkable progress, the design of the universal ship detection (USD) system is rarely studied. The main challenge of USD lies in the notorious domain bias and shift problem across multiple domains. This article implements USD based on domain-invariant representations to alleviate this issue. Specifically, the proposed method integrates a multilevel domain classification network (MDCN) and a domain-centric cut-paste module (DCM). First, the backbone network is facilitated to learn domain-independent image features from multiple domains through MDCN, thereby reducing the disturbance of domain-specific features to universal detector. Furthermore, the proposed method combines the domain-related synthetic samples generated by DCM to provide MDCN with strong supervision information, which further motivates the network to be more attentive to the domain-invariant representations at the instance level. Finally, we conduct experiments on multiple ship datasets in the synthetic aperture radar (SAR) and optical domains to verify the effectiveness of the method. The results show that the proposed method outperforms baseline by around 2.95% average precision (AP50), which achieves an effective USD system by complementing the information between domain-invariant representations. [ABSTRACT FROM AUTHOR]

The current regional-scale process-based photosynthesis models use biome-specified values of maximum carboxylation rate at 25 °C ($V_{m25}$) in simulating ecosystem gross primary productivity (GPP). These models ignore the variations in $V_{m25}$ over time and space, resulting in substantial errors in regional estimates of cropland GPP. Thus, to resolve this problem, we used the ensemble Kalman filter (EnKF) to assimilate tower-based GPP from five maize flux sites into a process-based mode to obtain the “apparent” value of $V_{m25}$ and then modeled this parameter using machine learning (ML) algorithms. The results showed that $V_{m25}$ increased during the early growing season and then decreased after reaching a peak value in the middle of the growing season. The coefficient of determination ($R^{2}$) root mean square error (RMSE) for satellite-driven coupled photosynthesis and evapotranspiration simulator (SCOPES)-Crop with EnKF-derived varied $V_{m25}$ in simulating daily GPP across all site-days increased (decreased) by 0.17 (5.63 $\mu \text {mol}\,\text {m}^{-2}\,\text {s}^{-1}$) on average compared to that for the model with fixed $V_{m25}$. We used four ML algorithms, namely artificial neural network, random forest, extreme gradient enhancement, and convolutional neural network (CNN), to model the $V_{m25}$ of maize. The CNN algorithm yielded the best results. The average of the $R^{2}$ (RMSE) values of simulated GPP using CNN-based $V_{m25}$ over the three flux sites is 0.93 (1.95 $\mu \text {mol}\,\text {m}^{-2}\,\text {s}^{-1}$), higher (smaller) than that using fixed $V_{m25}$. This study implies that representing the seasonal variations in $V_{m25}$ can facilitate improved estimates of GPP and the ML methods are useful tools for modeling the variation in $V_{m25}$. [ABSTRACT FROM AUTHOR]

Object detection in remote sensing images (RSIs) poses great difficulties due to arbitrary orientations, various scales, and dense location of the targets over the ground. Recent evidence suggests that encoding the orientation information is of great use for training an accurate object detector for oriented object detection (OOD). In this article, we propose a new frequency-domain orientation learning (FDOL) module with two main components: the frequency-domain feature extraction (FFE) network and an orientation enhanced self-attention layer (OES-Layer). The FFE network models the interactions among spatial locations in the frequency domain to determine the frequency of spatial features. Then, these features are fed into our OES-Layer to learn the orientation information. Moreover, the orientation weights are adopted to guide the feature selection in a self-attention (SA) architecture, using them as a control gate to emphasize the spatial responses of target instances. Considering that the original similarity weights (calculated by the SA algorithm) do not distinctly model the orientation variation, the considered orientation weights provide an efficient asset to emphasize the orientation of objects. Extensive experiments on the DOTA and HRSC2016 datasets demonstrate that our method achieves state-of-the-art performance among single-scale methods while achieving competitive performance over multiscale methods. [ABSTRACT FROM AUTHOR]

Radar sounders (RSs) are gaining importance in planetary missions thanks to their unique capability of providing direct measurements of subsurface (SS) structures. To support their design and data interpretation, several electromagnetic (e.m.) simulation techniques have been developed with enhanced capabilities for emulating the RS acquisition process. However, the raw simulated radargrams obtained from e.m. simulators are difficult to interpret and analyze without a focusing operation, which results in an underestimation of the RS detection performance. While frequency methods for range and azimuth compression of real RS data are well-established, their use on simulated data is not addressed in the literature and requires major modifications. This article presents a novel method that implements azimuth compression using unfocused and focused processing on simulated raw data. The proposed method is based on an adaptation of the range-Doppler algorithm to the case of raw data generated by a coherent RS simulator. The method is demonstrated in three case studies to show the similarity between simulated and real data processing: 1) simple geometries; 2) a simulated SHAllow RADar (SHARAD) radargram compared with the real data product; and 3) a real application scenario for supporting the design of a new RS instrument. The results indicate higher fidelity of the focused simulated data with the real data product and the target structure, confirming the usefulness of the proposed approach in obtaining realistic processing of simulated radargrams. [ABSTRACT FROM AUTHOR]

Damaged building detection from remote sensing imagery helps to quickly and rapidly assess losses after an earthquake. In recent years, deep learning technology has become a favorable tool for remote sensing image information detection. Based on the characteristics of damaged buildings in remote sensing images, in this article, a framework for damaged building detection that considers heterogeneity characteristics is proposed. First, a local–global context attention module is proposed to improve the feature detection ability of the network, which can extract the features of damaged buildings from different directions and effectively aggregate global and local features. In addition, the module takes the correlation between feature maps at different scales into account while extracting information. Second, a feature fusion module with self-attention is established to replace the simple connection between the encoding and decoding processes, which improves the detail feature recovery ability of the network during the upsampling process. Finally, to fully aggregate semantic and detailed features at different scales, a multibranch auxiliary classifier is established by adding two separate branches in the prediction stage. The effectiveness of the proposed approach is verified based on the data from the 2010 Haiti earthquake, and comparisons with three object-oriented methods and 16 existing excellent deep learning models are performed. The intersection over union (IOU) increase of 0.03%–7.39% is achieved using the proposed approach compared with excellent deep learning models. [ABSTRACT FROM AUTHOR]

Different-resolution change detection (DRCD) is now becoming an urgent problem to be solved, which is of great potential in rapid monitoring, such as disaster assessment and urban expansion. In DRCD tasks, bitemporal inputs are given in the form of different resolutions, and thus, conventional change detection (CD) methods cannot be applied directly. Previous studies have attempted to deal with this problem by reconstructing the low-resolution (LR) image into a high-resolution (HR) one, including interpolation and super-resolution (SR). However, these solutions are limited by the availability of training data, making it hard to meet different kinds of needs. Besides, these image-level strategies have also ignored the interaction and alignment of high-level features. Therefore, we propose a new approach based on multimodel Transformers (MM-Trans), which solves the resolution gaps of bitemporal inputs in DRCD tasks from the perspective of feature alignment. In the MM-Trans, a weight-unshared feature extractor is first utilized to precisely capture the features of the different-resolution inputs. Then, a spatial-aligned Transformer (sp-Trans) is introduced to align the LR-image features to the same size of the HR-image ones, which can be optimized in a learnable way by an auxiliary token loss. After that, a semantic-aligned Transformer (se-Trans) is adopted, in which the bitemporal features can be further interacted and aligned semantically. Finally, a prediction head is employed to obtain fine-grained change results. Experiments conducted on three common CD datasets, CDD, S2Looking, and HTCD dataset, have shown the advancement of the MM-Trans and fully demonstrated its potential in DSCD tasks. [ABSTRACT FROM AUTHOR]

By reason of factors such as terrains, weather conditions, sensor imaging methods, and cultural and economic development, there is a large shift between the remote sensing imagery collected from different geographic locations and different sensors, which makes the state-of-the-art semantic segmentation models trained on source domain (an image set gathered from specific geographic locations and sensors) difficult to generalize to target domain (another image set collected from other geographic locations and sensors). Currently, unsupervised domain adaptation (UDA) using adversarial training, whose purpose is to align the marginal distribution in the output space between the source and target domains, is the most explored and practical approach to address this issue. However, this global alignment approach does not take into account diversities of different regions in a specific image nor the category-level distribution, which leads to the consequence that some regions and categories which are already well aligned between the source and target domains may be incorrectly remapped. Therefore, we propose a region and category adaptive domain discriminator (RCA-DD), aiming to emphasize the differences in regions and categories during the process of alignment. Specifically, on the one hand, we propose an entropy-based regional attention module (ERAM) in domain discriminator to emphasize the importance of difficult-to-align regions. On the other hand, we propose a class-clear module (CCM) to update only the distribution of existing categories in one iteration without affecting all categories. Finally, a lot of experiments are introduced to indicate that the proposed method can obtain better results when compared with other state-of-the-art UDA methods using adversarial training. [ABSTRACT FROM AUTHOR]

The persistent scatterer interferometry (PSI) is a powerful remote sensing technique to monitor displacements of the Earth’s surface. It is based on identifying and analyzing phase-stable scatterers undergoing marginal decorrelation over time. Most PSI approaches constrain the analysis to scatterers which are coherent over the whole considered time series persistent scatterer (PS) and neglect scatterers which are only temporary persistent (TPS). Here, we propose a method to fully integrate TPS into an existing PSI approach which is characterized by an iterative parameter estimation with subsequent phase unwrapping. We build on a previously published method to identify TPS and estimate their coherent lifetime based on amplitude statistics. A phase-based likelihood ratio test is proposed to iteratively refine the appearing/fading dates of TPS during parameter estimation. Finally, we jointly unwrap the phase observations of PS and TPS to receive displacement time series. The temporal data of TPS are redefined if their lifetime does not cover the selected master scene. Experimental results based on Sentinel-1 data in the Vietnamese city of Ca Mau show that the change date refinement significantly increases the average coherence and number of identified TPS. The densification of the observation point network by incorporating TPS helps better detect and understand displacement phenomena. The displacement time series of TPS are beneficial to analyze the nonlinear motion in connection with urban development, like initial settlement of newly constructed buildings. [ABSTRACT FROM AUTHOR]

There is a need to map underground pipelines due to nonavailable existing pipeline maps caused by poor management of statutory records and insufficient updating of documentation whenever pipeline construction or rerouting occurs. By fusing multisource data, a novel method to map underground pipelines is proposed in this article. Statutory records of the underground pipelines are converted into the initial pipeline map. Pipeline information obtained from manhole covers and remote sensing technologies are normalized into the pipeline dataset composed of detected points. The probabilistic pipeline mapping model (PPMM) is then proposed to map the buried pipelines from the conducted pipeline dataset, with or without statutory pipeline records. In this model, each detected point is classified into the specific pipeline that most likely generates the data of this point, and detected points generated from the same pipeline are fit to revise the pipelines’ locations and directions. The above classification and fitting operations are performed iteratively, and PPMM would output the pipeline map with the highest probability. Experimental studies on real-world datasets are conducted and analyzed, and the obtained results demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Remote sensing change detection (RSCD), which identifies the changed and unchanged pixels from a registered pair of remote sensing images, has enjoyed remarkable success recently. However, locating changed objects with fine structural details is still a challenging problem in RSCD. In this article, we propose a novel RSCD network via temporal feature interaction and guided refinement (TFI-GR) to solve this issue. Specifically, unlike previous methods, which just employ one single concatenation or subtraction operation for bi-temporal feature fusion, we design a temporal feature interaction module (TFIM) to enhance interaction between bi-temporal features and capture temporal difference information at diverse feature levels. Afterward, a guided refinement modules (GRMs), which aggregates both low- and high-level temporal difference representations to polish the location information of high-level features and filter the background clutters of low-level features, is repeatedly performed. Finally, the multilevel temporal difference features are progressively fused to generate change maps for change detection. To demonstrate the effectiveness of the proposed TFI-GR, comprehensive experiments are performed on three high spatial resolution RSCD datasets. Experimental results indicate that the proposed method is superior to other state-of-the-art change detection methods. The demo code of this work is publicly available at https://github.com/guanyuezhen/TFI-GR. [ABSTRACT FROM AUTHOR]

High-resolution (HR) remote sensing imagery plays a critical role in remote sensing image interpretation, and single image super-resolution (SISR) reconstruction technology is becoming increasingly valuable and significant. The state-of-the-art deep-learning-based SISR methods have demonstrated remarkable advantages while reconstructing complex texture details still remains a big challenge. Besides, as a typical ill-posed inverse problem, how to determine the optimal solution is another important topic. To address these problems, in this work, a dual learning-based graph neural network (DLGNN) is proposed, in which the graph neural network (GNN) is utilized to consider the self-similarity patches in remote sensing imagery by aggregating cross-scale neighboring feature patches, and dual learning strategy is adopted to refine the reconstruction results by constraining the mapping process in terms of the loss function, transferring the typical ill-posed problem to a well-posed one. Abundant experiments on 3K VEHICLE_SR datasets and Massachusetts Roads demonstrate the validity and outstanding performance for remote sensing image super-resolution (SR) tasks compared with other state-of-the-art SR construction methods. Code is available at https://github.com/CUG-RS/DLGNN [ABSTRACT FROM AUTHOR]

Segmentation incremental learning (SIL) has received a lot of attention in recent years due to the ability to overcome the problem of catastrophic forgetting. Our study found that differences in label number distribution (LAND) affect the performance of SIL. Because the labels for pixels of the old category are marked as background when the model is trained on the new tasks, the LAND is inconsistent with static learning that is considered to be the upper bound on incremental learning, which hinders the mitigation of the catastrophic forgetting problem. In response to the above problems, we propose an incremental learning method named SIL-LAND, which improves the accuracy by making the LAND of our method close to that of static learning. From the perspective of high-level semantic labels, we propose the prototype update mechanism for the problem that nonadaptive representative prototypes ignore the sample diversity of semantic categories in remote sensing images. By compensating for the difference in LAND at the feature level, the distance between the prototype and the actual class center is reduced; aiming at the lack of semantic consistency between feature vectors and prototypes, we propose a similarity measure module to increase the intraclass similarity between the prototype and the corresponding feature vectors. From the perspective of one-hot labels, we propose label reconstruction, including foreground screening and background padding to make the number distribution of one-hot labels as close as possible to that of static learning. A series of experimental results demonstrates the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural networks (CNNs) have demonstrated impressive capabilities in the representation and classification of hyperspectral remote sensing images. Traditional CNNs require massive data to sufficiently train the network. To tackle this problem, graph convolutional network (GCN) has been introduced for hyperspectral image classification. GCN methods usually construct the graph from either spectral or spatial domain, which has not adequately explored the information in the joint spectral–spatial domain. In this article, we propose a superpixel spectral–spatial feature fusion graph convolution network for hyperspectral image classification (S3FGCN). S3FGCN can comprehensively use information in spectral, spatial, and spectral–spatial domains with limited data. Moreover, to enhance the performance, we explore a shared weights’ GCN in the spectral–spatial domain. To further improve the efficiency, superpixels are used to construct the adjacency matrix. Finally, dynamic sampling is adopted to make the model focus more on difficult samples. In the experiments on four datasets, S3FGCN demonstrates better accuracy compared with the state-of-the-art hyperspectral image classification methods. [ABSTRACT FROM AUTHOR]

The performance of a deep-learning-based model primarily relies on the diversity and size of the training dataset. However, obtaining such a large amount of labeled data for practical remote sensing (RS) applications is expensive and labor-intensive. Training protocols have been previously proposed for few-shot learning (FSL) and zero-shot learning (ZSL). However, FSL is not compatible with handling unobserved class data at the inference phase, while ZSL requires many training samples of the seen classes. In this work, we propose a novel training protocol for image retrieval and name it as label-deficit zero-shot learning (LDZSL). We use this novel LDZSL training protocol for the challenging task of cross-sensor data retrieval in RS. This protocol uses very few labeled data samples of the seen classes during training and interprets unobserved class data samples at the inference phase. This strategy is critical as some data modalities are hard to annotate without domain experts. This work proposes a novel bilevel Siamese network to perform the LDZSL cross-sensor retrieval of multispectral and synthetic aperture radar (SAR) images. We use the available georeferenced SAR and multispectral data to domain align the embedding features of the two modalities. We experimentally demonstrate the proposed model’s efficacy using the So2Sat dataset compared with the existing state-of-the-art models of the ZSL framework trained under a reduced training set. We also show the generalizability of the proposed model using a sketch-based image retrieval task. Experimental results on the Earth on the Canvas dataset exhibit comparative performance over the literature. [ABSTRACT FROM AUTHOR]

We investigate the problem of efficiently localizing sketch depicted scenes in a remote sensing image dataset. We pose the problem as that of remote sensing image retrieval with sketch queries and explore the use of hashing techniques to achieve efficient retrieval. Given two training datasets of sketches and remote sensing images that have a common set of class labels, we develop a hashing strategy that coconstructs two hash code books for the sketches and the remote sensing images separately. The hash code book coconstruction strategy encourages hash codes for the sketches and remote sensing images from different classes to be far away from one another and those from the same class to be close. This property is maintained by two cohesion intensive cues: 1) an interclass pairwise disperse cue (InterPDC) and 2) an intraclass pairwise balance cue (IntraPBC). We use the two coconstructed hash code books for training two linear mapping models that generate hash codes for sketches and remote sensing images separately. Sorting the Hamming distance between the sketch hash codes and the remote sensing image hash codes renders efficient remote sensing image retrieval with sketch queries. This enables localizing the sketch depicted scenes in the remote sensing image dataset. In addition, our method can also be used for fast localizing sketch depicted scenes in a remote sensing image of large size. Extensive experiments on public datasets validate the effectiveness and efficiency of our method. [ABSTRACT FROM AUTHOR]

With satellite platforms gazing at a target territory, the captured satellite videos exhibit local misalignment and local intensity variation on some stationary objects that can be mistakenly extracted as moving objects and increase false alarm rates. Typical approaches for mitigating the effect of moving cameras in moving object detection (MOD) follow domain transformation technique, where the misalignment between consecutive frames is restricted to the image planar. However, such technique cannot properly handle satellite videos, as the local misalignment on them is caused by the varying projections from the 3D objects on the Earth's surface to 2D image planar. In order to suppress the effect of moving satellite platform in MOD, we propose a Moving-Confidence-Assisted Matrix Decomposition (MCMD) model, where foreground regularization is designed to promote real moving objects and ignore system movements with the assistance of a moving-confidence score estimated from dense optical flows. For solving the convex optimization problem in MCMD, both batch processing and online solutions are developed in this study, by adopting the alternating direction method and the stochastic optimization strategy, respectively. Experimental results on the videos captured by SkySat and Jilin-1 show that MCMD outperforms the state-of-the-art techniques with improved precision by suppressing effect of nonstationary satellite platforms. [ABSTRACT FROM AUTHOR]

Land cover change detection (LCCD) using remote sensing images (RSIs) plays an important role in natural disaster evaluation, forest deformation monitoring, and wildfire destruction detection. However, bitemporal images are usually acquired at different atmospheric conditions, such as sun height and soil moisture, which usually cause pseudo and noise change in the change detection map. Changed areas on the ground also generally have various shapes and sizes, consequently making the utilization of spatial contextual information a challenging task. In this article, we design a novel neural network with a spatial–spectral attention mechanism and multiscale dilation convolution modules. This work is based on the previously demonstrated promising performance of convolutional neural network for LCCD with RSIs and attempts to capture more positive changes and further enhance the detection accuracies. The learning of the proposed neural network is guided with a change magnitude image. The performance and feasibility of the proposed network are validated with four pairs of RSIs that depict real land cover change events on the Earth’s surface. Comparison of the performance of the proposed approach with that of five state-of-art methods indicates the superiority of the proposed network in terms of ten quantitative evaluation metrics and visual performance. Such as, the proposed network achieved an improvement of about 0.08%–14.87% in terms of overall accuracy (OA) for Dataset-A. [ABSTRACT FROM AUTHOR]

Synthetic aperture radar (SAR), an active microwave remote sensing equipment, can provide all-day, all-weather, and high-resolution images for Earth observation. Ship monitoring using SAR is significant to fishery application, marine transportation, maritime security, and so on. Most traditional SAR ship detection methods use the amplitude information of the SAR imagery and employ the constant false alarm rate (CFAR), deep learning, and other technologies to detect the ship targets. However, in addition to the amplitude image, the SAR single-look complex (SLC) data can also be utilized for ship detection since its complex-valued information is beneficial to characterize the ship’s feature in the transforming domains. In this article, the strong scattering feature of ships in the image domain and low-rank feature in the transforming domain is studied, and on this basis, a two-step detector is proposed to detect ship targets in SAR SLC image. The first step is to identify the presence of ships, which is based on the eigen-subspace projection (ESP) technology to detect the occurrences of the ship targets in azimuth cells of the SAR data. The second step is to locate the position of ships, which is based on the low-rank and sparse matrix decomposition (LRSMD) to obtain the instantaneous frequency information of the ship targets in the time–frequency domain. Finally, the ship targets can be located in a line-by-line manner in the azimuth direction. Experimental results via European Sentinel-1A and Chinese Gaofen-3 single-pol SAR SLC data demonstrate the performance of the proposed detector. [ABSTRACT FROM AUTHOR]

Region of interest (ROI) extraction plays a significant role in the field of remote sensing image (RSI) processing. Recently, weakly supervised ROI extraction methods have attracted considerable attention due to low labeling cost. Most of them follow the pipeline of first generating pseudo labels and then using the pseudo labels to train a segmentation model. However, there remain problems to be solved: 1) the unbalanced distribution of foreground and background samples in the RSI dataset influences the network performance; 2) the pseudo labels mainly cover the most discriminative part of object regions that are incomplete; and 3) training with pseudo labels inevitably causes noise issues that degrade the model performance. To solve these issues, we propose a weakly supervised uncertainty-aware self-refinement learning (UASRL) method, where the initial unbalanced image-level labels are progressively refined to high-quality pixel-level annotations. In the proposed UASRL, we first present a deep generative model combined with self-attention modules to improve the unbalanced distribution in the weakly labeled dataset. Then, we design a confidence-weighted complementary erasing-based weakly supervised method to generate pseudo labels with high integrity. Finally, for training with noisy pseudo labels, we develop an uncertainty-aware joint optimization (UAJO) training strategy to reduce the negative effect caused by noisy labels and further refine pixelwise labels in a coarse-to-accurate manner, which in turn jointly promotes the model’s performance. Extensive experiments on three types of RSI datasets reveal that our proposed method is superior to other competing methods and shows a preferable tradeoff between annotation cost and detection performance. [ABSTRACT FROM AUTHOR]

Although semantic segmentation models based on deep neural networks (DNNs) have achieved excellent results, generalizing well from one remote sensing dataset (source domain) to another dataset with different acquisition conditions (target domain) remains a major challenge. Many domain adaptation (DA) approaches have been proposed to address this problem. DA aims to help DNNs learn a generalizable representation space in which source and target domains have similar feature distributions, but most of the existing DA approaches have difficulty in aligning the high-dimensional image representations of two domains directly. In this study, we proposed a model integrating contrastive learning and adversarial learning in a unified framework for aligning two domains in both representation space and spatial layout. Specifically, the model consists of a semantic segmentation network for feature extraction and two branches for DA. The first branch is used for adaptation in representation space directly by a proposed pixelwise contrastive loss, while the second branch is used for adaptation in predicted results to help two domains have similar spatial layouts through a novel but simple entropy-based similarity discriminator. Additionally, a training strategy called category similarity matching sampling was proposed to provide source and target image pairs with similar category composition for each training iteration, which can help the two branches work better. Extensive experiments indicated that the two branches can benefit each other to gain a superior performance and DA pretraining by our methods can achieve impressive results with only a small number of target labeled samples. [ABSTRACT FROM AUTHOR]

Current remote sensing hardware technology is not yet able to acquire multiband remote sensing images with both high spatial and spectral resolution. As an important tool to compensate for the lack of spatial information acquisition of multispectral (MS) images, pan-sharpening has been an important and continuously active research area in remote sensing image processing. Although many methods have emerged, the problem of how to obtain high spatial resolution while effectively maintaining the spectral information of MS images has not been well solved. Many aspects still need further research. In this article, we first investigate the essential properties and rationality of two common framework types in the multiresolution analysis (MRA) sharpening method of pan-sharpening from the source perspective—the identical-resolution framework (IRF) derived from the generalized fusion application and the different-resolution framework (DRF) exclusive to the sharpening application, and show that the core difference between the two frameworks lies in the different ideas of utilizing the multiscale transformation, i.e., they tend to expand the scale space and model the spatially blurred degradation relationship between the sources, respectively. Both of them have their own advantages and disadvantages in handling detailed information, and neither of them can effectively deal with the “detail exclusivity” problem. Based on this, the idea of “entropy level matching” (ELM) of pan-sharpening is presented, and a comprehensive framework that can combine the advantages of the two types of frameworks is constructed, namely, the multiscale ELM framework. Furthermore, as an application of this framework, we propose a sharpening method shearlet transform-based entropy matching (STEM) built on the nonsubsampled shearlet as a multiscale transformation method. According to the difference in detail injection mode in it, it can be further divided into two sharpening methods based on additive mode and substitutive mode. The comparison experiments with 11 popular methods show that the proposed two sharpening methods can effectively improve the spatial resolution of MS images while keeping the spectral information well, and the comprehensive performance advantage is obvious. The source code of the proposed method can be downloaded from https://github.com/JZ-Tao/STEM/. [ABSTRACT FROM AUTHOR]

Point cloud completion aims to infer the complete point clouds from incomplete ones, which is used in remote sensing applications such as reconstructing and autonomous driving. However, most existing methods cannot recover accurate structure details of the object. In this article, we propose point shift network (PS-Net). Our main contributions lie in the following three-folds. First, we propose a multiresolution encoder, which extracts and fuses multiresolution point cloud features hierarchically, thus avoiding information loss caused by a single global feature. Second, we design a multiresolution point cloud generation structure, which can be combined with the multiresolution encoder to generate gradually dense point clouds, avoiding the problem of nonuniformly density of the single-layer decoder. Third, we design the shift network (SN), which is used to generate shift vectors to shift the coordinates of each point cloud, so as to further fine-tune the coordinate positions of point clouds, achieving more accurate prediction. We conduct comprehensive experiments on the ShapeNet, KITTI, ScanObjectNN, and ModelNet40 datasets, which demonstrate that the proposed PS-Net achieves better performance than the existing methods and verify the robustness of the proposed method. This article contributes a new method to point cloud completion, realizes fine point cloud shape completion, and brings new possibilities to the research of autonomous driving, registration, and reconstruction. [ABSTRACT FROM AUTHOR]

It is promising to improve Moderate Resolution Imaging Spectroradiometer (MODIS) snow cover mapping by training effective machine learning models. However, considering the strong spatiotemporal heterogeneity of snow cover, the representativeness of training samples becomes a critical issue to ensure the model effectiveness over large areas and long terms. To deal with this issue, we propose the $m$ -day dynamic training strategy. The core of this strategy is to divide a long-term snow cover mapping task into multiple short-term tasks with consecutive $m$ days ($m$ is greater or equal to the number of days that higher resolution remote sensing images used as equivalent ground references cover the study area once). It can ensure the spatial representativeness of training samples by utilizing all available equivalent ground references data, and reduce the issue caused by snow cover changing over time. We apply this strategy to random forest (RF) for both binary snow cover mapping (BSC) and fractional snow cover mapping (FSC) using Landsat-8 data as equivalent ground reference and validate its effect on the Tibetan Plateau. Results show that this strategy achieves higher accuracies than other training strategies, with $F1$ of 0.9596 for BSC mapping, $R$ and mean square error (MSE) of 0.9037 and 0.0260 for FSC mapping, respectively. In addition, spatiotemporal analysis of results further demonstrates that this strategy holds advantages for snow cover mapping in areas of complex terrain and in periods when snow cover changes rapidly. In conclusion, this strategy can effectively ensure the representativeness of training samples generated from Landsat-8 data and thus improve MODIS snow cover mapping through machine learning. [ABSTRACT FROM AUTHOR]

Pansharpening is an image fusion procedure, which aims to produce a high spatial resolution multispectral (MS) image by combining a low spatial resolution MS image and a high spatial resolution panchromatic image. The most popular and successful paradigm for pansharpening is the framework known as detail injection, while it cannot fully exploit complex and nonlinear complementary features of both images. In this article, we propose a detail-injection-model-inspired deep fusion network for pansharpening (DIM-FuNet). First, by treating pansharpening as a complicated and nonlinear detail learning and injection problem, we establish a unified optimizing detail-injection model with triple detail fidelity terms: 1) a band-dependent spatial detail fidelity term; 2) a local detail fidelity term; and 3) a complicated detail synthesis term. Second, the model is optimized via the iterative gradient descent and unfolded into a deep convolutional neural network. Subsequently, the unrolling network has triple branches, in which a point-wise convolutional subnetwork and a depth-wise convolutional subnetwork are corresponding to the former two detail-constrained terms, and an adaptive weighted reconstruction module with a fusion subnetwork to aggregate details of two branches and to synthesize the final complicated details. Finally, the deep unrolling network is trained in an end-to-end manner. Different from traditional deep fusion networks, the architecture design of DIM-FuNet is guided by the optimizing model and thus promotes better interpretability. Experimental results on reduced and full-resolution demonstrate the effectiveness of the proposed DIM-FuNet which achieves the best performance compared with the state-of-the-art pansharpening method. [ABSTRACT FROM AUTHOR]

The automatic extraction of lake water is one of the research hotspots in the field of remote sensing image processing. Due to the small interclass variance between lakes and other ground objects, and the complex texture characteristics of lake boundaries, existing methods often have problems such as over-segmentation and inaccurate boundary segmentation when segmenting lake water bodies. To alleviate these problems, this article designs an end-to-end semantic segmentation network [noise-canceling transformer network (NT-Net)] for the automatic extraction of lake water bodies from remote sensing images. Aiming at the problem of over-segmentation caused by nonlake objects, an interference attenuation module is designed in the network. This module can model the key features that are distinguishable and suitable for segmenting lake water by analyzing the difference in feature representation between lakes and other ground objects, thus suppressing the feature representation of nonlake objects. To more accurately segment the lake boundary, a multilevel transformer module is designed. This module can capture the context association of boundary information and enhance the feature representation of boundary information by using the self-attention mechanism. The comparative experimental results show that, compared with the current mainstream semantic segmentation networks, the method in this article has advantages in extracting lake water bodies comprehensively and coherently. [ABSTRACT FROM AUTHOR]

Novel view synthesis of remote sensing (RS) scenes is of great significance for scene visualization, human–computer interaction, and various downstream applications. Despite the recent advances in computer graphics and photogrammetry technology, generating novel views is still challenging particularly for RS images due to its high complexity, view sparsity, and limited view-perspective variations. In this article, we propose a novel RS view synthesis method by leveraging the recent advances in implicit neural representations. Considering the overhead and far depth imaging of RS images, we represent the 3-D space by combining implicit multiplane images (ImMPI) representation and deep neural networks. The 3-D scene is reconstructed under a self-supervised optimization paradigm through a differentiable multiplane renderer with multiview input constraints. Images from any novel views thus can be freely rendered on the basis of the reconstructed model. As a by-product, the depth maps corresponding to the given viewpoint can be generated along with the rendering output. We refer to our method as ImMPIs. To further improve the view synthesis under sparse-view inputs, we explore the learning-based initialization of RS 3-D scenes and proposed a neural-network-based prior extractor to accelerate the optimization process. In addition, we propose a new dataset for RS novel view synthesis with multiview real-world Google Earth images. Extensive experiments demonstrate the superiority of the ImMPI over previous state-of-the-art methods in terms of reconstruction accuracy, visual fidelity, and time efficiency. Ablation experiments also suggest the effectiveness of our methodology design. [ABSTRACT FROM AUTHOR]

Since remote sensing images of post-fire vegetation are characterized by high resolution, multiple interferences, and high similarities between the background and the target area, it is difficult for existing methods to detect and segment the burned area in these images with sufficient speed and accuracy. In this article, we apply salient object detection (SOD) to burned area segmentation (BAS), the first time this has been done, and propose an efficient burned area segmentation network (BASNet) to improve the performance of unmanned aerial vehicle (UAV) high-resolution image segmentation. BASNet comprises positioning module and refinement module. The positioning module efficiently extracts high-level semantic features and general contextual information via global average pooling layer and convolutional block (CB) to determine the coarse location of the salient region. The refinement module adopts the CB attention module to effectively discriminate the spatial location of objects. In addition, to effectively combine edge information with spatial location information in the lower layer of the network and the high-level semantic information in the deeper layer, we design the residual fusion module to perform feature fusion by level to obtain the prediction results of the network. Extensive experiments on two UAV datasets collected from Chongli in China and Andong in South Korea, demonstrate that our proposed BASNet significantly outperforms the state-of-the-art SOD methods quantitatively and qualitatively. BASNet also achieves a promising prediction speed for processing high-resolution UAV images, thus providing wide-ranging applicability in post-disaster monitoring and management. [ABSTRACT FROM AUTHOR]

The few-shot scene classification is dedicated to identifying unseen remote sensing classes when only a very small number of labeled samples are available for reference. Most of the existing few-shot scene classification methods are based on meta-learning and use the episodic learning for training, which lacks the consideration for the utilization of data efficiency. In this article, instead of designing sophisticated meta-learning-based algorithms, we are committed to training a feature extractor with good generalization performance and strong feature extraction capability. Specifically, we propose a novel two-path aggregation attention network with quad-patch data augmentation, called data architecture network (DANet), to solve the problem of few-shot scene classification from both data and architecture aspects. In terms of data, we design a new data augmentation strategy named quad-patch augmentation. We use the characteristics of remote sensing images to chunk and reassemble any existing data, thereby generating pseudo-new data to enrich the training set. In terms of architecture, we present a two-path aggregation attention module that makes it easier for the model to focus on the key clues in a targeted manner. The comparative experiments in natural image datasets and remote sensing image datasets demonstrate the effectiveness of our two innovations. In addition, DANet achieves competitive or state-of-the-art (SOTA) results on three benchmark scene classification datasets. [ABSTRACT FROM AUTHOR]

Targeting at depicting land covers with pixelwise semantic categories, semantic segmentation in remote sensing images needs to portray diverse distributions over vast geographical locations, which is difficult to be achieved by the homogeneous pixelwise forward paths in the architectures of existing deep models. Although specific algorithms have been designed to select pixelwise adaptive forward paths for natural image analysis, it still lacks theoretical supports on how to obtain optimal selections. In this article, we provide mathematical analyses in terms of the parameter optimization, which guides us to design a method called hidden path selection network (HPS-Net). With the help of hidden variables deriving from an extra mini-branch, HPS-Net is able to tackle the inherent problem about inaccessible global optimums by adjusting the direct relationships between feature maps and pixelwise path selections in existing algorithms, which we call hidden path selection. For the better training and evaluation, we further refine and expand the 5-class Gaofen image dataset (GID-5) to a new one with 15 land-cover categories, i.e., GID-15. The experimental results on both GID-5 and GID-15 demonstrate that the proposed modules can stably improve the performance of different deep structures, which validates the proposed mathematical analyses. [ABSTRACT FROM AUTHOR]

Extracting roads from high-resolution remote sensing images (HRSIs) is vital in a wide variety of applications, such as autonomous driving, path planning, and road navigation. Due to the long and thin shape as well as the shades induced by vegetation and buildings, small-sized roads are more difficult to discern. In order to improve the reliability and accuracy of small-sized road extraction when roads of multiple sizes coexist in an HRSI, an enhanced deep neural network model termed dual-decoder-U-net (DDU-Net) is proposed in this article. Motivated by the U-Net model, a small decoder is added to form a dual-decoder structure for more detailed features. In addition, we introduce the dilated convolution attention module (DCAM) between the encoder and decoders to increase the receptive field as well as to distill multiscale features through cascading dilated convolution and global average pooling. The convolutional block attention module (CBAM) is also embedded in the parallel dilated convolution and pooling branches to capture more attention-aware features. Extensive experiments are conducted on the Massachusetts Roads dataset with experimental results showing that the proposed model outperforms the state-of-the-art DenseUNet, DeepLabv3+, and D-LinkNet by 6.5%, 3.3%, and 2.1% in the mean intersection over union (mIoU), and by 4%, 4.8%, and 3.1% in the F1 score, respectively. Both ablation and heatmap analysis are presented to validate the effectiveness of the proposed model. Moreover, the designed small decoder and introduced DCAM can be used as a portable module to be embedded in other U-Net-like models with encoder–decoder structure to enhance the road detection performance, especially for small-sized roads. The high portability of the designed module is validated by embedding it in the LinkNet, which greatly improves the road segmentation performance. [ABSTRACT FROM AUTHOR]

Deep-learning-based super-resolution (SR) methods have been extensively studied and have achieved significant performance with deep convolutional neural networks. However, the results still suffer from the ringing effect, especially in satellite image SR tasks, due to the loss of image details in the satellite degradation process. In this article, we build a novel satellite SR framework by decomposing a high-resolution image into three components, i.e., low-resolution (LR), artifact, and high-frequency information. Specifically, we propose an artifact removal network with a self-adaption difference convolution (SDC) to fully exploit the structure prior in the LR image and predict the artifact map. Considering that the artifact map and the high-frequency map share a similar pattern, we introduce the supervised structure correction (SSC) block that establishes a bridge between the high-frequency generation process and the artifact removal process. Experimental results on satellite images demonstrate that the proposed method owns an improved tradeoff between the performance and the computational cost compared to existing state-of-the-art satellite and natural SR methods. The source code is available at https://github.com/jiaming-wang/ARSRN. [ABSTRACT FROM AUTHOR]

Change detection (CD) is among the most important applications in remote sensing (RS) that allows identifying the changes that occurred in a given geographical area across different times. Even though CD systems have seen a lot of progress in RS, their output is either a binary map highlighting the changing area or a semantic change map that indicates the type of change for each pixel. The change maps are often difficult to interpret by end users, and they omit important information such as relationships and attributes of the changed areas. Motivated by the recent advancement of image captioning in the RS community, in this article, we propose to describe the changes over bitemporal images through change sentence descriptions. The aim of this article is to provide a user-friendly interpretation of the occurred changes. To this end, we propose two change captioning (CC) systems that take bitemporal images as input and generate coherent sentence descriptions of the occurred changes. Convolutional neural networks (CNNs) are used to extract discriminative features from the bitemporal images and recurrent neural networks (RNNs) or support vector machines (SVMs) are exploited to generate coherent change descriptions. Furthermore, in the absence of a CC dataset to test our systems, we propose two new datasets. One is based on very high-resolution RGB images, and the other one is based on multispectral RS images. The obtained experimental results show promising capabilities of the proposed systems to generate coherent change descriptions from the bitemporal images. The datasets are available at the following link: https://disi.unitn.it/~melgani/datasets.html. [ABSTRACT FROM AUTHOR]

Pansharpening is a special image fusion task of reconstructing a high-resolution multispectral (HRMS) image by integrating a panchromatic (PAN) image of high spatial resolution and a low-resolution multispectral (LRMS) image. To handle such an ill-posed multimodal fusion task, in this article, we propose a novel pansharpening method, referred to as model-driven and data-driven network (MD3Net), which combines model-driven and data-driven approaches. The architecture design of MD3Net is inspired from the traditional model constructed based on domain knowledge and thus making its network topology explainable and its input-output predictable. To further explore the powerful learning ability of deep-learning-based approaches, we introduce the deep prior into the MD3Net as its implicit regularization, thus improving its data adaptability and representation capability. Comprehensive experiments conducted on both reduced and full resolution of several acknowledged datasets have qualitatively and quantitatively verified the superiority of our network compared with a benchmark consisting of several state-of-the-art approaches. The code can be downloaded from https://github.com/YinsongYan/M3DNet. [ABSTRACT FROM AUTHOR]

Deep descriptors have advantages over handcrafted descriptors on local image patch matching. However, due to the complex imaging mechanism of remote sensing images and the significant differences in appearance between multi-modal images, existing deep learning descriptors are unsuitable for multi-modal remote sensing image registration directly. To solve this problem, this article proposes a deep feature correlation learning network (Cnet) for multi-modal remote sensing image registration. First, Cnet builds a feature learning network based on the deep convolutional network with the attention learning module, to enhance feature representation by focusing on meaningful features. Second, this article designs a novel feature correlation loss function for Cnet optimization. It focuses on the relative feature correlation between matching and nonmatching samples, which can improve the stability of network training and decrease the risk of overfitting. In addition, the proposed feature correlation loss with a scale factor can further enhance network training and accelerate network convergence. Extensive experimental results on image patch matching (Brown, HPatches), cross-spectral image registration (VIS-NIR), multi-modal remote sensing image registration, and single-modal remote sensing image registration have demonstrated the effectiveness and robustness of the proposed method. [ABSTRACT FROM AUTHOR]

As a typical application of remote sensing technology, change detection can find the ground information changes by acquiring the images of the same region at different times. The change detection using the synthetic aperture radar (SAR) with the advantages of all day and all-weather usually monitors the significant surface change, such as flood disasters and earthquake deformation. However, when it comes to detecting subtle changes such as vehicle traces, the traditional methods ignoring the phase coherence between image pairs cannot intensify these faint changes in the difference image. The SAR coherent change detection (CCD) based on repeat-pass repeat-geometry complex images utilizing both the intensity and phase fraction could exhibit the subtle vehicle trace in the difference image. However, the complicated background and decorrelation factors significantly affect the quality of difference images, further causing great trouble for automatic trace detection. This article proposes the spatial feature enhanced Unet and adaptive data augmentation to realize vehicle trace detection. More specifically, the pseudocolor image is first synthesized based on a two-stage coherence estimation method. Then, considering the long-continuity and parallel distribution of vehicle trace samples, the enhanced Unet is constructed by fusing spatial convolutional neural network and spatial attention mechanism. After that, the adaptation data augmentation strategy is presented by introducing manual registration errors and multiple estimation windows. Finally, the experimental results on the Sandia CCD data and our measured data demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Change detection for remote sensing images involves detecting regional surface changes of interest between two images taken of the same geographical area but at different times. In image processing, the spatial domain uses grayscale values to describe an image. The frequency is directly related to the spatial change rate, so the frequency domain can be intuitively associated with patterns of intensity variations in the image. These two domains provide different perspectives for image interpretation. Most existing deep-learning-based methods formulate change detection as a pixel-wise binary classification problem and utilize various strategies to extract information in the spatial domain. However, they rarely pay attention to the rich information in the frequency domain. To address this problem, we propose an end-to-end joint frequency-spatial domain network (JFSDNet) to implement remote sensing optical image change detection. Specifically, we introduce frequency information into the change detection to supplement the loss of image details caused by downsampling. In addition, we employ a frequency selection module to adaptively discriminate and choose frequency clues by reducing the complexity of the frequency features. The JFSDNet is applied to two publicly available datasets: the change-detection dataset (CDD) dataset and the LEarning VIsion and Remote sensing Change Detection (LEVIR-CD) dataset. Compared with other methods, both visual interpretation and quantitative assessment confirmed that our proposed method achieved a favorable performance. [ABSTRACT FROM AUTHOR]

Sea fog significantly threatens the safety of maritime activities. This article develops a sea fog detection dataset (SFDD) and a dual-branch sea fog detection network (DB-SFNet). We investigate all the observed sea fog events in the Yellow Sea and the Bohai Sea (118.1°E–128.1°E, 29.5°N–43.8°N) from 2010 to 2020 and collect the sea fog images for each event from the Geostationary Ocean Color Imager (GOCI) to comprise the dataset SFDD. The location of the sea fog in each image in SFDD is accurately marked. The proposed dataset is characterized by a long-time span, a large number of samples, and accurate labeling, which can substantially improve the robustness of various sea fog detection models. Furthermore, this article proposes a DB-SFNet to achieve accurate and holistic sea fog detection. The proposed DB-SFNet is composed of a knowledge extraction module and a dual-branch optional encoding decoding module. The two modules jointly extract discriminative features from both visual and statistical domains. Experiments show promising sea fog detection results with an F1-score of 0.77 and a critical success index of 0.63. Compared with existing advanced deep learning networks, DB-SFNet is superior in detection performance and stability, particularly in the mixed cloud and fog areas. [ABSTRACT FROM AUTHOR]

Remote sensing images (RSIs) have been applied to many fields, such as environmental monitoring, urban planning, and military defense. All-day thermal infrared imaging observation system can finely portray the trajectory of human activities and provide data support for United Nations sustainable development planning. However, in the process of RSIs, image quality degradation containing image blurring is caused by optical system aberration, satellite platform vibration, imaging system out-of-focus, and atmospheric turbulence. Moreover, stripe noise in the image is produced by the nonuniformity of infrared sensors. Image deblurring and destriping are classical tasks in RSIs, but the above two problems are discussed separately in almost all research, i.e., denoising after adding stripe noise on clear images or deblurring under the assumption that only random noise exists. In this article, stripe noise and blur are jointly removed from on-orbit RSIs acquired by the thermal infrared spectrometer (TIS) on the SDGSAT-1 satellite, using a method based on stripe component residuals and gradient property. According to the experimental results, the performance of the proposed method is greatly improved compared with processing these two tasks separately, which can provide a valuable reference for the study of high-resolution thermal infrared RSIs recovery. [ABSTRACT FROM AUTHOR]