Search

Pansharpening is an important technology for obtaining high-resolution multispectral (HRMS) images by fusing low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images. Although many pansharpening models have emerged by taking advantage of deep learning (DL) technology, there remains a pressing need to further assess pansharpening accuracy and stability when LRMS images with complex land-cover types. What's more, these models often overlook the exploitation of PAN images' inherent high-frequency information. To address these issues, we propose a pansharpening model combining multi-level and multi-scale network architectures. The multi-level network architecture is used to build spatial-spectral dependence on LRMS-PAN pairs, and strengthen the network's feature capture capability by keeping the multi-level texture details. The multi-scale architecture is subsequently used to extract the spatial structure and deep texture of the PAN images at different scales. Downsampled experiments and real experiments in four standard datasets show that the proposed model achieves a state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) satellites like PRISMA (PRecursore IperSpettrale della Missione Applicativa) offer remarkable capabilities, yet they are constrained by a relatively coarse spatial resolution, curbing their efficacy in those applications that require pinpoint accuracy. Here we propose a fusion process, aimed at the enhancement of PRISMA HS spatial resolution by using the spatial and spectral information of Sentinel-2 multispectral (MS) data (HS-MS fusion process), validated against four airborne HS flights simultaneous to satellite overpasses on different land use distributions. Adopting the PRISMA panchromatic (PAN) image, the proposed solution was also compared with the results of a HS-PAN pansharpening process. A two-steps operational workflow is proposed, based on two state-of-the-art and open-source algorithms. The first step consisted of the geocoding of PRISMA L2 products using Senintel-2 as reference and was accomplished with the phase-based algorithm implemented in AROSICS (Automated and Robust Open-Source Image Co-registration Software). The geometric displacement in L2 data was found to be between 80 m and 250 m, irregularly spatially distributed throughout the same scene and among scenes, and it was corrected by means of thousands of regularly spatially distributed tie points. A second-order polynomial transformation function was integrated in the algorithm. The second step consisted of employing the HySure (HS Super resolution) fusion algorithm to perform both the HS-MS fusion and the HS-PAN pansharpening, returning a PRISMA HS improved dataset with a spatial resolution of 10 m and 5 m, respectively. Four different per-band accuracy metrics were used to evaluate the accuracy of both products against airborne data. Overall, HS-MS data achieved increased accuracy in all validation metrics, i.e. + 28 % (root mean square error, RMSE), +23 % (spectral angle mapper, SAM), +7% (peak signal-to-noise ratio, PSNR) and + 11 % (universal image quality index, UIQI), with respect of HS-PAN data. These outcomes showed that using the spectral information of Sentinel-2 both spectral and spatial patterns were reconstructed more consistently in three different urban and rural scenarios, avoiding the presence of blur and at-edge artefacts as opposed to HS-PAN pansharpening, therefore suggesting an optimal strategy for satellite HS data resolution enhancement. [ABSTRACT FROM AUTHOR]

This study aimed to propose an improved Gram–Schmidt adaptive (GSA) pansharpening method using the support vector regression (SVR) and Markov random field (MRF) models in the cases of high ratios between spatial resolutions of LRMS and PAN images. In the present study, the SVR model was used to model the nonlinear relationship between the original LRMS images and the corresponding downsampled PAN image, thereby aiming to obtain the intensity component ( I L ) of the upsampled MS image. Then, the initial pansharpened HRMS image was generated from the GSA pansharpening method with I L calculated by the SVR model, which is denoted as GSA–SVR in this study. Finally, the quality of the initial pansharpened image was further improved by using the MRF model, which is denoted as GSA–SVR–MRF. A performance comparison of the GSA–SVR–MRF method with competitive pansharpening techniques as well as the GSA–SVR method demonstrated its superiority in maintaining the spatial and spectral details of the PAN and original LRMS images. The GSA–SVR–MRF method was found to be the best in terms of most quality indices. [ABSTRACT FROM AUTHOR]

The pansharpening method based on spatial injection is prone to spectral distortion due to the spatial detail injection amount mismatch. For this reason, this paper proposes a method that combines adaptive intensity component extraction and spatial detail injection optimisation. Firstly, a new method for extracting the I component of multispectral (MS) image is proposed, which decomposes each band of MS image and panchromatic (PAN) image according to the structure-energy mechanism through the joint bilateral filter and then determines the weight coefficient of each band of MS image to extract the I component by solving the minimum optimisation problem of the structure layer of PAN image and the energy layer of MS image. Secondly, a spatial detail extraction method combining local spatial frequency and local energy is proposed, and the method can fully extract the fine detail features of PAN images and MS images. In this paper, many experiments are conducted on QuickBird, GeoEye-1 and WorldView-4 datasets, and quantitative and qualitative comparisons are made with eight advanced methods, and qualitative and quantitative analyses of the experimental results show that the proposed method has certain superiority. [ABSTRACT FROM AUTHOR]

In recent years, deep learning techniques for pansharpening multiresolution images have gained increasing interest. Due to the lack of ground truth data, most deep learning solutions rely on synthetic reduced-resolution data for supervised training. This approach has limitations due to the statistical mismatch between real full-resolution and synthetic reduced-resolution data, which affects the models' generalization capacity. Consequently, there has been a shift towards unsupervised learning frameworks for pansharpening deep learning-based techniques. Unsupervised schemes require defining sophisticated loss functions with at least two components: one for spectral quality, ensuring consistency between the pansharpened image and the input multispectral component, and another for spatial quality, ensuring consistency between the output and the panchromatic input. Despite promising results, there has been limited investigation into the interaction and balance of these loss terms to ensure stability and accuracy. This work explores how unsupervised spatial and spectral consistency losses can be reliably combined preserving the outocome quality. By examining these interactions, we propose a general rule for balancing the two loss components to enhance the stability and performance of unsupervised pansharpening models. Experiments on three state-of-the-art algorithms using WorldView-3 images demonstrate that methods trained with the proposed framework achieve good performance in terms of visual quality and numerical indexes. [ABSTRACT FROM AUTHOR]

Pansharpening refers to the process of fusing multispectral (MS) images with panchromatic (PAN) images to obtain high-resolution multispectral (HRMS) images. However, due to the low correlation and similarity between MS and PAN images, as well as inaccuracies in spatial information injection, HRMS images often suffer from significant spectral and spatial distortions. To address these issues, a pansharpening method based on multimodal texture correction and adaptive edge detail fusion is proposed in this paper. To obtain a texture-corrected ( T C ) image that is highly correlated and similar to the MS image, the target-adaptive CNN-based pansharpening (A-PNN) method is introduced. By constructing a multimodal texture correction model, intensity, gradient, and A-PNN-based deep plug-and-play correction constraints are established between the T C and source images. Additionally, an adaptive degradation filter algorithm is proposed to ensure the accuracy of these constraints. Since the T C image obtained can effectively replace the PAN image and considering that the MS image contains valuable spatial information, an adaptive edge detail fusion algorithm is also proposed. This algorithm adaptively extracts detailed information from the T C and MS images to apply edge protection. Given the limited spatial information in the MS image, its spatial information is proportionally enhanced before the adaptive fusion. The fused spatial information is then injected into the upsampled multispectral (UPMS) image to produce the final HRMS image. Extensive experimental results demonstrated that compared with other methods, the proposed algorithm achieved superior results in terms of both subjective visual effects and objective evaluation metrics. [ABSTRACT FROM AUTHOR]

Pansharpening, a pivotal task in remote sensing, involves integrating low-resolution multispectral images with high-resolution panchromatic images to synthesize an image that is both high-resolution and retains multispectral information. These pansharpened images enhance precision in land cover classification, change detection, and environmental monitoring within remote sensing data analysis. While deep learning techniques have shown significant success in pansharpening, existing methods often face limitations in their evaluation, focusing on restricted satellite data sources, single scene types, and low-resolution images. This paper addresses this gap by introducing PanBench, a high-resolution multi-scene dataset containing all mainstream satellites and comprising 5898 pairs of samples. Each pair includes a four-channel (RGB + near-infrared) multispectral image of 256 × 256 pixels and a mono-channel panchromatic image of 1024 × 1024 pixels. To avoid irreversible loss of spectral information and achieve a high-fidelity synthesis, we propose a Cascaded Multiscale Fusion Network (CMFNet) for pansharpening. Multispectral images are progressively upsampled while panchromatic images are downsampled. Corresponding multispectral features and panchromatic features at the same scale are then fused in a cascaded manner to obtain more robust features. Extensive experiments validate the effectiveness of CMFNet. [ABSTRACT FROM AUTHOR]

A novel technique utilizing a convolutional autoencoder (CAE) is introduced with the aim of enhancing the spatial resolution of multispectral (MS) images while concurrently mitigating spectral distortion. First, an original panchromatic (PAN) image is constructed from its spatially degraded version. Then, the relationship between the original PAN image and its degraded version is utilized to reconstruct the high-resolution MS image; in addition, an intensity component of MS image, which is obtained using an adaptive intensity-hue-saturation (AIHS), is reconstructed by utilizing the aforementioned relationship. Two types of remote sensing datasets are adopted, and the effect of the patch size with the overlapping pixel on spectral and spatial distortion is considered. After training CAE, the low-resolution MS image and its intensity component are given to the trained network as input to obtain the MS image and intensity component with better details. Eventually, the fused image is obtained by using a component substitution (CS) framework. Experimental findings corroborate that the proposed method yields superior outcomes compared with several existing approaches, demonstrating advantages in both objective metrics and visual fidelity. [ABSTRACT FROM AUTHOR]

Abstract Pansharpening is a vital technique in remote sensing that combines a low‐resolution multi‐spectral image with its corresponding panchromatic image to obtain a high‐resolution multi‐spectral image. Despite its potential benefits, the challenge lies in extracting features from the source images and eliminating artefacts in the fused images. In response to the challenge, a hybrid generative adversarial network‐based model, termed SWPanGAN, is proposed. For better feature extraction, the conventional convolution neural network is replaced with a Swin transformer in the generator, which provides the generator with the ability to model long‐range dependencies. Additionally, to suppress artefacts, a wavelet‐based discriminator is proposed for effectively distinguishing the frequency discrepancy. With these modifications, both the generator and discriminator networks of SWPanGAN are enhanced. Extensive experiments illustrate that our SWPanGAN can generate high‐quality pansharpening images and surpass other state‐of‐the‐art methods.

Pansharpening involves the fusion of panchromatic (PAN) and multispectral (MS) images to obtain a high-resolution image with enhanced spectral and spatial information. Assessing the quality of the resulting fused image poses a challenge due to the absence of a high-resolution reference image. Numerous methods have been proposed to address this, from assessing quality at reduced resolution to full-resolution evaluations. Many existing approaches are pixel-based, where quality metrics are applied and averaged on individual pixels. In this article, we introduce a novel object-based method for assessing the quality of pansharpened images at full resolution. In object-based quality assessment methods, the reaction of different areas of the fused image to the fusion process is reflected. Our approach revolves around extracting objects from the given image and evaluating extracted objects. By doing so, the distinct responses of different objects within the fused image to the fusion process are captured. The proposed method leverages a unique object extraction technique known as segmentation by nearest neighbor (SNN) to extract objects of the MS image. This method extracts the objects based on the image’s characteristics without any requirement for parameter tuning. These extracted objects are then mapped onto both PAN and fused images. The proposed spectral index measures the spectral homogeneity of the fused image’s objects and the proposed spatial index measures the injected spatial content from the PAN image to the fused image’s objects. Experimental results underscore the robustness and reliability of the proposed method. Additionally, by visualizing distortion values on object-maps, we gain insights into fusion quality across diverse areas within the scene.

The objective of the pansharpening task is to integrate multispectral (MS) images with low spatial resolution (LR) and to integrate panchromatic (PAN) images with high spatial resolution (HR) to generate HRMS images. Recently, deep learning-based pansharpening methods have been widely studied. However, traditional deep learning methods lack transparency while deep unrolling methods have limited performance when using one implicit prior for HRMS images. To address this issue, we incorporate one implicit prior with a semi-implicit prior and propose a double prior deep unrolling network (DPDU-Net) for pansharpening. Specifically, we first formulate the objective function based on observation models of PAN and LRMS images and two priors of an HRMS image. In addition to the implicit prior in the image domain, we enforce the sparsity of the HRMS image in a certain multi-scale implicit space; thereby, the feature map can obtain better sparse representation ability. We optimize the proposed objective function via alternating iteration. Then, the iterative process is unrolled into an elaborate network, with each iteration corresponding to a stage of the network. We conduct both reduced-resolution and full-resolution experiments on two satellite datasets. Both visual comparisons and metric-based evaluations consistently demonstrate the superiority of the proposed DPDU-Net. [ABSTRACT FROM AUTHOR]

Hyperspectral pansharpening is crucial for the improvement of the usability of images in various applications. However, it remains underexplored due to a scarcity of data. The primary goal of pansharpening is to enhance the spatial resolution of hyperspectral images by reconstructing missing spectral information without compromising consistency with the original data. This paper addresses the data gap by presenting a new hyperspectral dataset specifically designed for pansharpening and the evaluation of several deep learning strategies using this dataset. The new dataset has two crucial features that make it invaluable for deep learning hyperspectral pansharpening research. (1) It presents the highest cardinality of images in the state of the art, making it the first statistically relevant dataset for hyperspectral pansharpening evaluation, and (2) it includes a wide variety of scenes, ensuring robust generalization capabilities for various approaches. The data, collected by the ASI PRISMA satellite, cover about 262,200 km2 and their heterogeneity is ensured by a random sampling of the Earth's surface. The analysis of the deep learning methods consists in the adaptation of these approaches to the PRISMA hyperspectral data and the quantitative and qualitative evaluation of their performance in this new scenario. The investigation included two settings: Reduced Resolution (RR) to evaluate the techniques in a controlled environment and Full Resolution (FR) for a real-world evaluation. In addition, for the sake of completeness, we have also included machine-learning-free approaches in both scenarios. Our comprehensive analysis reveals that data-driven neural network methods significantly outperform traditional approaches, demonstrating a superior adaptability and performance in hyperspectral pansharpening under both RR and FR protocols. [ABSTRACT FROM AUTHOR]

Pansharpening is a vital technique in remote sensing that combines a low‐resolution multi‐spectral image with its corresponding panchromatic image to obtain a high‐resolution multi‐spectral image. Despite its potential benefits, the challenge lies in extracting features from the source images and eliminating artefacts in the fused images. In response to the challenge, a hybrid generative adversarial network‐based model, termed SWPanGAN, is proposed. For better feature extraction, the conventional convolution neural network is replaced with a Swin transformer in the generator, which provides the generator with the ability to model long‐range dependencies. Additionally, to suppress artefacts, a wavelet‐based discriminator is proposed for effectively distinguishing the frequency discrepancy. With these modifications, both the generator and discriminator networks of SWPanGAN are enhanced. Extensive experiments illustrate that our SWPanGAN can generate high‐quality pansharpening images and surpass other state‐of‐the‐art methods. [ABSTRACT FROM AUTHOR]

Pansharpening involves the fusion of panchromatic (PAN) and multispectral (MS) images to obtain a high-resolution image with enhanced spectral and spatial information. Assessing the quality of the resulting fused image poses a challenge due to the absence of a high-resolution reference image. Numerous methods have been proposed to address this, from assessing quality at reduced resolution to full-resolution evaluations. Many existing approaches are pixel-based, where quality metrics are applied and averaged on individual pixels. In this article, we introduce a novel object-based method for assessing the quality of pansharpened images at full resolution. In object-based quality assessment methods, the reaction of different areas of the fused image to the fusion process is reflected. Our approach revolves around extracting objects from the given image and evaluating extracted objects. By doing so, the distinct responses of different objects within the fused image to the fusion process are captured. The proposed method leverages a unique object extraction technique known as segmentation by nearest neighbor (SNN) to extract objects of the MS image. This method extracts the objects based on the image's characteristics without any requirement for parameter tuning. These extracted objects are then mapped onto both PAN and fused images. The proposed spectral index measures the spectral homogeneity of the fused image's objects and the proposed spatial index measures the injected spatial content from the PAN image to the fused image's objects. Experimental results underscore the robustness and reliability of the proposed method. Additionally, by visualizing distortion values on object-maps, we gain insights into fusion quality across diverse areas within the scene. [ABSTRACT FROM AUTHOR]

The remote sensing applications demand the imaging products composed of rich spatial details and high spectral resolution, namely high resolution multispectral (HRMS) images which can not be generated by current generation satellite sensors due to technological and physical constraints. Pansharpening is an image fusion approach that combines the rich geometric details of the panchromatic image and the color information from the multispectral image into a single product. Inspite of the availability of numerous pansharpening techniques, there is a necessity to develop an effective mechanism for the balanced enhancement of spatial and spectral quality in the fused HRMS image. In this paper, a constrained optimization approach for multispectral and PAN image fusion (COAMP) is proposed. The total variation is an efficient operator to extract the geometric features of an image. The spatial differences are exported into the HRMS image with the aid of TV based gradient operator. The spectral information is transferred to the fused image using spectral vectors obtained by taking gradients in the third dimension along the bands of the multispectral image. The optimization problem is formulated and minimized using an efficient solver to get the HRMS image. The comprehensive experiments performed on three different datasets at both reduced scale and full scale resolutions. Additionally, quantitative assessment is also performed using six well-known metrics. The experiments approves the efficacy of the proposed COAMP model. [ABSTRACT FROM AUTHOR]

Copyright of Remote Sensing for Natural Resources is the property of Remote Sensing for Natural Resources Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The analysis and policymaking of national salt production require quality Salt Land maps with more detailed object information. This study successfully classified Salt Land into Active and Inactive based on pansharpening images using random forest algorithm. The results show: 1) the best pansharpening method is UNB PanSharp; 2) the texture variables have more significant contributions to enhance classification accuracy; 3) scheme 5 is the most efficient and recommended scheme in research, achieves an OA of 66.699% and a Kappa coefficient of 39.4%. This study can provide a good reference for developing Salt Land mapping and policies for practitioners or stakeholders. [ABSTRACT FROM AUTHOR]

Existing deep learning techniques for image fusion either learn image mapping (LIM) directly, which renders them ineffective at preserving details due to the equal consideration to each pixel, or learn detail mapping (LDM), which only attains a limited level of performance because only details are used for reasoning. The recent lossless invertible network (INN) has demonstrated its detail-preserving ability. However, the direct applicability of INN to the image fusion task is limited by the volume-preserving constraint. Additionally, there is the lack of a consistent detail-preserving image fusion framework to produce satisfactory outcomes. To this aim, we propose a general paradigm for image fusion based on a novel conditional INN (named DCINN). The DCINN paradigm has three core components: a decomposing module that converts image mapping to detail mapping; an auxiliary network (ANet) that extracts auxiliary features directly from source images; and a conditional INN (CINN) that learns the detail mapping based on auxiliary features. The novel design benefits from the advantages of INN, LIM, and LDM approaches while avoiding their disadvantages. Particularly, using INN to LDM can easily meet the volume-preserving constraint while still preserving details. Moreover, since auxiliary features serve as conditional features, the ANet allows for the use of more than just details for reasoning without compromising detail mapping. Extensive experiments on three benchmark fusion problems, i.e., pansharpening, hyperspectral and multispectral image fusion, and infrared and visible image fusion, demonstrate the superiority of our approach compared with recent state-of-the-art methods. The code is available at https://github.com/wwhappylife/DCINN [ABSTRACT FROM AUTHOR]

Pansharpening is a crucial step in various remote sensing tasks, aimed at generating high-resolution multispectral images from panchromatic and low-resolution multispectral images. While deep learning has shown promising results in improving the accuracy of pansharpening, previous models often enhanced accuracy by stacking a large number of trainable parameters, making model training and application challenging. In this article, we propose a pansharpening network based on detail guided and global scale convolution, which can balance the parameter quantity of the model and its accuracy. Specifically, our model utilizes the global convolutional neural network (GCNN) module, which has favorable time complexity and, to some extent, alleviates issues such as insufficient receptive fields and excessive compression of long-distance information found in traditional convolutional neural networks. GCNN enables our model to capture global information effectively. In addition, we introduce a detail guided residual learning module that uses high-resolution image information to enhance details and compensate for the loss of high-frequency information during forward propagation. Furthermore, we design a lightweight convolutional module named channel aggregation learning that utilizes partial convolution for efficient interaction of interchannel feature information. Moreover, we introduce fast Fourier transform loss in the loss function to capture frequency domain information loss, further improving model performance. Extensive experiments on multiple datasets demonstrate the effectiveness of our proposed method.

In the field of remote sensing, it is not feasible to obtain high spatial resolution multispectral (HRMS) images from a single satellite sensor. The existing methods use pansharpening techniques to obtain HRMS images by fusing panchromatic (PAN) and multispectral (MS) images. However, due to the scale difference between PAN and MS images, most pansharpening methods often use explicit sampling methods to integrate features at different scales. These explicit-based sampling techniques represent pixels as discrete points through predefined functions, rendering it challenging to fit the distribution among diverse modal data, this results in the loss of image texture details during the fusion process. Implicit neural networks can enhance the generative capability of images by incorporating pixel coordinate information, which is crucial for the fusion of remote sensing images with different spatial resolutions. Inspired by implicit neural representation, we propose a progressive implicit neural feature fusion network (PINFNet) for remote sensing images. A progressive implicit neural feature fusion is proposed; it establishes a coordinate modal relationship between the spatial and spectral information through the guidance of the high spatial features in PAN images. This enables the proposed PINFNet to progressively learn and integrate spatial and spectral information at different scales. Our method, as opposed to discrete sampling techniques, is capable of establishing a continuous representation between diverse modal data, which in turn preserves more texture detail information. Extensive experiments have shown that this approach outperforms state-of-the-art methods while maintaining high efficiency.

The goal of pansharpening is to fuse a high spatial resolution panchromatic (PAN) image with a lower spatial resolution multispectral (MS) image to produce a high-resolution multispectral image. Most deep learning-based methods consider only local or global features, and focusing solely on one type of feature may limit the network's representational capacity. In addition, the fusion process often overlooks the heterogeneous and complementary information unique to PAN and MS images. Therefore, we propose a prompt-driven dynamic adaptive fusion network. To better combine the advantages of local and global features, we introduce a local and global adaptive modulation module. We also innovatively propose a prompt-driven dynamic fusion module that effectively integrates unique heterogeneous information while reconstructing corresponding complementary information. Finally, an expert mixing mechanism is employed to enhance the fused features, achieving superior fusion results. Our proposed method outperforms recent pansharpening methods, as demonstrated by reduced-resolution experiments and full-resolution validation.

Multispectral images captured by remote sensing systems usually have low spatial resolution. Pansharpening offers a promising solution by enhancing the resolution of these low-resolution multispectral images to a high-resolution multispectral without the need for costly hardware upgrades. Existing methods employ either CNN or Transformer as the feature extractor backbone, however, CNN-based methods are weak in capturing long-distance correlation, and Transformer-based methods are limited to extracting fine-grain detail. Moreover, these models achieve impressive results with numerous learnable parameters, which makes them impractical for integration into remote sensing systems. In this work, a parameter-efficient pansharpening model, named joint-modality association network, is built by leveraging complementary information from multiple modalities and recursive training. It aims to improve the resolution of remote-sensing images. Specifically, we efficiently leverage the complementary information from different modalities, including the transformer and CNN joint block, and employ a hierarchical association mechanism to create a more distinctive and informative representation by associating intramodality and cross-modality. Furthermore, the parameter-sharing mechanism of recursive training can effectively reduce the number of parameters in the model. Benefiting from its lightweight design and effective information fusion strategy, the proposed method can generate faithful super-resolved multispectral images that excel in both spectral and spatial resolution. Experimental results show the superiority of the proposed method over extensive benchmarks.

Full-resolution quality evaluation model for pansharpened images is significant for remote sensing applications, yet presents a challenge of the absence of reference compared with the reduced-resolution approach. To predict the image quality accurately, it is necessary to consider the distortion during the pansharpening process. Based on an observation that the quality of pairwise images can more easily be ranked, we propose a rank learning based full-resolution quality evaluation method for pansharpened images. Our approach begins with the synthesizing of ranked distortion images in spatial and spectral domains. Then, we develop a pansharpening distortion-perceiving model. This model employs spatial and spectral Siamese networks to perceive distortions and applies a pair-wise learning strategy for ranked images. Consequently, we establish a distortion-guided full-resolution quality evaluation framework for pansharpening. This framework integrates the spatial and spectral distortion-perceiving network and is enhanced with a dimension alignment module and a discrepancy Rrpresentation module, enabling effective distortion extraction among high-resolution multispectral, panchromatic, and low-resolution multispectral images. We conducted a series of experiments on a large-scale public pansharpened database. The experimental results demonstrate the effectiveness of our proposed approach.

Pansharpening is a process that fuses a multispectral (MS) image with a panchromatic (PAN) image to generate a high-resolution multispectral (HRMS) image. Current methods often overlook scale inconsistency and the correlation within and between a window domain, resulting in suboptimal outcomes. In addition, the use of deep convolutional neural network or transformer often leads to high computational expenses. To address these challenges, we present a lightweight pansharpening network that leverages cross-scale interaction and spatial-spectral enhanced window attention. We first design a spatial-spectral enhanced window transformer (SEWformer) to effectively capture crucial attention within and between interleaved windows. To improve scale consistency, we develop a cross-scale interactive encoder that interacts with different scale attentions derived from the SEWformer. Furthermore, a multiscale residual network with channel attention is constructed as a decoder, which, in conjunction with the encoder, ensures precise detail extraction. The final HRMS image is obtained by combining the extracted details with the UPMS image. Extensive experimental validation on diverse datasets showcases the superiority of our approach over state-of-the-art pansharpening techniques in terms of both performance and efficiency. Compared to the second-best comparison approach, our method achieves significant improvements in the ERGAS metric: 29.6$\%$ on IKONOS, 43.8$\%$ on Pléiades, and 27.6$\%$ on WorldView-3 datasets.

Pansharpening is a fundamental and crucial image processing task for many remote sensing applications, which generates a high-resolution multispectral image by fusing a low-resolution multispectral image and a high-resolution panchromatic image. Recently, vision transformers have been introduced into the pansharpening task for utilizing global contextual information. However, long-range and local dependencies modeling and multiscale feature learning are all essential to the pansharpening task. Learning and exploiting these various information raises a big challenge and limits the performance and efficiency of existing pansharpening methods. To solve this issue, we propose a pansharpening network based on multiscale embedding and dual attention transformers (MDPNet). Specifically, a multiscale embedding block is proposed to embed multiscale information of the images into vectors. Thus, transformers only need to process a multispectral embedding sequence and a panchromatic embedding sequence to efficiently use multiscale information. Furthermore, an additive hybrid attention transformer is proposed to fuse the embedding sequences in an additive injection manner. Finally, a channel self-attention transformer is proposed to utilize channel correlations for high-quality detail generation. Experiments over QuickBird and WorldView-3 datasets demonstrate the proposed MDPNet outperforms state-of-the-art methods visually and quantitatively with low running time. Ablation studies further verify the effectiveness of the proposed multiscale embedding and transformers in pansharpening.

Hypersharpening is one of the fusion-based superresolution approaches in remote sensing that improves the spatial and spectral resolution of a hyperspectral image (HSI) with a low spatial resolution. This requirement is achieved by fusing the HSI with a panchromatic image that has high spatial resolution to generate a newly combined variant, which has high spatial quality and high spectral resolution. While several studies in the literature applied neural networks for hypersharpening, there exist unsolved issues such as how to deeply discover the spatial–spectral correlation and inject geometric details without distortion. To address these issues, we propose a hypersharpening technique by applying a multicomponent-based hierarchical fusion network called (HS-McHF), which hierarchically learns the low and high-frequency spatial–spectral features. We then suggest an optimization model to discover the correlation between low-resolution HSI and high-resolution panchromatic images and solve it by stochastic gradient descent through a neural network. Moreover, we decrease the band overlapping in the initial HSI by combining a deconvolution model to prevent spectral distortion and reduce the noise in the panchromatic geometric details injection by deploying an encoder–decoder network. Our extensive experiments demonstrate that the HS-McHF provides superior efficiency compared to state-of-the-art fusion based superresolution approaches.

Multispectral (MS) image pansharpening is a significant technology for remote sensing image analysis, aiming to restore a high-resolution MS image by merging a high-resolution panchromatic image with a low-resolution MS image. While the convolutional neural networks (CNNs) have garnered considerable attention for their exceptional fusion capabilities in recent years, the existing CNN-based methods cannot effectively integrate spectral–spatial information. In this article, we introduce a novel MS pansharpening framework that integrates spectral and spatial networks in a bilateral pyramid form, allowing for the extraction of hierarchical spectral–spatial information. The proposed reduced residual dense (RRD) module serves as the fundamental building block of the spatial network. The RRD module gradually reduces the dimension of feature maps and employs the concatenation of RRD with global residual learning for comprehensive feature representation. In the spectral network, we present a cooperative attention fusion module to further enhance the correlation between spectral and spatial features. Through extensive experiments conducted on benchmarking simulated and real datasets, our proposed framework consistently outperforms state-of-the-art methods, demonstrating its effectiveness in MS image pansharpening applications.

Hyperspectral (HS) pansharpening refers to fusing low spatial resolution HS (LRHS) images with the corresponding panchromatic (PAN) images to create high spatial resolution HS (HRHS) images. Most of the existing HS pansharpening methods overlook the spatial and spectral imbalance of the ground objects of different types in the observed scenes. To address the dilemma, in this article we develop a novel tree-structured neural network (Tree-SNet) to form an adaptive spatial-spectral processing for HS pansharpening. The Tree-SNet method maps a convolutional neural network (CNN) onto a hierarchical tree structure, where routing nodes automatically tune the data distributed to tree paths, which is adaptive to the local characteristics of the data, while spatial enhancement (SpatE) and spectral enhancement (SpecE) modules are dynamically performed in the tree paths to further strengthen the adaptive processing. The proposed Tree-SNet is evaluated on several datasets, and the experimental results verify its superiority.

Due to the limitations of satellite sensors, we can only obtain MS images and PAN images separately. The focus of our attention is to utilize the pan-sharpening method to generate the high-resolution multispectral (HRMS) images. In this article, we proposed the dual-branch multiscale fusion network, which based on the spatial-spectral transformer to comprehensively capture the information contained in MS images and PAN images at different scales. The architecture of our network consists of three parts: during the feature extraction and image fusion stage, we first independently apply upscaling and downscaling operations to the MS and PAN images. Subsequently, we concatenate the images from the two distinct branches and input them into the shallow feature extraction module individually. And then we input them into our adaptive feature extraction block to further extract the crucial details of the images using the attention mechanism. The images a various scales in different branches are then passed through three spectral transformer and three spatial transformer modules to perform a comprehensive extraction of both spatial and spectral characteristics. Finally, the residual local feature module is utilized during the image reconstruction part to deeply extract intricate information from the images and obtain the final HRMS fused image. We have conducted both simulated and real experiments on the benchmark datasets QB and WV2. The final qualitative and quantitative comparative results demonstrate that our innovative method outperforms the current SOTA methods.

Image preprocessing and fusion are commonly used for enhancing remote-sensing images, but the resulting images often lack useful spatial features. As the majority of research on image fusion has concentrated on the satellite domain, the image-fusion task for Unmanned Aerial Vehicle (UAV) images has received minimal attention. This study investigated an image-improvement strategy by integrating image preprocessing and fusion tasks for UAV images. The goal is to improve spatial details and avoid color distortion in fused images. Techniques such as image denoising, sharpening, and Contrast Limited Adaptive Histogram Equalization (CLAHE) were used in the preprocessing step. The unsharp mask algorithm was used for image sharpening. Wiener and total variation denoising methods were used for image denoising. The image-fusion process was conducted in two steps: (1) fusing the spectral bands into one multispectral image and (2) pansharpening the panchromatic and multispectral images using the PanColorGAN model. The effectiveness of the proposed approach was evaluated using quantitative and qualitative assessment techniques, including no-reference image quality assessment (NR-IQA) metrics. In this experiment, the unsharp mask algorithm noticeably improved the spatial details of the pansharpened images. No preprocessing algorithm dramatically improved the color quality of the enhanced images. The proposed fusion approach improved the images without importing unnecessary blurring and color distortion issues. [ABSTRACT FROM AUTHOR]

Pansharpening is an established remote sensing image fusion technique that yields a high-resolution multispectral (HRMS) image. Despite the fact that the advanced technologies like sparse coding and deep learning have achieved a remarkable improvement in solving the pansharpening problem, a unified model is required to further enhance the fusion quality. The variational optimisation (VO) mechanism has gained interest of most of the researchers in recent years. In this article, pansharpening is designated as a constrained optimisation problem with a data generative term and two regularisers to ameliorate spatial details and spectral information. The gradient information is exploited to impart the spatial details from the panchromatic image to the fused image. The correlation among multispectral image bands inspired to promote the spectral quality of the HRMS image as well as to reduce the distortion. Consequently, the optimisation problem is efficiently solved for the required HRMS image using the operator splitting approach. The extensive experiments performed in accordance with the well-known protocols rationalise that the proposed model outperforms most of the state-of-the-art methods in terms of objective metrics and visual outcomes. [ABSTRACT FROM AUTHOR]

High spatial resolution multispectral (HMS) images can provide sufficient information for researchers to analyze the potential disasters in the living environment. However, an original multispectral (MS) image is with low-space-resolution and high-spectrum-resolution, while an original panchromatic (PAN) image has the opposite property. Pansharpening aims at obtaining HMS image by retaining spectrum of the MS image and injecting details of the PAN image simultaneously. In this paper, we present a new pansharpening method. First, we use a bilateral filter (BF) to obtain the low-frequency-component (LFC) of PAN and MS images, respectively. Then the high-frequency-component (HFC) of PAN and MS images are readily obtained. Second, an adaptive intensity-hue-saturation (AIHS) based method is applied to generate the HFC of intensity. Finally, a dual-scale guided image filter (GIF) is utilized to calculate the difference between HFC of intensity and PAN to get the detail images. And then, these detail images are injected into the original MS image to achieve the HMS image. The proposed method is applied into testing various satellite data sets, and performs better effect on both visual quality and objective indictors than the existing methods. [ABSTRACT FROM AUTHOR]

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods. [ABSTRACT FROM AUTHOR]

Joint super-resolution and pansharpening (JSP) brings new insight into the spatial improvement of multispectral images. How to efficiently balance the spatial and spectral qualities in JSP is important for deep learning-based approaches. To address this problem, we propose a unified interpretable deep network for JSP, named UIJSP-Net. First, we formulate the JSP problem as an optimization problem in a specially designed physical model based on the relationship among the JSP result, the multispectral image, and the panchromatic image. In particular, two deep priors are utilized to describe latent distributions of different variables, which can improve the accuracy of the physical model. Furthermore, we adopt the alternating direction method of multipliers to solve the above optimization problem, where a series of iterative steps are generated. Finally, we design UIJSP-Net by unfolding these iterative steps into multiple corresponding stages in a unified network. Because UIJSP-Net has clear physical meanings, the spatial resolution of multispectral images can be efficiently improved while the spectral information can be kept as well. Extensive experimental results are carried out on both simulated and real datasets to demonstrate the superiority of UIJSP-Net over other state-of-the-art methods from qualitative and quantitative aspects. [ABSTRACT FROM AUTHOR]

The Sustainable Development Science Satellite 1 (SDGSAT-1) satellite, launched in November 2021, is dedicated to providing data detailing the "traces of human activities" for the implementation of the United Union's 2030 Agenda for Sustainable Development and global scientific research. The glimmer imager (GI) that is equipped on SDGSAT-1 can provide nighttime light (NL) data with a 10 m panchromatic (PAN) band and red, green, and blue (RGB) bands of 40 m resolution, which can be used for a wide range of applications, such as in urban expansion, population studies of cities, and economics of cities, as well as nighttime aerosol thickness monitoring. The 10 m PAN band can be fused with the 40 m RGB bands to obtain a 10 m RGB NL image, which can be used to identify the intensity and type of night lights and the spatial distribution of road networks and to improve the monitoring accuracy of sustainable development goal (SDG) indicators related to city developments. Existing remote sensing image fusion algorithms are mainly developed for daytime optical remote sensing images. Compared with daytime optical remote sensing images, NL images are characterized by a large amount of dark (low-value) pixels and high background noises. To investigate whether daytime optical image fusion algorithms are suitable for the fusion of GI NL images and which image fusion algorithms are the best choice for GI images, this study conducted a comprehensive evaluation of thirteen state-of-the-art pansharpening algorithms in terms of quantitative indicators and visual inspection using four GI NL datasets. The results showed that PanNet, GLP_HPM, GSA, and HR outperformed the other methods and provided stable performances among the four datasets. Specifically, PanNet offered UIQI values ranging from 0.907 to 0.952 for the four datasets, whereas GSA, HR, and GLP_HPM provided UIQI values ranging from 0.770 to 0.856. The three methods based on convolutional neural networks achieved more robust and better visual effects than the methods using multiresolution analysis at the original scale. According to the experimental results, PanNet shows great potential in the fusion of SDGSAT-1 GI imagery due to its robust performance and relatively short training time. The quality metrics generated at the degraded scale were highly consistent with visual inspection, but those used at the original scale were inconsistent with visual inspection. [ABSTRACT FROM AUTHOR]

The multispectral image combines monochrome and multiple bands from a sensor capturing the same area, with lower spectral resolution than the panchromatic (PAN) band. To boost the resolution, a pan-sharpening technique fuses the low-resolution bands with the high-resolution PAN band. This study focuses on using IHS transformation with cylindrical and cubical models to fuse multi-sensor data. The resulting hybrid images are well suited for human and machine analysis, enabling tasks like segmentation, feature extraction, and object identification. The fusion process's effectiveness is evaluated using various metrics. A modified Otsu method is introduced, detecting multiple objects instead of just one, optimizing computation resources for efficient feature extraction from pan-sharpened imagery. This research advances image processing techniques, particularly in land cover and land use (LCLU) classification. The study's primary contribution lies in comparing the effectiveness of the IHS method with the novel Otsu method tailored for LCLU classification. This provides a deeper understanding of multiple objects in imagery, enhancing land cover and land use analysis capabilities. The methodology proves valuable for resource-limited systems and lower-end configurations. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) pansharpening refers to fusing low spatial resolution HS (LRHS) images with the corresponding panchromatic (PAN) images to create high spatial resolution HS (HRHS) images. Most of the existing HS pansharpening methods overlook the spatial and spectral imbalance of the ground objects of different types in the observed scenes. To address the dilemma, in this article we develop a novel tree-structured neural network (Tree-SNet) to form an adaptive spatial-spectral processing for HS pansharpening. The Tree-SNet method maps a convolutional neural network (CNN) onto a hierarchical tree structure, where routing nodes automatically tune the data distributed to tree paths, which is adaptive to the local characteristics of the data, while spatial enhancement (SpatE) and spectral enhancement (SpecE) modules are dynamically performed in the tree paths to further strengthen the adaptive processing. The proposed Tree-SNet is evaluated on several datasets, and the experimental results verify its superiority. [ABSTRACT FROM AUTHOR]

Pansharpening is a fundamental and crucial image processing task for many remote sensing applications, which generates a high-resolution multispectral image by fusing a low-resolution multispectral image and a high-resolution panchromatic image. Recently, vision transformers have been introduced into the pansharpening task for utilizing global contextual information. However, long-range and local dependencies modeling and multiscale feature learning are all essential to the pansharpening task. Learning and exploiting these various information raises a big challenge and limits the performance and efficiency of existing pansharpening methods. To solve this issue, we propose a pansharpening network based on multiscale embedding and dual attention transformers (MDPNet). Specifically, a multiscale embedding block is proposed to embed multiscale information of the images into vectors. Thus, transformers only need to process a multispectral embedding sequence and a panchromatic embedding sequence to efficiently use multiscale information. Furthermore, an additive hybrid attention transformer is proposed to fuse the embedding sequences in an additive injection manner. Finally, a channel self-attention transformer is proposed to utilize channel correlations for high-quality detail generation. Experiments over QuickBird and WorldView-3 datasets demonstrate the proposed MDPNet outperforms state-of-the-art methods visually and quantitatively with low running time. Ablation studies further verify the effectiveness of the proposed multiscale embedding and transformers in pansharpening. [ABSTRACT FROM AUTHOR]

Pansharpening, a pivotal task in remote sensing, involves integrating low-resolution multispectral images with high-resolution panchromatic images to synthesize an image that is both high-resolution and retains multispectral information. These pansharpened images enhance precision in land cover classification, change detection, and environmental monitoring within remote sensing data analysis. While deep learning techniques have shown significant success in pansharpening, existing methods often face limitations in their evaluation, focusing on restricted satellite data sources, single scene types, and low-resolution images. This paper addresses this gap by introducing PanBench, a high-resolution multi-scene dataset containing all mainstream satellites and comprising 5898 pairs of samples. Each pair includes a four-channel (RGB + near-infrared) multispectral image of 256 × 256 pixels and a mono-channel panchromatic image of 1024 × 1024 pixels. To avoid irreversible loss of spectral information and achieve a high-fidelity synthesis, we propose a Cascaded Multiscale Fusion Network (CMFNet) for pansharpening. Multispectral images are progressively upsampled while panchromatic images are downsampled. Corresponding multispectral features and panchromatic features at the same scale are then fused in a cascaded manner to obtain more robust features. Extensive experiments validate the effectiveness of CMFNet.

Pansharpening refers to the process of fusing multispectral (MS) images with panchromatic (PAN) images to obtain high-resolution multispectral (HRMS) images. However, due to the low correlation and similarity between MS and PAN images, as well as inaccuracies in spatial information injection, HRMS images often suffer from significant spectral and spatial distortions. To address these issues, a pansharpening method based on multimodal texture correction and adaptive edge detail fusion is proposed in this paper. To obtain a texture-corrected (TC) image that is highly correlated and similar to the MS image, the target-adaptive CNN-based pansharpening (A-PNN) method is introduced. By constructing a multimodal texture correction model, intensity, gradient, and A-PNN-based deep plug-and-play correction constraints are established between the TC and source images. Additionally, an adaptive degradation filter algorithm is proposed to ensure the accuracy of these constraints. Since the TC image obtained can effectively replace the PAN image and considering that the MS image contains valuable spatial information, an adaptive edge detail fusion algorithm is also proposed. This algorithm adaptively extracts detailed information from the TC and MS images to apply edge protection. Given the limited spatial information in the MS image, its spatial information is proportionally enhanced before the adaptive fusion. The fused spatial information is then injected into the upsampled multispectral (UPMS) image to produce the final HRMS image. Extensive experimental results demonstrated that compared with other methods, the proposed algorithm achieved superior results in terms of both subjective visual effects and objective evaluation metrics.

In recent years, deep learning techniques for pansharpening multiresolution images have gained increasing interest. Due to the lack of ground truth data, most deep learning solutions rely on synthetic reduced-resolution data for supervised training. This approach has limitations due to the statistical mismatch between real full-resolution and synthetic reduced-resolution data, which affects the models’ generalization capacity. Consequently, there has been a shift towards unsupervised learning frameworks for pansharpening deep learning-based techniques. Unsupervised schemes require defining sophisticated loss functions with at least two components: one for spectral quality, ensuring consistency between the pansharpened image and the input multispectral component, and another for spatial quality, ensuring consistency between the output and the panchromatic input. Despite promising results, there has been limited investigation into the interaction and balance of these loss terms to ensure stability and accuracy. This work explores how unsupervised spatial and spectral consistency losses can be reliably combined preserving the outocome quality. By examining these interactions, we propose a general rule for balancing the two loss components to enhance the stability and performance of unsupervised pansharpening models. Experiments on three state-of-the-art algorithms using WorldView-3 images demonstrate that methods trained with the proposed framework achieve good performance in terms of visual quality and numerical indexes.

Pansharpening strives to improve the spatial resolution of multi-spectral images while maintaining spectral fidelity. However, existing methods usually cannot guarantee a balance between spatial and spectral quality, and degrade when handling the desirable scale of full resolution. To address these challenges, this prospective study proposes a scale-transfer learning framework via estimating spectral observation model. Specifically, we design a cross-spectral transfer network to learn an expected spectral observation model by the cycle adversarial between spectral degradation and interpolation, which describes the accurate nonlinear mapping process from multi-spectral to panchromatic images. Having the favorable spectral observation model established, a scale-transfer pansharpening paradigm with co-learning of full and reduced resolutions can be constructed. First, we develop a spectral observation model-based spatial fidelity term at the reduced-resolution scale, which can alleviate the imbalance problem of spectral and spatial information widespread in current supervised paradigms. Second, we explore the reprojection regularization from full to reduced resolution based on the spectral observation model, which facilitates the ability of the pansharpening model to be extended to the scale of full resolution. Extensive experiments demonstrate the advantage of our method over the current state-of-the-arts in terms of information balance and scale transformation. We further apply our method to produce the high-resolution normalized difference vegetation index and achieve vegetation enhancement with competitive performance. Moreover, our method is lightweight and faster than other comparative methods. [ABSTRACT FROM AUTHOR]

Pansharpening refers to the fusion of a multispectral image (MS) and a panchromatic image (PAN) to obtain a new image with the same spatial resolution as the PAN image and the same spectral resolution as the MS image. This paper describes a new, efficient, and accurate pansharpening architecture. The Vector-Quantized Variational AutoEncoder (VQ-VAE) is the foundation of the proposed method. The VQ-VAE model is trained to learn the nonlinear mapping of degraded panchromatic image patches to highresolution patches. This approach ensures that high-resolution patches can be recovered from low-resolution ones. After training on PAN patches, the VQ-VAE estimates high-resolution multispectral patches for each band of the original multispectral image before reconstructing the high-resolution multispectral image from the patches. The original multispectral image, the panchromatic image, and the estimated high-resolution multispectral image are combined through a modified Component Substitution (CS) process to obtain the pansharpened image. Three large satellite datasets from urban areas with 4-band spectral resolution (blue, green, red, and near-infrared) were used to evaluate the proposed pansharpening method's performance. The effectiveness of the proposed method is demonstrated by the quantitative and visual results obtained compared to several literature approaches. [ABSTRACT FROM AUTHOR]

With the recent advancements in technology, there has been a tremendous growth in the usage of images captured using satellites in various applications, like defense, academics, resource exploration, land-use mapping, and so on. Certain mission-critical applications need images of higher visual quality, but the images captured by the sensors normally suffer from a tradeoff between high spectral and spatial resolutions. Hence, for obtaining images with high visual quality, it is necessary to combine the low resolution multispectral (MS) image with the high resolution panchromatic (PAN) image, and this is accomplished by means of pansharpening. In this paper, an efficient pansharpening technique is devised by using a hybrid optimized deep learning network. Zeiler and Fergus network (ZF Net) is utilized for performing the fusion of the sharpened and upsampled MS image with the PAN image. A novel Dingo coot (DICO) optimization is created for updating the learning parameters and weights of the ZF Net. Moreover, the devised DICO_ZF Net for pansharpening is examined for its effectiveness by considering measures, like Peak Signal To Noise Ratio (PSNR) and Degree of Distortion (DD) and is found to have attained values at 50.177 dB and 0.063 dB. [ABSTRACT FROM AUTHOR]

Pansharpening is a technique used in remote sensing to combine high-resolution panchromatic (PAN) images with lower resolution multispectral (MS) images to generate high-resolution multispectral images while preserving spectral characteristics. Recently, convolutional neural networks (CNNs) have been the mainstream in pansharpening by extracting the deep features of PAN and MS images and fusing these abstract features to reconstruct high-resolution details. However, they are limited by the short-range contextual dependencies of convolution operations. Although transformer models can alleviate this problem, they still suffer from weak capability in reconstructing high-resolution detailed information from global representations. To this end, a novel Swin-transformer-based pansharpening model named SwinPAN is proposed. Specifically, a detail reconstruction network (DRNet) is developed in an image difference and residual learning framework to reconstruct the high-resolution detailed information from the original images. DRNet is developed based on the Swin Transformer with a dynamic high-pass preservation module with adaptive convolution kernels. The experimental results on three remote sensing datasets with different sensors demonstrate that the proposed approach performs better than state-of-the-art networks through qualitative and quantitative analysis. Specifically, the generated pansharpening results contain finer spatial details and richer spectral information than other methods. [ABSTRACT FROM AUTHOR]

Pansharpening is the fusion of panchromatic images and multispectral images to obtain images with high spatial resolution and high spectral resolution, which have a wide range of applications. At present, methods based on deep learning can fit the nonlinear features of images and achieve excellent image quality; however, the images generated with supervised learning approaches lack real-world applicability. Therefore, in this study, we propose an unsupervised pansharpening method based on a generative adversarial network. Considering the fine tubular structures in remote sensing images, a dense connection attention module is designed based on dynamic snake convolution to recover the details of spatial information. In the stage of image fusion, the fusion of features in groups is applied through the cross-scale attention fusion module. Moreover, skip layers are implemented at different scales to integrate significant information, thus improving the objective index values and visual appearance. The loss function contains four constraints, allowing the model to be effectively trained without reference images. The experimental results demonstrate that the proposed method outperforms other widely accepted state-of-the-art methods on the QuickBird and WorldView2 data sets.

The objective of the pansharpening task is to integrate multispectral (MS) images with low spatial resolution (LR) and to integrate panchromatic (PAN) images with high spatial resolution (HR) to generate HRMS images. Recently, deep learning-based pansharpening methods have been widely studied. However, traditional deep learning methods lack transparency while deep unrolling methods have limited performance when using one implicit prior for HRMS images. To address this issue, we incorporate one implicit prior with a semi-implicit prior and propose a double prior deep unrolling network (DPDU-Net) for pansharpening. Specifically, we first formulate the objective function based on observation models of PAN and LRMS images and two priors of an HRMS image. In addition to the implicit prior in the image domain, we enforce the sparsity of the HRMS image in a certain multi-scale implicit space; thereby, the feature map can obtain better sparse representation ability. We optimize the proposed objective function via alternating iteration. Then, the iterative process is unrolled into an elaborate network, with each iteration corresponding to a stage of the network. We conduct both reduced-resolution and full-resolution experiments on two satellite datasets. Both visual comparisons and metric-based evaluations consistently demonstrate the superiority of the proposed DPDU-Net.