Search

Due to the difficulty of collecting a sufficient number of photons within a narrow spectral window, hyperspectral images often have low spatial resolution. In order to solve this problem without increasing hardware costs, a large number of image super-resolution algorithms have been proposed and have shown promising results. However, the performance of these algorithms is inconsistent in different datasets. To fix this situation, we proposed a multi-model deep prior regularization method for multispectral and hyperspectral image fusion. This method leverages mechanistic knowledge to combine various super-resolution results into a single algorithm, significantly improving the quality of the hyperspectral images. Specifically, we first proposed a variable residual mixed attention network. This network takes a low-resolution hyperspectral image and a high-resolution multispectral image as inputs, producing deep prior results from three different modes by adjusting spatial and channel attention. Then, we reconstructed the prior results along the spectral dimension and used them as regularization terms in our mechanistic model. We transformed the problem of finding the optimal solution of the model into solving the Sylvester equation and applied a fast solver for this equation to create the high-resolution hyperspectral images. Additionally, we used a genetic algorithm to determine the best hyperparameters for different deep prior results. Through validation on three datasets, we have demonstrated that our method performs better than existing methods in both quantitative and qualitative comparisons. [ABSTRACT FROM AUTHOR]

The fusion of low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) provides a cost-effective approach to obtaining high-resolution hyperspectral images (HR-HSI). Existing methods primarily based on convolutional neural networks (CNNs) struggle to capture global features and do not adequately address the significant scale and spectral resolution differences between LR-HSI and HR-MSI. To tackle these challenges, our novel FCSwinU network leverages the spectral fast Fourier convolution (SFFC) module for spectral feature extraction and utilizes the Swin Transformer's self-attention mechanism for multi-scale global feature fusion. FCSwinU employs a UNet-like encoder–decoder framework to effectively merge spatiospectral features. The encoder integrates the Swin Transformer feature abstraction module (SwinTFAM) to encode pixel correlations and perform multi-scale transformations, facilitating the adaptive fusion of hyperspectral and multispectral data. The decoder then employs the Swin Transformer feature reconstruction module (SwinTFRM) to reconstruct the fused features, restoring the original image dimensions and ensuring the precise recovery of spatial and spectral details. Experimental results from three benchmark datasets and a real-world dataset robustly validate the superior performance of our method in both visual representation and quantitative assessment compared to existing fusion methods. [ABSTRACT FROM AUTHOR]

Multispectral and hyperspectral image fusion (MS/HS fusion) aims to generate a high-resolution hyperspectral (HRHS) image by fusing a high-resolution multispectral (HRMS) and a low-resolution hyperspectral (LRHS) images. The deep unfolding-based MS/HS fusion method is a representative deep learning paradigm due to its excellent performance and sufficient interpretability. However, existing deep unfolding-based MS/HS fusion methods only rely on a fixed linear degradation model, which focuses on modeling the relationships between HRHS and HRMS, as well as HRHS and LRHS. In this paper, we break free from this observation model framework and propose a new observation model. Firstly, the proposed observation model is built based on the convolutional sparse coding (CSC) technique, and then a proximal gradient algorithm is designed to solve this model. Secondly, we unfold the iterative algorithm into a deep network, dubbed as MHF-CSCNet, where the proximal operators are learned using convolutional neural networks. Finally, all trainable parameters can be automatically learned end-to-end from the training pairs. Experimental evaluations conducted on various benchmark datasets demonstrate the superiority of our method both quantitatively and qualitatively compared to other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Multispectral pedestrian detection with visible light and infrared images is robust to changes in lighting conditions and therefore is of great importance to numerous applications that require all‐day environmental perception. This paper proposes a novel method named FCE‐RCNN, which integrates saliency detection as a sub‐task and utilizes global information for enhanced feature representation. The approach enhances thermal inputs by incorporating gradients at the raw‐data level before feature extraction. Utilizing a dual‐stream backbone, a global semantic information extraction module is introduced that combines pooling with horizontal–vertical attention mechanisms, capturing high‐quality global semantic information for lower‐level feature enrichment and guidance. Additionally, the pedestrian locality enhancement module is designed to enhance spatial locality information of pedestrians through saliency detection. Furthermore, to alleviate the challenges posed by positional shifts between cross‐spectral features, deformable convolution is innovatively employed. Experimental results on the KAIST dataset demonstrate that FCE‐RCNN significantly improves nighttime detection, achieving a log‐average miss rate of 6.92%, outperforming the new method ICAFusion by 0.93%. These results underscore the effectiveness of FCE‐RCNN, and the method also maintains competitive inference speed, making it suitable for real‐time applications. [ABSTRACT FROM AUTHOR]

Pansharpening refers to enhancing the spatial resolution of multispectral images through panchromatic images while preserving their spectral features. However, existing traditional methods or deep learning methods always have certain distortions in the spatial or spectral dimensions. This paper proposes a remote sensing spatial–spectral fusion method based on a multi-scale dual-stream convolutional neural network, which includes feature extraction, feature fusion, and image reconstruction modules for each scale. In terms of feature fusion, we propose a multi cascade module to better fuse image features. We also design a new loss function aim at enhancing the high degree of consistency between fused images and reference images in terms of spatial details and spectral information. To validate its effectiveness, we conduct thorough experimental analyses on two widely used remote sensing datasets: GeoEye-1 and Ikonos. Compared with the nine leading pansharpening techniques, the proposed method demonstrates superior performance in multiple key evaluation metrics. [ABSTRACT FROM AUTHOR]

As we take stock of the contemporary issue, remote sensing images are gradually advancing towards hyperspectral–high spatial resolution (H2) double-high images. However, high resolution produces serious spatial heterogeneity and spectral variability while improving image resolution, which increases the difficulty of feature recognition. So as to make the best of spectral and spatial features under an insufficient number of marking samples, we would like to achieve effective recognition and accurate classification of features in H2 images. In this paper, a cross-hop graph network for H2 image classification(H2-CHGN) is proposed. It is a two-branch network for deep feature extraction geared towards H2 images, consisting of a cross-hop graph attention network (CGAT) and a multiscale convolutional neural network (MCNN): the CGAT branch utilizes the superpixel information of H2 images to filter samples with high spatial relevance and designate them as the samples to be classified, then utilizes the cross-hop graph and attention mechanism to broaden the range of graph convolution to obtain more representative global features. As another branch, the MCNN uses dual convolutional kernels to extract features and fuse them at various scales while attaining pixel-level multi-scale local features by parallel cross connecting. Finally, the dual-channel attention mechanism is utilized for fusion to make image elements more prominent. This experiment on the classical dataset (Pavia University) and double-high (H2) datasets (WHU-Hi-LongKou and WHU-Hi-HongHu) shows that the H2-CHGN can be efficiently and competently used in H2 image classification. In detail, experimental results showcase superior performance, outpacing state-of-the-art methods by 0.75–2.16% in overall accuracy. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) satellites like PRISMA (PRecursore IperSpettrale della Missione Applicativa) offer remarkable capabilities, yet they are constrained by a relatively coarse spatial resolution, curbing their efficacy in those applications that require pinpoint accuracy. Here we propose a fusion process, aimed at the enhancement of PRISMA HS spatial resolution by using the spatial and spectral information of Sentinel-2 multispectral (MS) data (HS-MS fusion process), validated against four airborne HS flights simultaneous to satellite overpasses on different land use distributions. Adopting the PRISMA panchromatic (PAN) image, the proposed solution was also compared with the results of a HS-PAN pansharpening process. A two-steps operational workflow is proposed, based on two state-of-the-art and open-source algorithms. The first step consisted of the geocoding of PRISMA L2 products using Senintel-2 as reference and was accomplished with the phase-based algorithm implemented in AROSICS (Automated and Robust Open-Source Image Co-registration Software). The geometric displacement in L2 data was found to be between 80 m and 250 m, irregularly spatially distributed throughout the same scene and among scenes, and it was corrected by means of thousands of regularly spatially distributed tie points. A second-order polynomial transformation function was integrated in the algorithm. The second step consisted of employing the HySure (HS Super resolution) fusion algorithm to perform both the HS-MS fusion and the HS-PAN pansharpening, returning a PRISMA HS improved dataset with a spatial resolution of 10 m and 5 m, respectively. Four different per-band accuracy metrics were used to evaluate the accuracy of both products against airborne data. Overall, HS-MS data achieved increased accuracy in all validation metrics, i.e. + 28 % (root mean square error, RMSE), +23 % (spectral angle mapper, SAM), +7% (peak signal-to-noise ratio, PSNR) and + 11 % (universal image quality index, UIQI), with respect of HS-PAN data. These outcomes showed that using the spectral information of Sentinel-2 both spectral and spatial patterns were reconstructed more consistently in three different urban and rural scenarios, avoiding the presence of blur and at-edge artefacts as opposed to HS-PAN pansharpening, therefore suggesting an optimal strategy for satellite HS data resolution enhancement. [ABSTRACT FROM AUTHOR]

Pan-sharpening is a technique used to generate high-resolution multi-spectral (HRMS) images by merging high-resolution panchromatic (PAN) images with low-resolution multi-spectral (LRMS) images. Many existing methods face challenges in effectively balancing the trade-off between spectral and spatial information, leading to spectral and spatial structural distortion. In order to effectively tackle these issues, we propose a dual-branch and triple attention (DBTA) network. The proposed DBTA network consists of two essential modules: the Channel-spatial Attention (CSA) module and the Spectral Attention (SPA) module. The CSA module effectively captures the spatial structural information of the images by jointly using spatial and channel attention units. Meanwhile, the SPA module improves the expressive capacity of spectral information by dynamically adjusting channel weights. These two modules work in synergy to achieve comprehensive extraction and fusion of spectral and spatial information, thus resulting in more accurate and clearer reconstructed images. Extensive experiments have been conducted on various satellite datasets to evaluate the performance of the proposed DBTA method outperforms the state-of-the-art competitors in both qualitative and quantitative evaluations. [ABSTRACT FROM AUTHOR]

Efficient diagnosis of apple diseases and pests is crucial to the healthy development of the apple industry. However, the existing single-source image-based classification methods have limitations due to the constraints of single-source input image information, resulting in low classification accuracy and poor stability. Therefore, a classification method for apple disease and pest areas based on multi-source image fusion is proposed in this paper. Firstly, RGB images and multispectral images are obtained using drones to construct an apple diseases and pests canopy multi-source image dataset. Secondly, a vegetation index selection method based on saliency attention is proposed, which uses a multi-label ReliefF feature selection algorithm to obtain the importance scores of vegetation indices, enabling the automatic selection of vegetation indices. Finally, an apple disease and pest area multi-label classification model named AMMFNet is constructed, which effectively combines the advantages of RGB and multispectral multi-source images, performs data-level fusion of multi-source image data, and combines channel attention mechanisms to exploit the complementary aspects between multi-source data. The experimental results demonstrated that the proposed AMMFNet achieves a significant subset accuracy of 92.92%, a sample accuracy of 85.43%, and an F1 value of 86.21% on the apple disease and pest multi-source image dataset, representing improvements of 8.93% and 10.9% compared to prediction methods using only RGB or multispectral images. The experimental results also proved that the proposed method can provide technical support for the coarse-grained positioning of diseases and pests in apple orchards and has good application potential in the apple planting industry. [ABSTRACT FROM AUTHOR]

Copyright of Remote Sensing for Natural Resources is the property of Remote Sensing for Natural Resources Editorial Office and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Visual object tracking, pivotal for applications like earth observation and environmental monitoring, encounters challenges under adverse conditions such as low light and complex backgrounds. Traditional tracking technologies often falter, especially when tracking dynamic objects like aircraft amidst rapid movements and environmental disturbances. This study introduces an innovative adaptive multimodal image object-tracking model that harnesses the capabilities of multispectral image sensors, combining infrared and visible light imagery to significantly enhance tracking accuracy and robustness. By employing the advanced vision transformer architecture and integrating token spatial filtering (TSF) and crossmodal compensation (CMC), our model dynamically adjusts to diverse tracking scenarios. Comprehensive experiments conducted on a private dataset and various public datasets demonstrate the model's superior performance under extreme conditions, affirming its adaptability to rapid environmental changes and sensor limitations. This research not only advances visual tracking technology but also offers extensive insights into multisource image fusion and adaptive tracking strategies, establishing a robust foundation for future enhancements in sensor-based tracking systems. [ABSTRACT FROM AUTHOR]

From the perspective of optimization, most of the current mainstream remote sensing data fusion methods are based on traditional mathematical optimization or single objective optimization. The former requires manual parameter tuning and easily falls into local optimum. Although the latter can overcome the shortcomings of traditional methods, the single optimization objective makes it unable to combine the advantages of multiple models, which may lead to distortion of the fused image. To address the problems of missing multi-model combination and parameters needing to be set manually in the existing methods, a pansharpening method based on multi-model collaboration and multi-objective optimization is proposed, called MMCMOO. In the proposed new method, the multi-spectral image fusion problem is transformed into a multi-objective optimization problem. Different evolutionary strategies are used to design a variety of population generation mechanisms, and a non-dominated sorting genetic algorithm (NSGA-II) is used to optimize the two proposed target models, so as to obtain the best pansharpening quality. The experimental results show that the proposed method is superior to the traditional methods and single objective methods in terms of visual comparison and quantitative analysis on our datasets. [ABSTRACT FROM AUTHOR]

The objective of pan-sharpening is to effectively fuse high-resolution panchromatic (PAN) images with limited spectral information and low-resolution multispectral (LR-MS) images, thereby generating a fused image with a high spatial resolution and rich spectral information. However, current fusion techniques face significant challenges, including insufficient edge detail, spectral distortion, increased noise, and limited robustness. To address these challenges, we propose a multi-frequency spectral–spatial interaction enhancement network (MFSINet) that comprises the spectral–spatial interactive fusion (SSIF) and multi-frequency feature enhancement (MFFE) subnetworks. The SSIF enhances both spatial and spectral fusion features by optimizing the characteristics of each spectral band through band-aware processing. The MFFE employs a variant of wavelet transform to perform multiresolution analyses on remote sensing scenes, enhancing the spatial resolution, spectral fidelity, and the texture and structural features of the fused images by optimizing directional and spatial properties. Moreover, qualitative analysis and quantitative comparative experiments using the IKONOS and WorldView-2 datasets indicate that this method significantly improves the fidelity and accuracy of the fused images. [ABSTRACT FROM AUTHOR]

Remote sensing applications require high-resolution images to obtain precise information about the Earth’s surface. Multispectral images have high spatial resolution but low spectral resolution. Hyperspectral images have high spectral resolution but low spatial resolution. This study proposes a residual learning and attention-based parallel network based on residual network and channel attention. The network performs image fusion of a high spatial resolution multispectral image and a low spatial resolution hyperspectral image. The network training and fusion experiments are conducted on four public benchmark data sets to show the effectiveness of the proposed model. The fusion performance is compared with classical signal processing–based image fusion techniques. Four image metrics are used for the quantitative evaluation of the fused images. The proposed network improved fusion ability by reducing the root mean square error and relative dimensionless global error in synthesis and increased the peak signal-to-noise ratio when compared to other state-of-the-art models. [ABSTRACT FROM AUTHOR]

Multispectral image fusion plays a crucial role in smart city environment safety. In the domain of visible and infrared image fusion, object vanishment after fusion is a key problem which restricts the fusion performance. To address this problem, a novel Salient Regional Generative Adversarial Network GAN (SaReGAN) is presented for infrared and VIS image fusion. The SaReGAN consists of three parts. In the first part, the salient regions of infrared image are extracted by visual saliency map and the information of these regions is preserved. In the second part, the VIS image, infrared image and salient information are merged thoroughly in the generator to gain a pre-fused image. In the third part, the discriminator attempts to differentiate the pre-fused image and VIS image, in order to learn details from VIS image based on the adversarial mechanism. Experimental results verify that the SaReGAN outperforms other state-of-the-art methods in quantitative and qualitative evaluations. [ABSTRACT FROM AUTHOR]

The fusion of hyperspectral and multispectral images presents a challenge as it involves blending a low-resolution hyperspectral image (HSI) with a corresponding multispectral image (MSI) to produce a high-resolution hyperspectral image (HRI). A number of existing techniques have limitations; for instance, matrix decomposition-based approaches fail to retain adequate spatial and spectral image information during fusion, while tensor decomposition-based processes have high computational overhead. In this paper, we propose a novel method for fusing hyperspectral and multispectral images. Our method leverages the strong correlation among the spectral bands of hyperspectral images and employs the SVD technique to extract spectral feature subspaces. This approach results in a more compact and representative feature space for fusion. Secondly, the proposed method utilizes Elastic Net regularization in combination with ${L_1}$ L 1 and ${L_2}$ L 2 regularization for effective feature selection of highly covariate features. Weighted group sparse regularization is employed to enhance the fusion effect, enabling better representation of the image's structure and features. The algorithm is subsequently evaluated on multiple datasets to confirm its effectiveness. The results of the experiment suggest that the suggested algorithm greatly enhances the spatial resolution and visual clarity of HSI images while preserving the spectral characteristics when compared to conventional methods for fusion of hyperspectral images. Additionally, the constraint for regulation can competently reduce any noise or artefacts, thereby boosting image discrimination. To summarize, the utilization of a sparse tensor-based hyperspectral image fusion algorithm with subspace learning provides an efficacious approach for processing hyperspectral imagery. This method is capable of improving spatial resolution, extracting advantageous features from hyperspectral imagery, and ultimately supports the process of remote sensing image analysis and application. [ABSTRACT FROM AUTHOR]

Despite natural image super-resolution (SR) methods have achieved great success, super-resolution methods for hyperspectral image (HSI) with rich spectral features are still a very challenging task. Furthermore, due to the diversity of HSI captured by different cameras, their degradation conditions vary. Currently available HSI SR methods are mainly based on fixed degradation models, and their performance is severely affected when the actual degradation does not match the assumed degradation model. To address this issue, this paper proposes a single-band hyperspectral image super-resolution (SBHSR) method for HSI. This method assumes that the degradation Gaussian blur kernel parameters of the HSI are unknown, and adapts to various degradation conditions by conducting blind super-resolution on the image. It also combines another band with better spatial structure as an auxiliary band for feature fusion. To address the spectral differences between the super-resolution band and the auxiliary band, we use the Spatial Adaptation Module (SAM) to map the feature distribution, ensuring consistent spectral brightness between the auxiliary band and the super-resolution band. Due to the use of blind degradation super-resolution method, our method is more robust compared to previous HSI SR methods. Additionally, as it is specifically designed for single-band hyperspectral image super-resolution, it has the advantage of being faster and more efficient. Experimental validation on two hyperspectral datasets demonstrates the superiority of our method, as it improves the spatial resolution of the image while preserving the spectral features, and performs better than existing blind super-resolution methods. [ABSTRACT FROM AUTHOR]

Multispectral image (MSI) and hyperspectral image (HSI) fusion is a popular method for HSI super-resolution reconstruction (HSI-SR). MSI-HSI fusion problem is ill-posed and demands several image priors or regularization terms to solve accurately, which is a challenging issue. In this paper, we propose an MSI-HSI fusion model via subspace representation and nonlocal low-rank regularization (SRNLRR). SRNLRR model incorporates the global spectral correlations and spatial nonlocal similarities of HSI to improve the fusion results, where the priors complement each other. First, we use the mode-n tensor-matrix product to project latent high spatial resolution HSI (HR-HSI) into spectral subspace, which can capture spectral low-rankness and reduce computational complexity. Then, based on low-rank representation (LRR) and nonlocal processing strategy, we design a spatial nonlocal LRR regularization (spa-NLRR) and a spectral global LRR regularization (spe-GLRR). These two regularizations analyze the spatial nonlocal similarities and spectral global correlations from intermediate-level vision. Finally, we use the residual regularization program to obtain more image information and input it into the fusion model. We use alternate minimization (AM) methods to optimize the SRNLRR model and employ the alternating direction method (ADM) on spatial/spectral LRR learning. Comparison experiments between the SRNLRR method and six advanced methods on three HSI datasets indicate the superiority of our method. [ABSTRACT FROM AUTHOR]

Various remote sensing images seek more attention with their high temporal and spatial resolution in the applications of earth science. Conversely, it is a difficult task for a single satellite to enhance high-quality images owing to its cost and technical constraints. It aims to suggest a novel remote sensing image fusion model for overcoming the limitations of the existing fusion approaches, where effective fusion rules were generated. Initially, the low frequency of MS image is used to enhance the PAN image using the optimized wavelet transform. Then, the enhanced PAN and MS images are used by the Improved Deep Convolutional Neural Network with Atrous Convolutions (IDCNN-AC) for getting the high-quality fused images. Further, the parameters in the IDCNN-AC are optimized using Hybrid Harris Hawks Dingo Optimization (HHHDO) for enhancing the fusion performance. Finally, the simulation outcome shows the efficient performance of the proposed image fusion model using different quantitative measures. [ABSTRACT FROM AUTHOR]

Image fusion refers to the process of synergistic combination of useful sensory information from multiple images to synthesize a composite image with greater information content and increased practical value. It aims to maximize pertinent information specific to a sensor while minimizing uncertainty and redundancy in the fused output. In this paper, the authors have proposed a simple yet cohesive framework for 2D greyscale image fusion using morphological differential features. The features are extracted using morphological open–close filters applied at multiple scales using an isotropic structuring element which brings out categorical bright and dark features from the source images. At each scale, the bright (and dark) differential features are mutually compared using higher-valued AC coefficients obtained in the DCT domain within a block. The scale-specific fused features are recursively added to form an image containing high-frequency information from all conceivable scales. The fused image is achieved by superimposing the cumulative feature image onto a suitable base image. The base image is obtained by using a morphological weighted version of pseudomedian filter over the source images using the largest homothetic of the structuring element. The superiority of the framework is empirically verified in different domains of fusion, i.e. multi-focus, multi-sensor, multi-exposure, and multi-spectral image fusion. The proposed approach has surpassed the state-of-the-art unified fusion algorithms in terms of qualitative and quantitative evaluation with a perfect resource-time trade-off. Furthermore, the proposed method has been extended to greyscale–colour and colour–colour image pairs qualifying it for anatomical–functional image fusion. [ABSTRACT FROM AUTHOR]

The efficient monitoring of foliar nutrient such as the C:N ratio is important for sustainable rangeland productivity. Recent advances in Earth observation technology provide multispectral sensors with high spatial, spectral and temporal resolution suitable for modelling foliar nutrients. This study explored the fusion of Sentinel-2 and PlanetScope datasets using the pixel fusion technique and the random forest algorithm. The result in this study is vital in understanding the value of image fusion in enhancing rangeland nutrient estimation for optimal productivity. The study concludes that the fusion of optical sensors requires further investigation. [ABSTRACT FROM AUTHOR]

In this letter, based on the tensor representation modelling, we propose a multi-mode and multi-order gradient tensor-based non-convex model (M2GTNM) for Bayesian fusion of multispectral (MS) and panchromatic (Pan) images, which aims at producing the high-resolution MS (HRMS) images. Specifically, by modelling the MS image as the order-3 tensor, we mainly develop the multi-mode and multi-order gradient tensor sparse priors of MS image for fusion. For the spectral preservation of low-resolution MS (LRMS) image, the spectral fidelity constraint between HRMS and LRMS images is imposed. For the spatial-mode prior modelling, the multi-order spatial gradient tensor-based non-convex ${l_{1/2}}$ l 1 / 2 sparse prior between HRMS and Pan images is particularly imposed. Moreover, for the spectral-mode prior modelling, the spectral gradient tensor-based non-convex ${l_{1/2}}$ l 1 / 2 sparse prior between HRMS and upsampled LRMS images is further imposed. Then, we apply the alternating direction method of multipliers to optimize the proposed model. Finally, the reduced-scale and full-scale fusion experiments both validate the effectiveness of M2GTNM method. [ABSTRACT FROM AUTHOR]

Detecting pedestrians in varying illumination conditions poses a significant challenge, necessitating the development of innovative solutions. In response to this, we introduce Prior-AttentionNet, a pedestrian detection model featuring a Prior-Attention mechanism. This model leverages the stark contrast between thermal objects and their backgrounds in far-infrared (FIR) images by employing saliency attention derived from FIR images via UNet. However, extracting salient regions of diverse scales from FIR images poses a challenge for saliency attention. To address this, we integrate Simple Linear Iterative Clustering (SLIC) superpixel segmentation, embedding the segmentation feature map as prior knowledge into UNet's decoding stage for comprehensive end-to-end training and detection. This integration enhances the extraction of focused attention regions, with the synergy of segmentation prior and saliency attention forming the core of Prior-AttentionNet. Moreover, to enrich pedestrian details and contour visibility in low-light conditions, we implement multispectral image fusion. Experimental evaluations were conducted on the KAIST and OTCBVS datasets. Applying Prior-Attention mode to FIR-RGB images significantly improves the delineation and focus on multi-scale pedestrians. Prior-AttentionNet's general detector demonstrates the capability of detecting pedestrians with minimal computational resources. The ablation studies indicate that the FIR-RGB+ Prior-Attention mode markedly enhances detection robustness over other modes. When compared to conventional multispectral pedestrian detection models, Prior-AttentionNet consistently surpasses them by achieving higher mean average precision and lower miss rates in diverse scenarios, during both day and night. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) data, encompassing hundreds of spectral channels for the same area, offer a wealth of spectral information and are increasingly utilized across various fields. However, their limitations in spatial resolution and imaging width pose challenges for precise recognition and fine classification in large scenes. Conversely, multispectral (MS) data excel in providing spatial details for vast landscapes but lack spectral precision. In this article, we proposed an adaptive learning-based mapping model, including an image fusion module, spectral super-resolution network, and adaptive learning network. Spectral super-resolution networks learn the mapping between multispectral and hyperspectral images based on the attention mechanism. The image fusion module leverages spatial and spectral consistency in training data, providing pseudo labels for spectral super-resolution training. And the adaptive learning network incorporates spectral response priors via unsupervised learning, adjusting the output of the super-resolution network to preserve spectral information in reconstructed data. Through the experiment, the model eliminates the need for the manual setting of image prior information and complex parameter selection, and can adjust the network structure and parameters dynamically, eventually enhancing the reconstructed image quality, and enabling the fine classification of large-scale scenes with high spatial resolution. Compared with the recent dictionary learning and deep learning spectral super-resolution methods, our approach exhibits superior performance in terms of both image similarity and classification accuracy. [ABSTRACT FROM AUTHOR]

Panchromatic and multi-spectral image fusion, called panchromatic sharpening, is the process of combining the spatial and spectral information of the source image into the fused image to give the image a higher spatial and spectral resolution. In order to improve the spatial resolution and spectral information quality of the image, an adaptive multi-spectral image fusion method based on an improved arithmetic optimization algorithm is proposed. This paper proposed improved arithmetic optimization algorithm, which uses dynamic stochastic search technique and oppositional learning operator, to perform local search and behavioral complementation of population individuals, and to improve the ability of population individuals to jump out of the local optimum. The method combines adaptive methods to calculate the weights of linear combinations of panchromatic and multi-spectral gradients to improve the quality of fused images. This study not only improves the quality and effect of image fusion, but also focuses on optimizing the operation efficiency of the algorithm to have real-time and high efficiency. Experimental results show that the proposed method exhibits strong performance on different datasets, improves the spatial resolution and spectral information quality of the fused images, and has good adaptability and robustness. The source code is available at: . [ABSTRACT FROM AUTHOR]

Although the fusion of multispectral (MS) and hyperspectral (HS) images in remote sensing has become relatively mature, and different types of fusion methods have their own characteristics in terms of fusion effect, data dependency, and computational efficiency, few studies have focused on the impact of radiance extreme areas, which widely exist in real remotely sensed scenes. To this end, this paper proposed a novel method called radiance extreme area compensation fusion (RECF). Based on the architecture of spectral unmixing fusion, our method uses the reconstruction of error map to construct local smoothing constraints during unmixing and utilizes the nearest-neighbor multispectral data to achieve optimal replacement compensation, thereby eliminating the impact of overexposed and underexposed areas in hyperspectral data on the fusion effect. We compared the RECF method with 11 previous published methods on three sets of airborne hyperspectral datasets and HJ2 satellite hyperspectral data and quantitatively evaluated them using 5 metrics, including PSNR and SAM. On the test dataset with extreme radiance interference, the proposed RECF method achieved well in the overall evaluation results; for instance, the PSNR metric reached 47.6076 and SAM reached 0.5964 on the Xiong'an dataset. In addition, the result shows that our method also achieved better visual effects on both simulation and real datasets. [ABSTRACT FROM AUTHOR]

Existing deep learning techniques for image fusion either learn image mapping (LIM) directly, which renders them ineffective at preserving details due to the equal consideration to each pixel, or learn detail mapping (LDM), which only attains a limited level of performance because only details are used for reasoning. The recent lossless invertible network (INN) has demonstrated its detail-preserving ability. However, the direct applicability of INN to the image fusion task is limited by the volume-preserving constraint. Additionally, there is the lack of a consistent detail-preserving image fusion framework to produce satisfactory outcomes. To this aim, we propose a general paradigm for image fusion based on a novel conditional INN (named DCINN). The DCINN paradigm has three core components: a decomposing module that converts image mapping to detail mapping; an auxiliary network (ANet) that extracts auxiliary features directly from source images; and a conditional INN (CINN) that learns the detail mapping based on auxiliary features. The novel design benefits from the advantages of INN, LIM, and LDM approaches while avoiding their disadvantages. Particularly, using INN to LDM can easily meet the volume-preserving constraint while still preserving details. Moreover, since auxiliary features serve as conditional features, the ANet allows for the use of more than just details for reasoning without compromising detail mapping. Extensive experiments on three benchmark fusion problems, i.e., pansharpening, hyperspectral and multispectral image fusion, and infrared and visible image fusion, demonstrate the superiority of our approach compared with recent state-of-the-art methods. The code is available at https://github.com/wwhappylife/DCINN [ABSTRACT FROM AUTHOR]

In the field of digital image processing, high dynamic range (HDR) image technology is widely used in digital photography, medical imaging, virtual reality and other fields. However, inconsistent motion areas may occur when shooting dynamic scenes. Therefore, ghost detection needs to be performed on the image sequence. When an image sequence with moving areas is processed using the static multi‐exposure method and fused to generate an HDR image, since the moving area occupies a certain proportion, the moving area in the final image will have overlapping images, unclear details, and reduced image quality. In response to the above phenomenon, this paper proposes a ghost removal fusion algorithm based on enhanced reference images. While ensuring more texture details in the image, it also reduces the fusion time of the image. The fused image has natural colors and can effectively solve the ghost phenomenon that occurs during the processing of HDR images. [ABSTRACT FROM AUTHOR]

Scanning micrο X-ray fluorescence (μ-XRF) and multispectral imaging (MSI) were applied to study philately stamps, selected for their small size and intricate structures. The μ-XRF measurements were accomplished using the M6 Jetstream Bruker scanner under optimized conditions for spatial resolution, while the MSI measurements were performed employing the XpeCAM-X02 camera. The datasets were acquired asynchronously. Elemental distribution maps can be extracted from the μ-XRF dataset, while chemical distribution maps can be obtained from the analysis of the multispectral dataset. The objective of the present work is the fusion of the datasets from the two spectral imaging modalities. An algorithmic co-registration of the two datasets is applied as a first step, aiming to align the multispectral and μ-XRF images and to adapt to the pixel sizes, as small as a few tens of micrometers. The dataset fusion is accomplished by applying k-means clustering of the multispectral dataset, attributing a representative spectrum to each pixel, and defining the multispectral clusters. Subsequently, the μ-XRF dataset within a specific multispectral cluster is analyzed by evaluating the mean XRF spectrum and performing k-means sub-clustering of the μ-XRF dataset, allowing the differentiation of areas with variable elemental composition within the multispectral cluster. The data fusion approach proves its validity and strength in the context of philately stamps. We demonstrate that the fusion of two spectral imaging modalities enhances their analytical capabilities significantly. The spectral analysis of pixels within clusters can provide more information than analyzing the same pixels as part of the entire dataset. [ABSTRACT FROM AUTHOR]

The fusion of infrared and visible images is a well-researched task in computer vision. These fusion methods create fused images replacing the manual observation of single sensor image, often deployed on edge devices for real-time processing. However, there is an issue of information imbalance between infrared and visible images. Existing methods often fail to emphasize temperature and edge texture information, potentially leading to misinterpretations. Moreover, these methods are computationally complex, and challenging for edge device adaptation. This paper proposes a method that calculates the distribution proportion of infrared pixel values, allocating fusion weights to adaptively highlight key information. It introduces a weight allocation mechanism and MobileBlock with a multispectral information complementary module, innovations which strengthened the model's fusion capabilities, made it more lightweight, and ensured information compensation. Training involves a temperature-color-perception loss function, enabling adaptive weight allocation based on image pair information. Experimental results show superiority over mainstream fusion methods, particularly in the electric power equipment scene and publicly available datasets. [ABSTRACT FROM AUTHOR]

The fusion of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI) is a crucial technology for producing high-resolution hyperspectral images. Most existing image fusion algorithms based on deep learning do not fully utilize the ability of neural network to extract and process multi-scale features, which leads to the problem of difficulty in fully learning features and ambiguity of features. In order to overcome these issues, a residual selective kernel attention-based U-net named RSKAU-net is designed for LR-HSI and HR-MSI fusion. RSKAU-net is constructed by a residual selective kernel module with an attention mechanism and a channel attention block. The residual selective kernel attention-based (RSKA) module is designed to process images of different resolutions, which adaptively extracts multi-scale features and efficiently emphasizes significant features through the attention mechanism. The channel attention (CA) module retains important spectral information by assigning different weights to each channel of LR-HSI. The proposed network can enhance the spatial information of LR-HSI while preserving its spectral information. Meanwhile, it effectively fuses the features from the source image to obtain the HR-HSI with rich details. The experimental results demonstrate that the proposed network has advantages in terms of both visual effect and objective quantitative indices when compared to existing HSI-MSI fusion approaches. [ABSTRACT FROM AUTHOR]

Beijing Satellite 3 is a high-performance optical remote sensing satellite with a spatial resolution of 0.3–0.5 m. It can provide timely and independent ultra-high-resolution spatial big data and comprehensive spatial information application services. At present, there is no relevant research on the fusion method of BJ-3A satellite images. In many applications, high-resolution panchromatic images alone are insufficient. Therefore, it is necessary to fuse them with multispectral images that contain spectral color information. Currently, there is a lack of research on the fusion method of BJ-3A satellite images. This article explores six traditional pixel-level fusion methods (HPF, HCS, wavelet, modified-IHS, PC, and Brovey) for fusing the panchromatic image and multispectral image of the BJ-3A satellite. The fusion results were analyzed qualitatively from two aspects: spatial detail enhancement capability and spectral fidelity. Five indicators, namely mean, standard deviation, entropy, correlation coefficient, and average gradient, were used for quantitative analysis. Finally, the fusion results were comprehensively evaluated from three aspects: spectral curves of ground objects, absolute error figure, and object-oriented classification effects. The findings of the research suggest that the fusion method known as HPF is the optimum and appropriate technique for fusing panchromatic and multispectral images obtained from BJ-3A. These results can be utilized as a guide for the implementation of BJ-3A panchromatic and multispectral data fusion in real-world scenarios. [ABSTRACT FROM AUTHOR]

The definition and calculation of a spectral index suitable for characterizing vegetated landscapes depend on the number and widths of the bands of the imaging instrument. Here, we point out the advantages of performing the fusion of hyperspectral (HS) satellite data with the multispectral (MS) bands of Sentinel-2 to calculate such vegetation indexes as the normalized area over reflectance curve (NAOC) and the red-edge inflection point (REIP), which benefit from the availability of quasi-continuous pixel spectra. Unfortunately, MS data may be acquired from satellite platforms with very high spatial resolution; HS data may not. Despite their excellent spectral resolution, satellite imaging spectrometers currently resolve areas not greater than 30 × 30 m2, where different thematic classes of landscape may be mixed together to form a unique pixel spectrum. A way to resolve mixed pixels is to perform the fusion of the HS dataset with the same dataset produced by an MS scanner that images the same scene with a finer spatial resolution. The HS dataset is sharpened from 30 m to 10 m by means of the Sentinel-2 bands that have all been previously brought to 10 m. To do so, the hyper-sharpening protocol, that is, m : n fusion, is exploited in two nested steps: the first one to bring the 20 m bands of Sentinel-2 all to 10 m, the second one to sharpen all the 30 m HS bands to 10 m by using the Sentinel-2 bands previously hyper-sharpened to 10 m. Results are presented on an agricultural test site in The Netherlands imaged by Sentinel-2 and by the satellite imaging spectrometer recently launched as a part of the environmental mapping and analysis program (EnMAP). Firstly, the excellent match of statistical consistency of the fused HS data to the original MS and HS data is evaluated by means of analysis tools, existing and developed ad hoc for this specific case. Then, the spatial and radiometric accuracy of REIP and NAOC calculated from fused HS data are analyzed on the classes of pure and mixed pixels. On pure pixels, the values of REIP and NAOC calculated from fused data are consistent with those calculated from the original HS data. Conversely, mixed pixels are spectrally unmixed by the fusion process to resolve the 10 m scale of the MS data. How the proposed method can be used to check the temporal evolution of vegetation indexes when a unique HS image and many MS images are available is the object of a final discussion. [ABSTRACT FROM AUTHOR]

Image preprocessing and fusion are commonly used for enhancing remote-sensing images, but the resulting images often lack useful spatial features. As the majority of research on image fusion has concentrated on the satellite domain, the image-fusion task for Unmanned Aerial Vehicle (UAV) images has received minimal attention. This study investigated an image-improvement strategy by integrating image preprocessing and fusion tasks for UAV images. The goal is to improve spatial details and avoid color distortion in fused images. Techniques such as image denoising, sharpening, and Contrast Limited Adaptive Histogram Equalization (CLAHE) were used in the preprocessing step. The unsharp mask algorithm was used for image sharpening. Wiener and total variation denoising methods were used for image denoising. The image-fusion process was conducted in two steps: (1) fusing the spectral bands into one multispectral image and (2) pansharpening the panchromatic and multispectral images using the PanColorGAN model. The effectiveness of the proposed approach was evaluated using quantitative and qualitative assessment techniques, including no-reference image quality assessment (NR-IQA) metrics. In this experiment, the unsharp mask algorithm noticeably improved the spatial details of the pansharpened images. No preprocessing algorithm dramatically improved the color quality of the enhanced images. The proposed fusion approach improved the images without importing unnecessary blurring and color distortion issues. [ABSTRACT FROM AUTHOR]

Several transformations are used in image fusion to extract the different image spatial details. In this paper, an improved version of the BEMD (Bi-dimensional Empirical Mode Decomposition) is proposed and is used to propose an image fusion method aiming to have the high spatial resolution of the panchromatic image PAN and the spectral resolution of the multispectral image MS in the same fused image. The proposed improved BEMD permits to avoid using injection models that are used in image fusion for preserving the spectral signatures in the fused image. Mainly, we propose a new 2D extrema points extraction for BEMD. One of the most important characteristics of the proposed BEMD components is that they are more faithful to the EMD components; their local behaviour being of pure oscillating 2D mono-components with zero mean values. Comparisons, among predecessor methods with qualitative evaluation and various spectral and spatial quantitative measures, show the effectiveness and efficiency of the proposed image fusion method. In addition, this fusion method is computationally fast and can be used to quickly merge a massive volume of data. [ABSTRACT FROM AUTHOR]

Preserving spectral and spatial information in the satellite image fusion problem is one of the essential challenges. This paper presents a GAN-based method for fusing panchromatic and multispectral satellite images. The proposed method utilises the idea of Cycle-GAN with two generators, for spectral and spatial information preservation, based on residual in residual dense block super-resolution architecture. First, generator-1 translates panchromatic and multispectral satellite images to a high-resolution fused image and then preserves details by utilising generator-2. The goal is to reach the spatial and spectral details of panchromatic and multispectral satellite images, respectively. Two discriminators are employed, one for spectral and the other for spatial transformations, and weighted L1 loss is used for both as cycle losses. By leveraging the unique capabilities of the two generators, the proposed method achieves high-quality image fusion results with improved spectral and spatial resolutions that are evaluated over two different datasets. The experimental results demonstrate the effectiveness of the proposed method, with enhancements ranging from approximately 2% to 30% and 0.5% to 35% in WorldView2 and GeoEye-1 images, respectively, in five metrics including PSNR. Moreover, we show significant enhancements of approximately 7% to 50% and 22% to 29% in the same datasets in metrics like SAM and ERGAS. [ABSTRACT FROM AUTHOR]

Pansharpening is an established remote sensing image fusion technique that yields a high-resolution multispectral (HRMS) image. Despite the fact that the advanced technologies like sparse coding and deep learning have achieved a remarkable improvement in solving the pansharpening problem, a unified model is required to further enhance the fusion quality. The variational optimisation (VO) mechanism has gained interest of most of the researchers in recent years. In this article, pansharpening is designated as a constrained optimisation problem with a data generative term and two regularisers to ameliorate spatial details and spectral information. The gradient information is exploited to impart the spatial details from the panchromatic image to the fused image. The correlation among multispectral image bands inspired to promote the spectral quality of the HRMS image as well as to reduce the distortion. Consequently, the optimisation problem is efficiently solved for the required HRMS image using the operator splitting approach. The extensive experiments performed in accordance with the well-known protocols rationalise that the proposed model outperforms most of the state-of-the-art methods in terms of objective metrics and visual outcomes. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) and multispectral image (MSI) fusion aims at producing a super-resolution image (SRI). In this paper, we establish a nonconvex optimization model for image fusion problem through low rank tensor triple decomposition. Using the limited memory BFGS (L-BFGS) approach, we develop a first-order optimization algorithm for obtaining the desired super-resolution image (TTDSR). Furthermore, two detailed methods are provided for calculating the gradient of the objective function. With the aid of the Kurdyka-Łojasiewicz property, the iterative sequence is proved to converge to a stationary point. Finally, experimental results on different datasets show the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

An adequate fusion of the most significant salient information from multiple input channels is essential for many aerial imaging tasks. While multispectral recordings reveal features in various spectral ranges, synthetic aperture sensing makes occluded features visible. We present a first and hybrid (model- and learning-based) architecture for fusing the most significant features from conventional aerial images with the ones from integral aerial images that are the result of synthetic aperture sensing for removing occlusion. It combines the environment's spatial references with features of unoccluded targets that would normally be hidden by dense vegetation. Our method outperforms state-of-the-art two-channel and multi-channel fusion approaches visually and quantitatively in common metrics, such as mutual information, visual information fidelity, and peak signal-to-noise ratio. The proposed model does not require manually tuned parameters, can be extended to an arbitrary number and arbitrary combinations of spectral channels, and is reconfigurable for addressing different use cases. We demonstrate examples for search and rescue, wildfire detection, and wildlife observation. [ABSTRACT FROM AUTHOR]

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data for classification has received widespread attention and has led to significant progress in research and remote sensing applications. However, existing common CNN architectures suffer from the significant drawback of not being able to model remote sensing images globally, while transformer architectures are not able to capture local features effectively. To address these bottlenecks, this paper proposes a classification framework for multisource remote sensing image fusion. First, a spatial and spectral feature projection network is constructed based on parallel feature extraction by combining HSI and LiDAR data, which is conducive to extracting joint spatial, spectral, and elevation features from different source data. Furthermore, in order to construct local–global nonlinear feature mapping more flexibly, a network architecture coupling together multiscale convolution and a multiscale vision transformer is proposed. Moreover, a plug-and-play nonlocal feature token aggregation module is designed to adaptively adjust the domain offsets between different features, while a class token is employed to reduce the complexity of high-dimensional feature fusion. On three open-source remote sensing datasets, the performance of the proposed multisource fusion classification framework improves about 1% to 3% over other state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Images obtained in low-light scenes are often accompanied by problems such as low visibility, blurred details, and color distortion, enhancing them can effectively improve the visual effect and provide favorable conditions for advanced visual tasks. In this study, we propose a Multi-Technology Fusion of Low-light Image Enhancement Network (MTIE-Net) that modularizes the enhancement task. MTIE-Net consists of a residual dense decomposition network (RDD-Net) based on Retinex theory, an encoder-decoder denoising network (EDD-Net), and a parallel mixed attention-based self-calibrated illumination enhancement network (PCE-Net). The low-light image is first decomposed by RDD-Net into a lighting map and reflectance map; EDD-Net is used to process noise in the reflectance map; Finally, the lighting map is fused with the denoised reflectance map as an input to PCE-Net, using the Fourier transform for illumination enhancement and detail recovery in the frequency domain. Numerous experimental results show that MTIE-Net outperforms the comparison methods in terms of image visual quality enhancement improvement, denoising, and detail recovery. The application in nighttime face detection also fully demonstrates its promise as a pre-processing means in practical applications. [ABSTRACT FROM AUTHOR]

In large-scale remote sensing scenarios characterized by intricate terrain, the straightforward road imaging features in synthetic aperture radar (SAR) images make them susceptible to interference from other elements such as ridges, compromising the robustness of conventional SAR image road extraction methods. This paper introduces a method that integrates Gaofen-3 (GF-3) with a resolution of 3.0 m, Digital Elevation Models (DEMs), and Gaofen-2 (GF-2) remote sensing image data with a resolution of 4.0 m, aiming to improve the performance of road extraction in complex terrain. Leveraging DEMs, this study addresses the limitations in feature-based SAR algorithms, extending their application to complex remote sensing scenarios. Decision-level fusion, integrating SAR and multispectral images, further refines road extraction precision. To overcome issues related to terrain interference, including fragmented road segments, an adaptive rotated median filter and graph-theory-based optimization are introduced. These advancements collectively enhance road recognition accuracy and topological precision. The experimental results validate the effectiveness of the multi-source remote sensing image fusion and optimization methods. Compared to road extraction from multispectral images, the F1-score of the proposed method on the test images increased by 2.18%, 4.22%, and 1.4%, respectively. [ABSTRACT FROM AUTHOR]

The appearance of tea leaves is the most intuitive factor in assessing their quality and is an important indicator of the price. The study utilizes machine vision and hyperspectral image technology to develop a rapid method for assessing the appearance quality of seven grades of Dianhong Congou black tea samples based on shape, color, texture, and feature fusion. Using the shape metrics of length, width, perimeter, narrowness, area, and rectangularity, 88 texture values, and nine color values as features, the effectiveness of intelligent recognition algorithms such as support vector machine, random forest, and least squares support vector machine (LSSVM) in classifying single features (shape, color, and texture) and fused data models of the categories of Dianhong tea samples was explored. The results showed that the accuracy of the feature fusion-based appearance quality evaluation models was all higher than the modeling results of the corresponding single features, and the accuracy of the LSSVM model based on the fused data was 100%. This study showed that feature data fusion reflects the external information of tea leaves more comprehensively and that intelligent classification algorithms combined with image analysis techniques were an effective strategy for the identification of regional black tea. [ABSTRACT FROM AUTHOR]

Multimodal medical image fusion (MMIF) integrates the advantages of multiple source images to assist clinical diagnosis. Existing image fusion methods need help to distinguish the importance between features and often define features to be retained subjectively, which leads to global structure loss and limits the performance of fusion. To overcome these restrictions, we propose a novel self-supervised tensor low-rank decomposition fusion network that can effectively extract global information from high-rank to low-rank conversion processes. Specifically, the compensation of textural features is performed by employing a self-supervised auxiliary task, and the whole network is dynamically fine-tuned according to a hybrid loss. In our model, an enhanced weights (EW) estimation method based on the global luminance contrast is developed, and a structure tensor loss with constraints is introduced to improve the robustness of the fusion results. Moreover, extensive experiments on six types of multimodal medical images show that visual and qualitative results are superior to competitors, validating the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

Pansharpening aims at fusing a panchromatic (PAN) image and a low‐resolution multispectral (LRMS) image into a high‐resolution multispectral (HRMS) image. In recent years, GAN‐based pansharpening methods have achieved excellent results, but they suffer from inadequate feature preservation and unstable training. To address these issues, a novel GAN‐based model named TriLossGAN is proposed. This method constructs three loss components with the help of the generator and the dual‐discriminator, which are calculated in both the original spatial domain and the transform domain to better preserve high‐frequency and low‐frequency information in the fused image. Additionally, a new training strategy is designed to stabilize the training process. In extensive experiments, the proposed method achieved satisfactory results on three datasets with QNR values of 0.9584 on GaoFen‐2, 0.9601 on QuickBird, and 0.9138 on WorldView‐3. Qualitative and quantitative comparisons demonstrate that TriLossGAN outperforms other state‐of‐the‐art methods. [ABSTRACT FROM AUTHOR]

Multispectral image (MSI) and hyperspectral image (HSI) fusion (MHIF) aims to address the challenge of acquiring high-resolution (HR) HSI images. This field combines a low-resolution (LR) HSI with an HR-MSI to reconstruct HR-HSIs. Existing methods directly utilize transformers to perform feature extraction and fusion. Despite the demonstrated success, there exist two limitations: (1) Employing the entire transformer model for feature extraction and fusion fails to fully harness the potential of the transformer in integrating the spectral information of the HSI and spatial information of the MSI. (2) HSIs have a strong spectral correlation and exhibit sparsity in the spatial domain. Existing transformer-based models do not optimize this physical property, which makes their methods prone to spectral distortion. To accomplish these issues, this paper introduces a novel framework for MHIF called a Sparse Mix-Attention Transformer (SMAformer). Specifically, to fully harness the advantages of the transformer architecture, we propose a Spectral Mix-Attention Block (SMAB), which concatenates the keys and values extracted from LR-HSIs and HR-MSIs to create a new multihead attention module. This design facilitates the extraction of detailed long-range information across spatial and spectral dimensions. Additionally, to address the spatial sparsity inherent in HSIs, we incorporated a sparse mechanism within the core of the SMAB called the Sparse Spectral Mix-Attention Block (SSMAB). In the SSMAB, we compute attention maps from queries and keys and select the K highly correlated values as the sparse-attention map. This approach enables us to achieve a sparse representation of spatial information while eliminating spatially disruptive noise. Extensive experiments conducted on three synthetic benchmark datasets, namely CAVE, Harvard, and Pavia Center, demonstrate that the SMAformer method outperforms state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Achieving a balance between spectral resolution and spatial resolution in multi-spectral remote sensing images is challenging due to physical constraints. Consequently, pan-sharpening technology was developed to address this challenge. While significant progress was recently achieved in deep-learning-based pan-sharpening techniques, most existing deep learning approaches face two primary limitations: (1) convolutional neural networks (CNNs) struggle with long-range dependency issues, and (2) significant detail loss during deep network training. Moreover, despite these methods' pan-sharpening capabilities, their generalization to full-sized raw images remains problematic due to scaling disparities, rendering them less practical. To tackle these issues, we introduce in this study a multi-spectral remote sensing image fusion network, termed TAMINet, which leverages a two-stream coordinate attention mechanism and multi-detail injection. Initially, a two-stream feature extractor augmented with the coordinate attention (CA) block is employed to derive modal-specific features from low-resolution multi-spectral (LRMS) images and panchromatic (PAN) images. This is followed by feature-domain fusion and pan-sharpening image reconstruction. Crucially, a multi-detail injection approach is incorporated during fusion and reconstruction, ensuring the reintroduction of details lost earlier in the process, which minimizes high-frequency detail loss. Finally, a novel hybrid loss function is proposed that incorporates spatial loss, spectral loss, and an additional loss component to enhance performance. The proposed methodology's effectiveness was validated through experiments on WorldView-2 satellite images, IKONOS, and QuickBird, benchmarked against current state-of-the-art techniques. Experimental findings reveal that TAMINet significantly elevates the pan-sharpening performance for large-scale images, underscoring its potential to enhance multi-spectral remote sensing image quality. [ABSTRACT FROM AUTHOR]

This paper proposes an efficient and high-fidelity image fusion method based on adaptive smoothing filtering for panchromatic (PAN) and multispectral (MS) image fusion. The scale ratio reflects the ratio of spatial resolution between the panchromatic image and the multispectral image. When facing a multiscale fusion task, traditional methods are unable to simultaneously handle the problems of spectral resolution loss resulting from high scale ratios and the issue of reduced spatial resolution due to low scale ratios. To adapt to the fusion of panchromatic and multispectral satellite images of different scales, this paper improves the problem of the insufficient filtering of high-frequency information of remote sensing images of different scales by the classic filter-based intensity modulation (SFIM) model. It uses Gaussian convolution kernels instead of traditional mean convolution kernels and builds a Gaussian pyramid to adaptively construct convolution kernels of different scales to filter out high-frequency information of high-resolution images. It can adaptively process panchromatic multispectral images of different scales, iteratively filter the spatial information in panchromatic images, and ensure that the scale transformation is consistent with the definition of multispectral images. Using 15 common fusion methods, this paper compares the experimental results of ZY-3 with scale ratio 2.7 and SV-1 with scale ratio 4 data. The results show that the method proposed in this paper retains good spatial information for image fusion at different scales and has good spectral preservation. [ABSTRACT FROM AUTHOR]

Multi-exposure image fusion as a technical means to bridge the dynamic range gap between real scenes and image acquisition devices, which makes the fused images better quality and more realistic and vivid simulation of real scenes, has been widely concerned by scholars from various countries. In order to improve the adaptive fusion effect of multi-exposure images, this paper proposes a fusion algorithm based on multilayer perceptron (MLP) based on the perceptron model and verifies the feasibility of the algorithm by the peak signal-to-noise ratio (PSNR), correlation coefficient (PCC), structural similarity (SSMI) and HDR-VDR-2, an evaluation index of HDR image quality. Comparison with other algorithms revealed that the average PSNR of the MLP algorithm improved by 4.43% over the Ma algorithm, 7.88% over the Vanmail algorithm, 10.30% over the FMMR algorithm, 11.19% over the PMF algorithm, and 11.19% over the PMF algorithm. For PCC, the MLP algorithm improves by 20.14%, 17.46%, 2.31%, 11.24%, and 15.36% over the other algorithms in that order. For SSMI, the MLP algorithm improved by 16.99%, 8.96%, 17.17%, 14.41%, and 4.85% over the other algorithms, in that order. For HDR-VDR-2, the MLP algorithm improved by 3.02%, 2.79%, 6.84%, 4.90%, and 6.55% over the other algorithms, in that order. The results show that the MLP algorithm can avoid image artifacts while retaining more details. The MLP-based adaptive fusion method is a step further in the theoretical study of multi-exposure image fusion, which is of great significance for subsequent research and practical application by related technology vendors. [ABSTRACT FROM AUTHOR]