Search

Spatio-spectral fusion of panchromatic (PAN) and hyperspectral (HS) images is of great importance in improving spatial resolution of images acquired by many commercial HS sensors. DenseNets have recently achieved great success for image super-resolution because they facilitate gradient flow by concatenating all the feature outputs in a feedforward manner. In this article, we propose a residual hyper-dense network (RHDN) that extends the DenseNet to solve the spatio-spectral fusion problem. The overall structure of the proposed RHDN method is a two-branch network, which allows the network to capture the features of HS images within and outside the visible range separately. At each branch of the network, a two-stream strategy of feature extraction is designed to process PAN and HS images individually. A convolutional neural network (CNN) with cascade residual hyper-dense blocks (RHDBs), which allows direct connections between the pairs of layers within the same stream and those across different streams, is proposed to learn more complex combinations between the HS and PAN images. The residual learning is adopted to make the network efficient. Extensive benchmark evaluations well demonstrate that the proposed RHDN fusion method yields significant improvements over many widely accepted state-of-the-art approaches.

The joint classification methods of hyperspectral image (HSI) and light detection and ranging (LiDAR) data based on deep learning have demonstrated exceptional classification performance with sufficient labeled samples. However, it is expensive and time-consuming to acquire labeled data. To address this limitation, we propose a shared-private decoupling-based multilevel feature alignment semisupervised (SASS) learning method for HSI and LiDAR classification, which introduces the idea of domain adaptation (DA) to capture the shared features of labeled and unlabeled data for classification and circularly selects reliable pseudolabels based on these features to retrain the model. Specifically, we treat labeled data as the source domain (SD) and unlabeled data as the target domain (TD) and propose a shared-private feature decoupling (SPFD) module to acquire shared representations of SD and TD by separating domain private features. The multilevel shared feature alignment (MSFA) strategy is designed to synthetically consider both spatial details and semantic information by minimizing the maximum mean discrepancy (MMD) between these shared features. In addition, we design a graph transformer-based class-balanced pseudolabel generation (GBPG) strategy for iterative model training with reliable pseudolabels, which exploits the graph transformer network-based sample acquisition (GTSA) strategy to select valuable samples and generate corresponding pseudolabels using the adaptive class-specific threshold-based sample annotation (ATSA) strategy. Experimental results on three public datasets validate the effectiveness of the proposed method. The code is available at https://github.com/Jiahuiqu/SASS.

Hyperspectral target detection is a binary classification problem of detecting targets by utilizing the spectral characteristics of hyperspectral images (HSIs). However, in real-world applications, there is more than one class of interest in the same scene. The accuracy of detection for the previously learned target classes may be decreased when the model is retrained for detecting new target classes in scenes, which is called catastrophic forgetting. Consequently, how to ensure that the model has high detection performance for previously learned targets while learning new ones has become a key challenge for hyperspectral multitarget detection tasks. In this article, we propose an incremental detection of hyperspectral targets (IDHTs) method based on continual learning. IDHT decomposes the multitarget detection task into a series of independent subtasks and learns them sequentially. Within each subtask, our proposed incremental spectral detector (ISD) enables the training and learning of new class targets. Simultaneously, we introduce the previous label replay strategy (PLRS), which synthesizes fused labels for the current task training by combining detection outcomes from the previous model with pseudo-labels of the current target. PLRS effectively bridges the knowledge gap across various subtasks in hyperspectral multitarget detection tasks. The proposed IDHT can flexibly and dynamically adapt to new categories and overcome the limitations of fixed-category feature learning. In addition, two hyperspectral datasets are disclosed to evaluate the proposed method. Our method demonstrates significant effectiveness and superiority on both public datasets and two self-collected datasets. Our code and dataset are available at https://github.com/Jiahuiqu/IDHT.

Hyperspectral image (HSI) super-resolution (SR) employing the denoising diffusion probabilistic model (DDPM) holds significant promise with its remarkable performance. However, existing relevant works exhibit two limitations: 1) directly applying DDPM to fusion-based HSI-SR ignores the physical mechanism of HSI-SR and unique characteristics of HSI, resulting in less interpretability and 2) scale-invariant DDPM suffers from a time-consuming inference. To tackle these issues, we propose an interpretable scale-propelled diffusion (ISPDiff) model for HSI-SR, which combines the underlying principles of HSI-SR with DDPM for progressively unrolling reconstruction by learning its distribution at various scales, enhancing the transparency significantly and reducing the inference time prominently. Concretely, we destroy and downsample HSI into Gaussian noise in the forward process of ISPDiff. Then we design a unified scale-flexible model in the backward process to iteratively refine HSI in a coarse-to-fine manner through scale-matched reconstruction and cross-scale upsampling, which can be unfolded with optimization algorithms. These solved equations are one-to-one corresponding unrolled into two deep neural networks, called progressive perceptual model-driven scale-matched restoration network (P2MSRN) and cross-scale model-driven upsampling network (CMUN). Through end-to-end training, the proposed ISPDiff implements HSI-SR with a scale-propelled unrolling diffusion characterized by enhanced interpretability, stronger task orientation, and reduced time consumption. Systematic experiments have been conducted on three public datasets, demonstrating that ISPDiff outperforms state-of-the-art methods. The code is available at https://github.com/Jiahuiqu/ISPDiff.

Convolutional neural networks (CNNs) with good feature learning capabilities are widely used in hyperspectral image change detection (HSI-CD) tasks. However, most existing CNN-based HSI-CD methods face two inherent challenges: 1) the lack of available labeled datasets and 2) the limited receptive field that cannot capture the long-distance dependence between the spectral sequences of HSIs. In this article, we propose a parallel compared-and-stacked pyramid transformer network (PCPTNet) for unsupervised HSI-CD, which can model the context information of spectral sequences in the input multi-temporal HSI patch without real labeled data. Specifically, a superpixel-level joint decision-based training samples selection strategy is presented that fully considers the correlation between pixels to improve the reliability of training samples. Then, taking advantage of transformer in context information modeling, PCPTNet is proposed to capture sufficient difference features and stacked features with different scales for CD, which can effectively reduce missed and false detection. The multiscale features containing sufficient low-level detail information and high-level semantic features are fused hierarchically to classify changed and unchanged pixels. Extensive experiments on three real HSI datasets demonstrate that the PCPTNet outperforms other state-of-the-art HSI-CD methods in both visual and quantitative results.

Existing fusion-based hyperspectral image super-resolution (fusion-based HSI-SR) methods usually reconstruct high-resolution hyperspectral image (HR-HSI) by integrating the complementary information of low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (HR-MSI). However, most of such methods rely on accurately registered images or consider registration and fusion as a two-stage task, which means that fusion must tolerate the accumulation of errors due to misregistration. In this article, we propose a progressive multi-iteration registration-fusion co-optimization network (PMI-RFCoNet) for unregistered HSI-SR, which progressively refines the registration and fusion result over multiple levels to reconstruct registered HR-HSI. To achieve registration-fusion co-optimization, the registration-fusion co-optimization block (Co-RFB) is designed to iterate continuously over multiple levels. We embed the interactive registration module (IRM) and the spectral recalibration and fusion module (SRFU) in Co-RFB, which can facilitate the network utilizing spatial and spectral features at different levels to generate more accurate HR-HSI. Specifically, IRM generates deformation field based on spatial correlations captured at long distances to repair non-rigid pixel offsets, and SRFU further performs adaptive high-fidelity spectral correction and spatial information fusion on the registration results. We conduct experimental verification on four widely used datasets, and the results show that PMI-RFCoNet can flexibly cope with different types and degrees of non-rigid deformation and achieve superior performance. Code is available at https://github.com/Jiahuiqu/PMI-RFCoNet.

The joint classification of hyperspectral images (HSIs) and LiDAR data plays a crucial role in Earth observation missions. Most advanced methods are based on discrete label supervision. However, since discrete labels only convey limited information that a sample belongs to a single definite class and lack of prior information, it is difficult to supervise the model to capture rich inherent semantic information in complex data distributions, hindering the classification performance. To this end, we propose a text-supervised multidimensional contrastive fusion network (TMCFN), which leverages class text information to guide the learning of visual representations while establishing a semantic association of text and visual features for classification using multidimensionally incorporated contrastive learning (CL) paradigms. Specifically, TMCFN is composed of text information encoding (TIE), visual features representation (VFR), and text–visual features alignment and classification (TVFAC). TIE is employed to extract semantic information from class text extended from class names, intrinsic attributes and inter-class relationships. VFR mainly comprises a new fusion-based contrastive feature learning module (FCFLM) to extract discriminative visual features and a text-guided attention feature fusion module (TAF $^{2}\text{M}$ ) to fuse visual features under the guidance of text information. TVFAC optimizes the learning of visual features under the supervision of text information while using a CL paradigm to align text and visual features for establishing the semantic association, and achieves the classification by directly computing the similarity between the visual features and each text feature without an additional classifier. Experiments with three standard datasets verify the effectiveness of TMCFN.