Search

Many mid-level and high-level computer vision tasks, such as object segmentation, detection and recognition, image retrieval and classification, and video tracking and understanding, still have not leveraged the advantages offered by spectral information. The increasing accessibility and affordability of spectral imaging technology have revolutionised computer vision, allowing for data capture across various wavelengths beyond the visual spectrum. [Extracted from the article]

Deep learning (DL) based hyperspectral images (HSIs) denoising approaches directly learn the nonlinear mapping between noisy and clean HSI pairs. They usually do not consider the physical characteristics of HSIs. This drawback makes the models lack interpretability that is key to understanding their denoising mechanism and limits their denoising ability. In this paper, we introduce a novel model-guided interpretable network for HSI denoising to tackle this problem. Fully considering the spatial redundancy, spectral low-rankness, and spectral-spatial correlations of HSIs, we first establish a subspace-based multidimensional sparse (SMDS) model under the umbrella of tensor notation. After that, the model is unfolded into an end-to-end network named SMDS-Net, whose fundamental modules are seamlessly connected with the denoising procedure and optimization of the SMDS model. This makes SMDS-Net convey clear physical meanings, i.e., learning the low-rankness and sparsity of HSIs. Finally, all key variables are obtained by discriminative training. Extensive experiments and comprehensive analysis on synthetic and real-world HSIs confirm the strong denoising ability, strong learning capability, promising generalization ability, and high interpretability of SMDS-Net against the state-of-the-art HSI denoising methods. The source code and data of this article will be made publicly available at https://github.com/bearshng/smds-net for reproducible research. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) denoising is an ill-posed inverse problem. The underlying physical model is always important to tackle this problem, which is unfortunately ignored by most of the current deep learning (DL)-based methods, producing poor denoising performance. To address this issue, this article introduces an end-to-end model-aided nonlocal neural network (MAC-Net) which simultaneously takes the spectral low-rank model and spatial deep prior into account for HSI noise reduction. Specifically, motivated by the success of the spectral low-rank model in depicting the strong spectral correlations and the nonlocal similarity prior in capturing spatial long-range dependencies, we first build a spectral low-rank model and then integrate a nonlocal U-Net into the model. In this way, we obtain a hybrid model-based and DL-based HSI denoising method where the spatial local and nonlocal multi-scale and spectral low-rank structures are effectively exploited. After that, we cast the optimization and denoising procedure of the hybrid method as a forward process of a neural network and introduce a set of learnable modules to yield our MAC-Net. Compared with traditional model-based methods, our MAC-Net overcomes the difficulties of accurate modeling, thanks to the strong learning and representation ability of DL. Unlike most “black-box” DL-based methods, the spectral low-rank model is beneficial to increase the generalization ability of the network and decrease the requirement of training samples. Experimental results on the natural and remote-sensing HSIs show that MAC-Net achieves state-of-the-art performance over both model-based and DL-based methods. The source code and data of this article will be made publicly available at https://github.com/bearshng/mac-net for reproducible research. [ABSTRACT FROM AUTHOR]

Hyperspectral unmixing is recognized as an important tool to learn the constituent materials and corresponding distribution in a scene. The physical spectral mixture model is always important to tackle this problem because of its highly ill-posed nature. In this article, we introduce a linear spectral mixture model (LMM)-based end-to-end deep neural network named SNMF-Net for hyperspectral unmixing. SNMF-Net shares an alternating architecture and benefits from both model-based methods and learning-based methods. On the one hand, SNMF-Net is of high physical interpretability as it is built by unrolling $L_{p}$ sparsity constrained nonnegative matrix factorization ($L_{p}$ -NMF) model belonging to LMM families. On the other hand, all the parameters and submodules of SNMF-Net can be seamlessly linked with the alternating optimization algorithm of $L_{p}$ -NMF and unmixing problem. This enables us to reasonably integrate the prior knowledge on unmixing, the optimization algorithm, and the sparse representation theory into the network for robust learning, so as to improve unmixing. Experimental results on the synthetic and real-world data show the advantages of the proposed SNMF-Net over many state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In many statistical hyperspectral unmixing approaches, the unmixing task is essentially an optimization problem given a defined linear or nonlinear spectral mixture model. However, most of the model inference algorithms require a time-consuming iterative procedure. On the other hand, neural networks have been recently used to estimate abundances given some training samples, or directly estimate endmembers and abundances simultaneously in an unsupervised setting. However, their disadvantages are clear: lack of interpretability and reliance on the large training set. Model-inspired neural networks are constructed by the problem model and its corresponding inference algorithm. It incorporates the prior knowledge of physical model and algorithm into network architecture, combining the advantages of model-based and learning-based methods. This article deeply unfolds the linear mixture model and the corresponding iterative shrinkage-thresholding algorithm (ISTA) to build two unmixing network architectures. The first assumes that the set of endmembers are known, and the deep unfolded ISTA model is only for abundance estimation; and the second is used for blind unmixing to estimate both endmembers and abundances at the same time. The networks can be trained by supervised and unsupervised schemes, respectively, with a small-size training set, and then, unmixing becomes a feedforward process, which is very fast since no iteration is required. The experimental results show their competitive performance compared with the state-of-the-art unmixing approaches. [ABSTRACT FROM AUTHOR]

Traditional color images only depict color intensities in red, green and blue channels, often making object trackers fail in challenging scenarios, e.g., background clutter and rapid changes of target appearance. Alternatively, material information of targets contained in large amount of bands of hyperspectral images (HSI) is more robust to these difficult conditions. In this paper, we conduct a comprehensive study on how material information can be utilized to boost object tracking from three aspects: dataset, material feature representation and material based tracking. In terms of dataset, we construct a dataset of fully-annotated videos, which contain both hyperspectral and color sequences of the same scene. Material information is represented by spectral-spatial histogram of multidimensional gradients, which describes the 3D local spectral-spatial structure in an HSI, and fractional abundances of constituted material components which encode the underlying material distribution. These two types of features are embedded into correlation filters, yielding material based tracking. Experimental results on the collected dataset show the potentials and advantages of material based object tracking. [ABSTRACT FROM AUTHOR]

Due to the mechanism of the data acquisition process, hyperspectral imagery (HSI) are usually contaminated by various noises, e.g., Gaussian noise, impulse noise, strips, and dead lines. In this article, a spectral–spatial $L_{0}$ gradient regularized low-rank tensor factorization (LRTF $L_{0}$) method is proposed for hyperspectral denoising, in which the restored HSI is approximated by low-rank block term decomposition (BTD). BTD factorizes a tensor into the sum of a series of component tensors, each of which is represented by the outer product of a matrix and a vector. From subspace learning point of view, the vector and matrix can be considered as a spectral atom and its corresponding coding coefficients. In the proposed method, the correlations in both spectral and spatial domains are taken into account via the small size of atom set and low-rankness of coding matrices. In addition, HSIs also have the local structure of piecewise smoothness in both spectral and spatial domains. Motivated by the supreme virtues of $L_{0}$ gradient regularization in image structure exploitation, we develop a spectral–spatial $L_{0}$ gradient regularization and embed it into BTD to explore the spectral–spatial texture information. The proposed method can simultaneously remove various types of noises, and the experimental results on both synthetic data and real-world data show its superiority when compared with several state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Hyperspectral unmixing decomposes a hyperspectral imagery (HSI) into a number of constituent materials and associated proportions. Recently, nonnegative tensor factorization (NTF)-based methods have been proposed for hyperspectral unmixing thanks to their capability in representing an HSI without any information loss. However, tensor factorization-based HSI processing approaches often suffer from low-signal-to-noise ratio condition of HSI and nonuniqueness of the solution. This problem can be effectively alleviated by introducing various spatial constraints into tensor factorization to suppress the noise and decrease the number of extreme, stationary, and saddle points. On the other hand, total variation (TV) adaptively promotes piecewise smoothness while preserving edges. In this paper, we propose a TV regularized matrix–vector NTF method. It takes advantage of tensor factorization in preserving global spectral–spatial information and the merits of TV in exploiting local spatial information, thus generating smooth abundance maps with preserved edges. Experimental results on synthetic and real-world data show that the proposed method outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Many spectral unmixing approaches ranging from geometry, algebra to statistics have been proposed, in which nonnegative matrix factorization (NMF)-based ones form an important family. The original NMF-based unmixing algorithm loses the spectral and spatial information between mixed pixels when stacking the spectral responses of the pixels into an observed matrix. Therefore, various constrained NMF methods are developed to impose spectral structure, spatial structure, and spectral-spatial joint structure into NMF to enforce the estimated endmembers and abundances preserve these structures. Compared with matrix format, the third-order tensor is more natural to represent a hyperspectral data cube as a whole, by which the intrinsic structure of hyperspectral imagery can be losslessly retained. Extended from NMF-based methods, a matrix-vector nonnegative tensor factorization (NTF) model is proposed in this paper for spectral unmixing. Different from widely used tensor factorization models, such as canonical polyadic decomposition CPD) and Tucker decomposition, the proposed method is derived from block term decomposition, which is a combination of CPD and Tucker decomposition. This leads to a more flexible frame to model various application-dependent problems. The matrix-vector NTF decomposes a third-order tensor into the sum of several component tensors, with each component tensor being the outer product of a vector (endmember) and a matrix (corresponding abundances). From a formal perspective, this tensor decomposition is consistent with linear spectral mixture model. From an informative perspective, the structures within spatial domain, within spectral domain, and cross spectral-spatial domain are retreated interdependently. Experiments demonstrate that the proposed method has outperformed several state-of-the-art NMF-based unmixing methods. [ABSTRACT FROM AUTHOR]

Object detection in remote sensing (RS) images is a challenging task due to the difficulties of small size, varied appearance, and complex background. Although a lot of methods have been developed to address this problem, many of them cannot fully exploit multilevel context information or handle cluttered background in RS images either. To this end, in this paper, we propose a feature fusion and filtration network ( F 3 -Net) to improve object detection in RS images, which has higher capacity of combining the context information at multiple scales while suppressing the interference from the background. Specifically, F 3 -Net leverages a feature adaptation block with a residual structure to adjust the backbone network in an end-to-end manner, better considering the characteristics of RS images. Afterward, the network learns the context information of the object at multiple scales by hierarchically fusing the feature maps from different layers. In order to suppress the interference from cluttered background, the fused feature is then projected into a low-dimensional subspace by an additional feature filtration module. As a result, more relevant and accurate context information is extracted for further detection. Extensive experiments on DOTA, NWPU VHR-10, and UCAS AOD datasets demonstrate that the proposed detector achieves very promising detection performance. [ABSTRACT FROM AUTHOR]

Hyperspectral image (HSI) classification is a small-sample-size problem due to the expensive cost of labeling. As a novel approach to this problem, cross-scene HSI classification has become a hot research topic in recent years. In cross-scene HSI classification, the scene containing enough labeled samples (called source scene) is used to benefit the classification in another scene containing a small number of training samples (called target scene). Transfer learning is a typical solution for cross-scene classification. However, many transfer learning algorithms assume an identical feature space for source and target scenes, which violates the fact that source and target scenes often lie in different feature spaces with various dimensions due to different HSI sensors. Aiming at the different feature spaces between the two scenes, we propose an end-to-end heterogeneous deep transfer learning algorithm, namely, cross-domain variational autoencoder (CDVAE). This algorithm is mainly composed of two key parts: 1) the features of the two scenes are embedded into the shared feature subspace through the two-stream variational autoencoder (VAE) to ensure that the output feature dimensions of the two scenes are identical and 2) graph regularization is used to establish the manifold constraints between source and target scenes in the shared subspace, so as to align the feature spaces. Experiments on two different cross-scene HSI datasets have proved the superior performance of the proposed CDVAE algorithm. [ABSTRACT FROM AUTHOR]

Small‐sample‐size problem is always a challenge for hyperspectral image (HSI) classification. Considering the co‐occurrence of land‐cover classes between similar scenes, transfer learning can be performed, and cross‐scene classification is deemed a feasible approach proposed in recent years. In cross‐scene classification, the source scene which possesses sufficient labelled samples is used for assisting the classification of the target scene that has a few labelled samples. In most situations, different HSI scenes are imaged by different sensors resulting in their various input feature dimensions (i.e. number of bands), hence heterogeneous transfer learning is desired. An end‐to‐end heterogeneous transfer learning algorithm namely domain‐invariant attention network (DIAN) is proposed to solve the cross‐scene classification problem. The DIAN mainly contains two modules. (1) A feature‐alignment CNN (FACNN) is applied to extract features from source and target scenes, respectively, aiming at projecting the heterogeneous features from two scenes into a shared low‐dimensional subspace. (2) A domain‐invariant attention block is developed to gain cross‐domain consistency with a specially designed class‐specific domain‐invariance loss, thus further eliminating the domain shift. The experiments on two different cross‐scene HSI datasets show that the proposed DIAN achieves satisfying classification results. [ABSTRACT FROM AUTHOR]

As single-pixel imaging technology is highly sensitive and low-cost, a large number of hyperspectral imaging methods based on single-pixel have been proposed. Although the image of each band can be reconstructed independently, the parameters that need to be computed for reconstructing hyperspectral images will increase in parallel with the number of bands. Therefore, in this paper, we propose a redundant compressed single-pixel hyperspectral imaging system (RCSHI), which takes full advantage of the spectral redundancy of hyperspectral images by jointly reconstructing images from different bands and compressing the spectral dimension of the data. RCSHI reduces the amount of redundant hyperspectral information stacking and the number of parameters that need to be computed to reconstruct hyperspectral images. In addition, we use progressive fusion enhancement to recover the compressed spectral dimension and enhance the spatial dimension of the images. Experimental results show that RCSHI is able to acquire hyperspectral data with high spatial and spectral resolution and performs well on various datasets and actual scenarios, especially at lower compressed sampling rates. [ABSTRACT FROM AUTHOR]

• We design an undersampling rapid coded aperture spectrometer system. This system refers to the spatial information of RGB to select the sampling order and uses the DCT matrix as the coding patterns. The designed imaging system can better address the contradiction between image resolution and imaging speed to a certain extent, so as to achieve high speed and high resolution hyperspectral imaging. • A spectral image enhancement system is designed to improve the image spatial detail recognition capability that consists of a CMMI information fusion function and a PMS-net-based feature extraction module. By improving accuracy at a lower cost, the network structure is made simpler, which improves the generalization and robustness of the network to some extent. • Based on the fast conversion of the micromirror unit in DMD, we built a DMD-based hyperspectral imaging system. We obtained real hyperspectral datasets in reality by this system, and verified that our proposed algorithm has good performance on real spectral datasets. In recent years, Coded Aperture Snapshot Spectroscopic Imaging System (CASSI) has attracted more and more attention. However, in the case of undersampling, the effect of the reconstructed image is often unsatisfactory. For spectral imaging system, the slower the imaging speed, the higher the image resolution of hyperspectral images, and vice versa. To solve this problem, in this paper, we design an undersampling rapid coded aperture spectrometer system. Based on the analysis of the internal relationship between coded spectral imaging and single pixel imaging, the order of the discrete cosine transform (DCT) coding patterns is selected by using the energy concentration property of the DCT and the rich spatial information contained in the RGB images. In addition, we built a spectral image enhancement network to improve the quality of the reconstructed images. At the sampling rate of 0.0625, the PSNR, SSIM and SAM of the spectral images reconstructed based on our system are 43.8341, 0.9869 and 2.0862, respectively, the reconstruction effect of this method is significantly superior to other methods, and this method provides an idea to address the contradiction between image resolution and imaging speed to a certain extent, so as to achieve high speed and high resolution hyperspectral imaging. [ABSTRACT FROM AUTHOR]

• In existing multiplexed spectral imaging methods based on matrix operations (especially HTS), the reconstructed results are similar to miscalculations in the case of under-sampling. In contrast, our FCTS based on frequency domain calculation discards only the high-frequency components and has higher coding efficiency and physical interpretability. • Compared with existing snapshot systems (especially CASSI), FCTS has no limitation on the number of dispersion channels and can use high dispersion elements such as gratings according to practical needs. The experimental system in this paper uses an Amici prism with a spectral resolution of about 4 nm, which is tested to be lower than 1 nm if a 300-line grating is used, and has an unparalleled advantage in spectral resolution for snapshot systems. • Real systems and simulations show that the FCTS frequency domain calculation method is less affected by optical misalignment, diffraction and distortion in optical systems. It is more robust to optical distortions. • Because of the Fourier transform process of FCTS under-sampling we only take the low frequency part of the Fourier spectrum. So, there are some high frequency details missing in the reconstruction. We can also use the common super resolution reconstruction or image enhancement methods to complete the image details to improve the image quality. We consider using Fourier coded aperture transform to solve the acquisition and reconstruction problem of three-dimensional (3-D) hyperspectral imaging. In recent years, Coded aperture snapshot spectral imaging (CASSI) system has received increasing attention. However, reconstruction errors are unavoidable due to its ill-posed problems and the number of dispersion channels is limited. The Hadamard transform spectral (HTS) imaging method has no limitation on the number of dispersion channels, but it usually requires a long data acquisition time. By exploring the process of coded spectral imaging, in this paper, we propose Fourier coded aperture transform spectral imaging (FCTS). We compared the performance of HTS and FCTS by theoretical analysis and experiments. The results show that FCTS has better concentrated energy than HTS on natural hyperspectral data. Compared with the Hadamard transform method, the Fourier basis mode has better physical correlation with the natural image, and the reconstruction quality of FCTS is better than that of HTS in the case of under-sampling. [ABSTRACT FROM AUTHOR]