Search

Hypersharpening is one of the fusion-based superresolution approaches in remote sensing that improves the spatial and spectral resolution of a hyperspectral image (HSI) with a low spatial resolution. This requirement is achieved by fusing the HSI with a panchromatic image that has high spatial resolution to generate a newly combined variant, which has high spatial quality and high spectral resolution. While several studies in the literature applied neural networks for hypersharpening, there exist unsolved issues such as how to deeply discover the spatial–spectral correlation and inject geometric details without distortion. To address these issues, we propose a hypersharpening technique by applying a multicomponent-based hierarchical fusion network called (HS-McHF), which hierarchically learns the low and high-frequency spatial–spectral features. We then suggest an optimization model to discover the correlation between low-resolution HSI and high-resolution panchromatic images and solve it by stochastic gradient descent through a neural network. Moreover, we decrease the band overlapping in the initial HSI by combining a deconvolution model to prevent spectral distortion and reduce the noise in the panchromatic geometric details injection by deploying an encoder–decoder network. Our extensive experiments demonstrate that the HS-McHF provides superior efficiency compared to state-of-the-art fusion based superresolution approaches.

Supervised pansharpening methods require the ground truth, which is generally unavailable. Therefore, the popularity of unsupervised pansharpening methods has increased. Generative adversarial networks (GANs) are often employed for unsupervised pansharpening, although achieving precise control over the generation process to capture rich spatial and spectral details is challenging. CycleGAN introduces cycle consistency loss and utilizes the cooperative training of two generators and two discriminators to learn the mapping between different domains. This approach partially addresses the issue of limited control over the generated results in traditional GANs. Therefore, CycleGAN also can be employed to accomplish unsupervised pansharpening tasks. However, it is complicated to directly apply the network structure of CycleGAN to pansharpening. To address this issue, we integrate a process model capable of simulating spatial and spectral degradations into a single-generator CycleGAN, which can learn the target distribution. Specifically, we propose an unsupervised CycleGAN for pansharpening based on spatial and spectral degradations and consists of one lightweight generator and two discriminators. Then, the low-resolution multispectral and panchromatic images are considered as the spatial and spectral degradations of the high-resolution multispectral images. Besides, unsupervised loss functions consisting of cycle consistency, adversarial, spectral angle mapper, and edge enhancement losses are designed to preserve spectral and spatial information. The experimental results on the QuickBird, GeoEye-1, and GF-2 datasets show that the qualitative and quantitative analysis of the proposed method is comparable with most supervised methods and superior to most unsupervised methods.

Change detection refers to revealing the surface changes from multitemporal images of a given scene, where changed regions of different size and shape may appear anywhere. Convolutional neural network, as one of the most widely used deep learning architectures, has shown good performance in change detection task recently. However, due to multiple convolution and pooling operators in deep architectures, it often happens that small changes are missed and sharp edges are blurred. In this article, an adaptive spatial and difference learning method is proposed to tackle this problem, which mainly contains a feature-adapted difference learning (FADL) module and a difference recalibration (DR) module. In FADL, a kernel-adapted spatial learning part is constructed to capture varying spatial information, which selects proper kernels to adapt different shape and size of changed regions; a joint feature refinement part is employed to learn comprehensive features between bitemporal data, which are further enhanced by difference information. In addition, DR with shallow features is designed to recalibrate spatial and difference details further. From the experiments on three public datasets, our proposed method can relieve the spatial ambiguity problem and obtains the optimal results among the compared change detection methods.

Due to the physical boundaries, fusing low spatial resolution hyperspectral (LrHSI) with high spatial resolution multispectral (HrMSI) images is a hot and promising area for obtaining hyperspectral that have high spatial-spectral resolution images (HrHSI). Effectively formulating the fundamental features of hyperspectral images (HSI), such as global spectral correlation, nonlocal spatial correlation, as well spatial-spectral correlation, is complex in HSI-MSI fusion. Moreover, the fusion process is highly affected by the degradation systems, where these systems are not known in real scenarios. To this end, in this article, we proposed a model-guided deep unfolded fusion network with non-local spatial-spectral priors (MGDuNLSS-net) that can maintain the essential features of the HSIs and implicitly estimates the degradation process in an adequate running time. Specifically, the proposed method is designed based on subspace representation in an iterative manner and unrolling its steps toward a deep network as an end-to-end framework. This approach contains two submodules, fusion [nonlocal spatial-spectral block (NLSSB)] and imaging system submodules. The former submodule is proposed to exploit the images' intrinsic characteristics to improve the preservation of spectral and spatial details. NLSSB contains two nonlocal self-similarity (NLSS) layers embedded between two bidirectional simple recurrent unit (BSRU) layers. The recurrent calculation, as well as refined components to maintain the global spectral correlation, are the light recurrence operation and highway network, while 3-D convolutions in the BSRU can retain the spatial-spectral correlation. The NLSS layer can efficiently and effectively model long-range spatial contexts, which is designed based on criss-cross attention. The later submodule is used to refine the prediction of the degradation process at any iteration via backprojecting the estimated fused image to the observed pair, which can ensure the good performance of fusion. Compared with state-of-the-art fusion approaches, three remote sensing datasets are used to validate the proposed approach's performance.

At present, an emerging technique called the algorithm unrolling approach has attracted wide attention, because it is capable of developing efficient and interpretable layers to eliminate the black-box nature of deep learning (DL). In this article, inspired by the sparse unmixing model, we propose a model-driven DL approach, namely, an implicit variable iterative unrolling network (IVIU-Net). First of all, the unmixing performance and adaptive ability of the model are enhanced by introducing learnable parameters into the sparse unmixing algorithm. Then, a specific spatial convolution module is integrated into the network to promote the smoothness of the latent abundance map. Finally, a comprehensive loss function with three terms such as average spectral angle distance, hyperspectral images reconstruction error, and spectral information divergence, is presented to train the IVIU-Net in an unsupervised way. Compared to the unmixing results of most existing data-driven DL algorithms, our network has significant advantages in two folds: it is able to achieve better stability instead of relying heavily on the endmember initialization results and it has better interpretability and robustness in the unmixing procedure. Experimental results on synthetic and real data show that the proposed network outperforms the state-of-the-art in terms of better convergence, faster unmixing speed as well as better accuracy.

The fusion-based super-resolution of hyperspectral images (HSIs) draws more and more attention in order to surpass the hardware constraints intrinsic to hyperspectral imaging systems in terms of spatial resolution. Low-resolution (LR)-HSI is combined with a high-resolution multispectral image (HR-MSI) to achieve HR-HSI. In this article, we propose multiresolution details enhanced attentive dual-UNet to improve the spatial resolution of HSI. The entire network contains two branches. The first branch is the wavelet detail extraction module, which performs discrete wavelet transform on MSI to extract spatial detail features and then passes through the encoding–decoding. Its main purpose is to extract the spatial features of MSI at different scales. The latter branch is the spatio-spectral fusion module, which aims to inject the detail features of the wavelet detail extraction network into the HSI to reconstruct the HSI better. Moreover, this network uses an asymmetric feature selective attention model to focus on important features at different scales. Extensive experimental results on both simulated and real data show that the proposed network architecture achieves the best performance compared with several leading HSI super-resolution methods in terms of qualitative and quantitative aspects.

The hyperspectral image (HSI) is easily contaminated by various kinds of mixed noise (such as Gaussian noise, impulse noise, stripes, and deadlines) during the process of data acquisition and conversion, which significantly affect the quality and applications of HSI. As an important and effective scheme for the quality improvement of HSI, the HSI restoration problem aims to recover a clean HSI from the noisy HSI with mixed noise. Thus, based on the tensor modeling of HSI, we propose a novel tensor-based HSI restoration model with low-rank modeling in gradient domains in a unified tensor representation framework in this article. First, for the spectral low-rank modeling of HSI in spectral gradient domain, we particularly exploit the low-rank property of spectral gradient, and propose the spectral gradient-based weighted nuclear norm low-rank prior term. Second, for the spatial-mode low-rank modeling of HSI in spatial gradient domain, we particularly exploit the low-rank property of spatial gradient tensors via the discrete Fourier transform, and propose the spatial gradient-based tensor nuclear norm low-rank prior term. Then, we use the alternative direction method of multipliers to solve the proposed model. Finally, the restoration results on both the simulated and real HSI datasets demonstrate that the proposed method is superior to many state-of-the-art methods in the aspects of visual and quantitative comparisons.

The fusion of high-resolution multispectral image (HrMSI) and low-resolution hyperspectral image (LrHSI) has been acknowledged as a promising method for generating a high-resolution hyperspectral image (HrHSI), which is also termed to be an essential part for precise recognition and cataloguing of the underlying materials. In order to improve the fusion of the LrHSI and HrMSI performance, in this article, we propose a novel nonnegative matrix factorization inspired deep unrolling networks (NMF-DuNet) for fusing LrHSI and HrMSI. For this aim, initially, a variational fusion model regularized by nonnegative sparse prior is proposed and then is solved through the gradient descent optimization method and unrolled towards the deep network. The nonnegative coefficient matrices and orthogonal of the proposed transform coefficients constraints are both incorporated into the proposed method. Moreover, the fusion of HrMSI and LrHSI heavily depends on an imaging model that explains the degeneracy of HSI in the spectral and spatial regions. Practically, the imaging model is often unknown. The degradation model is represented implicitly via a proposed network, and both the degradation model and sparse priors are jointly optimized through the training process of the proposed network. Instead of being hand-crafted, all the parameters of NMF-DuNet are learned end-to-end. Compared to the previous state-of-the-art model-based and learning-based fusion approaches, the hardware-friendly proposed NMF-DuNet outperforms both the model-based and learning-based fusion approaches and requires a far smaller number of trainable parameters and storage space while preserving the real-time performance.

Hyperspectral and multispectral image fusion aims to fuse a low-spatial-resolution hyperspectral image (HSI) and a high-spatial-resolution multispectral image to form a high-spatial-resolution HSI. Motivated by the success of model- and deep learning-based approaches, we propose a novel patch-aware deep fusion approach for HSI by unfolding a subspace-based optimization model, where moderate-sized patches are used in both training and test phases. The goal of this approach is to make full use of the information of patch under subspace representation, restrict the scale and enhance the interpretability of the deep network, thereby improving the fusion. First, a subspace-based fusion model was built with two regularization terms to localize pixels and extract texture. Then, the subspace-based fusion model was solved by the alternating direction method of multipliers algorithm, and the model was divided into one fidelity-based problem and two regularization-based problems. Finally, a structured deep fusion network was proposed by unfolding all steps of the algorithm as network layers. Specifically, the fidelity-based problem was solved by a gradient descent algorithm and implemented by a network. The two regularization-based problems were described by proximal operators and learnt by two u-shaped architectures. Moreover, an aggregation fusion technique was proposed to improve the performance by averaging the fused images in all iterations and aggregating the overlapping patches in the test phase. Experimental results, conducted on both synthetic and real datasets, demonstrated the effectiveness of the proposed approach.

Wireless body area networks that support fast-growing healthcare applications have to control the access of the sensors in the dynamic network and channel states. In this paper, we propose a sensor access control scheme based on reinforcement learning that enables the coordinator to choose the access time and transmit power of the sensors based on the state that consists of the signal-to-interference plus noise ratio, the transmission priority, the battery level, and the transmission delay of the sensors. This scheme is proved to improve the quality-of-service, save the energy consumption of the sensors, and enhance the transmission reliability. The simulation results show that this scheme reduces the bit error rate, saves the energy consumption, decreases the transmission delay, and increases the overall utility of the sensors compared with the benchmark schemes.

In this paper, we propose a reinforcement learning-based adaptive modulation and coding scheme for underwater communications; more specifically, based on the network states such as the quality of service requirement of the sensing message, the previous transmission quality, and the energy consumption. This scheme applies reinforcement learning to choose the modulation and coding policy in a dynamic underwater communication system. We provide the performance bound of this scheme and perform experiments in both pool and sea environments. The experimental data were collected and post-processed. Compared with the benchmark schemes, this scheme can improve the throughputs and reduce the BER with less energy consumption.

High-precision real-time 3D object detection based on the LiDAR point cloud is an important task for autonomous driving. Most existing methods utilize grid-based convolutional networks to handle sparse and cluttered point clouds. However, the performance of object detection is limited by the coarse grid quantization and expensive computational cost. In this paper, we propose a more efficient representation of 3D point clouds and propose SCNet, a single-stage, end-to-end 3D subdivision coding network that learns finer feature representations for vertical grids. SCNet divides each grid into smaller sub-grids to preserve more point cloud information and converts points in the grid to a uniform feature representation through 2D convolutional neural networks. The 3D point cloud is encoded as the fine 2D sub-grid representation, which helps to reduce the computational cost. We validate our SCNet on the KITTI object benchmark in which we show that the proposed object detector produces state-of-the-art results with more than 20 FPS.

Recent progress of self-supervised visual representation learning has achieved remarkable success on many challenging computer vision benchmarks. However, whether these techniques can be used for domain adaptation has not been explored. In this work, we propose a generic method for self-supervised domain adaptation, using object recognition and semantic segmentation of urban scenes as use cases. Focusing on simple pretext/auxiliary tasks (e.g. image rotation prediction), we assess different learning strategies to improve domain adaptation effectiveness by self-supervision. Additionally, we propose two complementary strategies to further boost the domain adaptation accuracy on semantic segmentation within our method, consisting of prediction layer alignment and batch normalization calibration. The experimental results show adaptation levels comparable to most studied domain adaptation methods, thus, bringing self-supervision as a new alternative for reaching domain adaptation. The code is available at this link. https://github.com/Jiaolong/self-supervised-da.

Spectral-spatial feature extraction is of great importance to hyperspectral image (HSI) classification. Different from the traditional feature extraction methods, deep learning models such as convolutional neural network (CNN) can learn the spectral-spatial discriminative feature automatically. However, deep learning models usually need to construct a large and complicated network and the training is time-consuming. To deal with these issues, in this paper, a spectral-spatial domain-specific convolutional deep extreme learning machine (ELM), named S2CDELM, is proposed for HSI classification. At first, by using the conception of local receptive filed (LRF), a spectral-spatial convolutional learning module with two branches is constructed for spectral and spatial feature extraction respectively. Specifically, the convolutional learning module is constructed by using random convolutional nodes but without back propagation, in which a spectral branch and a spatial branch are designed respectively. Then the extracted features are concatenated and fed to a fully connected stacked ELM network to further exploit spectral-spatial information for classification. As the convolutional filters and input weights of ELM are randomly generated, the whole framework is compact, simple and fast to construct. Experimental results on popular HSI benchmark data sets demonstrate that S2CDELM can provide satisfactory classification performance and a fast learning speed in comparison with several state-of-the-art classifiers.

Markov random field (MRF) based methods have been widely used in high spatial resolution (HSR) image classification. However, many existing MRF-based methods put more emphasis on pixel level contexts while less on superpixel level contextual information. To cope with this issue, this article presents a novel bilevel contextual MRF framework, named BLC-MRF, for HSR imagery classification. Specifically, pixel and superpixel level dependence are incorporated into the proposed MRF model to fully exploit spectral-spatial contextual information and preserve object boundaries in HSR images. In BLC-MRF, a pixel level MRF model is first performed and then cascaded as an input of a superpixel level MRF. In superpixel level, unary and pairwise potential terms are constructed by using the superpixel probability estimation method and spectral histogram distance, respectively. At last, a contextual MRF model is conducted and the final classification map can be computed by using α-expansion algorithm. The benefits of BLCMRF are twofold: first, the pixel and superpixel level contextual information can be exploited under MRF framework to preserve object boundaries for improving the classification performance, and, second, the algorithm can provide promising results with a small number of training samples. Experimental results on three HSR datasets demonstrate that the proposed approach outperforms several state-of-the-art methods in terms of the classification performance.

Respiratory motion causes difficulty in locating tumours in the thorax and upper abdomen for image-guided radiotherapy. Precisely predicting the respiratory-induced organ motion is still a challenging problem at present. In this paper, to predict the motion of lungs in a respiratory cycle, we propose a novel method comprising Bayesian registration and trajectory modelling based on cine four-dimensional computer tomography (4DCT) images. Specifically, we take the CT image captured at the end-inhale phase as the source image and those captured at other phases as the moving images. We then align the source image to each moving-phase image to generate the displacement fields using the Bayesian registration method. The lung-motion trajectory is then modelled based on a continuous time-related displacement field by linking the displacement fields at discrete phases. The results indicate that any point in the lungs at any given time is accurately predicted using the proposed method, which provides an alternative method of estimating the lung and tumour motions for radiation therapy.

Sequential data, such as video frames and event data, have been widely applied in the realworld. As a special kind of sequential data, hyperspectral images (HSIs) can be regarded as a sequence of 2-D images in the spectral dimension, which can be effectively utilized for distinguishing different landcovers according to the spectral sequences. This paper presents a novel noise reduction method based on superpixel-based subspace low rank representation for hyperspectral imagery. First, under the framework of a linear mixture model, the original hyperspectral cube is assumed to be low rank in the spectral domain, which could be represented by decomposing HSI data into two sub-matrices of lower ranks. Meanwhile, due to the high correlation of neighboring pixels, the spectra within each neighborhood would also promote low rankness, and the local spatial low rankness could be exploited by enforcing the nuclear norm within superpixel-based regions in the decomposed subspace. The superpixels are easily and effectively generated by utilizing state-of-the-art superpixel segmentation algorithms in the first principal component of the original HSI. Moreover, benefiting from the subspace decomposition, the proposed method has an overwhelming superiority in computational cost than the state-of-the-art LR-based methods. The final model could be efficiently solved by the augmented Lagrangian method. Experimental results on simulated and real hyperspectral data sets validate that the proposed method produces superior performance than other stateof-the-art denoising methods in terms of quantitative assessment and visual quality.

The extension of supervised extreme learning machine (ELM) to unsupervised one, which involves discriminative and manifold regularization, is increasingly gaining attention in hyperspectral image (HSI) clustering. This is due to the fact that HSI clustering problem requires a spectral-spatial feature extraction mechanism that must fully exploit local spectral-spatial contexts and global discriminative information to reduce the misclassification while improve the robustness in clustering procedural. In this paper, we propose a novel context-aware unsupervised discriminative ELM method for HSI clustering. The main novelty of the proposed method are twofold:1) a local spectral-spatial context integration and reshaping mechanism is incorporated into the hidden layer feature representation by using a context-aware propagation filtering procedure; and 2) both local manifold and global discriminative regularization are integrated into unsupervised ELM framework to learn an effective data representation. The most important advantage of the proposed method is that it efficiently exploits the spatial contextual information of HSI through a propagation filtering procedural; furthermore, the learned data representation can capture the intrinsic structure by exploiting the local manifold and global information by discriminative regularization. Experimental results show that the proposed algorithm obtains a competitive performance and outperforms other state of the art ELM-based methods and the other unsupervised methods.

Wireless body area networks (WBANs) have to address jamming attacks to support healthcare applications. In this paper, we present a reinforcement learning-based power control scheme for the communication between the in-body sensors and the WBAN coordinator to resist jamming attacks. This scheme applies Q-learning to guide the coordinator to achieve an optimal power control strategy without being aware of the in-body sensor's transmission parameters and the WBAN model of the other sensors in the dynamic anti-jamming transmission. In addition, a transfer learning method is adopted to accelerate the learning speed. Stackelberg equilibria and their existence conditions are deduced in a single time slot to upper bound the performance of the learning-based sensor power control scheme. Simulation results show that the proposed scheme can efficiently increase the utilities and decrease the transmission energy consumptions for the in-body sensors and the WBAN coordinator, and simultaneously reduce the attack possibility of the jammer compared with a standard Q-learning-based sensor power control scheme.

The popular intensity-hue-saturation (IHS) pan-sharpening method can provide high spatial quality while suffering from some spectral distortion, mainly due to the fact that it cannot estimate an accurate intensity image to substitute the original intensity image in the IHS space. To overcome this drawback, in this paper, we particularly use the modeling idea of variational complementary data fusion, and propose a novel yet highly effective pan-sharpening method with variational Hessian transferring in the generalized IHS (GIHS) transform domain, which aims to estimate a more accurate intensity image. More specifically, the novelty of proposed method consists in building a variational Hessian transferring model in the GIHS transform domain to transfer the Hessian-based spatial geometric information of panchromatic (Pan) image to the new intensity image and meanwhile take the spectral information preserving into consideration. Finally, the experimental results demonstrate the effectiveness of the proposed method which can perform higher spectral and spatial qualities, and higher efficiency than some state-of-the-art methods.

Many classical visual odometry and simultaneous localization and mapping methods are able to achieve excellent performance, but mainly are restricted on the static scenes and suffer degeneration when there are many dynamic objects. In this paper, an efficient coarse-to-fine algorithm is proposed for moving object detection in dynamic scenes for autonomous driving. A motion-based conditional random field for this task is modeled. Particularly, for initial dynamic-static segmentation, a superpixel-based binary segmentation is processed, and further for refinement, a pixel-level object segmentation in local region is performed. Additionally, to reduce the projection noise caused by disparity estimation, an approximate Mahalanobis normalization is provided. Finally, in order to evaluate the proposed method, two relative methods are compared as baseline on the public KITTI data set for visual odometry and moving object detection separately. The experiments show the effectiveness and improvement on odometry when the dynamic region is removed and also on moving objects detection.

Advanced Persistent Threats (APTs) represent stealthy, powerful, long-term, and well-funded attacks against cyber systems, such as data centers and cloud storage. Evolutionary game theory is used to capture the long-term continuous behavior of the APTs on the cloud storage devices. Two APT defense games with discrete strategies are formulated, in which both an APT attacker and a defender compete to control one or multiple storage devices regarding their attack or defense intervals. The dynamical stability of each defense and attack strategy pair is studied according to the replicator dynamics criteria to characterize the locally asymptotically stable equilibrium strategies. The evolutionary stable strategy is discussed in each game, which is a subset of the asymptotically stable Nash equilibrium (NE). The phase portraits provide the locally asymptotically stable points of the APT defense game, which represent the NE showing the relationship between the asymptotic stability and evolutionary stability.

Mobile devices, such as smartphones, can offload applications and data to the cloud via access points or base stations to reduce energy consumption and improve user experience. However, mobile offloading is vulnerable to smart attackers that use smart and programmable radio devices, such as universal software radio peripherals, to perform multiple types of attacks, such as spoofing and jamming, based on the radio environment and offloading transmissions. In this paper, a mobile offloading game is investigated that consists of three players: a mobile device that chooses its offloading rate, a smart attacker that determines its attack mode, and a security agent that decides whether or not to initiate full protection for the serving access point during the offloading. Nash and Stackelberg equilibria of the offloading game are derived and their existence conditions are discussed. A Q-learning-based mobile offloading strategy is proposed for mobile devices that are unaware of system parameters, such as the channel conditions, in dynamic radio environments. Simulation results show that the proposed offloading strategy can improve the utility of the mobile device and reduce the attack rate of smart attackers.