A Twice Optimizing Net With Matrix Decomposition for Hyperspectral and Multispectral Image Fusion

Fusing a low-resolution hyperspectral (LRHS) image and a high-resolution multispectral (HRMS) image to generate a high-resolution hyperspectral (HRHS) image has grown a significant and attractive application in remote sensing fields. Recently, the popularization of deep learning has injected more possibilities into the fusion work. However, there still exists a difficulty that is how to make the best of the acquired LRHS and HRMS images. In this article, we present a twice optimizing net with matrix decomposition to fulfill the fusion task, which can be roughly divided into three stages: pre-optimization, deep prior learning, post-optimization. Specifically, we first transform this fusion problem into a spectral optimization problem and a spatial optimization problem with the help of matrix decomposition. These two optimization problems can be handled sequentially by solving a linear equation, respectively, and then we can obtain the initial HRHS image by multiplying the two solutions. Next, we establish the mapping between the initial image and the reference image through an end-to-end deep residual network based on local and nonlocal connectivity. In order to get better performance, we have customized a loss function specifically for the fusion task as well. Finally, we return the predicted result again to the optimization procedure to get the final fusion image. After the evaluation on three simulated datasets and one real dataset, it illustrates that the proposed method outperforms many state-of-the-art ones.