[1]WANG Wenqing,ZHANG Xiaoqiao,HE Ji,et al.Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks[J].CAAI Transactions on Intelligent Systems,2025,20(3):649-657.[doi:10.11992/tis.202401003]
Copy

Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 20 Number of periods: 2025 3 Page number: 649-657 Column: 学术论文—机器感知与模式识别 Public date: 2025-05-05
Title:
Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks
Author(s):
WANG Wenqing1; 2;  ZHANG Xiaoqiao1;  HE Ji1;  LIU Han1; 2;  LIU Ding1; 2
1. School of Automation and Information Engineering, Xi’an University of Technology, Xi’an 710048, China;
2. Shaanxi Key Laboratory of Complex System Control and Intelligent Information Processing, Xi’an University of Technology, Xi’an 710048, China
Keywords:
image fusion; remote sensing; image processing; deep learning; convolutional neural network; machine learning; feature extraction; image reconstruction
CLC:
TP751
DOI:
10.11992/tis.202401003
Abstract:
The pansharpening of multispectral images represents a trending research topic in remote sensing image processing and interpretation. Moreover, compared with traditional pansharpening methods, deep learning-based pansharpening methods mainly extract deep features, thereby greatly improving the quality of fused images. Here, a method based on hybrid dual-branch convolutional neural network (CNN) and graph convolutional neural network (GCNN) is proposed to simultaneously extract spectral information, spatial information, and non-geometric structural information and improve the spatial and spectral resolutions of fused images. This hybrid method comprises the construction of a multi-resolution analysis fusion framework, followed by the construction of a feature extraction module, a feature fusion module, and an image reconstruction module based on deep neural networks. First, the hybrid dual-branch network module was constructed using 2D and 3D CNNs that focus on extracting spatial and spectral features, respectively. Second, GCNN was introduced to capture the spatial relationships of the nodes in the graph structure of the image and integrate non-local information. Afterward, the spatial, spectral, and non-geometric features extracted from multispectral and panchromatic images were fused by the feature fusion module. Finally, the fused features were input into the image reconstruction network to reconstruct the high-quality multispectral images. The proposed method was experimentally validated using GeoEye-1 and IKONOS remote sensing data. Compared with other methods, the experimental results obtained by the proposed method reveal its excellent performance in subjective and objective vision evaluations.
References:
[1] 杨勇，苏昭，黄淑英，等. 基于深度学习的像素级全色图像锐化研究综述[J]. 遥感学报, 2022, 26(12): 2411-2432.
YANG Yong, SU Zhao, HUANG Shuying, et al. Survey of deep-learning approaches for pixel-level pansharpening[J]. National remote sensing bulletin, 2022, 26(12): 2411-2432.
[2] 赵其昌, 吴一全, 苑玉彬. 光学遥感图像舰船目标检测与识别方法研究进展[J]. 航空学报, 2024, 45(8): 029025.
ZHAO Qichang, WU Yiquan, YUAN Yubin. Progress of ship detection and recognition methods in optical remote sensing images[J]. Acta aeronautica et astronautica sinica, 2024, 45(8): 029025.
[3] LI Zhi, PENG Zhenming, CAO Siying, et al. High-resolution remote sensing change detection based on inverse correction and density peak clustering[C]// 2023 IEEE International Geoscience and Remote Sensing Symposium. Pasadena: IEEE, 2023: 3028-3031.
[4] BAI Jing, LIU Ruotong, ZHAO Haisheng, et al. Hyperspectral image classification using geometric spatial-spectral feature integration: a class incremental learning approach[J]. IEEE transactions on geoscience and remote sensing, 2023, 61: 5531215.
[5] 胡建文, 汪泽平, 胡佩. 基于深度学习的空谱遥感图像融合综述[J]. 自然资源遥感, 2023, 35(1): 1-14.
HU Jianwen, WANG Zeping, HU Pei. A review of spatial-spectral remote sensing image fusion based on deep learning[J]. Remote sensing for natural resources, 2023, 35(1): 1-14.
[6] 徐佳, 关泽群, 何秀凤, 胡俊伟，等. 基于传感器光谱特性的全色与多光谱图像融合[J]. 遥感学报, 2009, 13(1): 97-102.
XU Jia, GUAN Zequn, HE Xiufeng, et al. Panchromatic and multispectral image fusion based on sensor spectral characteristics[J]. Journal of remote sensing, 2009, 13(1): 97-102.
[7] 肖亮, 刘鹏飞, 李恒. 多源空——谱遥感图像融合方法进展与挑战[J]. 中国图象图形学报, 2020, 25(5): 851-863.
XIAO Liang, LIU Pengfei, LI Heng. Advances and challenges in multi-source spatial-spectral remote sensing image fusion methods[J]. Journal of image and graphics, 2020, 25(5): 851-863.
[8] TU Teming, SU Shunchi, SHYU H C, et al. A new look at IHS-like image fusion methods[J]. Information fusion, 2001, 2(3): 177-186.
[9] KWARTENG P, CHAVEZ A. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis[J]. Photogrammetric engineering & remote sensing, 1989, 55: 339-348.
[10] MALLAT S G. A theory for multiresolution signal decomposition: the wavelet representation[J]. IEEE transactions on pattern analysis and machine intelligence, 1989, 11(7): 674-693.
[11] LIU Pengfei. Pansharpening with spatial hessian non-convex sparse and spectral gradient low rank priors[J]. IEEE transactions on image processing, 2023, 32: 2120-2131.
[12] ZHU Xiaoxiang, GROHNFELDT C, BAMLER R. Exploiting joint sparsity for pansharpening: the J-SparseFI algorithm[J]. IEEE transactions on geoscience and remote sensing, 2016, 54(5): 2664-2681.
[13] LIU Pengfei, TANG Songze, HUANG Lili. Pansharpening with spatial hyper-Laplacian and spectral sparse constraints[C]// 2022 IEEE International Geoscience and Remote Sensing Symposium. Kuala Lumpur: IEEE, 2022: 3762-3765.
[14] MASI G, COZZOLINO D, VERDOLIVA L, et al. Pansharpening by convolutional neural networks[J]. Remote sensing, 2016, 8(7): 594.
[15] LIU Xiangyu, LIU Qingjie, WANG Yunhong. Remote sensing image fusion based on two-stream fusion network[J]. Information fusion, 2020, 55: 1-15.
[16] YANG Junfeng, FU Xueyang, HU Yuwen, et al. PanNet: a deep network architecture for pan-sharpening[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 1753-1761.
[17] LIU Xiangyu, WANG Yunhong, LIU Qingjie. Psgan: a generative adversarial network for remote sensing image pan-sharpening[C]//2018 25th IEEE International Conference on Image Processing. Athens: IEEE, 2018: 873-877.
[18] SHI Cheng, PUN C M. Superpixel-based 3D deep neural networks for hyperspectral image classification[J]. Pattern recognition, 2018, 74: 600-616.
[19] AHMAD M, KHAN A M, MAZZARA M, et al. A fast and compact 3-D CNN for hyperspectral image classification[J]. IEEE geoscience and remote sensing letters, 2020, 19: 5502205.
[20] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-09-09)[2024-01-01]. https://arxiv.org/abs/1609.02907v4.
[21] 侯磊, 刘金环, 于旭, 等. 图神经网络研究综述[J]. 计算机科学, 2024, 51(6): 282-298.
HOU Lei, LIU Jinhuan, YU Xu, et al. Review of graph neural networks[J]. Computer science, 2024, 51(6): 282-298.
[22] XU Boyan, YIN Hujun. Graph convolutional networks in feature space for image deblurring and super-resolution[C]//2021 International Joint Conference on Neural Networks. Shenzhen: IEEE, 2021: 1-8.
[23] YAN Keyu, ZHOU Man, LIU Liu, et al. When pansharpening meets graph convolution network and knowledge distillation[J]. IEEE transactions on geoscience and remote sensing, 2022, 60: 5408915.
[24] WANG Xiaolong, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7794-7803.
[25] PALSSON F, SVEINSSON J R, ULFARSSON M O, et al. Quantitative quality evaluation of pansharpened imagery: consistency versus synthesis[J]. IEEE transactions on geoscience and remote sensing, 2016, 54(3): 1247-1259.
[26] WANG Wenqing, ZHOU Zhiqiang, ZHANG Xiaoqiao, et al. DiTBN: detail injection-based two-branch network for pansharpening of remote sensing images[J]. Remote sensing, 2022, 14(23): 6120.
[27] VIVONE G, ALPARONE L, CHANUSSOT J, et al. A critical comparison among pansharpening algorithms[J]. IEEE transactions on geoscience and remote sensing, 2015, 53(5): 2565-2586.
[28] KALLEL A. MTF-adjusted pansharpening approach based on coupled multiresolution decompositions[J]. IEEE transactions on geoscience and remote sensing, 2015, 53(6): 3124-3145.
[29] VIVONE G. Robust band-dependent spatial-detail approaches for panchromatic sharpening[J]. IEEE transactions on geoscience and remote sensing, 2019, 57(9): 6421-6433.
[30] ZHOU J, CIVCO D L, SILANDER J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International journal of remote sensing, 1998, 19(4): 743-757.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems