[1]王文卿,张小乔,何霁,等.基于混合双分支卷积神经网络和图卷积神经网络的全色锐化方法[J].智能系统学报,2025,20(3):649-657.[doi:10.11992/tis.202401003]
WANG Wenqing,ZHANG Xiaoqiao,HE Ji,et al.Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks[J].CAAI Transactions on Intelligent Systems,2025,20(3):649-657.[doi:10.11992/tis.202401003]
点击复制
WANG Wenqing,ZHANG Xiaoqiao,HE Ji,et al.Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks[J].CAAI Transactions on Intelligent Systems,2025,20(3):649-657.[doi:10.11992/tis.202401003]
基于混合双分支卷积神经网络和图卷积神经网络的全色锐化方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第3期
页码:
649-657
栏目:
学术论文—机器感知与模式识别
出版日期:
2025-05-05
- Title:
- Pansharpening based on hybrid dual-branch convolutional and graph convolutional neural networks
- Keywords:
- image fusion; remote sensing; image processing; deep learning; convolutional neural network; machine learning; feature extraction; image reconstruction
- 分类号:
- TP751
- 摘要:
- 多光谱图像全色锐化是遥感影像处理与解译领域的热点问题。相较于传统全色锐化方法，基于深度学习的全色锐化方法聚焦于图像深层次特征的提取，大幅提升了融合图像的质量。本文提出一种基于混合双分支卷积神经网络和图卷积神经网络的全色锐化方法，旨在同时挖掘图像的光谱、空间与非几何结构信息，以提升融合图像空间分辨率和光谱分辨率。本方法建立在多分辨率分析融合框架的基础上，利用深度神经网络构建了特征提取、特征融合和图像重构模块。混合双分支网络模块是由2D和3D卷积神经网络构建，其中，2D卷积神经网络负责挖掘多光谱图像与全色图像的空间特征，3D卷积神经网络负责挖掘图像的光谱特征。引入了图卷积神经网络以捕捉图像图结构中节点的空间关系，从而整合非局部信息。将多光谱图像与全色图像的空间、光谱和非几何特征通过特征融合模块进行融合。将融合特征输入图像重构网络重建高质量多光谱图像。本文算法在GeoEye-1和IKONOS遥感数据上进行了实验验证，实验结果表明：与其他方法相比，本文算法在主观视觉和客观评价指标上均表现出优秀性能。
- Abstract:
- The pansharpening of multispectral images represents a trending research topic in remote sensing image processing and interpretation. Moreover, compared with traditional pansharpening methods, deep learning-based pansharpening methods mainly extract deep features, thereby greatly improving the quality of fused images. Here, a method based on hybrid dual-branch convolutional neural network (CNN) and graph convolutional neural network (GCNN) is proposed to simultaneously extract spectral information, spatial information, and non-geometric structural information and improve the spatial and spectral resolutions of fused images. This hybrid method comprises the construction of a multi-resolution analysis fusion framework, followed by the construction of a feature extraction module, a feature fusion module, and an image reconstruction module based on deep neural networks. First, the hybrid dual-branch network module was constructed using 2D and 3D CNNs that focus on extracting spatial and spectral features, respectively. Second, GCNN was introduced to capture the spatial relationships of the nodes in the graph structure of the image and integrate non-local information. Afterward, the spatial, spectral, and non-geometric features extracted from multispectral and panchromatic images were fused by the feature fusion module. Finally, the fused features were input into the image reconstruction network to reconstruct the high-quality multispectral images. The proposed method was experimentally validated using GeoEye-1 and IKONOS remote sensing data. Compared with other methods, the experimental results obtained by the proposed method reveal its excellent performance in subjective and objective vision evaluations.
- 参考文献/References:
-
[1] 杨勇，苏昭，黄淑英，等. 基于深度学习的像素级全色图像锐化研究综述[J]. 遥感学报, 2022, 26(12): 2411-2432.
YANG Yong, SU Zhao, HUANG Shuying, et al. Survey of deep-learning approaches for pixel-level pansharpening[J]. National remote sensing bulletin, 2022, 26(12): 2411-2432.
[2] 赵其昌, 吴一全, 苑玉彬. 光学遥感图像舰船目标检测与识别方法研究进展[J]. 航空学报, 2024, 45(8): 029025.
ZHAO Qichang, WU Yiquan, YUAN Yubin. Progress of ship detection and recognition methods in optical remote sensing images[J]. Acta aeronautica et astronautica sinica, 2024, 45(8): 029025.
[3] LI Zhi, PENG Zhenming, CAO Siying, et al. High-resolution remote sensing change detection based on inverse correction and density peak clustering[C]// 2023 IEEE International Geoscience and Remote Sensing Symposium. Pasadena: IEEE, 2023: 3028-3031.
[4] BAI Jing, LIU Ruotong, ZHAO Haisheng, et al. Hyperspectral image classification using geometric spatial-spectral feature integration: a class incremental learning approach[J]. IEEE transactions on geoscience and remote sensing, 2023, 61: 5531215.
[5] 胡建文, 汪泽平, 胡佩. 基于深度学习的空谱遥感图像融合综述[J]. 自然资源遥感, 2023, 35(1): 1-14.
HU Jianwen, WANG Zeping, HU Pei. A review of spatial-spectral remote sensing image fusion based on deep learning[J]. Remote sensing for natural resources, 2023, 35(1): 1-14.
[6] 徐佳, 关泽群, 何秀凤, 胡俊伟，等. 基于传感器光谱特性的全色与多光谱图像融合[J]. 遥感学报, 2009, 13(1): 97-102.
XU Jia, GUAN Zequn, HE Xiufeng, et al. Panchromatic and multispectral image fusion based on sensor spectral characteristics[J]. Journal of remote sensing, 2009, 13(1): 97-102.
[7] 肖亮, 刘鹏飞, 李恒. 多源空——谱遥感图像融合方法进展与挑战[J]. 中国图象图形学报, 2020, 25(5): 851-863.
XIAO Liang, LIU Pengfei, LI Heng. Advances and challenges in multi-source spatial-spectral remote sensing image fusion methods[J]. Journal of image and graphics, 2020, 25(5): 851-863.
[8] TU Teming, SU Shunchi, SHYU H C, et al. A new look at IHS-like image fusion methods[J]. Information fusion, 2001, 2(3): 177-186.
[9] KWARTENG P, CHAVEZ A. Extracting spectral contrast in Landsat Thematic Mapper image data using selective principal component analysis[J]. Photogrammetric engineering & remote sensing, 1989, 55: 339-348.
[10] MALLAT S G. A theory for multiresolution signal decomposition: the wavelet representation[J]. IEEE transactions on pattern analysis and machine intelligence, 1989, 11(7): 674-693.
[11] LIU Pengfei. Pansharpening with spatial hessian non-convex sparse and spectral gradient low rank priors[J]. IEEE transactions on image processing, 2023, 32: 2120-2131.
[12] ZHU Xiaoxiang, GROHNFELDT C, BAMLER R. Exploiting joint sparsity for pansharpening: the J-SparseFI algorithm[J]. IEEE transactions on geoscience and remote sensing, 2016, 54(5): 2664-2681.
[13] LIU Pengfei, TANG Songze, HUANG Lili. Pansharpening with spatial hyper-Laplacian and spectral sparse constraints[C]// 2022 IEEE International Geoscience and Remote Sensing Symposium. Kuala Lumpur: IEEE, 2022: 3762-3765.
[14] MASI G, COZZOLINO D, VERDOLIVA L, et al. Pansharpening by convolutional neural networks[J]. Remote sensing, 2016, 8(7): 594.
[15] LIU Xiangyu, LIU Qingjie, WANG Yunhong. Remote sensing image fusion based on two-stream fusion network[J]. Information fusion, 2020, 55: 1-15.
[16] YANG Junfeng, FU Xueyang, HU Yuwen, et al. PanNet: a deep network architecture for pan-sharpening[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 1753-1761.
[17] LIU Xiangyu, WANG Yunhong, LIU Qingjie. Psgan: a generative adversarial network for remote sensing image pan-sharpening[C]//2018 25th IEEE International Conference on Image Processing. Athens: IEEE, 2018: 873-877.
[18] SHI Cheng, PUN C M. Superpixel-based 3D deep neural networks for hyperspectral image classification[J]. Pattern recognition, 2018, 74: 600-616.
[19] AHMAD M, KHAN A M, MAZZARA M, et al. A fast and compact 3-D CNN for hyperspectral image classification[J]. IEEE geoscience and remote sensing letters, 2020, 19: 5502205.
[20] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-09-09)[2024-01-01]. https://arxiv.org/abs/1609.02907v4.
[21] 侯磊, 刘金环, 于旭, 等. 图神经网络研究综述[J]. 计算机科学, 2024, 51(6): 282-298.
HOU Lei, LIU Jinhuan, YU Xu, et al. Review of graph neural networks[J]. Computer science, 2024, 51(6): 282-298.
[22] XU Boyan, YIN Hujun. Graph convolutional networks in feature space for image deblurring and super-resolution[C]//2021 International Joint Conference on Neural Networks. Shenzhen: IEEE, 2021: 1-8.
[23] YAN Keyu, ZHOU Man, LIU Liu, et al. When pansharpening meets graph convolution network and knowledge distillation[J]. IEEE transactions on geoscience and remote sensing, 2022, 60: 5408915.
[24] WANG Xiaolong, GIRSHICK R, GUPTA A, et al. Non-local neural networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7794-7803.
[25] PALSSON F, SVEINSSON J R, ULFARSSON M O, et al. Quantitative quality evaluation of pansharpened imagery: consistency versus synthesis[J]. IEEE transactions on geoscience and remote sensing, 2016, 54(3): 1247-1259.
[26] WANG Wenqing, ZHOU Zhiqiang, ZHANG Xiaoqiao, et al. DiTBN: detail injection-based two-branch network for pansharpening of remote sensing images[J]. Remote sensing, 2022, 14(23): 6120.
[27] VIVONE G, ALPARONE L, CHANUSSOT J, et al. A critical comparison among pansharpening algorithms[J]. IEEE transactions on geoscience and remote sensing, 2015, 53(5): 2565-2586.
[28] KALLEL A. MTF-adjusted pansharpening approach based on coupled multiresolution decompositions[J]. IEEE transactions on geoscience and remote sensing, 2015, 53(6): 3124-3145.
[29] VIVONE G. Robust band-dependent spatial-detail approaches for panchromatic sharpening[J]. IEEE transactions on geoscience and remote sensing, 2019, 57(9): 6421-6433.
[30] ZHOU J, CIVCO D L, SILANDER J A. A wavelet transform method to merge Landsat TM and SPOT panchromatic data[J]. International journal of remote sensing, 1998, 19(4): 743-757.
- 相似文献/References:
-
[1]李洋,焦淑红,孙新童.基于IHS和小波变换的可见光与红外图像融合[J].智能系统学报,2012,7(6):554.
LI Yang,JIAO Shuhong,SUN Xintong.Fusion of visual and infrared images based on IHS and wavelet transforms[J].CAAI Transactions on Intelligent Systems,2012,7():554.
[2]任晓霞,孙秀明,耿鹏,等.多小波和NSDFB组合域递归滤波多聚焦图像融合[J].智能系统学报,2016,11(2):241.[doi:10.11992/tis.201509017]
REN Xiaoxia,SUN Xiuming,GENG Peng,et al.Multifocus image fusion using a recursive filter in the combined domain of multiwavelets and NSDFB[J].CAAI Transactions on Intelligent Systems,2016,11():241.[doi:10.11992/tis.201509017]
[3]王建,吴锡生.基于改进的稀疏表示和PCNN的图像融合算法研究[J].智能系统学报,2019,14(5):922.[doi:10.11992/tis.201805045]
WANG Jian,WU Xisheng.Image fusion based on the improved sparse representation and PCNN[J].CAAI Transactions on Intelligent Systems,2019,14():922.[doi:10.11992/tis.201805045]
[4]姜义,吕荣镇,刘明珠,等.基于生成对抗网络的人脸口罩图像合成[J].智能系统学报,2021,16(6):1073.[doi:10.11992/tis.202012010]
JIANG Yi,LYU Rongzhen,LIU Mingzhu,et al.Masked face image synthesis based on a generative adversarial network[J].CAAI Transactions on Intelligent Systems,2021,16():1073.[doi:10.11992/tis.202012010]
[5]曲海成,王宇萍,谢梦婷,等.结合亮度感知与密集卷积的红外与可见光图像融合[J].智能系统学报,2022,17(3):643.[doi:10.11992/tis.202104004]
QU Haicheng,WANG Yuping,XIE Mengting,et al.Infrared and visible image fusion combined with brightness perception and dense convolution[J].CAAI Transactions on Intelligent Systems,2022,17():643.[doi:10.11992/tis.202104004]
[6]王瑞婷,王海燕,陈晓,等.基于混合卷积与三重注意力的高光谱图像分类网络[J].智能系统学报,2023,18(2):260.[doi:10.11992/tis.202204002]
WANG Ruiting,WANG Haiyan,CHEN Xiao,et al.Hyperspectral image classification based on hybrid convolutional neural network with triplet attention[J].CAAI Transactions on Intelligent Systems,2023,18():260.[doi:10.11992/tis.202204002]
[7]张心祎,谭耀,邢向磊.基于物理先验的深度特征融合水下图像复原[J].智能系统学报,2023,18(6):1185.[doi:10.11992/tis.202304038]
ZHANG Xinyi,TAN Yao,XING Xianglei.Deep feature fusion for underwater-image restoration based on physical priors[J].CAAI Transactions on Intelligent Systems,2023,18():1185.[doi:10.11992/tis.202304038]
[8]陶岩,张辉,黄志鸿,等.面向配电网典型部件的热故障精准判别方法[J].智能系统学报,2025,20(2):506.[doi:10.11992/tis.202311035]
TAO Yan,ZHANG Hui,HUANG Zhihong,et al.Accurate identification of thermal faults for typical components of distribution networks[J].CAAI Transactions on Intelligent Systems,2025,20():506.[doi:10.11992/tis.202311035]
备注/Memo
收稿日期:2024-1-2。
基金项目:国家自然科学基金项目(62376214, 92270117)；陕西省自然科学基础研究计划项目(2023- JC-YB-533).
作者简介:王文卿，副教授，主要研究方向为遥感影像处理与解译、智能信息处理、机器学习。主持国家自然科学基金项目2项，发表学术论文30余篇。E-mail：wangwenqing@xaut.edu.cn。;张小乔，硕士研究生，主要研究方向为遥感图像融合、深度学习、图像处理。E-mail：2210320138@stu.xaut.edu.cn。;刘涵，教授，主要研究方向为复杂工业过程建模与控制、机器学习、人工智能、智能信息处理。主持国家自然科学基金项目3项，发表学术论文160余篇。E-mail: liuhan@xaut.edu.cn。
通讯作者:刘涵. E-mail：liuhan@xaut.edu.cn
更新日期/Last Update:
1900-01-01