[1]曲海成,王宇萍,谢梦婷,等.结合亮度感知与密集卷积的红外与可见光图像融合[J].智能系统学报,2022,17(3):643-652.[doi:10.11992/tis.202104004]
QU Haicheng,WANG Yuping,XIE Mengting,et al.Infrared and visible image fusion combined with brightness perception and dense convolution[J].CAAI Transactions on Intelligent Systems,2022,17(3):643-652.[doi:10.11992/tis.202104004]
点击复制
QU Haicheng,WANG Yuping,XIE Mengting,et al.Infrared and visible image fusion combined with brightness perception and dense convolution[J].CAAI Transactions on Intelligent Systems,2022,17(3):643-652.[doi:10.11992/tis.202104004]
结合亮度感知与密集卷积的红外与可见光图像融合
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
17
期数:
2022年第3期
页码:
643-652
栏目:
人工智能院长论坛
出版日期:
2022-05-05
- Title:
- Infrared and visible image fusion combined with brightness perception and dense convolution
- Keywords:
- image fusion; brightness perception; dense convolution network; GAN; infrared image and visible image; information entropy; mutual information; sum of difference and correlation
- 分类号:
- TP391
- 摘要:
- 为解决弱光照条件下红外与可见光图像融合质量差的问题，提出一种结合亮度感知与密集卷积的红外与可见光图像融合方法（brightness perception and dense convolution，BPD-Fusion）。首先，对可见光图像进行亮度计算，得到亮度权重并对其暗区域进行亮度增强；然后，将增强的可见光图像与红外图像级联输入生成器，在其Conv1阶段后嵌入密集卷积以获取更丰富的图像特征；最后，为了达到较强的图像重构与生成能力，建立多损失函数构建端到端的图像融合过程。在TNO和KAIST数据集上进行融合质量测评：主观评价上，提出的方法视觉效果良好；客观评价上，差异相关和、信息熵、互信息和平均梯度指标均优于对比方法。
- Abstract:
- To solve the problem of poor fusion quality of infrared and visible images under weak illumination, we propose an infrared and visible image fusion method (BPD-Fusion) combining brightness perception and DenseNet. First, the brightness of the visible image is evaluated to obtain its weight and enhance the brightness of the dark area. Then, the enhanced visible and infrared images are input into the generator after concatenating, and DenseNet is embedded after the Conv1 stage to obtain more abundant source image features. Finally, to achieve stronger image reconstruction and generation ability, a multi-loss function is established to construct the end-to-end image fusion process. The fusion quality is evaluated on the TNO dataset and the challenging KAIST dataset. In subjective evaluation, a good visual effect is observed in the proposed method. In objective evaluation, the difference correlation sum, information entropy, mutual information, and average gradient of our method are better than those of the contrast method.
- 参考文献/References:
-
[1] 刘帅, 李士进, 冯钧. 多特征融合的遥感图像分类[J]. 数据采集与处理, 2014, 29(1): 108–115
LIU Shuai, LI Shijin, FENG Jun. Remote sensing image classification based on adaptive fusion of multiple features[J]. Journal of data acquisition and processing, 2014, 29(1): 108–115
[2] 孙洁, 黄承宁, 王玉祥. 基于多模态聚类及决策融合的SAR图像分类方法[J]. 现代雷达, 2020, 42(12): 66–71
SUN Jie, HUANG Chengning, WANG Yuxiang. Target classification of SAR images based on clustering of multiple modes and decision fusion[J]. Modern radar, 2020, 42(12): 66–71
[3] 白玉, 侯志强, 刘晓义, 等. 基于可见光图像和红外图像决策级融合的目标检测算法[J]. 空军工程大学学报(自然科学版), 2020, 21(6): 53–59,100
BAI Yu, HOU Zhiqiang, LIU Xiaoyi, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared image[J]. Journal of Air Force Engineering University (natural science edition), 2020, 21(6): 53–59,100
[4] 李舒涵, 许宏科, 武治宇. 基于红外与可见光图像融合的交通标志检测[J]. 现代电子技术, 2020, 43(3): 45–49
LI Shuhan, XU Hongke, WU Zhiyu. Traffic sign detection based on infrared and visible image fusion[J]. Modern electronics technique, 2020, 43(3): 45–49
[5] 马瞳宇, 崔静, 储鼎. 基于WebGL的实景三维场景与视频监控图像融合技术研究[J]. 测绘与空间地理信息, 2020, 43(S1): 80–83
MA Tongyu, CUI Jing, CHU Ding. Research on fusion technology of real 3D scene and video surveillance image based on WebGL[J]. Geomatics & spatial information technology, 2020, 43(S1): 80–83
[6] BAVIRISETTI D P, DHULI R. Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-loeve transform[J]. IEEE sensors journal, 2016, 16(1): 203–209.
[7] Kumar B K S. Image fusion based on pixel significance using cross bilateral filter[J]. Signal, image and video processing, 2015, 9(5): 1193–1204.
[8] MA Jinlei, ZHOU Zhiqiang, WANG Bo, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared physics & technology, 2017, 82: 8–17.
[9] LIU Yu, WANG Zengfu. Simultaneous image fusion and denoising with adaptive sparse representation[J]. IET image processing, 2015, 9(5): 347–357.
[10] Burt P, Adelson E. The laplacian pyramid as a compact image code[J]. IEEE transactions on communications, 1983, 31(4): 532–540.
[11] LI Shutao, YANG Bin, HU Jianwen. Performance comparison of different multi-resolution transforms for image fusion[J]. Information fusion, 2011, 12(2): 74–84.
[12] LI Shutao, YIN Haitao, FANG Leyuan. Group-sparse representation with dictionary learning for medical image denoising and fusion[J]. IEEE transactions on biomedical engineering, 2012, 59(12): 3450–3459.
[13] MA Jiayi, MA Yong, LI Chang. Infrared and visible image fusion methods and applications: a survey[J]. Information fusion, 2019, 45: 153–178.
[14] PRABHAKAR K R, SRIKAR V S, BABU R V. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image Pairs[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 4724–4732.
[15] MA Jiayi, YU Wei, LIANG Pengwei, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information fusion, 2019, 48: 11–26.
[16] LI Hui, WU Xiaojun, KITTLER J. Infrared and visible image fusion using a deep learning framework[C]//Proceedings of 24th International Conference on Pattern Recognition (ICPR). Beijing: IEEE, 2018: 2705–2710.
[17] LI Hui, WU Xiaojun. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE transactions on image processing, 2019, 28(5): 2614–2623.
[18] XU Han, MA Jiayi, JIANG Junjun, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(1): 502–518.
[19] HUANG Gao, LIU Zhuang, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). Honolulu: IEEE, 2017: 2261–2269.
[20] 刘万军, 佟畅, 曲海成. 空洞卷积与注意力融合的对抗式图像阴影去除算法[J]. 智能系统学报, 2021, 16(6): 1081–1089
LIU Wanjun, TONG Chang, QU Haicheng. An antagonistic image shadow removal algorithm based on dilated convolution and attention mechanism[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1081–1089
[21] Alexander T. TNO image fusion dataset[EB/OL]. (2018-09-15)[2021-04-02]https://flgshare.com/articles/TNO Image Fusion Dataset/1008029.
[22] HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston: IEEE, 2015: 1037–1045.
[23] ROBERTS J W, VAN AARDT J, AHMED F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification[J]. Journal of applied remote sensing, 2008, 2(1): 023522.
[24] QU Guihong, ZHANG Dali, YAN Pingfan. Information measure for performance of image fusion[J]. Electronics letters, 2002, 38(7): 313–315.
[25] ASLANTAS V, BENDES E. A new image quality metric for image fusion: the sum of the correlations of differences[J]. AEU-international journal of electronics and communications, 2015, 69(12): 1890–1896.
- 相似文献/References:
-
[1]李洋,焦淑红,孙新童.基于IHS和小波变换的可见光与红外图像融合[J].智能系统学报,2012,7(6):554.
LI Yang,JIAO Shuhong,SUN Xintong.Fusion of visual and infrared images based on IHS and wavelet transforms[J].CAAI Transactions on Intelligent Systems,2012,7():554.
[2]任晓霞,孙秀明,耿鹏,等.多小波和NSDFB组合域递归滤波多聚焦图像融合[J].智能系统学报,2016,11(2):241.[doi:10.11992/tis.201509017]
REN Xiaoxia,SUN Xiuming,GENG Peng,et al.Multifocus image fusion using a recursive filter in the combined domain of multiwavelets and NSDFB[J].CAAI Transactions on Intelligent Systems,2016,11():241.[doi:10.11992/tis.201509017]
[3]王建,吴锡生.基于改进的稀疏表示和PCNN的图像融合算法研究[J].智能系统学报,2019,14(5):922.[doi:10.11992/tis.201805045]
WANG Jian,WU Xisheng.Image fusion based on the improved sparse representation and PCNN[J].CAAI Transactions on Intelligent Systems,2019,14():922.[doi:10.11992/tis.201805045]
[4]姜义,吕荣镇,刘明珠,等.基于生成对抗网络的人脸口罩图像合成[J].智能系统学报,2021,16(6):1073.[doi:10.11992/tis.202012010]
JIANG Yi,LYU Rongzhen,LIU Mingzhu,et al.Masked face image synthesis based on a generative adversarial network[J].CAAI Transactions on Intelligent Systems,2021,16():1073.[doi:10.11992/tis.202012010]
[5]张心祎,谭耀,邢向磊.基于物理先验的深度特征融合水下图像复原[J].智能系统学报,2023,18(6):1185.[doi:10.11992/tis.202304038]
ZHANG Xinyi,TAN Yao,XING Xianglei.Deep feature fusion for underwater-image restoration based on physical priors[J].CAAI Transactions on Intelligent Systems,2023,18():1185.[doi:10.11992/tis.202304038]
备注/Memo
收稿日期:2021-04-02。
基金项目:辽宁工程技术大学学科创新团队资助项目（LNTU20TD-23）；辽宁省教育厅一般项目（LJ2019JL010）.
作者简介:曲海成，副教授，博士，辽宁工程技术大学软件学院副院长，主要研究方向为图像与智能信息处理。主持省自然科学基金项目1项、省教育厅面上项目2项。发表学术论文60余篇;王宇萍，硕士研究生，主要研究方向为图像与智能信息处理;谢梦婷，本科生，主要研究方向为图像处理技术研究
通讯作者:曲海成.E-mail：quhaicheng@lntu.edu.cn
更新日期/Last Update:
1900-01-01