[1]赵文清,许丽娇,陈昊阳,等.多层特征融合与语义增强的盲图像质量评价[J].智能系统学报,2024,19(1):132-141.[doi:10.11992/tis.202301007]
ZHAO Wenqing,XU Lijiao,CHEN Haoyang,et al.Blind image quality assessment based on multi-level feature fusion and semantic enhancement[J].CAAI Transactions on Intelligent Systems,2024,19(1):132-141.[doi:10.11992/tis.202301007]
点击复制
ZHAO Wenqing,XU Lijiao,CHEN Haoyang,et al.Blind image quality assessment based on multi-level feature fusion and semantic enhancement[J].CAAI Transactions on Intelligent Systems,2024,19(1):132-141.[doi:10.11992/tis.202301007]
多层特征融合与语义增强的盲图像质量评价
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第1期
页码:
132-141
栏目:
学术论文—智能系统
出版日期:
2024-01-05
- Title:
- Blind image quality assessment based on multi-level feature fusion and semantic enhancement
- Keywords:
- deep learning; image quality; convolution neural network; feature extraction; channel attention structure; multi-level feature fusion; dilated convolution; triplet loss function
- 分类号:
- TP391.41
- 文献标志码:
- 2023-07-31
- 摘要:
- 针对现有盲图像质量评价算法在面对真实失真图像时性能较差的问题，本文提出多层特征融合和语义信息增强相结合的无参考图像质量评价算法。提取图像的局部和全局失真特征，利用特征融合模块对特征进行多层融合；利用多层扩张卷积增强语义信息，进而指导失真图像到质量分数的映射过程；考虑预测分数和主观分数之间的相对排名关系，对L1损失函数和三元组排名损失函数进行融合，构建新的损失函数Lmix。为了验证本文方法的有效性，在野生图像质量挑战数据集上进行了验证和对比实验，该算法的斯皮尔曼等级相关系数与皮尔逊线性相关系数指标相比原算法分别提升2.3%和2.3%；在康斯坦茨真实图像质量数据数据集和野生图像质量挑战数据集上进行了跨数据集实验，该算法在面对真实失真图像时表现出了良好的泛化性能。
- Abstract:
- Aiming at the low performance of the existing blind image quality assessment algorithm when facing the real distorted images, the paper proposes a new no-reference image quality assessment algorithm, namely multi-level feature fusion and semantic enhancement for NR (MFFSE-NR), which combines multi-level feature fusion and semantic information enhancement. The local and global distortion features of an image are extracted, then a feature fusion module is used to fuse the features in layers. The multi-layer dilated convolution is employed to enhance semantic information and further direct the mapping process from distorted image to quality fraction. Finally, a novel loss function called Lmix is created by combining the triplet ranking loss function and the L1 loss function, taking account of the relative ranking relationship between the predicted score and the subjective score. Validation and comparison experiments carried out on LIVEC dataset show that both the SROCC and PLCC index are improved respectively by 2.3% than the original algorithm; cross-dataset validation on the KonIQ-10k dataset and LIVEC dataset confirm that the proposed algorithm has good generalization ability when dealing with the real distorted images.
- 参考文献/References:
-
[1] 方玉明, 眭相杰, 鄢杰斌, 等. 无参考图像质量评价研究进展[J]. 中国图象图形学报, 2021, 26(2): 265–286
FANG Yuming, SUI Xiangjie, YAN Jiebin, et al. Progress in no-reference image quality assessment[J]. Journal of image and graphics, 2021, 26(2): 265–286
[2] 曹玉东, 刘海燕, 贾旭, 等. 基于深度学习的图像质量评价方法综述[J]. 计算机工程与应用, 2021, 57(23): 27–36
CAO Yudong, LIU Haiyan, JIA Xu, et al. Overview of image quality assessment method based on deep learning[J]. Computer engineering and applications, 2021, 57(23): 27–36
[3] 毕晓君, 潘梦迪. 基于生成对抗网络的机载遥感图像超分辨率重建[J]. 智能系统学报, 2020, 15(1): 74–83
BI Xiaojun, PAN Mengdi. Super-resolution reconstruction of airborne remote sensing images based on the generative adversarial networks[J]. CAAI transactions on intelligent systems, 2020, 15(1): 74–83
[4] WANG Zhou, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE transactions on image processing:a publication of the IEEE signal processing society, 2004, 13(4): 600–612.
[5] GAO Xinbo, LU Wen, TAO Dacheng, et al. Image quality assessment based on multiscale geometric analysis[J]. IEEE transactions on image processing, 2009, 18(7): 1409–1423.
[6] 王志明. 无参考图像质量评价综述[J]. 自动化学报, 2015, 41(6): 1062–1079
WANG Zhiming. Review of no-reference image quality assessment[J]. Acta automatica sinica, 2015, 41(6): 1062–1079
[7] KANG Le, YE Peng, LI Yi, et al. Convolutional neural networks for no-reference image quality assessment[C]//2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2014: 1733?1740.
[8] XU Jiangtao, YE Peng, LI Qiaohong, et al. Blind image quality assessment based on high order statistics aggregation[J]. IEEE transactions on image processing:a publication of the IEEE signal processing society, 2016, 25(9): 4444–4457.
[9] 严浙平, 曲思瑜, 邢文. 水下图像增强方法研究综述[J]. 智能系统学报, 2022, 17(5): 860–873
YAN Zheping, QU Siyu, XING Wen. An overview of underwater image enhancement methods[J]. CAAI transactions on intelligent systems, 2022, 17(5): 860–873
[10] BOSSE S, MANIRY D, MüLLER K R, et al. Deep neural networks for no-reference and full-reference image quality assessment[J]. IEEE transactions on image processing, 2017, 27(1): 206–219.
[11] WU Jinjian, MA Jupo, LIANG Fuhu, et al. End-to-end blind image quality prediction with cascaded deep neural network[J]. IEEE transactions on image processing, 2020, 29: 7414–7426.
[12] ZHANG Weixia, MA Kede, YAN Jia, et al. Blind image quality assessment using a deep bilinear convolutional neural network[J]. IEEE transactions on circuits and systems for video technology, 2020, 30(1): 36–47.
[13] 鄢杰斌, 方玉明, 刘学林. 图像质量评价研究综述: 从失真的角度[J]. 中国图象图形学报, 2022, 27(5): 1430–1466
YAN Jiebin, FANG Yuming, LIU Xuelin. The review of distortion-related image quality assessment[J]. Journal of image and graphics, 2022, 27(5): 1430–1466
[14] SU Shaolin, YAN Qingsen, ZHU Yu, et al. Blindly assess image quality in the wild guided by a self-adaptive hyper network[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 3664?3673.
[15] 杨春玲, 杨雅静. 基于多尺度特征逐层融合深度神经网络的无参考图像质量评价方法[J]. 华南理工大学学报(自然科学版), 2022, 50(4): 81–89,141
YANG Chunling, YANG Yajing. A deep neural network based on layer-by-layer fusion of multi-scale features for no-reference image quality assessment[J]. Journal of South China University of Technology (natural science edition), 2022, 50(4): 81–89,141
[16] YING Zhenqiang, NIU Haoran, GUPTA P, et al. From patches to pictures (PaQ-2-PiQ): mapping the perceptual space of picture quality[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 3572?3582.
[17] GHADIYARAM D, BOVIK A C. Massive online crowdsourced study of subjective and objective picture quality[J]. IEEE transactions on image processing, 2015, 25(1): 372–387.
[18] HOSU V, LIN Hanhe, SZIRANYI T, et al. KonIQ-10k: an ecologically valid database for deep learning of blind image quality assessment[J]. IEEE transactions on image processing, 2020, 29: 4041–4056.
[19] 刘玉珍, 刘美怡, 林森等. 多尺度特征融合注意力网络的水下图像增强[J]. 计算机辅助设计与图形学学报, 2023, 35(5): 685–695
LIU Yuzhen, LIU Meiyi, LIN Sen, et al. Underwater image enhancement based on multi-scale feature fusion and attention network[J]. Journal of computer-aided design & computer graphics, 2023, 35(5): 685–695
[20] GOLESTANEH S A, DADSETAN S, KITANI K M. No-reference image quality assessment via transformers, relative ranking, and self-consistency[C]//2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Piscataway: IEEE, 2022: 3989?3999.
[21] GAO Shanghua, CHENG Mingming, ZHAO Kai, et al. Res2Net: a new multi-scale backbone architecture[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 43(2): 652–662.
[22] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2016: 770?778.
[23] SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet: a unified embedding for face recognition and clustering[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition . Piscataway: IEEE, 2015: 815?823.
[24] LI Dingquan, JIANG Tingting, JIANG Ming. Exploiting high-level semantics for no-reference image quality assessment of realistic blur images[C]//Proceedings of the 25th ACM international conference on Multimedia. New York: ACM, 2017: 378?386.
[25] 高涛, 杨朝晨, 陈婷, 等. 深度多尺度融合注意力残差人脸表情识别网络[J]. 智能系统学报, 2022, 17(2): 393–401
GAO Tao, YANG Zhaochen, CHEN Ting, et al. Deep multiscale fusion attention residual network for facial expression recognition[J]. CAAI transactions on intelligent systems, 2022, 17(2): 393–401
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]丁科,谭营.GPU通用计算及其在计算智能领域的应用[J].智能系统学报,2015,10(1):1.[doi:10.3969/j.issn.1673-4785.201403072]
DING Ke,TAN Ying.A review on general purpose computing on GPUs and its applications in computational intelligence[J].CAAI Transactions on Intelligent Systems,2015,10():1.[doi:10.3969/j.issn.1673-4785.201403072]
[3]马晓,张番栋,封举富.基于深度学习特征的稀疏表示的人脸识别方法[J].智能系统学报,2016,11(3):279.[doi:10.11992/tis.201603026]
MA Xiao,ZHANG Fandong,FENG Jufu.Sparse representation via deep learning features based face recognition method[J].CAAI Transactions on Intelligent Systems,2016,11():279.[doi:10.11992/tis.201603026]
[4]刘帅师,程曦,郭文燕,等.深度学习方法研究新进展[J].智能系统学报,2016,11(5):567.[doi:10.11992/tis.201511028]
LIU Shuaishi,CHENG Xi,GUO Wenyan,et al.Progress report on new research in deep learning[J].CAAI Transactions on Intelligent Systems,2016,11():567.[doi:10.11992/tis.201511028]
[5]马世龙,乌尼日其其格,李小平.大数据与深度学习综述[J].智能系统学报,2016,11(6):728.[doi:10.11992/tis.201611021]
MA Shilong,WUNIRI Qiqige,LI Xiaoping.Deep learning with big data: state of the art and development[J].CAAI Transactions on Intelligent Systems,2016,11():728.[doi:10.11992/tis.201611021]
[6]王亚杰,邱虹坤,吴燕燕,等.计算机博弈的研究与发展[J].智能系统学报,2016,11(6):788.[doi:10.11992/tis.201609006]
WANG Yajie,QIU Hongkun,WU Yanyan,et al.Research and development of computer games[J].CAAI Transactions on Intelligent Systems,2016,11():788.[doi:10.11992/tis.201609006]
[7]黄心汉.A3I:21世纪科技之光[J].智能系统学报,2016,11(6):835.[doi:10.11992/tis.201605022]
HUANG Xinhan.A3I: the star of science and technology for the 21st century[J].CAAI Transactions on Intelligent Systems,2016,11():835.[doi:10.11992/tis.201605022]
[8]宋婉茹,赵晴晴,陈昌红,等.行人重识别研究综述[J].智能系统学报,2017,12(6):770.[doi:10.11992/tis.201706084]
SONG Wanru,ZHAO Qingqing,CHEN Changhong,et al.Survey on pedestrian re-identification research[J].CAAI Transactions on Intelligent Systems,2017,12():770.[doi:10.11992/tis.201706084]
[9]杨梦铎,栾咏红,刘文军,等.基于自编码器的特征迁移算法[J].智能系统学报,2017,12(6):894.[doi:10.11992/tis.201706037]
YANG Mengduo,LUAN Yonghong,LIU Wenjun,et al.Feature transfer algorithm based on an auto-encoder[J].CAAI Transactions on Intelligent Systems,2017,12():894.[doi:10.11992/tis.201706037]
[10]王科俊,赵彦东,邢向磊.深度学习在无人驾驶汽车领域应用的研究进展[J].智能系统学报,2018,13(1):55.[doi:10.11992/tis.201609029]
WANG Kejun,ZHAO Yandong,XING Xianglei.Deep learning in driverless vehicles[J].CAAI Transactions on Intelligent Systems,2018,13():55.[doi:10.11992/tis.201609029]
备注/Memo
收稿日期:2023-01-09。
基金项目:国家自然科学基金项目(61773160, 61871182)；河北省自然科学基金项目(F2021502013)；中央高校基本科研业务费项目(2020MS153，2021PT018).
作者简介:赵文清，教授，主要研究方向为人工智能与图像处理。主持和参与国家自然科学基金项目、河北省自然科学基金项目、中央高校基本科研业务费专项资金资助项目和国家电网科技项目等10余项。获河北省科技进步二等奖、三等奖各1项。发表学术论文80余篇， 出版专著1部。E-mai：jbzwq@126.com。;许丽娇，硕士研究生，主要研究方向为图像质量评价。E-mai：xulijiao2021@163.com。;陈昊阳，硕士研究生，主要研究方向为图像质量评价。E-mai：870715478@qq.com。
通讯作者:赵文清. E-mail：jbzwq@126.com
更新日期/Last Update:
1900-01-01