[1]丛维仪,郑卓然,贾修一.一种用于联合低光增强和人脸超分的深度学习网络[J].智能系统学报,2025,20(1):109-117.[doi:10.11992/tis.202406029]
CONG Weiyi,ZHENG Zhuoran,JIA Xiuyi.A deep learning network for joint low-light enhancement and face spuer-resolution[J].CAAI Transactions on Intelligent Systems,2025,20(1):109-117.[doi:10.11992/tis.202406029]
点击复制
CONG Weiyi,ZHENG Zhuoran,JIA Xiuyi.A deep learning network for joint low-light enhancement and face spuer-resolution[J].CAAI Transactions on Intelligent Systems,2025,20(1):109-117.[doi:10.11992/tis.202406029]
一种用于联合低光增强和人脸超分的深度学习网络
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第1期
页码:
109-117
栏目:
学术论文—机器感知与模式识别
出版日期:
2025-01-05
- Title:
- A deep learning network for joint low-light enhancement and face spuer-resolution
- Keywords:
- face super resolution; low-light image enhancement; supervised learning; random mask; loss function; deep learning; local feature extraction; global feature extraction
- 分类号:
- TP391.4
- 摘要:
- 在低光环境下，人脸图像增强是许多任务的重要恢复方法。然而，现有的低光环境下人脸超分辨率方法通常依赖于低光增强和超分算法的序列建模。遗憾的是，由于优化目标之间的差异，使用这种方法来增强人脸图像很容易导致伪影或噪声。为了应对这一挑战，本文提出了一个端到端的低光人脸图像超分辨率网络（low-light face super resolution network, LFSRNet）。该网络由浅层特征提取、深层特征提取和特征过滤上采样3个模块组成。首先浅层特征模块将输入的低光、低分辨率人脸图像映射到特征空间。随后，深度特征提取模块对其进行亮度校正并细化结构。最后，特征过滤上采样模块处理提取到的特征并重建人脸图像。此外，为了更好地重建丢失的面部细节本文还设计了一个损失函数faceMaskLoss。大量实验证明了所提模型的有效性。
- Abstract:
- In low-light environments, face image enhancement is used as a vital recovery method for many tasks. However, existing methods for face super-resolution in low-light environments usually relied on sequence modeling that combines low-light enhancement and super-resolution algorithms. Unfortunately, using this method to enhance a face image easily led to artifacts or noise because of the differences between the optimization objectives. To tackle this challenge, we proposed LFSRNet, an end-to-end low-light face image super-resolution network. Our network consisted of three modules: shallow feature extraction, deep feature extraction, and feature filtering upsampling. The shallow feature module initially mapped the input low-light, low-resolution face image into feature space. Subsequently, the deep feature extraction module performed luminance correction and refined the structure. Finally, the feature filtering upsampling module processed the extracted features and reconstructed the face image. Additionally, in order to better reconstruct the lost facial details, we also designed a loss function faceMaskLoss. Extensive experiments demonstrate the effectiveness of our proposed model.
- 参考文献/References:
-
[1] KOWALSKI M, NARUNIEC J, TRZCINSKI T. Deep alignment network: a convolutional neural network for robust face alignment[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 2034-2043.
[2] 贺怀清, 陈琴, 惠康华. 基于深度可分离沙漏网络的快速人脸对齐[J]. 计算机工程与设计, 2021, 42(8): 2316-2323.
HE Huaiqing, CHEN Qin, HUI Kanghua. Fast face alignment based on deep separable hourglass network[J]. Computer engineering and design, 2021, 42(8): 2316-2323.
[3] LIU Yaojie, JOURABLOO A, REN W, et al. Dense face alignment[C]//2017 IEEE International Conference on Computer Vision Workshops. Venice: IEEE, 2017: 1619-1628.
[4] LIN Jinpeng, YANG Hao, CHEN Dong, et al. Face parsing with RoI tanh-warping[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5647-5656.
[5] LIU Yinglu, SHI Hailin, SHEN Hao, et al. A new dataset and boundary-attention semantic segmentation for face parsing[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2020, 34(7): 11637-11644.
[6] ZHENG Qingping, DENG Jiankang, ZHU Zheng, et al. Decoupled multi-task learning with cyclical self-regulation for face parsing[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 4146-4155.
[7] 屈东东, 贺利乐, 何林. 改进的轻量化人脸识别算法[J]. 智能系统学报, 2023, 18(3): 544-551.
QU Dongdong, HE Lile, HE Lin. Improved lightweight face recognition algorithm[J]. CAAI transactions on intelligent systems, 2023, 18(3): 544-551.
[8] 狄岚, 矫慧文, 梁久祯. 稀疏综合字典学习的小样本人脸识别[J]. 智能系统学报, 2021, 16(2): 218-227.
DI Lan, JIAO Huiwen, LIANG Jiuzhen. Sparse comprehensive dictionary learning for small-sample face recognition[J]. CAAI transactions on intelligent systems, 2021, 16(2): 218-227.
[9] 章毅, 吕嘉仪, 兰星, 等. 结合面部动作单元感知的三维人脸重建算法[J]. 软件学报, 2024, 35(5): 2176-2191.
ZHANG Yi, LYU Jiayi, LAN Xing, et al. AU-aware algorithm for 3D facial reconstruction[J]. Journal of software, 2024, 35(5): 2176-2191.
[10] 黄丰智, 范鹏, 范智瀚, 等. 基于散斑结构光编码的三维人脸重建方法研究[J]. 计算机应用与软件, 2023, 40(9): 218-221,228.
HUANG Fengzhi, FAN Peng, FAN Zhihan, et al. 3d face reconstruction method based on speckle structured light coding[J]. Computer applications and software, 2023, 40(9): 218-221,228.
[11] ZHANG Tianke, CHU Xuangeng, LIU Yunfei, et al. Accurate 3D face reconstruction with facial component tokens[C]//2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 8999-9008.
[12] CHEN Chaofeng, GONG Dihong, WANG Hao, et al. Learning spatial attention for face super-resolution[J]. IEEE transactions on image processing, 2021, 30: 1219-1231.
[13] CHEN Yu, TAI Ying, LIU Xiaoming, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2492-2501.
[14] GAO Guangwei, XU Zixiang, LI Juncheng, et al. CTCNet: a CNN-Transformer cooperation network for face image super-resolution[J]. IEEE transactions on image processing, 2023, 32: 1978-1991.
[15] GUO Kehua, HU Min, REN Sheng, et al. Deep illumination-enhanced face super-resolution network for low-light images[J]. ACM transactions on multimedia computing, communications, and applications, 2022, 18(3): 1-19.
[16] DING Xin, HU Ruimin, WANG Zhongyuan. Face enhancement and hallucination in the wild[J]. Neural computing and applications, 2023, 35(3): 2399-2412.
[17] ABDULLAH-AL-WADUD M, KABIR M H, ALI AKBER DEWAN M, et al. A dynamic histogram equalization for image contrast enhancement[C]//2007 Digest of Technical Papers International Conference on Consumer Electronics. Las Vegas: IEEE, 53(2): 593-600.
[18] GUO Xiaojie, LI Yu, LING Haibin. LIME: low-light image enhancement via illumination map estimation[J]. IEEE transactions on image processing, 2017, 26(2): 982-993.
[19] IBRAHIM H, PIK KONG N S. Brightness preserving dynamic histogram equalization for image contrast enhancement[J]. IEEE transactions on consumer electronics, 2007, 53(4): 1752-1758.
[20] LI Mading, LIU Jiaying, YANG Wenhan, et al. Structure-revealing low-light image enhancement via robust retinex model[J]. IEEE transactions on image processing, 2018, 27(6): 2828-2841.
[21] 赵明华, 汶怡春, 都双丽, 等. 基于照度与场景纹理注意力图的低光图像增强[J]. 中国图象图形学报, 2024, 29(4): 862-874.
ZHAO Minghua, WEN Yichun, DU Shuangli, et al. Low-light image enhancement algorithm based on illumination and scene texture attention map[J]. Journal of image and graphics, 2024, 29(4): 862-874.
[22] 吴巨峰, 赵训刚, 周强, 等. 基于对比学习的低光照图像增强[J]. 计算机科学, 2023, 50(S1): 525-530.
WU Jufeng, ZHAO Xungang, ZHOU Qiang, et al. Contrastive learning for low-light image enhancement[J]. Computer science, 2023, 50(S1): 525-530.
[23] ZHANG Yonghua, GUO Xiaojie, MA Jiayi, et al. Beyond brightening low-light images[J]. International journal of computer vision, 2021, 129(4): 1013-1037.
[24] ZHU Anqi, ZHANG Lin, SHEN Ying, et al. Zero-shot restoration of underexposed images via robust retinex decomposition[C]//2020 IEEE International Conference on Multimedia and Expo. London: IEEE, 2020: 1-6.
[25] LORE K G, AKINTAYO A, SARKAR S. LLNet: a deep autoencoder approach to natural low-light image enhancement[J]. Pattern recognition, 2017, 61: 650-662.
[26] WEI Chen, WANG Wenjing, YANG Wenhan, et al. Deep retinex decomposition for low-light enhancement[C]//Proceedings of 29th British Machine Vision Conference. London: BMVA, 2018: 155-155.
[27] WANG Liwen, LIU Zhisong, SIU Wanchi, et al. Lightening network for low-light image enhancement[J]. IEEE transactions on image processing, 2020, 29: 7984-7996.
[28] HE Jingwen, SHI Wu, CHEN Kai, et al. GCFSR: a generative and controllable face super resolution method without facial and GAN priors[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1879-1888.
[29] 姚肇亮, 刘宇男, 张姗姗, 等. 基于多尺度特征融合的属性已知人脸图像超分辨率[J]. 南京理工大学学报, 2022, 46(5): 571-578.
YAO Zhaoliang, LIU Yunan, ZHANG Shanshan, et al. Multi-scale feature fusion based attribute-aware face image super-resolution[J]. Journal of Nanjing University of Science and Technology, 2022, 46(5): 571-578.
[30] 陈泓佑, 陈帆, 和红杰, 等. 基于多任务对抗和抗噪对抗学习的人脸超分辨率算法[J]. 模式识别与人工智能, 2022, 35(10): 863-880.
CHEN Hongyou, CHEN Fan, HE Hongjie, et al. Face super-resolution algorithm based on multi-task adversarial and antinoise adversarial learning[J]. Pattern recognition and artificial intelligence, 2022, 35(10): 863-880.
[31] MENON S, DAMIAN A, HU Shijia, et al. PULSE: self-supervised photo upsampling via latent space exploration of generative models[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2434-2442.
[32] WANG Chenyang, JIANG Junjun, ZHONG Zhiwei, et al. Spatial-frequency mutual learning for face super-resolution[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22356-22366.
[33] LU Tao, WANG Yuanzhi, ZHANG Yanduo, et al. Face hallucination via split-attention in split-attention network[C]//Proceedings of the 29th ACM International Conference on Multimedia. Virtual Event: ACM, 2021: 5501-5509.
[34] YANG Tao, REN Peiran, XIE Xuansong, et al. GAN prior embedded network for blind face restoration in the wild[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 672-681.
[35] WANG Yuanzhi, LU Tao, YAO Yuan, et al. Learning to hallucinate face in the dark[J]. IEEE transactions on multimedia, 2023, 26: 2314-2326.
[36] HAN Kai, WANG Yunhe, CHEN Hanting, et al. A survey on vision transformer[J]. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(1): 87-110.
[37] NEWELL A, YANG Kaiyu, DENG Jia. Stacked hourglass networks for human pose estimation[C]//Proceedings of the 14th European Conference on Computer Vision. London: Springer, 2016: 483-499.
[38] TU Zhengzhong, TALEBI H, ZHANG Han, et al. MaxViT: multi-axis vision Transformer[C]//Proceedings of the 17th European Conference on Computer Vision. London: Springer, 2022: 459-479.
[39] KONG Fangyuan, LI Mingxi, LIU Songwei, et al. Residual local feature network for efficient super-resolution[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New Orleans: IEEE, 2022: 765-775.
[40] LIU Jie, ZHANG Wenjie, TANG Yuting, et al. Residual feature aggregation network for image super-resolution[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2356-2365.
[41] WANG Qilong, WU Banggu, ZHU Pengfei, et al. ECA-net: efficient channel attention for deep convolutional neural networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11531-11539.
[42] ZHANG Kaipeng, ZHANG Zhanpeng, LI Zhifeng, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE signal processing letters, 2016, 23(10): 1499-1503.
[43] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1132-1140.
[44] LIANG Jingyun, CAO Jiezhang, SUN Guolei, et al. SwinIR: image restoration using swin transformer[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021: 1833-1844.
[45] WANG Hang, CHEN Xuanhong, NI Bingbing, et al. Omni aggregation networks for lightweight image super-resolution[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22378-22387.
[46] RASHEED M T, SHI Daming. LSR: Lightening super-resolution deep network for low-light image enhancement[J]. Neurocomputing, 2022, 505: 263-275.
备注/Memo
收稿日期:2024-6-18。
基金项目:国家自然科学基金项目(62176123, 62476130).
作者简介:丛维仪，硕士研究生，主要研究方向为图像增强。E-mail：congweiyi@njust.edu.cn。;郑卓然，博士研究生，主要研究方向为深度学习和图像增强。E-mail：zhengzr@njust.edu.cn。;贾修一，教授，博士生导师， 中国计算机学会杰出会员，主要研究方向为机器学习、粒计算和计算机视觉。主持国家自然科学基金项目4项。发表学术论文100余篇。E-mail：jiaxy@njust.edu.cn。
通讯作者:贾修一. E-mail：jiaxy@njust.edu.cn
更新日期/Last Update:
2025-01-05