[1]CONG Weiyi,ZHENG Zhuoran,JIA Xiuyi.A deep learning network for joint low-light enhancement and face spuer-resolution[J].CAAI Transactions on Intelligent Systems,2025,20(1):109-117.[doi:10.11992/tis.202406029]
Copy

A deep learning network for joint low-light enhancement and face spuer-resolution

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 20 Number of periods: 2025 1 Page number: 109-117 Column: 学术论文—机器感知与模式识别 Public date: 2025-01-05
Title:
A deep learning network for joint low-light enhancement and face spuer-resolution
Author(s):
CONG Weiyi;  ZHENG Zhuoran;  JIA Xiuyi
School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China
Keywords:
face super resolution; low-light image enhancement; supervised learning; random mask; loss function; deep learning; local feature extraction; global feature extraction
CLC:
TP391.4
DOI:
10.11992/tis.202406029
Abstract:
In low-light environments, face image enhancement is used as a vital recovery method for many tasks. However, existing methods for face super-resolution in low-light environments usually relied on sequence modeling that combines low-light enhancement and super-resolution algorithms. Unfortunately, using this method to enhance a face image easily led to artifacts or noise because of the differences between the optimization objectives. To tackle this challenge, we proposed LFSRNet, an end-to-end low-light face image super-resolution network. Our network consisted of three modules: shallow feature extraction, deep feature extraction, and feature filtering upsampling. The shallow feature module initially mapped the input low-light, low-resolution face image into feature space. Subsequently, the deep feature extraction module performed luminance correction and refined the structure. Finally, the feature filtering upsampling module processed the extracted features and reconstructed the face image. Additionally, in order to better reconstruct the lost facial details, we also designed a loss function faceMaskLoss. Extensive experiments demonstrate the effectiveness of our proposed model.
References:
[1] KOWALSKI M, NARUNIEC J, TRZCINSKI T. Deep alignment network: a convolutional neural network for robust face alignment[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 2034-2043.
[2] 贺怀清, 陈琴, 惠康华. 基于深度可分离沙漏网络的快速人脸对齐[J]. 计算机工程与设计, 2021, 42(8): 2316-2323.
HE Huaiqing, CHEN Qin, HUI Kanghua. Fast face alignment based on deep separable hourglass network[J]. Computer engineering and design, 2021, 42(8): 2316-2323.
[3] LIU Yaojie, JOURABLOO A, REN W, et al. Dense face alignment[C]//2017 IEEE International Conference on Computer Vision Workshops. Venice: IEEE, 2017: 1619-1628.
[4] LIN Jinpeng, YANG Hao, CHEN Dong, et al. Face parsing with RoI tanh-warping[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 5647-5656.
[5] LIU Yinglu, SHI Hailin, SHEN Hao, et al. A new dataset and boundary-attention semantic segmentation for face parsing[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2020, 34(7): 11637-11644.
[6] ZHENG Qingping, DENG Jiankang, ZHU Zheng, et al. Decoupled multi-task learning with cyclical self-regulation for face parsing[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 4146-4155.
[7] 屈东东, 贺利乐, 何林. 改进的轻量化人脸识别算法[J]. 智能系统学报, 2023, 18(3): 544-551.
QU Dongdong, HE Lile, HE Lin. Improved lightweight face recognition algorithm[J]. CAAI transactions on intelligent systems, 2023, 18(3): 544-551.
[8] 狄岚, 矫慧文, 梁久祯. 稀疏综合字典学习的小样本人脸识别[J]. 智能系统学报, 2021, 16(2): 218-227.
DI Lan, JIAO Huiwen, LIANG Jiuzhen. Sparse comprehensive dictionary learning for small-sample face recognition[J]. CAAI transactions on intelligent systems, 2021, 16(2): 218-227.
[9] 章毅, 吕嘉仪, 兰星, 等. 结合面部动作单元感知的三维人脸重建算法[J]. 软件学报, 2024, 35(5): 2176-2191.
ZHANG Yi, LYU Jiayi, LAN Xing, et al. AU-aware algorithm for 3D facial reconstruction[J]. Journal of software, 2024, 35(5): 2176-2191.
[10] 黄丰智, 范鹏, 范智瀚, 等. 基于散斑结构光编码的三维人脸重建方法研究[J]. 计算机应用与软件, 2023, 40(9): 218-221,228.
HUANG Fengzhi, FAN Peng, FAN Zhihan, et al. 3d face reconstruction method based on speckle structured light coding[J]. Computer applications and software, 2023, 40(9): 218-221,228.
[11] ZHANG Tianke, CHU Xuangeng, LIU Yunfei, et al. Accurate 3D face reconstruction with facial component tokens[C]//2023 IEEE/CVF International Conference on Computer Vision. Paris: IEEE, 2023: 8999-9008.
[12] CHEN Chaofeng, GONG Dihong, WANG Hao, et al. Learning spatial attention for face super-resolution[J]. IEEE transactions on image processing, 2021, 30: 1219-1231.
[13] CHEN Yu, TAI Ying, LIU Xiaoming, et al. FSRNet: end-to-end learning face super-resolution with facial priors[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 2492-2501.
[14] GAO Guangwei, XU Zixiang, LI Juncheng, et al. CTCNet: a CNN-Transformer cooperation network for face image super-resolution[J]. IEEE transactions on image processing, 2023, 32: 1978-1991.
[15] GUO Kehua, HU Min, REN Sheng, et al. Deep illumination-enhanced face super-resolution network for low-light images[J]. ACM transactions on multimedia computing, communications, and applications, 2022, 18(3): 1-19.
[16] DING Xin, HU Ruimin, WANG Zhongyuan. Face enhancement and hallucination in the wild[J]. Neural computing and applications, 2023, 35(3): 2399-2412.
[17] ABDULLAH-AL-WADUD M, KABIR M H, ALI AKBER DEWAN M, et al. A dynamic histogram equalization for image contrast enhancement[C]//2007 Digest of Technical Papers International Conference on Consumer Electronics. Las Vegas: IEEE, 53(2): 593-600.
[18] GUO Xiaojie, LI Yu, LING Haibin. LIME: low-light image enhancement via illumination map estimation[J]. IEEE transactions on image processing, 2017, 26(2): 982-993.
[19] IBRAHIM H, PIK KONG N S. Brightness preserving dynamic histogram equalization for image contrast enhancement[J]. IEEE transactions on consumer electronics, 2007, 53(4): 1752-1758.
[20] LI Mading, LIU Jiaying, YANG Wenhan, et al. Structure-revealing low-light image enhancement via robust retinex model[J]. IEEE transactions on image processing, 2018, 27(6): 2828-2841.
[21] 赵明华, 汶怡春, 都双丽, 等. 基于照度与场景纹理注意力图的低光图像增强[J]. 中国图象图形学报, 2024, 29(4): 862-874.
ZHAO Minghua, WEN Yichun, DU Shuangli, et al. Low-light image enhancement algorithm based on illumination and scene texture attention map[J]. Journal of image and graphics, 2024, 29(4): 862-874.
[22] 吴巨峰, 赵训刚, 周强, 等. 基于对比学习的低光照图像增强[J]. 计算机科学, 2023, 50(S1): 525-530.
WU Jufeng, ZHAO Xungang, ZHOU Qiang, et al. Contrastive learning for low-light image enhancement[J]. Computer science, 2023, 50(S1): 525-530.
[23] ZHANG Yonghua, GUO Xiaojie, MA Jiayi, et al. Beyond brightening low-light images[J]. International journal of computer vision, 2021, 129(4): 1013-1037.
[24] ZHU Anqi, ZHANG Lin, SHEN Ying, et al. Zero-shot restoration of underexposed images via robust retinex decomposition[C]//2020 IEEE International Conference on Multimedia and Expo. London: IEEE, 2020: 1-6.
[25] LORE K G, AKINTAYO A, SARKAR S. LLNet: a deep autoencoder approach to natural low-light image enhancement[J]. Pattern recognition, 2017, 61: 650-662.
[26] WEI Chen, WANG Wenjing, YANG Wenhan, et al. Deep retinex decomposition for low-light enhancement[C]//Proceedings of 29th British Machine Vision Conference. London: BMVA, 2018: 155-155.
[27] WANG Liwen, LIU Zhisong, SIU Wanchi, et al. Lightening network for low-light image enhancement[J]. IEEE transactions on image processing, 2020, 29: 7984-7996.
[28] HE Jingwen, SHI Wu, CHEN Kai, et al. GCFSR: a generative and controllable face super resolution method without facial and GAN priors[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 1879-1888.
[29] 姚肇亮, 刘宇男, 张姗姗, 等. 基于多尺度特征融合的属性已知人脸图像超分辨率[J]. 南京理工大学学报, 2022, 46(5): 571-578.
YAO Zhaoliang, LIU Yunan, ZHANG Shanshan, et al. Multi-scale feature fusion based attribute-aware face image super-resolution[J]. Journal of Nanjing University of Science and Technology, 2022, 46(5): 571-578.
[30] 陈泓佑, 陈帆, 和红杰, 等. 基于多任务对抗和抗噪对抗学习的人脸超分辨率算法[J]. 模式识别与人工智能, 2022, 35(10): 863-880.
CHEN Hongyou, CHEN Fan, HE Hongjie, et al. Face super-resolution algorithm based on multi-task adversarial and antinoise adversarial learning[J]. Pattern recognition and artificial intelligence, 2022, 35(10): 863-880.
[31] MENON S, DAMIAN A, HU Shijia, et al. PULSE: self-supervised photo upsampling via latent space exploration of generative models[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2434-2442.
[32] WANG Chenyang, JIANG Junjun, ZHONG Zhiwei, et al. Spatial-frequency mutual learning for face super-resolution[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22356-22366.
[33] LU Tao, WANG Yuanzhi, ZHANG Yanduo, et al. Face hallucination via split-attention in split-attention network[C]//Proceedings of the 29th ACM International Conference on Multimedia. Virtual Event: ACM, 2021: 5501-5509.
[34] YANG Tao, REN Peiran, XIE Xuansong, et al. GAN prior embedded network for blind face restoration in the wild[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 672-681.
[35] WANG Yuanzhi, LU Tao, YAO Yuan, et al. Learning to hallucinate face in the dark[J]. IEEE transactions on multimedia, 2023, 26: 2314-2326.
[36] HAN Kai, WANG Yunhe, CHEN Hanting, et al. A survey on vision transformer[J]. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(1): 87-110.
[37] NEWELL A, YANG Kaiyu, DENG Jia. Stacked hourglass networks for human pose estimation[C]//Proceedings of the 14th European Conference on Computer Vision. London: Springer, 2016: 483-499.
[38] TU Zhengzhong, TALEBI H, ZHANG Han, et al. MaxViT: multi-axis vision Transformer[C]//Proceedings of the 17th European Conference on Computer Vision. London: Springer, 2022: 459-479.
[39] KONG Fangyuan, LI Mingxi, LIU Songwei, et al. Residual local feature network for efficient super-resolution[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New Orleans: IEEE, 2022: 765-775.
[40] LIU Jie, ZHANG Wenjie, TANG Yuting, et al. Residual feature aggregation network for image super-resolution[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 2356-2365.
[41] WANG Qilong, WU Banggu, ZHU Pengfei, et al. ECA-net: efficient channel attention for deep convolutional neural networks[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 11531-11539.
[42] ZHANG Kaipeng, ZHANG Zhanpeng, LI Zhifeng, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE signal processing letters, 2016, 23(10): 1499-1503.
[43] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu: IEEE, 2017: 1132-1140.
[44] LIANG Jingyun, CAO Jiezhang, SUN Guolei, et al. SwinIR: image restoration using swin transformer[C]//2021 IEEE/CVF International Conference on Computer Vision Workshops. Montreal: IEEE, 2021: 1833-1844.
[45] WANG Hang, CHEN Xuanhong, NI Bingbing, et al. Omni aggregation networks for lightweight image super-resolution[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22378-22387.
[46] RASHEED M T, SHI Daming. LSR: Lightening super-resolution deep network for low-light image enhancement[J]. Neurocomputing, 2022, 505: 263-275.
Similar References:

Memo

-

Last Update: 2025-01-05

Copyright © CAAI Transactions on Intelligent Systems