[1]高涛,杨朝晨,陈婷,等.深度多尺度融合注意力残差人脸表情识别网络[J].智能系统学报,2022,17(2):393-401.[doi:10.11992/tis.202107028]
GAO Tao,YANG Zhaochen,CHEN Ting,et al.Deep multiscale fusion attention residual network for facial expression recognition[J].CAAI Transactions on Intelligent Systems,2022,17(2):393-401.[doi:10.11992/tis.202107028]
点击复制
GAO Tao,YANG Zhaochen,CHEN Ting,et al.Deep multiscale fusion attention residual network for facial expression recognition[J].CAAI Transactions on Intelligent Systems,2022,17(2):393-401.[doi:10.11992/tis.202107028]
深度多尺度融合注意力残差人脸表情识别网络
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
17
期数:
2022年第2期
页码:
393-401
栏目:
学术论文—人工智能基础
出版日期:
2022-03-05
- Title:
- Deep multiscale fusion attention residual network for facial expression recognition
- Keywords:
- facial expression recognition; residual network; multiscale features; attention mechanism; occlusion of human faces; convolution neural network; feature fusion; deep learning
- 分类号:
- TP391
- 摘要:
- 针对人脸表情呈现方式多样化以及人脸表情识别易受光照、姿势、遮挡等非线性因素影响的问题，提出了一种深度多尺度融合注意力残差网络(deep multi-scale fusion attention residual network, DMFA-ResNet)。该模型基于ResNet-50残差网络，设计了新的注意力残差模块，由7个具有三条支路的注意残差学习单元构成，能够对输入图像进行并行多卷积操作，以获得多尺度特征，同时引入注意力机制，突出重点局部区域，有利于遮挡图像的特征学习。通过在注意力残差模块之间增加过渡层以去除冗余信息，简化网络复杂度，在保证感受野的情况下减少计算量，实现网络抗过拟合效果。在3组数据集上的实验结果表明，本文提出的算法均优于对比的其他先进方法。
- Abstract:
- This paper proposes a deep multiscale fusion attention residual network based on the ResNet-50 model to solve the problems of the diversification of facial expression presentation and the susceptibility of facial expression recognition to nonlinear factors, such as illumination, posture, and occlusion. A novel attention residual module consisting of seven attention residual learning units with three branches is designed to perform multiple convolution operations on the input image in parallel and obtain multiscale features. To highlight important local areas, the attention mechanism is introduced simultaneously, which is conducive to the feature learning of the occluded images. Furthermore, a novel transition layer is added between the attention residual modules to remove redundant information, simplify the network complexity, reduce the amount of calculation while ensuring the receptive field, and realize the anti-overfitting effect of the network. Experimental results on three datasets demonstrate that the proposed algorithm is superior to other advanced methods.
- 参考文献/References:
-
[1] BEN Xianye, REN Yi, ZHANG Junping, et al. Video-based Facial micro-expression analysis: a survey of datasets, features and algorithms[EB/OL].(2021-03-19)[2021-05-01].https://arxiv.org/abs/2201.12728v1.
[2] CHEN Boyu, GUAN Wenlong, LI Peixia, et al. Residual multi-task learning for facial landmark localization and expression recognition[EB/OL].(2021-07-01)[2021-07-05].https://www.sciencedirect.com/science/article/pii/S0031320321000807.
[3] LI Shan, DENG Weihong. Deep facial expression recognition: a survey[EB/OL].（2020-03-17）[2021-05-01]. https://ieeexplore.ieee.org/document/9039580.
[4] ZHAO Guoying, PIETIKAINEN M. Dynamic texture recognition using local binary patterns with an application to facial expressions[J]. IEEE transactions on pattern analysis and machine intelligence, 2007, 29(6): 915–928.
[5] WHITEHILL J, OMLIN C W. Haar features for FACS AU recognition[C]//Proceedings of the 7th International Conference on Automatic Face and Gesture Recognition. Southampton, UK, 2006: 5?101.
[6] BARTLETT M S, LITTLEWORT G, FRANK M, et al. Recognizing facial expression: machine learning and application to spontaneous behavior[C]//Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. San Diego, USA, 2005: 568?573.
[7] LI Xiaobai, PFISTER T, HUANG Xiaohua, et al. A spontaneous micro-expression database: inducement, collection and baseline[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). Shanghai, China, 2013: 1?6.
[8] RIVERA A R, CASTILLO J R, CHA E O O. Local directional number pattern for face analysis: face and expression recognition[J]. IEEE transactions on image processing, 2013, 22(5): 1740–1752.
[9] KIM T H, YU C, LEE S W. Facial expression recognition using feature additive pooling and progressive fine-tuning of CNN[J]. Electronics letters, 2018, 54(23): 1326–1328.
[10] AN Fengping, LIU Zhiwen. Facial expression recognition algorithm based on parameter adaptive initialization of CNN and LSTM[J]. The visual computer, 2020, 36(3): 483–498.
[11] XIE Siyue, HU Haifeng, WU Yongbo. Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition[J]. Pattern recognition, 2019, 92: 177–191.
[12] WANG Kai, PENG Xiaojiang, YANG Jianfei, et al. Suppressing uncertainties for large-scale facial expression recognition[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA, 2020: 6897?6906.
[13] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA, 2016: 770?778.
[14] LI Yong, ZENG Jiabei, SHAN Shiguang, et al. Occlusion aware facial expression recognition using CNN with attention mechanism[J]. IEEE transactions on image processing, 2019, 28(5): 2439–2450.
[15] LIU Yuanyuan, YUAN Xiaohui, GONG Xi, et al. Conditional convolution neural network enhanced random forest for facial expression recognition[J]. Pattern recognition, 2018, 84: 251–261.
[16] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 7132?7141.
[17] 江静, 邓伟洪. 持续学习改进的人脸表情识别[J]. 中国图象图形学报, 2020, 25(11): 2361–2369
JIANG Jing, DENG Weihong. Facial expression recognition improved by continual learning[J]. Journal of image and graphics, 2020, 25(11): 2361–2369
[18] 王善敏, 帅惠, 刘青山. 关键点深度特征驱动人脸表情识别[J]. 中国图象图形学报, 2020, 25(4): 813–823
WANG Shanmin, SHUAI Hui, LIU Qingshan. Facial expression recognition based on deep facial landmark features[J]. Journal of image and graphics, 2020, 25(4): 813–823
[19] 张文萍, 贾凯, 王宏玉, 等. 改进的Island损失函数在人脸表情识别上的应用[J]. 计算机辅助设计与图形学学报, 2020, 32(12): 1910–1917
ZHANG Wenping, JIA Kai, WANG Hongyu, et al. Application of improved Island loss in facial expression recognition[J]. Journal of computer-aided design & computer graphics, 2020, 32(12): 1910–1917
[20] LIU Xiaofeng, KUMAR B V K V, JIA Ping, et al. Hard negative generation for identity-disentangled facial expression recognition[J]. Pattern recognition, 2019, 88: 1–12.
[21] YANG Huiyuan, CIFTCI U, YIN Lijun. Facial expression recognition by de-expression residue learning[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018.
[22] XIE Siyue, HU Haifeng. Facial expression recognition using hierarchical features with deep comprehensive multipatches aggregation convolutional neural networks[J]. IEEE transactions on multimedia, 2019, 21(1): 211–220.
[23] LOPES A T, DE AGUIAR E, DE SOUZA A F, et al. Facial expression recognition with convolutional neural networks: coping with few data and the training sample order[J]. Pattern recognition, 2017, 61: 610–628.
[24] ZHANG Hepeng, HUANG Bin, TIAN Guohui. Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture[J]. Pattern recognition letters, 2020, 131: 128–134.
[25] YANG Biao, CAO Jinmeng, NI Rongrong, et al. Facial expression recognition using weighted mixture deep neural network based on double-channel facial images[J]. IEEE access, 2017, 6: 4630–4640.
[26] KIM J H, KIM B G, ROY P P, et al. Efficient facial expression recognition algorithm based on hierarchical deep neural network structure[J]. IEEE access, 2019, 7: 41273–41285.
- 相似文献/References:
-
[1]孙正兴,徐文晖.基于局部SVM分类器的表情识别方法[J].智能系统学报,2008,3(5):455.
SUN Zheng-xing,XU Wen-hui.Fac ial expression recogn ition based on local SVM classif iers[J].CAAI Transactions on Intelligent Systems,2008,3():455.
[2]彭程,刘帅师,万川,等.基于局部纹理ASM模型的人脸表情识别[J].智能系统学报,2011,6(3):231.
PENG Cheng,LIU Shuaishi,WAN Chuan,et al.An active shape model for facial expression recognition based on a local texture model[J].CAAI Transactions on Intelligent Systems,2011,6():231.
[3]童莹.一种方向性的局部二值模式在人脸表情识别中的应用[J].智能系统学报,2015,10(3):422.[doi:10.3969/j.issn.1673-4785.201405016]
TONG Ying.Local binary pattern based on the directions and its application in facial expression recognition[J].CAAI Transactions on Intelligent Systems,2015,10():422.[doi:10.3969/j.issn.1673-4785.201405016]
[4]曹锦纲,李金华,郑顾平.基于生成式对抗网络的道路交通模糊图像增强[J].智能系统学报,2020,15(3):491.[doi:10.11992/tis.201903041]
CAO Jin gang,LI Jinhua,ZHENG Guping.Enhancement of blurred road-traffic images based on generative adversarial network[J].CAAI Transactions on Intelligent Systems,2020,15():491.[doi:10.11992/tis.201903041]
[5]林丽惠,罗志明,王军政,等.融合整体与局部信息的武夷岩茶叶片分类方法[J].智能系统学报,2020,15(5):919.[doi:10.11992/tis.202003018]
LIN Lihui,LUO Zhiming,WANG Junzheng,et al.Classification of Wuyi rock tealeaves by integrating global and local information[J].CAAI Transactions on Intelligent Systems,2020,15():919.[doi:10.11992/tis.202003018]
[6]刘万军,佟畅,曲海成.空洞卷积与注意力融合的对抗式图像阴影去除算法[J].智能系统学报,2021,16(6):1081.[doi:10.11992/tis.202011022]
LIU Wanjun,TONG Chang,QU Haicheng.An antagonistic image shadow removal algorithm based on dilated convolution and attention mechanism[J].CAAI Transactions on Intelligent Systems,2021,16():1081.[doi:10.11992/tis.202011022]
[7]张欣培,周尧,章毅.用于胎儿超声切面识别的知识蒸馏方法[J].智能系统学报,2022,17(1):181.[doi:10.11992/tis.202105007]
ZHANG Xinpei,ZHOU Yao,ZHANG Yi.Knowledge distillation method for fetal ultrasound section identification[J].CAAI Transactions on Intelligent Systems,2022,17():181.[doi:10.11992/tis.202105007]
[8]李倩玉,王蓓,金晶,等.基于双向LSTM卷积网络与注意力机制的自动睡眠分期模型[J].智能系统学报,2022,17(3):523.[doi:10.11992/tis.202103013]
LI Qianyu,WANG Bei,JIN Jing,et al.Automatic sleep staging model based on the bi-directional LSTM convolutional network and attention mechanism[J].CAAI Transactions on Intelligent Systems,2022,17():523.[doi:10.11992/tis.202103013]
[9]刘万军,赵思琪,曲海成,等.结合前景特征增强与区域掩码自注意力的细粒度图像分类[J].智能系统学报,2022,17(6):1134.[doi:10.11992/tis.202109029]
LIU Wanjun,ZHAO Siqi,QU Haicheng,et al.Combining foreground feature reinforcement and region mask self-attention for fine-grained image classification[J].CAAI Transactions on Intelligent Systems,2022,17():1134.[doi:10.11992/tis.202109029]
[10]沈鑫,魏利胜.基于注意力残差U-Net的皮肤镜图像分割方法[J].智能系统学报,2023,18(4):699.[doi:10.11992/tis.202201030]
SHEN Xin,WEI Lisheng.Dermoscope image segmentation method based on ARB-UNet[J].CAAI Transactions on Intelligent Systems,2023,18():699.[doi:10.11992/tis.202201030]
[11]梁艳,温兴,潘家辉.融合全局与局部特征的跨数据集表情识别方法[J].智能系统学报,2023,18(6):1205.[doi:10.11992/tis.202212030]
LIANG Yan,WEN Xing,PAN Jiahui.Cross-dataset facial expression recognition method fusing global and local features[J].CAAI Transactions on Intelligent Systems,2023,18():1205.[doi:10.11992/tis.202212030]
备注/Memo
收稿日期:2021-07-16。
基金项目:国家重点研发计划项目(2019YFE0108300);国家自然科学基金项目(62001058);陕西省重点研发计划项目(2019GY-039);长安大学中央高校基本科研业务费专项资金项目(300102241201)
作者简介:高涛，教授，博士，主要研究方向为数字图像处理、模式识别。获得国家专利9项。发表学术论文16篇;杨朝晨，硕士研究生，主要研究方向为数字图像处理、深度学习;陈婷,副教授，博士，主要研究方向为图形图像处理、计算机视觉
通讯作者:陈婷.E-mail:tchenchd@126.com
更新日期/Last Update:
1900-01-01