[1]陈斌,朱晋宁.双流增强融合网络微表情识别[J].智能系统学报,2023,18(2):360-371.[doi:10.11992/tis.202109036]
CHEN Bin,ZHU Jinning.Micro-expression recognition based on a dual-stream enhanced fusion network[J].CAAI Transactions on Intelligent Systems,2023,18(2):360-371.[doi:10.11992/tis.202109036]
点击复制
CHEN Bin,ZHU Jinning.Micro-expression recognition based on a dual-stream enhanced fusion network[J].CAAI Transactions on Intelligent Systems,2023,18(2):360-371.[doi:10.11992/tis.202109036]
双流增强融合网络微表情识别
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
18
期数:
2023年第2期
页码:
360-371
栏目:
学术论文—机器感知与模式识别
出版日期:
2023-05-05
- Title:
- Micro-expression recognition based on a dual-stream enhanced fusion network
- Keywords:
- micro-expression; dual-stream network; generative adversarial network; data enhancement; fusion of features; pattern identification; convolutional neural network; cycle constraint
- 分类号:
- TP39; TH691.9
- 摘要:
- 为解决微表情识别领域数据集样本数量少，样本类型分布不均导致识别率鲁棒性差的问题，提出了一种基于双流增强网络的微表情识别模型。该模型基于单帧RGB图像流及光流图像流的双流卷积神经网络，以权威数据集为基础，数据增强为基准，构建微表情识别模型。通过在SoftMax逻辑回归层融合单帧空域信息和光流时域信息，对两个独立流的网络性能进行提升，并通过引入基于带循环约束的生成对抗网络的图像生成方式对数据集进行扩充。通过将输入微表情视频帧序列进行分解，将其分割为双流网络的灰度单帧序列与光流单帧序列，对两类序列图进行数据增强，再进行微表情识别模型构建的方法，有效提高了微表情识别率。基于双流增强网络的微表情识别模型可以较好提升微表情识别准确度，鲁棒性较好，泛化状态较稳定。
- Abstract:
- We propose a micro-expression recognition model based on a dual-stream enhanced network in order to address the issues of insufficient samples from the dataset of micro-expression and uneven distribution of sample types leading to a low rate of robustness. Targeting a dual-stream convolutional neural network of single-frame RGB image flow and optical image flow, a micro-expression recognition model is built based on a fundamental authoritative dataset and data enhancement. Single-frame airspace information and optical time flow domain information are incorporated in the SoftMax logistic regression layer to improve the network performance for two independent streams. The dataset is augmented by introducing a method for image generation based on a generative adversarial network with loop constraints. After segmenting the input micro-expression video frame sequence into greyscale single frame sequences and optical flow single frame sequences of a dual-stream sequence diagram, augmenting the data of the two sequences, and constructing the micro-expression recognition model, the input micro-expression video frame sequence is subdivided. The rate of micro-expression recognition has been significantly enhanced by this method. The micro-expression recognition model based on dual-stream enhanced networks can effectively improve the recognition accuracy of micro-expressions with improved robustness and generalization state.
- 参考文献/References:
-
[1] 朱伟杰, 陈莹. 双流网络信息交互机制下的微表情识别[J]. 计算机辅助设计与图形学学报, 2021, 33(4): 545–552
ZHU Weijie, CHEN Ying. Micro-expression recognition based on dual-stream networks information interaction[J]. Journal of computer-aided design & computer graphics, 2021, 33(4): 545–552
[2] HAGGARD E A, ISAACS K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy[M]//Methods of Research in Psychotherapy. Boston: Springer US, 1966: 154–165.
[3] EKMAN P, FRIESEN W V. Nonverbal leakage and clues to deception[J]. Psychiatry, 1969, 32(1): 88–106.
[4] KARIS M S, RAZIF N R A, ALI N M, et al. Local binary pattern (LBP) with application to variant object detection: a survey and method[C]//12th IEEE International Colloquium on Signal Processing & its Applications. Melaka：IEEE, 2016: 221?226.
[5] XU Feng, ZHANG Junping, WANG J Z. Microexpression identification and categorization using a facial dynamics map[J]. IEEE transactions on affective computing, 2017, 8(2): 254–267.
[6] WANG Sujing, CHEN Huiling, YAN Wenjing, et al. Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine[J]. Neural processing letters, 2014, 39(1): 25–43.
[7] PFISTER T, LI Xiaobai, ZHAO Guoying, et al. Recognising spontaneous facial micro-expressions[C]//2011 International Conference on Computer Vision. Barcelona: IEEE, 2011: 1449–1456.
[8] WANG Yandan, SEE J, PHAN R C W, et al. LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition[M]//Computer Vision-ACCV 2014. Cham: Springer International Publishing, 2015: 525–537.
[9] LIONG S T, SEE J, PHAN R C W, et al. Subtle expression recognition using optical strain weighted features[M]//Computer Vision-ACCV 2014 Workshops. Cham: Springer International Publishing, 2015: 644–657.
[10] LIU Yongjin, ZHANG Jinkai, YAN Wenjing, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE transactions on affective computing, 2016, 7(4): 299–310.
[11] HAPPY S L, ROUTRAY A. Fuzzy histogram of optical flow orientations for micro-expression recognition[J]. IEEE transactions on affective computing, 2019, 10(3): 394–406.
[12] GAN Y S, LIONG S T, YAU W C, et al. OFF-ApexNet on micro-expression recognition system[J]. Signal processing:image communication, 2019, 74: 129–139.
[13] 刘汝涵, 徐丹. 视频放大和深度学习在微表情识别任务上的应用[J]. 计算机辅助设计与图形学学报, 2019, 31(9): 1535–1541
LIU Ruhan, XU Dan. Video amplification and deep learning in micro-expression recognition[J]. Journal of computer-aided design & computer graphics, 2019, 31(9): 1535–1541
[14] 徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3): 333–348
XU Feng, ZHANG Junping. Facial microexpression recognition: a survey[J]. Acta automatica sinica, 2017, 43(3): 333–348
[15] 苏育挺, 王蒙蒙, 刘婧, 等. 基于多运动特征融合的微表情识别算法[J]. 激光与光电子学进展, 2020, 57(14): 353–358
SU Yuting, WANG Mengmeng, LIU Jing, et al. Micro-expression recognition algorithm based on multiple motive feature fusion[J]. Laser & optoelectronics progress, 2020, 57(14): 353–358
[16] 诗雨桐, 袁德成. 基于深度学习的面部微表情识别[J]. 沈阳化工大学学报, 2021, 35(4): 380–384
SHI yutong,YUANG decheng. Facial microexpression rcongnition based on deep learning[J]. Journal of Shenyang university of chemical technology, 2021, 35(4): 380–384
[17] 朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950–958
ZHU Weijie, CHEN Ying. Micro-expression recognition convolutional network for dual-stream temporal-domain information interaction[J]. Journal of frontiers of computer science and technology, 2022, 16(4): 950–958
[18] YAN Wenjing, LI Xiaobai, WANG Sujing, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS one, 2014, 9(1): e86041.
[19] LI Xiaobai, PFISTER T, HUANG Xiaohua, et al. A spontaneous micro-expression database: inducement, collection and baseline[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai: IEEE, 2013: 1–6.
[20] QU Fangbing, WANG Sujing, YAN Wenjing, et al. CAS(ME)2: A database of spontaneous macro-expressions and micro-expressions[C]//International Conference on Human-Computer Interaction. Cham: Springer, 2016: 48–59.
[21] 潘通, 李文国. 基于特征拼接的行人重识别[J]. 激光与光电子学进展, 2019, 56(16): 193–199
PAN Tong, LI Wenguo. Person re-identification based on feature stitching[J]. Laser & optoelectronics progress, 2019, 56(16): 193–199
[22] PAN Hang, XIE Lun, LI Juan, et al. Micro-expression recognition by two-stream difference network[J]. IET computer vision, 2021, 15(6): 440–448.
[23] 徐玮, 郑豪, 杨种学. 基于双注意力模型和迁移学习的Apex帧微表情识别[J]. 智能系统学报, 2021, 16(6): 1015?1020.
XU Wei, ZHENG Hao, YANG Zhongxue. Apex frame micro-expression recognition based on dual attention model and transfer learning[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1015?1020.
[24] LI Jing, WANG Yandan, SEE J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern analysis and applications, 2019, 22(4): 1331–1339.
[25] 张爱梅, 徐杨. 注意力分层双线性池化残差网络的表情识别[J]. 计算机工程与应用, 2020, 56(23): 161–166
ZHANG Aimei, XU Yang. Attention hierarchical bilinear pooling residual network for expression recognition[J]. Computer engineering and applications, 2020, 56(23): 161–166
[26] TEJA REDDY S P, TEJA KARRI S, DUBEY S R, et al. Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks[C]//2019 International Joint Conference on Neural Networks (IJCNN). Budapest: IEEE, 2019: 1–8.
[27] ZHOU Ling, MAO Qirong, XUE Luoyang. Dual-inception network for cross-database micro-expression recognition[C]//14th IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE, 2019: 1–5.
[28] 张艺涵, 张朝晖, 霍丽娜, 等. 结合双流特征融合及对抗学习的图像显著性检测[J]. 计算机辅助设计与图形学学报, 2021, 33(3): 376–384
ZHANG Yihan, ZHANG Zhaohui, HUO Lina, et al. Image saliency detection via two-stream feature fusion and adversarial learning[J]. Journal of computer-aided design & computer graphics, 2021, 33(3): 376–384
[29] 张红颖, 安征. 基于改进双流时空网络的人体行为识别[J]. 光学精密工程, 2021, 29(2): 420–429
ZHANG Hongying, AN Zheng. Human action recognition based on improved two-stream spatiotemporal network[J]. Optics and precision engineering, 2021, 29(2): 420–429
[30] YANG Dun, ZHOU Qing, WEN Ju. Interactive two-stream graph neural network for skeleton-based action recognition[J]. Journal of electronic imaging, 2021, 30: 033025.
[31] ZHU Dehui, DU Bo, ZHANG Liangpei. Two-stream convolutional networks for hyperspectral target detection[J]. IEEE transactions on geoscience and remote sensing, 2021, 59(8): 6907–6921.
[32] KIM N, CHO S, AHN C H, et al. Facial micro-expression recognition in video using squeezed landmark feature maps[C]//2021 International Conference on Information and Communication Technology Convergence. Jeju Island: ICT convergence, 2021: 1107-1110.
[33] ZACH C, POCK T, BISCHOF H. A duality based approach for realtime TV-L 1 optical flow[M]//Lecture Notes in Computer Science. Berlin: Springer Berlin Heidelberg, 2007: 214–223.
[34] 李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1): 176–182
LI Yong, LIN Xiaozhu, JIANG Mengying. Facial expression recognition with cross-connect LeNet-5 network[J]. Acta automatica sinica, 2018, 44(1): 176–182
[35] GOYAL S J, UPADHYAY A K, JADON R S, et al. Retraction note to: real-life facial expression recognition systems: a review[M]//Smart Computing and Informatics. Singapore: Springer Singapore, 2019: C1–C1.
[36] 陈斌, 朱晋宁, 东一舟. 基于残差整流增强卷积神经网络的表情识别[J]. 液晶与显示, 2020, 35(12): 1299–1308
CHEN Bin, ZHU Jinning, DONG Yizhou. Expression recognition based on residual rectifier enhanced convolution neural network[J]. Chinese journal of liquid crystals and displays, 2020, 35(12): 1299–1308
[37] 王秀友, 范建中, 刘华明, 等. 基于概率协作表示的多表情序列融合识别[J]. 激光与光电子学进展, 2019, 56(13): 87–93
WANG Xiuyou, FAN Jianzhong, LIU Huaming, et al. Multi-expression sequence fusion recognition based on probabilistic cooperative representation[J]. Laser & optoelectronics progress, 2019, 56(13): 87–93
[38] 张红颖, 王汇三, 胡文博. 基于双模型的相关滤波跟踪[J]. 光学精密工程, 2019, 27(11): 2450–2458
ZHANG Hongying, WANG Huisan, HU Wenbo. Correlation filter tracking algorithm based on double model[J]. Optics and precision engineering, 2019, 27(11): 2450–2458
[39] HUU P N, NGOC T L, MINH Q T. Proposing gesture recognition algorithm using two-stream convolutional network and LSTM[C]//IEEE Eighth International Conference on Communications and Electronics. Phu Quoc Island: IEEE, 2020: 427–432.
[40] 宋立飞, 翁理国, 汪凌峰, 等. 多尺度输入3D卷积融合双流模型的行为识别方法[J]. 计算机辅助设计与图形学学报, 2018, 30(11): 2074–2083
SONG Lifei, WENG Liguo, WANG Lingfeng, et al. Multi-scale 3D convolution fusion two-stream networks for action recognition[J]. Journal of computer-aided design & computer graphics, 2018, 30(11): 2074–2083
[41] 卢官明, 杨成, 杨文娟, 等. 基于LBP-TOP特征的微表情识别[J]. 南京邮电大学学报(自然科学版), 2017, 37(6): 1–7
LU Guanming, YANG Cheng, YANG Wenjuan, et al. Micro-expression recognition based on LBP-TOP features[J]. Journal of Nanjing university of posts and telecommunications (natural science edition), 2017, 37(6): 1–7
[42] 姜万, 周晓彦, 徐华南, 等. 基于LBP与双时空神经网络的微表情识别[J]. 信息与控制, 2020, 49(6): 673–679
JIANG Wan, ZHOU Xiaoyan, XU Huanan, et al. Micro-expression recognition based on LBP and two-stream spatial-temporal neural network[J]. Information and control, 2020, 49(6): 673–679
[43] KHOR H Q, SEE J, PHAN R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//13th IEEE International Conference on Automatic Face & Gesture Recognition. Xi’an: IEEE, 2018: 667–674.
[44] 刘德志, 梁正友, 孙宇. 结合空间注意力机制与光流特征的微表情识别方法[J]. 计算机辅助设计与图形学学报, 2021, 33(10): 1541–1552
LIU Dezhi, LIANG Zhengyou, SUN Yu. Micro-expression recognition method based on spatial attention mechanism and optical flow features[J]. Journal of computer-aided design & computer graphics, 2021, 33(10): 1541–1552
[45] OH Y H, LE NGO A C, SEE J, et al. Monogenic Riesz wavelet representation for micro-expression recognition[C]//IEEE International Conference on Digital Signal Processing. Singapore: IEEE, 2015: 1237–1241.
[46] HUANG Xiaohua, ZHAO Guoying, HONG Xiaopeng, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175: 564–578.
[47] 张学森, 贾静平. 基于三维卷积神经网络和峰值帧光流的微表情识别算法[J]. 模式识别与人工智能, 2021, 34(5): 423–433
ZHANG Xuesen, JIA Jingping. Micro-expression recognition algorithm based on 3D convolutional neural network and optical flow fields from neighboring frames of apex frame[J]. Pattern recognition and artificial intelligence, 2021, 34(5): 423–433
[48] LIONG S T, SEE J, WONG K, et al. Less is more: micro-expression recognition from video using apex frame[J]. Signal processing:image communication, 2018, 62: 82–92.
- 相似文献/References:
-
[1]颜文靖,蒋柯,傅小兰.心理学视角下的自动表情识别[J].智能系统学报,2022,17(5):1039.[doi:10.11992/tis.202112056]
YAN Wenjing,JIANG Ke,FU Xiaolan.Automatic facial expression recognition from a psychological perspective[J].CAAI Transactions on Intelligent Systems,2022,17():1039.[doi:10.11992/tis.202112056]
备注/Memo
收稿日期:2021-09-19。
基金项目:江苏省现代教育技术研究2021年度智慧校园专项课题（2021-R-96609）.
作者简介:陈斌,高级工程师,博士,主要研究方向为模式识别、机器学习、大数据分析。主持江苏省高等学校信息化重点项目2项和厅局级项目3项。发表学术论文20余篇;朱晋宁,高级工程师,主要研究方向为大数据分析、智能系统
通讯作者:陈斌. E-mail：60167@njnu.edu.cn
更新日期/Last Update:
1900-01-01