[1]CHEN Bin,ZHU Jinning.Micro-expression recognition based on a dual-stream enhanced fusion network[J].CAAI Transactions on Intelligent Systems,2023,18(2):360-371.[doi:10.11992/tis.202109036]
Copy

Micro-expression recognition based on a dual-stream enhanced fusion network

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 18 Number of periods: 2023 2 Page number: 360-371 Column: 学术论文—机器感知与模式识别 Public date: 2023-05-05
Title:
Micro-expression recognition based on a dual-stream enhanced fusion network
Author(s):
CHEN Bin;  ZHU Jinning
Informatization Office, Nanjing Normal University, Nanjing 210046, China
Keywords:
micro-expression; dual-stream network; generative adversarial network; data enhancement; fusion of features; pattern identification; convolutional neural network; cycle constraint
CLC:
TP39; TH691.9
DOI:
10.11992/tis.202109036
Abstract:
We propose a micro-expression recognition model based on a dual-stream enhanced network in order to address the issues of insufficient samples from the dataset of micro-expression and uneven distribution of sample types leading to a low rate of robustness. Targeting a dual-stream convolutional neural network of single-frame RGB image flow and optical image flow, a micro-expression recognition model is built based on a fundamental authoritative dataset and data enhancement. Single-frame airspace information and optical time flow domain information are incorporated in the SoftMax logistic regression layer to improve the network performance for two independent streams. The dataset is augmented by introducing a method for image generation based on a generative adversarial network with loop constraints. After segmenting the input micro-expression video frame sequence into greyscale single frame sequences and optical flow single frame sequences of a dual-stream sequence diagram, augmenting the data of the two sequences, and constructing the micro-expression recognition model, the input micro-expression video frame sequence is subdivided. The rate of micro-expression recognition has been significantly enhanced by this method. The micro-expression recognition model based on dual-stream enhanced networks can effectively improve the recognition accuracy of micro-expressions with improved robustness and generalization state.
References:
[1] 朱伟杰, 陈莹. 双流网络信息交互机制下的微表情识别[J]. 计算机辅助设计与图形学学报, 2021, 33(4): 545–552
ZHU Weijie, CHEN Ying. Micro-expression recognition based on dual-stream networks information interaction[J]. Journal of computer-aided design & computer graphics, 2021, 33(4): 545–552
[2] HAGGARD E A, ISAACS K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy[M]//Methods of Research in Psychotherapy. Boston: Springer US, 1966: 154–165.
[3] EKMAN P, FRIESEN W V. Nonverbal leakage and clues to deception[J]. Psychiatry, 1969, 32(1): 88–106.
[4] KARIS M S, RAZIF N R A, ALI N M, et al. Local binary pattern (LBP) with application to variant object detection: a survey and method[C]//12th IEEE International Colloquium on Signal Processing & its Applications. Melaka：IEEE, 2016: 221?226.
[5] XU Feng, ZHANG Junping, WANG J Z. Microexpression identification and categorization using a facial dynamics map[J]. IEEE transactions on affective computing, 2017, 8(2): 254–267.
[6] WANG Sujing, CHEN Huiling, YAN Wenjing, et al. Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine[J]. Neural processing letters, 2014, 39(1): 25–43.
[7] PFISTER T, LI Xiaobai, ZHAO Guoying, et al. Recognising spontaneous facial micro-expressions[C]//2011 International Conference on Computer Vision. Barcelona: IEEE, 2011: 1449–1456.
[8] WANG Yandan, SEE J, PHAN R C W, et al. LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition[M]//Computer Vision-ACCV 2014. Cham: Springer International Publishing, 2015: 525–537.
[9] LIONG S T, SEE J, PHAN R C W, et al. Subtle expression recognition using optical strain weighted features[M]//Computer Vision-ACCV 2014 Workshops. Cham: Springer International Publishing, 2015: 644–657.
[10] LIU Yongjin, ZHANG Jinkai, YAN Wenjing, et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE transactions on affective computing, 2016, 7(4): 299–310.
[11] HAPPY S L, ROUTRAY A. Fuzzy histogram of optical flow orientations for micro-expression recognition[J]. IEEE transactions on affective computing, 2019, 10(3): 394–406.
[12] GAN Y S, LIONG S T, YAU W C, et al. OFF-ApexNet on micro-expression recognition system[J]. Signal processing:image communication, 2019, 74: 129–139.
[13] 刘汝涵, 徐丹. 视频放大和深度学习在微表情识别任务上的应用[J]. 计算机辅助设计与图形学学报, 2019, 31(9): 1535–1541
LIU Ruhan, XU Dan. Video amplification and deep learning in micro-expression recognition[J]. Journal of computer-aided design & computer graphics, 2019, 31(9): 1535–1541
[14] 徐峰, 张军平. 人脸微表情识别综述[J]. 自动化学报, 2017, 43(3): 333–348
XU Feng, ZHANG Junping. Facial microexpression recognition: a survey[J]. Acta automatica sinica, 2017, 43(3): 333–348
[15] 苏育挺, 王蒙蒙, 刘婧, 等. 基于多运动特征融合的微表情识别算法[J]. 激光与光电子学进展, 2020, 57(14): 353–358
SU Yuting, WANG Mengmeng, LIU Jing, et al. Micro-expression recognition algorithm based on multiple motive feature fusion[J]. Laser & optoelectronics progress, 2020, 57(14): 353–358
[16] 诗雨桐, 袁德成. 基于深度学习的面部微表情识别[J]. 沈阳化工大学学报, 2021, 35(4): 380–384
SHI yutong,YUANG decheng. Facial microexpression rcongnition based on deep learning[J]. Journal of Shenyang university of chemical technology, 2021, 35(4): 380–384
[17] 朱伟杰, 陈莹. 双流时间域信息交互的微表情识别卷积网络[J]. 计算机科学与探索, 2022, 16(4): 950–958
ZHU Weijie, CHEN Ying. Micro-expression recognition convolutional network for dual-stream temporal-domain information interaction[J]. Journal of frontiers of computer science and technology, 2022, 16(4): 950–958
[18] YAN Wenjing, LI Xiaobai, WANG Sujing, et al. CASME II: an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS one, 2014, 9(1): e86041.
[19] LI Xiaobai, PFISTER T, HUANG Xiaohua, et al. A spontaneous micro-expression database: inducement, collection and baseline[C]//2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai: IEEE, 2013: 1–6.
[20] QU Fangbing, WANG Sujing, YAN Wenjing, et al. CAS(ME)2: A database of spontaneous macro-expressions and micro-expressions[C]//International Conference on Human-Computer Interaction. Cham: Springer, 2016: 48–59.
[21] 潘通, 李文国. 基于特征拼接的行人重识别[J]. 激光与光电子学进展, 2019, 56(16): 193–199
PAN Tong, LI Wenguo. Person re-identification based on feature stitching[J]. Laser & optoelectronics progress, 2019, 56(16): 193–199
[22] PAN Hang, XIE Lun, LI Juan, et al. Micro-expression recognition by two-stream difference network[J]. IET computer vision, 2021, 15(6): 440–448.
[23] 徐玮, 郑豪, 杨种学. 基于双注意力模型和迁移学习的Apex帧微表情识别[J]. 智能系统学报, 2021, 16(6): 1015?1020.
XU Wei, ZHENG Hao, YANG Zhongxue. Apex frame micro-expression recognition based on dual attention model and transfer learning[J]. CAAI transactions on intelligent systems, 2021, 16(6): 1015?1020.
[24] LI Jing, WANG Yandan, SEE J, et al. Micro-expression recognition based on 3D flow convolutional neural network[J]. Pattern analysis and applications, 2019, 22(4): 1331–1339.
[25] 张爱梅, 徐杨. 注意力分层双线性池化残差网络的表情识别[J]. 计算机工程与应用, 2020, 56(23): 161–166
ZHANG Aimei, XU Yang. Attention hierarchical bilinear pooling residual network for expression recognition[J]. Computer engineering and applications, 2020, 56(23): 161–166
[26] TEJA REDDY S P, TEJA KARRI S, DUBEY S R, et al. Spontaneous facial micro-expression recognition using 3D spatiotemporal convolutional neural networks[C]//2019 International Joint Conference on Neural Networks (IJCNN). Budapest: IEEE, 2019: 1–8.
[27] ZHOU Ling, MAO Qirong, XUE Luoyang. Dual-inception network for cross-database micro-expression recognition[C]//14th IEEE International Conference on Automatic Face & Gesture Recognition. Lille: IEEE, 2019: 1–5.
[28] 张艺涵, 张朝晖, 霍丽娜, 等. 结合双流特征融合及对抗学习的图像显著性检测[J]. 计算机辅助设计与图形学学报, 2021, 33(3): 376–384
ZHANG Yihan, ZHANG Zhaohui, HUO Lina, et al. Image saliency detection via two-stream feature fusion and adversarial learning[J]. Journal of computer-aided design & computer graphics, 2021, 33(3): 376–384
[29] 张红颖, 安征. 基于改进双流时空网络的人体行为识别[J]. 光学精密工程, 2021, 29(2): 420–429
ZHANG Hongying, AN Zheng. Human action recognition based on improved two-stream spatiotemporal network[J]. Optics and precision engineering, 2021, 29(2): 420–429
[30] YANG Dun, ZHOU Qing, WEN Ju. Interactive two-stream graph neural network for skeleton-based action recognition[J]. Journal of electronic imaging, 2021, 30: 033025.
[31] ZHU Dehui, DU Bo, ZHANG Liangpei. Two-stream convolutional networks for hyperspectral target detection[J]. IEEE transactions on geoscience and remote sensing, 2021, 59(8): 6907–6921.
[32] KIM N, CHO S, AHN C H, et al. Facial micro-expression recognition in video using squeezed landmark feature maps[C]//2021 International Conference on Information and Communication Technology Convergence. Jeju Island: ICT convergence, 2021: 1107-1110.
[33] ZACH C, POCK T, BISCHOF H. A duality based approach for realtime TV-L 1 optical flow[M]//Lecture Notes in Computer Science. Berlin: Springer Berlin Heidelberg, 2007: 214–223.
[34] 李勇, 林小竹, 蒋梦莹. 基于跨连接LeNet-5网络的面部表情识别[J]. 自动化学报, 2018, 44(1): 176–182
LI Yong, LIN Xiaozhu, JIANG Mengying. Facial expression recognition with cross-connect LeNet-5 network[J]. Acta automatica sinica, 2018, 44(1): 176–182
[35] GOYAL S J, UPADHYAY A K, JADON R S, et al. Retraction note to: real-life facial expression recognition systems: a review[M]//Smart Computing and Informatics. Singapore: Springer Singapore, 2019: C1–C1.
[36] 陈斌, 朱晋宁, 东一舟. 基于残差整流增强卷积神经网络的表情识别[J]. 液晶与显示, 2020, 35(12): 1299–1308
CHEN Bin, ZHU Jinning, DONG Yizhou. Expression recognition based on residual rectifier enhanced convolution neural network[J]. Chinese journal of liquid crystals and displays, 2020, 35(12): 1299–1308
[37] 王秀友, 范建中, 刘华明, 等. 基于概率协作表示的多表情序列融合识别[J]. 激光与光电子学进展, 2019, 56(13): 87–93
WANG Xiuyou, FAN Jianzhong, LIU Huaming, et al. Multi-expression sequence fusion recognition based on probabilistic cooperative representation[J]. Laser & optoelectronics progress, 2019, 56(13): 87–93
[38] 张红颖, 王汇三, 胡文博. 基于双模型的相关滤波跟踪[J]. 光学精密工程, 2019, 27(11): 2450–2458
ZHANG Hongying, WANG Huisan, HU Wenbo. Correlation filter tracking algorithm based on double model[J]. Optics and precision engineering, 2019, 27(11): 2450–2458
[39] HUU P N, NGOC T L, MINH Q T. Proposing gesture recognition algorithm using two-stream convolutional network and LSTM[C]//IEEE Eighth International Conference on Communications and Electronics. Phu Quoc Island: IEEE, 2020: 427–432.
[40] 宋立飞, 翁理国, 汪凌峰, 等. 多尺度输入3D卷积融合双流模型的行为识别方法[J]. 计算机辅助设计与图形学学报, 2018, 30(11): 2074–2083
SONG Lifei, WENG Liguo, WANG Lingfeng, et al. Multi-scale 3D convolution fusion two-stream networks for action recognition[J]. Journal of computer-aided design & computer graphics, 2018, 30(11): 2074–2083
[41] 卢官明, 杨成, 杨文娟, 等. 基于LBP-TOP特征的微表情识别[J]. 南京邮电大学学报(自然科学版), 2017, 37(6): 1–7
LU Guanming, YANG Cheng, YANG Wenjuan, et al. Micro-expression recognition based on LBP-TOP features[J]. Journal of Nanjing university of posts and telecommunications (natural science edition), 2017, 37(6): 1–7
[42] 姜万, 周晓彦, 徐华南, 等. 基于LBP与双时空神经网络的微表情识别[J]. 信息与控制, 2020, 49(6): 673–679
JIANG Wan, ZHOU Xiaoyan, XU Huanan, et al. Micro-expression recognition based on LBP and two-stream spatial-temporal neural network[J]. Information and control, 2020, 49(6): 673–679
[43] KHOR H Q, SEE J, PHAN R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//13th IEEE International Conference on Automatic Face & Gesture Recognition. Xi’an: IEEE, 2018: 667–674.
[44] 刘德志, 梁正友, 孙宇. 结合空间注意力机制与光流特征的微表情识别方法[J]. 计算机辅助设计与图形学学报, 2021, 33(10): 1541–1552
LIU Dezhi, LIANG Zhengyou, SUN Yu. Micro-expression recognition method based on spatial attention mechanism and optical flow features[J]. Journal of computer-aided design & computer graphics, 2021, 33(10): 1541–1552
[45] OH Y H, LE NGO A C, SEE J, et al. Monogenic Riesz wavelet representation for micro-expression recognition[C]//IEEE International Conference on Digital Signal Processing. Singapore: IEEE, 2015: 1237–1241.
[46] HUANG Xiaohua, ZHAO Guoying, HONG Xiaopeng, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns[J]. Neurocomputing, 2016, 175: 564–578.
[47] 张学森, 贾静平. 基于三维卷积神经网络和峰值帧光流的微表情识别算法[J]. 模式识别与人工智能, 2021, 34(5): 423–433
ZHANG Xuesen, JIA Jingping. Micro-expression recognition algorithm based on 3D convolutional neural network and optical flow fields from neighboring frames of apex frame[J]. Pattern recognition and artificial intelligence, 2021, 34(5): 423–433
[48] LIONG S T, SEE J, WONG K, et al. Less is more: micro-expression recognition from video using apex frame[J]. Signal processing:image communication, 2018, 62: 82–92.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems