[1]潘家辉,何志鹏,李自娜,等.多模态情绪识别研究综述[J].智能系统学报,2020,15(4):633-645.[doi:10.11992/tis.202001032]
PAN Jiahui,HE Zhipeng,LI Zina,et al.A review of multimodal emotion recognition[J].CAAI Transactions on Intelligent Systems,2020,15(4):633-645.[doi:10.11992/tis.202001032]
点击复制
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
15
期数:
2020年第4期
页码:
633-645
栏目:
综述
出版日期:
2020-07-05
- Title:
-
A review of multimodal emotion recognition
- 作者:
-
潘家辉1, 何志鹏1, 李自娜2, 梁艳1, 邱丽娜1
-
1. 华南师范大学 软件学院,广东 佛山 528225;
2. 华南师范大学 计算机学院,广东 广州 510641
- Author(s):
-
PAN Jiahui1, HE Zhipeng1, LI Zina2, LIANG Yan1, QIU Lina1
-
1. School of Software, South China Normal University, Foshan 528225, China;
2. School of Computer, South China Normal University, Guangzhou 510641, China
-
- 关键词:
-
情绪识别; 情绪描述模型; 情绪诱发方式; 信息融合; 融合策略; 情绪表征; 模态混合
- Keywords:
-
emotion recognition; emotion description model; emotion inducing mode; information fusion; fusion strategy; emotion representation; modality blend
- 分类号:
-
TP391.4
- DOI:
-
10.11992/tis.202001032
- 摘要:
-
本文针对多模态情绪识别这一新兴领域进行综述。首先从情绪描述模型及情绪诱发方式两个方面对情绪识别的研究基础进行了综述。接着针对多模态情绪识别中的信息融合这一重难点问题,从数据级融合、特征级融合、决策级融合、模型级融合4种融合层次下的主流高效信息融合策略进行了介绍。然后从多种行为表现模态混合、多神经生理模态混合、神经生理与行为表现模态混合这3个角度分别列举具有代表性的多模态混合实例,全面合理地论证了多模态相较于单模态更具情绪区分能力和情绪表征能力,同时对多模态情绪识别方法转为工程技术应用提出了一些思考。最后立足于情绪识别研究现状的分析和把握,对改善和提升情绪识别模型性能的方式和策略进行了深入的探讨与展望。
- Abstract:
-
This paper reviews the emerging field of multimodal emotion recognition. Firstly, the research foundation of emotion recognition is summarized from two aspects: emotion description model and emotion-inducing mode. Then, aiming at the key and difficult problem of information fusion in multi-modal emotion recognition, some mainstream and high-efficiency information fusion strategies are introduced from four fusion levels: data-level fusion, feature-level fusion, decision-level fusion, and model-level fusion. By exemplifying representative multi-modal mixing examples from three perspectives: the mixing of multiple external presentation modalities, the mixing of multiple neurophysiological modalities, and the mixing of neurophysiology and external presentation modalities, it fully demonstrates that multi-modality is more capable of emotional discrimination and emotional representation than single-modality. At the same time, some thoughts on the conversion of multi-modal recognition methods to engineering technology applications are put forward. Finally, based on the analysis and grasp of the current situation of emotion recognition research, the ways and strategies for improving and enhancing the performance of the emotion recognition models are discussed and prospected.
备注/Memo
收稿日期:2020-01-30。
基金项目:国家自然科学基金面上项目(61876067);广东省自然科学基金面上项目(2019A1515011375);广州市科技计划项目重点领域研发计划项目(202007030005)
作者简介:潘家辉,副教授,博士,广东医学会数字医学分会常务委员,主要研究方向为机器学习、脑机接口、模式识别与智能系统。广州市珠江科技新星,华南师范大学教学名师,曾两次获得广东省科学技术奖一等奖、中华医学科技奖三等奖等。主持国家自然科学基金项目2项、广东省自然科学基金项目2项、广州市重点研发领域项目1项、广州市科技创新人才项目1项。发表学术论文80余篇;何志鹏,硕士研究生,主要研究方向为情感计算、混合脑机接口;李自娜,硕士研究生,主要研究方向为机器学习、情感识别
通讯作者:潘家辉.E-mail:panjh82@qq.com
更新日期/Last Update:
2020-07-25