[1]曲海成,徐波.基于自适应图学习权重的多模态情感分析[J].智能系统学报,2025,20(2):516-528.[doi:10.11992/tis.202401001]
QU Haicheng,XU Bo.Multimodal sentiment analysis based on adaptive graph learning weight[J].CAAI Transactions on Intelligent Systems,2025,20(2):516-528.[doi:10.11992/tis.202401001]
点击复制
QU Haicheng,XU Bo.Multimodal sentiment analysis based on adaptive graph learning weight[J].CAAI Transactions on Intelligent Systems,2025,20(2):516-528.[doi:10.11992/tis.202401001]
基于自适应图学习权重的多模态情感分析
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第2期
页码:
516-528
栏目:
人工智能院长论坛
出版日期:
2025-03-05
- Title:
- Multimodal sentiment analysis based on adaptive graph learning weight
- Keywords:
- multimodal; sentiment analysis; modal differences; information redundancy; adaptive graph learning; cross modal attention; similarity constraints; information bottleneck
- 分类号:
- TP391
- 摘要:
- 在多模态情感分析任务中，由于不同模态表现方式的不一致性，模态间的情感信息密度具有较大的差异。为了平衡情感信息在不同模态中分布的不均匀性并减少多模态特征表示的冗余性，提出了一种基于自适应图学习权重的多模态情感分析方法。首先，采用不同的特征提取方法捕获单一模态内的特定信息；其次，将不同模态通过公共编码器映射到同一空间中，利用跨模态注意力机制来显式构建模态间的关联；然后，将每种模态对任务分类的预测值以及模态表示嵌入到自适应图中，通过模态标签学习不同模态对最终分类任务的贡献度来动态调整不同模态之间的权重，以适应主导模态的变化；最后，引入信息瓶颈机制进行去噪，旨在学习一种无冗余的多模态特征表示进行情感预测。在公开的多模态情感分析数据集上对所提出的模型进行了评估。实验结果表明，其有效提升了多模态情感分析的准确性。
- Abstract:
- The inconsistency in representing different modalities in multimodal sentiment analysis tasks results in significant differences in the density of emotional information between modalities. A multimodal sentiment analysis method based on adaptive graph learning weights is proposed to balance the uneven distribution of emotional information in different modalities and reduce the redundancy of multimodal feature representations. First, different feature extraction methods are used to capture specific information within each mode. Second, different modalities are mapped to the same space through a common encoder, and cross-modal attention mechanisms are used to explicitly construct correlations between modalities. Third, the predicted values and modal representations of each modality for task classification are embedded into the adaptive graph, and the contribution of different modalities to the final classification task is learned through modal labels to dynamically adjust the weights between different modalities for adapting to changes in the dominant modality. Finally, an information bottleneck mechanism is introduced for denoising, aiming to learn a nonredundant multimodal feature representation for sentiment prediction. The proposed model is evaluated on the publicly available multimodal sentiment analysis datasets. Experimental results show that its effectively improving the accuracy of multimodal sentiment analysis.
- 参考文献/References:
-
[1] PE?A D, AGUILERA A, DONGO I, et al. A framework to evaluate fusion methods for multimodal emotion recognition[J]. IEEE access, 2023, 11: 10218-10237.
[2] ZHANG Junling, WU Xuemei, HUANG Changqin. AdaMoW: multimodal sentiment analysis based on adaptive modality-specific weight fusion network[J]. IEEE access, 2023, 11: 48410-48420.
[3] 张亚洲, 戎璐, 宋大为, 等. 多模态情感分析研究综述[J]. 模式识别与人工智能, 2020, 33(5): 426-438.
ZHANG Yazhou, RONG Lu, SONG Dawei, et al. A survey on multimodal sentiment analysis[J]. Pattern recognition and artificial intelligence, 2020, 33(5): 426-438.
[4] GANDHI A, ADHVARYU K, PORIA S, et al. Multimodal sentiment analysis: a systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions[J]. Information fusion, 2023, 91: 424-444.
[5] MAI Sijie, HU Haifeng, XING Songlong. Modality to modality translation: an adversarial representation learning and graph fusion network for multimodal fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 164-172.
[6] ZADEH A, LIANG P P, PORIA S, et al. Multi-attention recurrent network for human communication comprehension[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans: AAAI, 2018: 5642-5649.
[7] HAN Wei, CHEN Hui, GELBUKH A, et al. Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis[C]//Proceedings of the 2021 International Conference on Multimodal Interaction. Montréal: ACM, 2021: 6-15.
[8] 刘颖, 王哲, 房杰, 等. 基于图文融合的多模态舆情分析[J]. 计算机科学与探索, 2022, 16(6): 1260-1278.
LIU Ying, WANG Zhe, FANG Jie, et al. Multi-modal public opinion analysis based on image and text fusion[J]. Journal of frontiers of computer science and technology, 2022, 16(6): 1260-1278.
[9] HUANG Changqin, ZHANG Junling, WU Xuemei, et al. TeFNA: text-centered fusion network with crossmodal attention for multimodal sentiment analysis[J]. Knowledge-based systems, 2023, 269: 110502.
[10] SUN Hao, LIU Jiaqing, CHEN Y W, et al. Modality-invariant temporal representation learning for multimodal sentiment classification[J]. Information fusion, 2023, 91: 504-514.
[11] MAI Sijie, ZENG Ying, HU Haifeng. Multimodal information bottleneck: learning minimal sufficient unimodal and multimodal representations[J]. IEEE transactions on multimedia, 2023, 25: 4121-4134.
[12] ZADEH A, CHEN Minghai, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[EB/OL]. (2017-07-23)[2024-01-02]. https://arxiv.org/abs/1707.07250.
[13] TSAI Y H H, BAI Shaojie, LIANG P P, et al. Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the Conference Association for Computational Linguistics Meeting. Florence: ACL, 2019: 6558-6569.
[14] SU Guixin, HE Junyi, LI Xia, et al. NFCMF: noise filtering and CrossModal fusion for multimodal sentiment analysis[C]//2021 International Conference on Asian Language Processing. Singapore: IEEE, 2021: 316-321.
[15] YANG Shuo, XU Zhaopan, WANG Kai, et al. BiCro: noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 19883-19892.
[16] 孙杰, 车文刚, 高盛祥. 面向多模态情感分析的多通道时序卷积融合[J]. 计算机科学与探索, 2024, 18(11): 3041-3050.
SUN Jie, CHE Wengang, GAO Shengxiang. Multi-channel temporal convolution fusion for multimodal sentiment analysis[J]. Journal of frontiers of computer science and technology, 2024, 18(11): 3041-3050.
[17] 鲍小异, 姜晓彤, 王中卿, 等. 基于跨语言图神经网络模型的属性级情感分类[J]. 软件学报, 2023, 34(2): 676-689.
BAO Xiaoyi, JIANG Xiaotong, WANG Zhongqing, et al. Cross-lingual aspect-level sentiment classification with graph neural network[J]. Journal of software, 2023, 34(2): 676-689.
[18] JANGRA A, MUKHERJEE S, JATOWT A, et al. A survey on multi-modal summarization[J]. ACM computing surveys, 2023, 55(13s): 1-36.
[19] MAJUMDER N, HAZARIKA D, GELBUKH A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling[J]. Knowledge-based systems, 2018, 161: 124-133.
[20] XU Nan, MAO Wenji. MultiSentiNet: a deep semantic network for multimodal sentiment analysis[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. Singapore: ACM, 2017: 2399-2402.
[21] AKHTAR M S, CHAUHAN D S, GHOSAL D, et al. Multi-task learning for multi-modal emotion recognition and sentiment analysis[EB/OL]. (2019-05-14) [2024-01-02]. https://arxiv.org/abs/1905.05812.
[22] HAZARIKA D, ZIMMERMANN R, PORIA S. MISA: modality-invariant and-specific representations for multimodal sentiment analysis[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle: ACM, 2020: 1122-1131.
[23] YU Wenmeng, XU Hua, YUAN Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2021: 10790-10797.
[24] ZUO Haolin, LIU Rui, ZHAO Jinming, et al. Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities[C]//2023 IEEE International Conference on Acoustics, Speech and Signal Processing. Rhodes Island: IEEE, 2023: 1-5.
[25] LIU A H, JIN S, LAI C I J, et al. Cross-modal discrete representation learning[EB/OL]. (2021-06-10) [2024-01-02]. https://arxiv.org/abs/2106.05438.
[26] 胡文彬, 陈龙, 黄贤波, 等. 融合交叉注意力的突发事件多模态中文反讽识别模型[J]. 智能系统学报, 2024, 19(2): 392-400.
HU Wenbin, CHEN Long, HUANG Xianbo, et al. A multimodal Chinese sarcasm detection model for emergencies based on cross attention[J]. CAAI transactions on intelligent systems, 2024, 19(2): 392-400.
[27] 李梦云, 张景, 张换香, 等. 基于跨模态语义信息增强的多模态情感分析[J]. 计算机科学与探索, 2024, 18(9): 2476-2486.
LI Mengyun, ZHANG Jing, ZHANG Huanxiang, et al. Multimodal sentiment analysis based on cross-modal semantic information enhancement[J]. Journal of frontiers of computer science and technology, 2024, 18(9): 2476-2486.
[28] 包广斌, 李港乐, 王国雄. 面向多模态情感分析的双模态交互注意力[J]. 计算机科学与探索, 2022, 16(4): 909-916.
BAO Guangbin, LI Gangle, WANG Guoxiong. Bimodal interactive attention for multimodal sentiment analysis[J]. Journal of frontiers of computer science and technology, 2022, 16(4): 909-916.
[29] TANG Jiajia, LIU Dongjun, JIN Xuanyu, et al. BAFN: bi-direction attention based fusion network for multimodal sentiment analysis[J]. IEEE transactions on circuits and systems for video technology, 2023, 33(4): 1966-1978.
[30] LYU Fengmao, CHEN Xiang, HUANG Yanyong, et al. Progressive modality reinforcement for human multimodal emotion recognition from unaligned multimodal sequences[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 2554-2562.
[31] RAHMAN W, HASAN M K, LEE Sangwu, et al. Integrating multimodal information in large pretrained transformers[C]//Proceedings of the conference. Association for Computational Linguistics. Online: ACL, 2020: 2359-2369.
[32] SANGWAN S, CHAUHAN D S, AKHTAR M S, et al. Multi-task gated contextual cross-modal attention framework for sentiment and emotion analysis[C]//Neural Information Processing. Sydney: Springer, 2019: 662-669.
[33] HUANG Yanping, PENG Hong, LIU Qian, et al. Attention-enabled gated spiking neural P model for aspect-level sentiment classification[J]. Neural networks, 2023, 157: 437-443.
[34] TISHBY N, PEREIRA F C, BIALEK W. The information bottleneck method[EB/OL]. (2000-04-24) [2024-01-02]. https://arxiv.org/abs/physics/0004057.
[35] ALEMI A A, FISCHER I, DILLON J V, et al. Deep variational information bottleneck[EB/OL]. (2016-12-01) [2024-01-02]. https://arxiv.org/abs/1612.00410.
[36] ZADEH A, ZELLERS R, PINCUS E, et al. Multimodal sentiment intensity analysis in videos: facial gestures and verbal messages[J]. IEEE intelligent systems, 2016, 31(6): 82-88.
[37] BAGHER ZADEH A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Melbourne: ACL, 2018: 2236-2246.
[38] LIU Zhun, SHEN Ying, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[EB/OL]. (2018-05-31) [2024-01-02]. https://arxiv.org/abs/1806.00064.
[39] HAN Wei, CHEN Hui, PORIA S. Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[EB/OL]. (2021-09-01) [2024-01-02]. https://arxiv.org/abs/2109.00412.
[40] WANG Di, GUO Xutong, TIAN Yumin, et al. TETFN: a text enhanced transformer fusion network for multimodal sentiment analysis[J]. Pattern recognition, 2023, 136: 109259.
[41] LI Yong, WANG Yuanzhi, CUI Zhen. Decoupled multimodal distilling for emotion recognition[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 6631-6640.
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]赵文清,侯小可,沙海虹.语义规则在微博热点话题情感分析中的应用[J].智能系统学报,2014,9(1):121.[doi:10.3969/j.issn.1673-4785.201208020]
ZHAO Wenqing,HOU Xiaoke,SHA Haihong.Application of semantic rules to sentiment analysis of microblog hot topics[J].CAAI Transactions on Intelligent Systems,2014,9():121.[doi:10.3969/j.issn.1673-4785.201208020]
[3]李雪,蒋树强.智能交互的物体识别增量学习技术综述[J].智能系统学报,2017,12(2):140.[doi:10.11992/tis.201701006]
LI Xue,JIANG Shuqiang.Incremental learning and object recognition system based on intelligent HCI: a survey[J].CAAI Transactions on Intelligent Systems,2017,12():140.[doi:10.11992/tis.201701006]
[4]侯涛,张强.基于T-S型模糊加权的多模软切换的风电机组变桨控制[J].智能系统学报,2018,13(4):625.[doi:10.11992/tis.201710009]
HOU Tao,ZHANG Qiang.Multi-mode soft-switch variable-pitch control of wind turbines based on T-S fuzzy weighting[J].CAAI Transactions on Intelligent Systems,2018,13():625.[doi:10.11992/tis.201710009]
[5]曾碧卿,韩旭丽,王盛玉,等.层次化双注意力神经网络模型的情感分析研究[J].智能系统学报,2020,15(3):460.[doi:10.11992/tis.201812017]
ZENG Biqing,HAN Xuli,WANG Shengyu,et al.Hierarchical double-attention neural networks for sentiment classification[J].CAAI Transactions on Intelligent Systems,2020,15():460.[doi:10.11992/tis.201812017]
[6]张新钰,邹镇洪,李志伟,等.面向自动驾驶目标检测的深度多模态融合技术[J].智能系统学报,2020,15(4):758.[doi:10.11992/tis.202002010]
ZHANG Xinyu,ZOU Zhenhong,LI Zhiwei,et al.Deep multi-modal fusion in object detection for autonomous driving[J].CAAI Transactions on Intelligent Systems,2020,15():758.[doi:10.11992/tis.202002010]
[7]肖宇晗,林慧苹,汪权彬,等.基于双特征嵌套注意力的方面词情感分析算法[J].智能系统学报,2021,16(1):142.[doi:10.11992/tis.202012024]
XIAO Yuhan,LIN Huiping,WANG Quanbin,et al.An algorithm for aspect-based sentiment analysis based on dual features attention-over-attention[J].CAAI Transactions on Intelligent Systems,2021,16():142.[doi:10.11992/tis.202012024]
[8]陈玲玲,毕晓君.多模态融合网络的睡眠分期研究[J].智能系统学报,2022,17(6):1194.[doi:10.11992/tis.202202018]
CHEN Lingling,BI Xiaojun.Sleep staging model based on multimodal fusion[J].CAAI Transactions on Intelligent Systems,2022,17():1194.[doi:10.11992/tis.202202018]
[9]张铭泉,周辉,曹锦纲.基于注意力机制的双BERT有向情感文本分类研究[J].智能系统学报,2022,17(6):1220.[doi:10.11992/tis.202112038]
ZHANG Mingquan,ZHOU Hui,CAO Jingang.Dual BERT directed sentiment text classification based on attention mechanism[J].CAAI Transactions on Intelligent Systems,2022,17():1220.[doi:10.11992/tis.202112038]
[10]丁维昌,施俊,王骏.自监督对比特征学习的多模态乳腺超声诊断[J].智能系统学报,2023,18(1):66.[doi:10.11992/tis.202111052]
DING Weichang,SHI Jun,WANG Jun.Multi-modality ultrasound diagnosis of the breast with self-supervised contrastive feature learning[J].CAAI Transactions on Intelligent Systems,2023,18():66.[doi:10.11992/tis.202111052]
[11]吴钟强,张耀文,商琳.基于语义特征的多视图情感分类方法[J].智能系统学报,2017,12(5):745.[doi:10.11992/tis.201706026]
WU Zhongqiang,ZHANG Yaowen,SHANG Lin.Multi-view sentiment classification of microblogs based on semantic features[J].CAAI Transactions on Intelligent Systems,2017,12():745.[doi:10.11992/tis.201706026]
[12]朱超杰,闫昱名,初宝昌,等.采用目标注意力的方面级多模态情感分析研究[J].智能系统学报,2024,19(6):1562.[doi:10.11992/tis.202404009]
ZHU Chaojie,YAN Yuming,CHU Baochang,et al.Aspect-level multimodal sentiment analysis via object-attention[J].CAAI Transactions on Intelligent Systems,2024,19():1562.[doi:10.11992/tis.202404009]
备注/Memo
收稿日期:2024-1-2。
基金项目:辽宁省高等学校基本科研项目(LIKMZ20220699).
作者简介:曲海成，副教授，辽宁工程技术大学软件学院副院长，中国计算机学会会员。主要研究方向为遥感影像高性能计算、视觉信息计算、目标检测与识别。主持辽宁省自然科学基金项目1项、辽宁省教育厅面上项目2项，发表学术论文60余篇。E-mail：quhai cheng@lntu.edu.cn;徐波，硕士研究生，主要研究方向为多模态情感分析。E-mail：lntu_xubo@163.com。
通讯作者:曲海成. E-mail：quhaicheng@lntu.edu.cn
更新日期/Last Update:
2025-03-05