[1]朱超杰,闫昱名,初宝昌,等.采用目标注意力的方面级多模态情感分析研究[J].智能系统学报,2024,19(6):1562-1572.[doi:10.11992/tis.202404009]
ZHU Chaojie,YAN Yuming,CHU Baochang,et al.Aspect-level multimodal sentiment analysis via object-attention[J].CAAI Transactions on Intelligent Systems,2024,19(6):1562-1572.[doi:10.11992/tis.202404009]
点击复制
ZHU Chaojie,YAN Yuming,CHU Baochang,et al.Aspect-level multimodal sentiment analysis via object-attention[J].CAAI Transactions on Intelligent Systems,2024,19(6):1562-1572.[doi:10.11992/tis.202404009]
采用目标注意力的方面级多模态情感分析研究
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第6期
页码:
1562-1572
栏目:
人工智能院长论坛
出版日期:
2024-12-05
- Title:
- Aspect-level multimodal sentiment analysis via object-attention
- Keywords:
- aspect-level sentiment analysis; multimodal; sentiment analysis; object detection; self-attention; natural language processing systems; deep learning; feature extraction
- 分类号:
- TP391
- 摘要:
- 方面级的多模态情感分析（aspect-level multimodal sentiment analysis，ALMSA）旨在识别出语句和图像信息在某个特定方面上所表现出的情感极性。该任务现有分析模型使用的均是图像的全局特征，并未考虑原始图像信息中的细节信息。针对这一问题，提出一种基于目标注意力的方面级多模态情感分析模型OAB-ALMSA（object-attention based aspect-level multimodal sentiment analysis）。采用目标检测算法捕获原始图像中目标的细节信息；引入目标注意力机制并构建迭代的融合层来完成多模态信息的充分融合；针对数据较高的复杂性所导致的训练困难问题，为模型制定课程式学习策略。经课程式学习训练的OAB-ALMSA模型在TWITTER-2015数据集上得到了最高的F1，这表明对图像中细节信息的利用能够提高模型对数据的综合理解，提升预测效果。
- Abstract:
- Aspect-level multimodal sentiment analysis (ALMSA) aims to identify the sentiment polarity of a specific aspect word using both sentence and image data. Current models often rely on the global features of images, overlooking the details in the original image. To address this issue, we propose an object attention-based aspect-level multimodal sentiment analysis model (OAB-ALMSA). This model first employs an object detection algorithm to capture the detailed information of the objects from the original image. It then applies an object-attention mechanism and builds an iterative fusion layer to fully fuse the multimodal information. Finally, a curriculum learning strategy is developed to tackle the challenges of training with complex samples. Experiments conducted on TWITTER-2015 data sets demonstrate that OAB-ALMSA, when combined with curriculum learning, achieves the highest F1. These results highlight that leveraging detailed image data enhances the model’s overall understanding and improves prediction accuracy.
- 参考文献/References:
-
[1] 中国互联网络信息中心. 中国互联网络发展状况统计报告[J]. 国家图书馆学刊, 2023, 32(2): 39.
[2] XU Nan, MAO Wenji, CHEN Guandan. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Hawaii: AAAI Press, 2019: 371-378.
[3] ZHANG Zhe, WANG Zhu, LI Xiaona, et al. ModalNet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network[J]. World wide web, 2021, 24(6): 1957-1974.
[4] KHAN Z, FU Yun. Exploiting BERT for multimodal target sentiment classification through input space translation[C]//Proceedings of the 29th ACM International Conference on Multimedia. Virtual Event China: ACM, 2021: 3034-3042.
[5] 王伟贤, 吴俊. 基于情感词典与语义规则集的微博文本情感分析[J]. 计算机科学与应用, 2023, 13(4): 754-763.
WANG Weixian, WU Jun. Sentiment analysis of microblog text based on sentiment dictionary and semantic rule set[J]. Computer science and application, 2023, 13(4): 754-763.
[6] SHANG Yongmin, ZHAO Yuqin. Sentiment analysis and implementation of online reviews based on machine learning[J]. Journal of Dali University, 2021, 6(12): 80-86.
[7] HUANG Jiaxin, MENG Yu, GUO Fang, et al. Weakly-supervised aspect-based sentiment analysis via joint aspect-sentiment topic embedding[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2020.
[8] KIRITCHENKO S, ZHU X, CHERRY C, et al. Detecting aspects and sentiment in customer reviews[C]//8th International Workshop on Semantic Evaluation. [S.l.]: [s.n.], 2014, 437-442.
[9] 王婷, 杨文忠. 文本情感分析方法研究综述[J]. 计算机工程与应用, 2021, 57(12): 11-24.
WANG Ting, YANG Wenzhong. Review of text sentiment analysis methods[J]. Computer engineering and applications, 2021, 57(12): 11-24.
[10] LONDHE A, RAO P V R D P. Aspect based sentiment analysis–an incremental model learning approach using LSTM-RNN[M]//Communications in Computer and Information Science. Cham: Springer International Publishing, 2021: 677-689.
[11] XUE Wei, LI Tao. Aspect based sentiment analysis with gated convolutional networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2018: 2514-2523.
[12] 张文轩, 殷雁君, 智敏. 用于方面级情感分析的情感增强双图卷积网络[J]. 计算机科学与探索, 2024, 18(1): 217-230.
ZHANG Wenxuan, YIN Yanjun, ZHI Min. Affection enhanced dual graph convolution network for aspect based sentiment analysis[J]. Journal of frontiers of computer science and technology, 2024, 18(1): 217-230.
[13] WANG Bailin, LU Wei. Learning latent opinions for aspect-level sentiment classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans: AAAI Press, 2018: 5537-5544.
[14] 曾锋, 曾碧卿, 韩旭丽, 等. 基于双层注意力循环神经网络的方面级情感分析[J]. 中文信息学报, 2019, 33(6): 108-115.
ZENG Feng, ZENG Biqing, HAN Xuli, et al. Double attention neural network for aspect-based sentiment analysis[J]. Journal of Chinese information processing, 2019, 33(6): 108-115.
[15] 谢珺, 王雨竹, 陈波, 等. 基于双指导注意力网络的属性情感分析模型[J]. 计算机研究与发展, 2022, 59(12): 2831-2843.
XIE Jun, WANG Yuzhu, CHEN Bo, et al. Aspect-based sentiment analysis model with Bi-guide attention network[J]. Journal of computer research and development, 2022, 59(12): 2831-2843.
[16] 李丽双, 周安桥, 刘阳, 等. 基于动态注意力GRU的特定目标情感分类[J]. 中国科学(信息科学), 2019, 49(8): 1019-1030.
LI Lishuang, ZHOU Anqiao, LIU Yang, et al. Aspect-based sentiment analysis based on dynamic attention GRU[J]. Scientia sinica (informationis), 2019, 49(8): 1019-1030.
[17] WU Zhengxuan, ONG D C. Context-guided BERT for targeted aspect-based sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI Press, 2021: 14094-14102.
[18] 曾凡旭, 李旭, 姚春龙, 等. 基于BERT的端到端方面级情感分析[J]. 大连工业大学学报, 2022, 41(3): 228-234.
ZENG Fanxu, LI Xu, YAO Chunlong, et al. End-to-end aspect-level sentiment analysis based on BERT[J]. Journal of Dalian Polytechnic University, 2022, 41(3): 228-234.
[19] GU Donghong, WANG Jiaqian, CAI Shaohua, et al. Targeted aspect-based multimodal sentiment analysis: an attention capsule extraction and multi-head fusion network[J]. IEEE access, 2021, 9: 157329-157336.
[20] 范东旭, 过弋. 基于可信细粒度对齐的多模态方面级情感分析[J]. 计算机科学, 2023, 50(12): 246-254.
FAN Dongxu, GUO Yi. Aspect-based multimodal sentiment analysis based on trusted fine-grained alignment[J]. Computer science, 2023, 50(12): 246-254.
[21] ZHOU Jie, ZHAO Jiabao, HUANG J X, et al. MASAD: a large-scale dataset for multimodal aspect-based sentiment analysis[J]. Neurocomputing, 2021, 455: 47-58.
[22] YU Jianfei, JIANG Jing. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: International Joint Conferences on Artificial Intelligence Organization, 2019: 5408-5414.
[23] LING Yan, YU Jianfei, XIA Rui. Vision-language pre-training for multimodal aspect-based sentiment analysis[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2022: 2149-2159.
[24] DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-11-11)[2021-01-01]. http://arxiv.org/abs/1810.04805.
[25] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(6): 1137-1149.
[26] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[27] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. (2017-06-12)[2021-01-01]. http://arxiv.org/abs/1706.03762.
[28] CHUNG J, GULCEHRE C, CHO K, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL]. (2014-12-11)[2021-01-01]. http://arxiv.org/abs/1412.3555.
[29] WANG Xin, CHEN Yudong, ZHU Wenwu. A survey on curriculum learning[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(9): 4555-4576.
[30] ZHANG Qi, FU Jinlan, LIU Xiaoyu, et al. Adaptive co-attention network for named entity recognition in tweets[J]. Proceedings of the AAAI conference on artificial intelligence. New Orleans: AAAI Press, 2018: 5674-5681.
[31] NGUYEN D Q, VU T, TUAN NGUYEN A. BERTweet: a pre-trained language model for English Tweets[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Stroudsburg: Association for Computational Linguistics, 2020: 9-14.
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]李雪,蒋树强.智能交互的物体识别增量学习技术综述[J].智能系统学报,2017,12(2):140.[doi:10.11992/tis.201701006]
LI Xue,JIANG Shuqiang.Incremental learning and object recognition system based on intelligent HCI: a survey[J].CAAI Transactions on Intelligent Systems,2017,12():140.[doi:10.11992/tis.201701006]
[3]吴钟强,张耀文,商琳.基于语义特征的多视图情感分类方法[J].智能系统学报,2017,12(5):745.[doi:10.11992/tis.201706026]
WU Zhongqiang,ZHANG Yaowen,SHANG Lin.Multi-view sentiment classification of microblogs based on semantic features[J].CAAI Transactions on Intelligent Systems,2017,12():745.[doi:10.11992/tis.201706026]
[4]侯涛,张强.基于T-S型模糊加权的多模软切换的风电机组变桨控制[J].智能系统学报,2018,13(4):625.[doi:10.11992/tis.201710009]
HOU Tao,ZHANG Qiang.Multi-mode soft-switch variable-pitch control of wind turbines based on T-S fuzzy weighting[J].CAAI Transactions on Intelligent Systems,2018,13():625.[doi:10.11992/tis.201710009]
[5]张新钰,邹镇洪,李志伟,等.面向自动驾驶目标检测的深度多模态融合技术[J].智能系统学报,2020,15(4):758.[doi:10.11992/tis.202002010]
ZHANG Xinyu,ZOU Zhenhong,LI Zhiwei,et al.Deep multi-modal fusion in object detection for autonomous driving[J].CAAI Transactions on Intelligent Systems,2020,15():758.[doi:10.11992/tis.202002010]
[6]陈玲玲,毕晓君.多模态融合网络的睡眠分期研究[J].智能系统学报,2022,17(6):1194.[doi:10.11992/tis.202202018]
CHEN Lingling,BI Xiaojun.Sleep staging model based on multimodal fusion[J].CAAI Transactions on Intelligent Systems,2022,17():1194.[doi:10.11992/tis.202202018]
[7]丁维昌,施俊,王骏.自监督对比特征学习的多模态乳腺超声诊断[J].智能系统学报,2023,18(1):66.[doi:10.11992/tis.202111052]
DING Weichang,SHI Jun,WANG Jun.Multi-modality ultrasound diagnosis of the breast with self-supervised contrastive feature learning[J].CAAI Transactions on Intelligent Systems,2023,18():66.[doi:10.11992/tis.202111052]
[8]亢洁,刘威.特征融合的装修案例跨模态检索方法[J].智能系统学报,2024,19(2):429.[doi:10.11992/tis.202207030]
KANG Jie,LIU Wei.A cross-modal retrieval algorithm of decoration cases on feature fusion[J].CAAI Transactions on Intelligent Systems,2024,19():429.[doi:10.11992/tis.202207030]
[9]段文杰,邓金科,张顺香,等.基于多层次知识增强的方面级情感分析模型[J].智能系统学报,2024,19(5):1287.[doi:10.11992/tis.202308044]
DUAN Wenjie,DENG Jinke,ZHANG Shunxiang,et al.Aspect-based sentiment analysis model based on multilevel knowledge enhancement[J].CAAI Transactions on Intelligent Systems,2024,19():1287.[doi:10.11992/tis.202308044]
[10]吴国栋,秦辉,胡全兴,等.大语言模型及其个性化推荐研究[J].智能系统学报,2024,19(6):1351.[doi:10.11992/tis.202309036]
WU Guodong,QIN Hui,HU Quanxing,et al.Research on large language models and personalized recommendation[J].CAAI Transactions on Intelligent Systems,2024,19():1351.[doi:10.11992/tis.202309036]
备注/Memo
收稿日期:2024-4-11。
基金项目:国家自然科学基金项目（U21B2009）；横向科技项目（2023110051000823）.
作者简介:朱超杰，硕士研究生，主要研究方向为多模态情感分析。E-mail：zcj1191098231@163.com;闫昱名，北京华电电子商务科技有限公司工程师，主要研究方向为信息化项目管理及实施。E-mail：yanyuming@chd.com.cn;高小燕，讲师，博士，主要研究方向为自然语言处理。E-mail：gaoxiaoyan@bjut.edu.cn
通讯作者:高小燕. E-mail：gaoxiaoyan@bjut.edu.cn
更新日期/Last Update:
2024-11-05