[1]ZHU Chaojie,YAN Yuming,CHU Baochang,et al.Aspect-level multimodal sentiment analysis via object-attention[J].CAAI Transactions on Intelligent Systems,2024,19(6):1562-1572.[doi:10.11992/tis.202404009]
Copy

Aspect-level multimodal sentiment analysis via object-attention

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 6 Page number: 1562-1572 Column: 人工智能院长论坛 Public date: 2024-12-05
Title:
Aspect-level multimodal sentiment analysis via object-attention
Author(s):
ZHU Chaojie1;  YAN Yuming2;  CHU Baochang2;  LI Gang2;  HUANG Heyan1;  GAO Xiaoyan3
1. School of Computer Science & Technology, Beijing Institute of Technology, Beijing 100081, China;
2. Beijing Huadian E-Commerce Technology Co., Ltd., Beijing 100073, China;
3. Faculty of Information Technology, Beijing University of Technology, Be
Keywords:
aspect-level sentiment analysis; multimodal; sentiment analysis; object detection; self-attention; natural language processing systems; deep learning; feature extraction
CLC:
TP391
DOI:
10.11992/tis.202404009
Abstract:
Aspect-level multimodal sentiment analysis (ALMSA) aims to identify the sentiment polarity of a specific aspect word using both sentence and image data. Current models often rely on the global features of images, overlooking the details in the original image. To address this issue, we propose an object attention-based aspect-level multimodal sentiment analysis model (OAB-ALMSA). This model first employs an object detection algorithm to capture the detailed information of the objects from the original image. It then applies an object-attention mechanism and builds an iterative fusion layer to fully fuse the multimodal information. Finally, a curriculum learning strategy is developed to tackle the challenges of training with complex samples. Experiments conducted on TWITTER-2015 data sets demonstrate that OAB-ALMSA, when combined with curriculum learning, achieves the highest F1. These results highlight that leveraging detailed image data enhances the model’s overall understanding and improves prediction accuracy.
References:
[1] 中国互联网络信息中心. 中国互联网络发展状况统计报告[J]. 国家图书馆学刊, 2023, 32(2): 39.
[2] XU Nan, MAO Wenji, CHEN Guandan. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Hawaii: AAAI Press, 2019: 371-378.
[3] ZHANG Zhe, WANG Zhu, LI Xiaona, et al. ModalNet: an aspect-level sentiment classification model by exploring multimodal data with fusion discriminant attentional network[J]. World wide web, 2021, 24(6): 1957-1974.
[4] KHAN Z, FU Yun. Exploiting BERT for multimodal target sentiment classification through input space translation[C]//Proceedings of the 29th ACM International Conference on Multimedia. Virtual Event China: ACM, 2021: 3034-3042.
[5] 王伟贤, 吴俊. 基于情感词典与语义规则集的微博文本情感分析[J]. 计算机科学与应用, 2023, 13(4): 754-763.
WANG Weixian, WU Jun. Sentiment analysis of microblog text based on sentiment dictionary and semantic rule set[J]. Computer science and application, 2023, 13(4): 754-763.
[6] SHANG Yongmin, ZHAO Yuqin. Sentiment analysis and implementation of online reviews based on machine learning[J]. Journal of Dali University, 2021, 6(12): 80-86.
[7] HUANG Jiaxin, MENG Yu, GUO Fang, et al. Weakly-supervised aspect-based sentiment analysis via joint aspect-sentiment topic embedding[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2020.
[8] KIRITCHENKO S, ZHU X, CHERRY C, et al. Detecting aspects and sentiment in customer reviews[C]//8th International Workshop on Semantic Evaluation. [S.l.]: [s.n.], 2014, 437-442.
[9] 王婷, 杨文忠. 文本情感分析方法研究综述[J]. 计算机工程与应用, 2021, 57(12): 11-24.
WANG Ting, YANG Wenzhong. Review of text sentiment analysis methods[J]. Computer engineering and applications, 2021, 57(12): 11-24.
[10] LONDHE A, RAO P V R D P. Aspect based sentiment analysis–an incremental model learning approach using LSTM-RNN[M]//Communications in Computer and Information Science. Cham: Springer International Publishing, 2021: 677-689.
[11] XUE Wei, LI Tao. Aspect based sentiment analysis with gated convolutional networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2018: 2514-2523.
[12] 张文轩, 殷雁君, 智敏. 用于方面级情感分析的情感增强双图卷积网络[J]. 计算机科学与探索, 2024, 18(1): 217-230.
ZHANG Wenxuan, YIN Yanjun, ZHI Min. Affection enhanced dual graph convolution network for aspect based sentiment analysis[J]. Journal of frontiers of computer science and technology, 2024, 18(1): 217-230.
[13] WANG Bailin, LU Wei. Learning latent opinions for aspect-level sentiment classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans: AAAI Press, 2018: 5537-5544.
[14] 曾锋, 曾碧卿, 韩旭丽, 等. 基于双层注意力循环神经网络的方面级情感分析[J]. 中文信息学报, 2019, 33(6): 108-115.
ZENG Feng, ZENG Biqing, HAN Xuli, et al. Double attention neural network for aspect-based sentiment analysis[J]. Journal of Chinese information processing, 2019, 33(6): 108-115.
[15] 谢珺, 王雨竹, 陈波, 等. 基于双指导注意力网络的属性情感分析模型[J]. 计算机研究与发展, 2022, 59(12): 2831-2843.
XIE Jun, WANG Yuzhu, CHEN Bo, et al. Aspect-based sentiment analysis model with Bi-guide attention network[J]. Journal of computer research and development, 2022, 59(12): 2831-2843.
[16] 李丽双, 周安桥, 刘阳, 等. 基于动态注意力GRU的特定目标情感分类[J]. 中国科学(信息科学), 2019, 49(8): 1019-1030.
LI Lishuang, ZHOU Anqiao, LIU Yang, et al. Aspect-based sentiment analysis based on dynamic attention GRU[J]. Scientia sinica (informationis), 2019, 49(8): 1019-1030.
[17] WU Zhengxuan, ONG D C. Context-guided BERT for targeted aspect-based sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI Press, 2021: 14094-14102.
[18] 曾凡旭, 李旭, 姚春龙, 等. 基于BERT的端到端方面级情感分析[J]. 大连工业大学学报, 2022, 41(3): 228-234.
ZENG Fanxu, LI Xu, YAO Chunlong, et al. End-to-end aspect-level sentiment analysis based on BERT[J]. Journal of Dalian Polytechnic University, 2022, 41(3): 228-234.
[19] GU Donghong, WANG Jiaqian, CAI Shaohua, et al. Targeted aspect-based multimodal sentiment analysis: an attention capsule extraction and multi-head fusion network[J]. IEEE access, 2021, 9: 157329-157336.
[20] 范东旭, 过弋. 基于可信细粒度对齐的多模态方面级情感分析[J]. 计算机科学, 2023, 50(12): 246-254.
FAN Dongxu, GUO Yi. Aspect-based multimodal sentiment analysis based on trusted fine-grained alignment[J]. Computer science, 2023, 50(12): 246-254.
[21] ZHOU Jie, ZHAO Jiabao, HUANG J X, et al. MASAD: a large-scale dataset for multimodal aspect-based sentiment analysis[J]. Neurocomputing, 2021, 455: 47-58.
[22] YU Jianfei, JIANG Jing. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: International Joint Conferences on Artificial Intelligence Organization, 2019: 5408-5414.
[23] LING Yan, YU Jianfei, XIA Rui. Vision-language pre-training for multimodal aspect-based sentiment analysis[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2022: 2149-2159.
[24] DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018-11-11)[2021-01-01]. http://arxiv.org/abs/1810.04805.
[25] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(6): 1137-1149.
[26] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[27] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. (2017-06-12)[2021-01-01]. http://arxiv.org/abs/1706.03762.
[28] CHUNG J, GULCEHRE C, CHO K, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL]. (2014-12-11)[2021-01-01]. http://arxiv.org/abs/1412.3555.
[29] WANG Xin, CHEN Yudong, ZHU Wenwu. A survey on curriculum learning[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(9): 4555-4576.
[30] ZHANG Qi, FU Jinlan, LIU Xiaoyu, et al. Adaptive co-attention network for named entity recognition in tweets[J]. Proceedings of the AAAI conference on artificial intelligence. New Orleans: AAAI Press, 2018: 5674-5681.
[31] NGUYEN D Q, VU T, TUAN NGUYEN A. BERTweet: a pre-trained language model for English Tweets[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Stroudsburg: Association for Computational Linguistics, 2020: 9-14.
Similar References:

Memo

-

Last Update: 2024-11-05

Copyright © CAAI Transactions on Intelligent Systems