[1]ZHAO Xuefeng,DI Hengxi,BAI Changze,et al.Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism[J].CAAI Transactions on Intelligent Systems,2025,20(6):1461-1473.[doi:10.11992/tis.202503032]
Copy

Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 20 Number of periods: 2025 6 Page number: 1461-1473 Column: 学术论文—机器感知与模式识别 Public date: 2025-11-05
Title:
Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism
Author(s):
ZHAO Xuefeng;  DI Hengxi;  BAI Changze;  ZHONG Zhaoman;  ZHONG Xiaomin
College of Computer Engineering, Jiangsu Ocean University, Lianyungang 222005, China
Keywords:
global feature; multimodal; aspect-based sentiment analysis; text description; gating mechanism; cross attention; image-prompt; pre-trained language model
CLC:
TP391
DOI:
10.11992/tis.202503032
Abstract:
Existing multimodal aspect-based sentiment analysis models only extract single global image features, thereby overlooking key detailed information. To address this issue, this study proposes a network model that combines multifaceted image feature extraction and a gated fusion mechanism. Specifically, a multifaceted image feature extraction module is constructed in the proposed model. By leveraging cross-modal translation technology, textual descriptions of scenes, human faces, objects, and colors are generated from multiple sentiment-related dimensions of the image. This process achieves detailed information extraction and cross-modal information alignment. Furthermore, a gated fusion interaction module has been developed, incorporating a gating mechanism and interactive attention to facilitate efficient fusion and interaction between features. In order to address the representation gap across different modalities, sequence information is integrated with image prompts to convert image features into the input space of the pre-trained language model (PLM). This facilitates more accurate sentiment classification. Experiments conducted on the Twitter-2015 and Twitter-2017 datasets demonstrate that compared with existing models, the proposed model achieves an average improvement of 0.93% in accuracy and 0.52% in F1-score, effectively enhancing the performance of sentiment classification.
References:
[1] WANG Hongbin, REN Chun, YU Zhengtao. Multimodal sentiment analysis based on multiple attention[J]. Engineering applications of artificial intelligence, 2025, 140: 109731.
[2] 曾子明, 孙守强, 李青青. 基于融合策略的突发公共卫生事件网络舆情多模态负面情感识别[J]. 情报学报, 2023, 42(5): 611-622.
ZENG Ziming, SUN Shouqiang, LI Qingqing. Multimodal negative sentiment recognition in online public opinion during public health emergencies based on fusion strategy[J]. Journal of the China society for scientific and technical information, 2023, 42(5): 611-622.
[3] XU Nan, MAO Wenji, CHEN Guandan. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI conference on artificial intelligence. Honolulu: AAAI Press, 2019: 371-378.
[4] WANG Jiawei, LIU Zhe, SHENG V, et al. SaliencyBERT: recurrent attention network for target-oriented multimodal sentiment classification[M]//Pattern Recognition and Computer Vision. Cham: Springer International Publishing, 2021: 3-15.
[5] KHAN Z, FU Yun. Exploiting BERT for multimodal target sentiment classification through input space translation[C]//Proceedings of the 29th ACM International Conference on Multimedia. [S.l.]: ACM, 2021: 3034-3042.
[6] YANG Hao, ZHAO Yanyan, QIN Bing. Face-sensitive image-to-emotional-text cross-modal translation for multimodal aspect-based sentiment analysis[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: USAACL, 2022: 3324-3335.
[7] 曾碧卿, 姚勇涛, 谢梁琦, 等. 结合局部感知与多层次注意力的多模态方面级情感分析[J]. 计算机工程, 2025, 51(9): 80-90.
ZENG Biqing, Yao Yongtao, Xie Liangqi, et al. Multimodal aspect level emotion Analysis combining local perception and multi-levelattention[J]. Computer engineering, 2025, 51(9): 80-90.
[8] 余本功, 陈明玥. 基于细粒度图像–方面的情感增强方面级情感分析[J]. 计算机应用研究, 2025, 42(4): 1073-1079.
YU Bengong, CHEN Mingyue. Aspect-oriented affective knowledge enhanced for aspect-based sentiment analysis[J]. Application research of computers, 2025, 42(4): 1073-1079.
[9] 曹银妮, 韩虎, 黄明伟, 等. 基于多视角融合表示的多模态方面级情感分析模型[J/OL]. 数据分析与知识发现, [2024-01-01]. https://link.cnki.net/urlid/10.1478.G2.20250305.1130.002.
CAO Yinni, HAN Hu, HUANG Mingwei, et al. Multimodal aspect level sentiment analysis model based on multi-perspective fusion representation [J/OL]. Data analysis and knowledge discovery, [2024-01-01]. https://link.cnki.net/urlid/10.1478.G2.20250305.1130.002.
[10] ?LVAREZ-L?PEZ T, JUNCAL-MART?NEZ J, FERN?NDEZ-GAVILANES M, et al. GTI at SemEval-2016 task 5: SVM and CRF for aspect detection and unsupervised aspect-based sentiment analysis[C]//Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). San Dieg: USAACL, 2016: 306-311.
[11] LIU Ning, SHEN Bo. Aspect-based sentiment analysis with gated alternate neural network[J]. Knowledge-based systems, 2020, 188: 105010.
[12] AN Wenbin, TIAN Feng, CHEN Ping, et al. Aspect-based sentiment analysis with heterogeneous graph neural network[J]. IEEE transactions on computational social systems, 2023, 10(1): 403-412.
[13] 段文杰, 邓金科, 张顺香, 等. 基于多层次知识增强的方面级情感分析模型[J]. 智能系统学报, 2024, 19(5): 1287-1297.
DUAN Wenjie, DENG Jinke, ZHANG Shunxiang, et al. Aspect-based sentiment analysis model based on multilevel knowledge enhancement[J]. CAAI transactions on intelligent systems, 2024, 19(5): 1287-1297.
[14] HE Bo, ZHAO Ruoyu, TANG Dali. CABiLSTM-BERT: Aspect-based sentiment analysis model based on deep implicit feature extraction[J]. Knowledge-based systems, 2025, 309: 112782.
[15] LI Xin, BING Lidong, WAI Lam, et al. Transformation networks for target-oriented sentiment classification[C]//Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 946-956.
[16] ZHAO Chuanjun, FENG Rong, SUN Xuzhuang, et al. Enhancing aspect-based sentiment analysis with BERT-driven context generation and quality filtering[J]. Natural language processing journal, 2024, 7: 100077.
[17] ALI KANDHRO I, ALI F, UDDIN M, et al. Exploring aspect-based sentiment analysis: an in-depth review of current methods and prospects for advancement[J]. Knowledge and information systems, 2024, 66(7): 3639-3669.
[18] 陈燕, 赖宇斌, 肖澳, 等. 基于CLIP和交叉注意力的多模态情感分析模型[J]. 郑州大学学报(工学版), 2024, 45(2): 42-50.
CHEN Yan, LAI Yubin, XIAO Ao, et al. Multimodal sentiment analysis model based on CLIP and cross-attention[J]. Journal of Zhengzhou University (engineering science), 2024, 45(2): 42-50.
[19] 张铭泉, 周辉, 曹锦纲. 基于注意力机制的双BERT有向情感文本分类研究[J]. 智能系统学报, 2022, 17(6): 1220-1227.
ZHANG Mingquan, ZHOU Hui, CAO Jingang. Dual BERT directed sentiment text classification based on attention mechanism[J]. CAAI transactions on intelligent systems, 2022, 17(6): 1220-1227.
[20] LIANG Bin, SU Hang, GUI Lin, et al. Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks[J]. Knowledge-based systems, 2022, 235: 107643.
[21] LEE C, LEE Hanyong, KIM K, et al. An efficient fine-tuning of generative language model for aspect-based sentiment analysis[C]//2024 IEEE International Conference on Consumer Electronics. Las Vegas: IEEE, 2024: 1-4.
[22] YU Jianfei, JIANG Jing. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: International Joint Conferences on Artificial Intelligence Organization, 2019: 5408–5414.
[23] LI You, DING Han, LIN Yuming, et al. Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis[J]. Artificial intelligence review, 2024, 57(4): 78.
[24] 杨颖, 钱馨雨, 王合宁. 结合多粒度视图动态融合的多模态方面级情感分析[J]. 计算机工程与应用, 2024, 60(22): 172-183.
YANG Ying, QIAN Xinyu, WANG Hening. Multimodal aspect-level sentiment analysis based on multi-granularity view dynamic fusion[J]. Computer engineering and applications, 2024, 60(22): 172-183.
[25] ZHAO Jun, YANG Fuping. Fusion with GCN and SE-ResNeXt network for aspect based multimodal sentiment analysis[C]//2023 IEEE 6th Information Technology, Networking, Electronic and Automation Control Conference. Chongqing: IEEE, 2023: 336-340.
[26] 朱超杰, 闫昱名, 初宝昌, 等. 采用目标注意力的方面级多模态情感分析研究[J]. 智能系统学报, 2024, 19(6): 1562-1572.
ZHU Chaojie, YAN Yuming, CHU Baochang, et al. Research on aspect-based multimodal sentiment Analysis using Target Attention[J]. CAAI transactions on intelligent systems, 2024, 19(6): 1562-1572.
[27] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[28] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-11-18)[2022-01-01]. https://arxiv.org/abs/1409.1556.
[29] XIAO Luwei, WU Xingjiao, YANG Shuwen, et al. Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis[J]. Information processing & management, 2023, 60(6): 103508.
[30] FAN Shaojing, SHEN Zhiqi, JIANG Ming, et al. Emotional attention: a study of image sentiment and visual attention[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7521-7531.
[31] SERENGIL S I, OZPINAR A. HyperExtended LightFace: a facial attribute analysis framework[C]//2021 International Conference on Engineering and Emerging Technologies. Istanbul: IEEE, 2021: 1-4.
[32] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of Machine Learning Research. Virtual: PMLR, 2021: 8748-8763.
[33] NGUYEN D Q, VU T, NGUYEN A T. Bertweet: a pretrained language model for English Tweets[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Online: EMNLP, 2020: 9-14.
[34] WANG Qianlong, XU Hongling, WEN Zhiyuan, et al. Image-to-text conversion and aspect-oriented filtration for multimodal aspect-based sentiment analysis[J]. IEEE transactions on affective computing, 2023, 15(3): 1264-1278.
[35] WANG Yequan, HUANG Minlie, ZHU Xiaoyan, et al. Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in NaturalLanguage Processing. Austin: USAACL, 2016: 606-615.
[36] CHEN Peng, SUN Zhongqian, BING Lidong, et al. Recurrent attention network on memory for aspect sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in NaturalLanguage Processing. Copenhagen: USAACL, 2017: 452-461.
[37] FAN Feifan, FENG Yansong, ZHAO Dongyan. Multi-grained attention network for aspect-level sentiment classification[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: USAACL, 2018: 3433-3442.
[38] YU Jianfei, JIANG Jing, XIA Rui. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM transactions on audio, speech, and language processing, 2019, 28: 429-439.
[39] ZHAO Fei, WU Zhen, LONG Siyu, et al. Learning from adjective-noun pairs: a knowledge-enhanced framework for target-oriented multimodal sentiment classification[C]// Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics 2022: 6784-6794.
[40] YU Jjianfei, WANG Jieming, XIA Rui, et al. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching[C]//Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. Vienna: IJCAI, 2022: 4482-4488.
[41] 张添植, 周刚, 刘洪波, 等. 基于文本和图像门控融合机制的多模态方面级情感分析[J]. 计算机科学, 2024, 51(9): 242-249.
ZHANG Tianzhi, ZHOU Gang, LIU Hongbo, et al. Text-image gated fusion mechanism for multimodal aspect-based sentiment analysis[J]. Computer science, 2024, 51(9): 242-249.
[42] ZHANG Tianzhi, ZHOU Gang, LU Jicang, et al. Text-image semantic relevance identification for aspect-based multimodal sentiment analysis[J]. PeerJ computer science, 2024, 10: e1904.
[43] 杨丽莎, 马常霞, 仲兆满, 等. 多尺度语义感知和注意力融合的多模态方面级情感分析模型[J]. 南京大学学报(自然科学), 2025, 61(2): 223-236.
YANG Lisha, MA Changxia, ZHONG Zhaoman, et al. Multiscale semantic perception and attention fusion for multimodal aspect-level sentiment analysis model[J]. Journal of Nanjing University (natural science), 2025, 61(2): 223-236.
[44] CHEN Yifan, XIONG Haoliang, LI Kuntao, et al. Relevance-aware visual entity filter network for multimodal aspect-based sentiment analysis[J]. International journal of machine learning and cybernetics, 2025, 16(2): 1389-1402.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems