[1]赵雪峰,狄恒西,柏长泽,等.结合多面图像特征提取和门控融合机制的多模态方面级情感分析[J].智能系统学报,2025,20(6):1461-1473.[doi:10.11992/tis.202503032]
ZHAO Xuefeng,DI Hengxi,BAI Changze,et al.Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism[J].CAAI Transactions on Intelligent Systems,2025,20(6):1461-1473.[doi:10.11992/tis.202503032]
点击复制
ZHAO Xuefeng,DI Hengxi,BAI Changze,et al.Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism[J].CAAI Transactions on Intelligent Systems,2025,20(6):1461-1473.[doi:10.11992/tis.202503032]
结合多面图像特征提取和门控融合机制的多模态方面级情感分析
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
20
期数:
2025年第6期
页码:
1461-1473
栏目:
学术论文—机器感知与模式识别
出版日期:
2025-11-05
- Title:
- Multimodal aspect-based sentiment analysis combining multifaceted image feature extraction and gated fusion mechanism
- Keywords:
- global feature; multimodal; aspect-based sentiment analysis; text description; gating mechanism; cross attention; image-prompt; pre-trained language model
- 分类号:
- TP391
- 摘要:
- 针对现阶段多模态方面级情感分析(multimodal aspect-based sentiment analysis, MABSA)模型仅提取单一图像全局特征、忽略关键细节信息的问题，提出一种结合多面图像特征提取和门控融合机制的网络模型。该模型通过构建多面图像特征提取模块，采用跨模态翻译技术，从图像中与情感相关的多个维度生成场景、人脸、物体和颜色文本描述，实现细节信息提取与跨模态信息对齐；设计门控融合交互模块，引入门控机制与交互注意力实现特征间的高效融合交互；为了弥补不同模态间的表示差距，构建融合图片提示的序列信息，将图像特征转换到预训练语言模型(pre-trained language model, PLM)的输入空间中，实现更准确的情感分类。在Twitter-2015和Twitter-2017数据集上的实验表明，该模型较现有模型在准确率和F1上平均提高0.93%和0.52%，能有效改善情感分类效果。
- Abstract:
- Existing multimodal aspect-based sentiment analysis models only extract single global image features, thereby overlooking key detailed information. To address this issue, this study proposes a network model that combines multifaceted image feature extraction and a gated fusion mechanism. Specifically, a multifaceted image feature extraction module is constructed in the proposed model. By leveraging cross-modal translation technology, textual descriptions of scenes, human faces, objects, and colors are generated from multiple sentiment-related dimensions of the image. This process achieves detailed information extraction and cross-modal information alignment. Furthermore, a gated fusion interaction module has been developed, incorporating a gating mechanism and interactive attention to facilitate efficient fusion and interaction between features. In order to address the representation gap across different modalities, sequence information is integrated with image prompts to convert image features into the input space of the pre-trained language model (PLM). This facilitates more accurate sentiment classification. Experiments conducted on the Twitter-2015 and Twitter-2017 datasets demonstrate that compared with existing models, the proposed model achieves an average improvement of 0.93% in accuracy and 0.52% in F1-score, effectively enhancing the performance of sentiment classification.
- 参考文献/References:
-
[1] WANG Hongbin, REN Chun, YU Zhengtao. Multimodal sentiment analysis based on multiple attention[J]. Engineering applications of artificial intelligence, 2025, 140: 109731.
[2] 曾子明, 孙守强, 李青青. 基于融合策略的突发公共卫生事件网络舆情多模态负面情感识别[J]. 情报学报, 2023, 42(5): 611-622.
ZENG Ziming, SUN Shouqiang, LI Qingqing. Multimodal negative sentiment recognition in online public opinion during public health emergencies based on fusion strategy[J]. Journal of the China society for scientific and technical information, 2023, 42(5): 611-622.
[3] XU Nan, MAO Wenji, CHEN Guandan. Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI conference on artificial intelligence. Honolulu: AAAI Press, 2019: 371-378.
[4] WANG Jiawei, LIU Zhe, SHENG V, et al. SaliencyBERT: recurrent attention network for target-oriented multimodal sentiment classification[M]//Pattern Recognition and Computer Vision. Cham: Springer International Publishing, 2021: 3-15.
[5] KHAN Z, FU Yun. Exploiting BERT for multimodal target sentiment classification through input space translation[C]//Proceedings of the 29th ACM International Conference on Multimedia. [S.l.]: ACM, 2021: 3034-3042.
[6] YANG Hao, ZHAO Yanyan, QIN Bing. Face-sensitive image-to-emotional-text cross-modal translation for multimodal aspect-based sentiment analysis[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: USAACL, 2022: 3324-3335.
[7] 曾碧卿, 姚勇涛, 谢梁琦, 等. 结合局部感知与多层次注意力的多模态方面级情感分析[J]. 计算机工程, 2025, 51(9): 80-90.
ZENG Biqing, Yao Yongtao, Xie Liangqi, et al. Multimodal aspect level emotion Analysis combining local perception and multi-levelattention[J]. Computer engineering, 2025, 51(9): 80-90.
[8] 余本功, 陈明玥. 基于细粒度图像–方面的情感增强方面级情感分析[J]. 计算机应用研究, 2025, 42(4): 1073-1079.
YU Bengong, CHEN Mingyue. Aspect-oriented affective knowledge enhanced for aspect-based sentiment analysis[J]. Application research of computers, 2025, 42(4): 1073-1079.
[9] 曹银妮, 韩虎, 黄明伟, 等. 基于多视角融合表示的多模态方面级情感分析模型[J/OL]. 数据分析与知识发现, [2024-01-01]. https://link.cnki.net/urlid/10.1478.G2.20250305.1130.002.
CAO Yinni, HAN Hu, HUANG Mingwei, et al. Multimodal aspect level sentiment analysis model based on multi-perspective fusion representation [J/OL]. Data analysis and knowledge discovery, [2024-01-01]. https://link.cnki.net/urlid/10.1478.G2.20250305.1130.002.
[10] ?LVAREZ-L?PEZ T, JUNCAL-MART?NEZ J, FERN?NDEZ-GAVILANES M, et al. GTI at SemEval-2016 task 5: SVM and CRF for aspect detection and unsupervised aspect-based sentiment analysis[C]//Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016). San Dieg: USAACL, 2016: 306-311.
[11] LIU Ning, SHEN Bo. Aspect-based sentiment analysis with gated alternate neural network[J]. Knowledge-based systems, 2020, 188: 105010.
[12] AN Wenbin, TIAN Feng, CHEN Ping, et al. Aspect-based sentiment analysis with heterogeneous graph neural network[J]. IEEE transactions on computational social systems, 2023, 10(1): 403-412.
[13] 段文杰, 邓金科, 张顺香, 等. 基于多层次知识增强的方面级情感分析模型[J]. 智能系统学报, 2024, 19(5): 1287-1297.
DUAN Wenjie, DENG Jinke, ZHANG Shunxiang, et al. Aspect-based sentiment analysis model based on multilevel knowledge enhancement[J]. CAAI transactions on intelligent systems, 2024, 19(5): 1287-1297.
[14] HE Bo, ZHAO Ruoyu, TANG Dali. CABiLSTM-BERT: Aspect-based sentiment analysis model based on deep implicit feature extraction[J]. Knowledge-based systems, 2025, 309: 112782.
[15] LI Xin, BING Lidong, WAI Lam, et al. Transformation networks for target-oriented sentiment classification[C]//Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 946-956.
[16] ZHAO Chuanjun, FENG Rong, SUN Xuzhuang, et al. Enhancing aspect-based sentiment analysis with BERT-driven context generation and quality filtering[J]. Natural language processing journal, 2024, 7: 100077.
[17] ALI KANDHRO I, ALI F, UDDIN M, et al. Exploring aspect-based sentiment analysis: an in-depth review of current methods and prospects for advancement[J]. Knowledge and information systems, 2024, 66(7): 3639-3669.
[18] 陈燕, 赖宇斌, 肖澳, 等. 基于CLIP和交叉注意力的多模态情感分析模型[J]. 郑州大学学报(工学版), 2024, 45(2): 42-50.
CHEN Yan, LAI Yubin, XIAO Ao, et al. Multimodal sentiment analysis model based on CLIP and cross-attention[J]. Journal of Zhengzhou University (engineering science), 2024, 45(2): 42-50.
[19] 张铭泉, 周辉, 曹锦纲. 基于注意力机制的双BERT有向情感文本分类研究[J]. 智能系统学报, 2022, 17(6): 1220-1227.
ZHANG Mingquan, ZHOU Hui, CAO Jingang. Dual BERT directed sentiment text classification based on attention mechanism[J]. CAAI transactions on intelligent systems, 2022, 17(6): 1220-1227.
[20] LIANG Bin, SU Hang, GUI Lin, et al. Aspect-based sentiment analysis via affective knowledge enhanced graph convolutional networks[J]. Knowledge-based systems, 2022, 235: 107643.
[21] LEE C, LEE Hanyong, KIM K, et al. An efficient fine-tuning of generative language model for aspect-based sentiment analysis[C]//2024 IEEE International Conference on Consumer Electronics. Las Vegas: IEEE, 2024: 1-4.
[22] YU Jianfei, JIANG Jing. Adapting BERT for target-oriented multimodal sentiment classification[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. Macao: International Joint Conferences on Artificial Intelligence Organization, 2019: 5408–5414.
[23] LI You, DING Han, LIN Yuming, et al. Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis[J]. Artificial intelligence review, 2024, 57(4): 78.
[24] 杨颖, 钱馨雨, 王合宁. 结合多粒度视图动态融合的多模态方面级情感分析[J]. 计算机工程与应用, 2024, 60(22): 172-183.
YANG Ying, QIAN Xinyu, WANG Hening. Multimodal aspect-level sentiment analysis based on multi-granularity view dynamic fusion[J]. Computer engineering and applications, 2024, 60(22): 172-183.
[25] ZHAO Jun, YANG Fuping. Fusion with GCN and SE-ResNeXt network for aspect based multimodal sentiment analysis[C]//2023 IEEE 6th Information Technology, Networking, Electronic and Automation Control Conference. Chongqing: IEEE, 2023: 336-340.
[26] 朱超杰, 闫昱名, 初宝昌, 等. 采用目标注意力的方面级多模态情感分析研究[J]. 智能系统学报, 2024, 19(6): 1562-1572.
ZHU Chaojie, YAN Yuming, CHU Baochang, et al. Research on aspect-based multimodal sentiment Analysis using Target Attention[J]. CAAI transactions on intelligent systems, 2024, 19(6): 1562-1572.
[27] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[28] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-11-18)[2022-01-01]. https://arxiv.org/abs/1409.1556.
[29] XIAO Luwei, WU Xingjiao, YANG Shuwen, et al. Cross-modal fine-grained alignment and fusion network for multimodal aspect-based sentiment analysis[J]. Information processing & management, 2023, 60(6): 103508.
[30] FAN Shaojing, SHEN Zhiqi, JIANG Ming, et al. Emotional attention: a study of image sentiment and visual attention[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7521-7531.
[31] SERENGIL S I, OZPINAR A. HyperExtended LightFace: a facial attribute analysis framework[C]//2021 International Conference on Engineering and Emerging Technologies. Istanbul: IEEE, 2021: 1-4.
[32] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of Machine Learning Research. Virtual: PMLR, 2021: 8748-8763.
[33] NGUYEN D Q, VU T, NGUYEN A T. Bertweet: a pretrained language model for English Tweets[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Online: EMNLP, 2020: 9-14.
[34] WANG Qianlong, XU Hongling, WEN Zhiyuan, et al. Image-to-text conversion and aspect-oriented filtration for multimodal aspect-based sentiment analysis[J]. IEEE transactions on affective computing, 2023, 15(3): 1264-1278.
[35] WANG Yequan, HUANG Minlie, ZHU Xiaoyan, et al. Attention-based LSTM for aspect-level sentiment classification[C]//Proceedings of the 2016 Conference on Empirical Methods in NaturalLanguage Processing. Austin: USAACL, 2016: 606-615.
[36] CHEN Peng, SUN Zhongqian, BING Lidong, et al. Recurrent attention network on memory for aspect sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in NaturalLanguage Processing. Copenhagen: USAACL, 2017: 452-461.
[37] FAN Feifan, FENG Yansong, ZHAO Dongyan. Multi-grained attention network for aspect-level sentiment classification[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: USAACL, 2018: 3433-3442.
[38] YU Jianfei, JIANG Jing, XIA Rui. Entity-sensitive attention and fusion network for entity-level multimodal sentiment classification[J]. IEEE/ACM transactions on audio, speech, and language processing, 2019, 28: 429-439.
[39] ZHAO Fei, WU Zhen, LONG Siyu, et al. Learning from adjective-noun pairs: a knowledge-enhanced framework for target-oriented multimodal sentiment classification[C]// Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics 2022: 6784-6794.
[40] YU Jjianfei, WANG Jieming, XIA Rui, et al. Targeted multimodal sentiment classification based on coarse-to-fine grained image-target matching[C]//Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. Vienna: IJCAI, 2022: 4482-4488.
[41] 张添植, 周刚, 刘洪波, 等. 基于文本和图像门控融合机制的多模态方面级情感分析[J]. 计算机科学, 2024, 51(9): 242-249.
ZHANG Tianzhi, ZHOU Gang, LIU Hongbo, et al. Text-image gated fusion mechanism for multimodal aspect-based sentiment analysis[J]. Computer science, 2024, 51(9): 242-249.
[42] ZHANG Tianzhi, ZHOU Gang, LU Jicang, et al. Text-image semantic relevance identification for aspect-based multimodal sentiment analysis[J]. PeerJ computer science, 2024, 10: e1904.
[43] 杨丽莎, 马常霞, 仲兆满, 等. 多尺度语义感知和注意力融合的多模态方面级情感分析模型[J]. 南京大学学报(自然科学), 2025, 61(2): 223-236.
YANG Lisha, MA Changxia, ZHONG Zhaoman, et al. Multiscale semantic perception and attention fusion for multimodal aspect-level sentiment analysis model[J]. Journal of Nanjing University (natural science), 2025, 61(2): 223-236.
[44] CHEN Yifan, XIONG Haoliang, LI Kuntao, et al. Relevance-aware visual entity filter network for multimodal aspect-based sentiment analysis[J]. International journal of machine learning and cybernetics, 2025, 16(2): 1389-1402.
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]孙劲光,孟凡宇.一种特征加权融合人脸识别方法[J].智能系统学报,2015,10(6):912.[doi:10.11992/tis.201509025]
SUN Jinguang,MENG Fanyu.Face recognition by weighted fusion of facial features[J].CAAI Transactions on Intelligent Systems,2015,10():912.[doi:10.11992/tis.201509025]
[3]李雪,蒋树强.智能交互的物体识别增量学习技术综述[J].智能系统学报,2017,12(2):140.[doi:10.11992/tis.201701006]
LI Xue,JIANG Shuqiang.Incremental learning and object recognition system based on intelligent HCI: a survey[J].CAAI Transactions on Intelligent Systems,2017,12():140.[doi:10.11992/tis.201701006]
[4]吴钟强,张耀文,商琳.基于语义特征的多视图情感分类方法[J].智能系统学报,2017,12(5):745.[doi:10.11992/tis.201706026]
WU Zhongqiang,ZHANG Yaowen,SHANG Lin.Multi-view sentiment classification of microblogs based on semantic features[J].CAAI Transactions on Intelligent Systems,2017,12():745.[doi:10.11992/tis.201706026]
[5]侯涛,张强.基于T-S型模糊加权的多模软切换的风电机组变桨控制[J].智能系统学报,2018,13(4):625.[doi:10.11992/tis.201710009]
HOU Tao,ZHANG Qiang.Multi-mode soft-switch variable-pitch control of wind turbines based on T-S fuzzy weighting[J].CAAI Transactions on Intelligent Systems,2018,13():625.[doi:10.11992/tis.201710009]
[6]张新钰,邹镇洪,李志伟,等.面向自动驾驶目标检测的深度多模态融合技术[J].智能系统学报,2020,15(4):758.[doi:10.11992/tis.202002010]
ZHANG Xinyu,ZOU Zhenhong,LI Zhiwei,et al.Deep multi-modal fusion in object detection for autonomous driving[J].CAAI Transactions on Intelligent Systems,2020,15():758.[doi:10.11992/tis.202002010]
[7]肖宇晗,林慧苹,汪权彬,等.基于双特征嵌套注意力的方面词情感分析算法[J].智能系统学报,2021,16(1):142.[doi:10.11992/tis.202012024]
XIAO Yuhan,LIN Huiping,WANG Quanbin,et al.An algorithm for aspect-based sentiment analysis based on dual features attention-over-attention[J].CAAI Transactions on Intelligent Systems,2021,16():142.[doi:10.11992/tis.202012024]
[8]陈玲玲,毕晓君.多模态融合网络的睡眠分期研究[J].智能系统学报,2022,17(6):1194.[doi:10.11992/tis.202202018]
CHEN Lingling,BI Xiaojun.Sleep staging model based on multimodal fusion[J].CAAI Transactions on Intelligent Systems,2022,17():1194.[doi:10.11992/tis.202202018]
[9]丁维昌,施俊,王骏.自监督对比特征学习的多模态乳腺超声诊断[J].智能系统学报,2023,18(1):66.[doi:10.11992/tis.202111052]
DING Weichang,SHI Jun,WANG Jun.Multi-modality ultrasound diagnosis of the breast with self-supervised contrastive feature learning[J].CAAI Transactions on Intelligent Systems,2023,18():66.[doi:10.11992/tis.202111052]
[10]亢洁,刘威.特征融合的装修案例跨模态检索方法[J].智能系统学报,2024,19(2):429.[doi:10.11992/tis.202207030]
KANG Jie,LIU Wei.A cross-modal retrieval algorithm of decoration cases on feature fusion[J].CAAI Transactions on Intelligent Systems,2024,19():429.[doi:10.11992/tis.202207030]
备注/Memo
收稿日期:2025-3-24。
基金项目:国家自然科学基金项目(72174079)；江苏省“青蓝工程”优秀教学团队项目(2022-29).
作者简介:赵雪峰，副教授，博士，江苏省科技副总，主要研究方向为多模态情感分析、数字图像处理与无损检测。作为主要成员主持、参与完成省市级项目4项，发表学术论文20余篇。E-mail：zhaoxf@jou.edu.cn。;狄恒西，硕士研究生, 主要研究方向为多模态情感分析、自然语言处理。E-mail：dihx@jou.edu.cn。;柏长泽，硕士研究生, 主要研究方向为多模态情感分析、自然语言处理。E-mail：2023220901@jou.edu。
通讯作者:赵雪峰. E-mail：zhaoxf@jou.edu.cn
更新日期/Last Update:
1900-01-01