[1]胡文彬,陈龙,黄贤波,等.融合交叉注意力的突发事件多模态中文反讽识别模型[J].智能系统学报,2024,19(2):392-400.[doi:10.11992/tis.202212011]
HU Wenbin,CHEN Long,HUANG Xianbo,et al.A multimodal Chinese sarcasm detection model for emergencies based on cross attention[J].CAAI Transactions on Intelligent Systems,2024,19(2):392-400.[doi:10.11992/tis.202212011]
点击复制
HU Wenbin,CHEN Long,HUANG Xianbo,et al.A multimodal Chinese sarcasm detection model for emergencies based on cross attention[J].CAAI Transactions on Intelligent Systems,2024,19(2):392-400.[doi:10.11992/tis.202212011]
融合交叉注意力的突发事件多模态中文反讽识别模型
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第2期
页码:
392-400
栏目:
学术论文—自然语言处理与理解
出版日期:
2024-03-05
- Title:
- A multimodal Chinese sarcasm detection model for emergencies based on cross attention
- Keywords:
- emergency; social media; multimodal comment; Chinese sarcasm detection; Chinese sarcasm dataset; cross-attention mechanism; attention mechanism; sentiment analysis
- 分类号:
- TP391
- 文献标志码:
- 2023-12-21
- 摘要:
- 网民在社交媒体参与突发事件讨论时，时常会采用反讽修辞方式表达对事件的看法，此举导致情感分析的难度增加，且已有中文反讽识别对社交媒体中网民发布的多模态评论研究较少，有必要对图文多模态中文反讽识别进行深入研究。运用交叉注意力机制捕捉模态间的不一致性表达，提出融合交叉注意力的多模态中文反讽识别模型(fuse cross attention model, FCAM)。在模型中，首先运用TextCNN(text convolutional neural networks) 和ResNet(deep residual network)分别提取中文文本浅层特征和图像特征，再运用交叉注意力机制分别得到文本层和图像层的注意力特征，按照残差方式分别实现文本浅层特征和文本层注意力特征的连接、图像特征和图像层注意力特征的连接，使用注意力机制融合2个特征表示，经过分类层得到反讽分类结果。基于某一地区新冠疫情期间相关话题的微博评论数据，构建出突发公共卫生事件多模态中文反讽数据集，在该数据集上试验验证，相较于基准模型，FCAM具有一定的优越性。
- Abstract:
- Internet users often use sarcasm when discussing emergencies on social media, which complicates emotional analysis. In addition, there is a lack of research on multimodal comments, particularly those in Chinese, and their use of sarcasm on social media platforms. Therefore, it is necessary to delve deeper into sarcasm detection in multimodal Chinese content, specifically within images and text. To address this need, we propose a multimodal Chinese sarcasm detection model called the fuse cross-attention model (FCAM). This model incorporates a cross-attention mechanism to identify inconsistencies between modes. The text convolutional neural network (TextCNN) is used to extract basic features of Chinese text, while the deep residential network (ResNet) is used to extract image features. The cross-attention mechanism is used to obtain attention features from the text and image layers. The residual method is employed to establish a connection between the basic text features and the text layer’s attention features, as well as a link between the image features and the image layer’s attention features. These two feature representations are fused using the attention mechanism, resulting in the sarcasm classification results through the classification layer. We have constructed a multimodal Chinese sarcasm data set based on Weibo comment data related to the COVID-19 pandemic in a specific region. Experimental testing on this data set confirms that FCAM holds certain advantages over the benchmark model.
- 参考文献/References:
-
[1] ZHANG M, ZHANG Y, FU G. Tweet sarcasm detection using deep neural network[C]// International Conference on Computational Linguistics. Osaka: The COLING 2016 Organizing Committee, 2016: 2449-2460.
[2] GHOSH A, VEALE D T. Fracking sarcasm using neural network[C]//Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Stroudsburg: Association for Computational Linguistics, 2016: 161-169.
[3] KHOTIJAH S, TIRTAWANGSA J, SURYANI A A. Using LSTM for context based approach of sarcasm detection in twitter[C]//Proceedings of the 11th International Conference on Advances in Information Technology. New York: ACM, 2020: 1-7.
[4] LOU Chenwei, LIANG Bin, GUI Lin, et al. Affective dependency graph for sarcasm detection[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2021: 1844-1849.
[5] 孙晓, 何家劲, 任福继. 基于多特征融合的混合神经网络模型讽刺语用判别[J]. 中文信息学报, 2016, 30(6): 215–223
SUN Xiao, HE Jiajin, REN Fuji. Pragmatic analysis of irony based on hybrid neural network model with multi-feature[J]. Journal of Chinese information processing, 2016, 30(6): 215–223
[6] 卢欣, 李旸, 王素格. 融合语言特征的卷积神经网络的反讽识别方法[J]. 中文信息学报, 2019, 33(5): 31–38
LU Xin, LI Yang, WANG Suge. Linguistic features enhanced convolutional neural networks for irony recognition[J]. Journal of Chinese information processing, 2019, 33(5): 31–38
[7] 樊小超, 杨亮, 林鸿飞, 等. 基于多语义融合的反讽识别[J]. 中文信息学报, 2021, 35(6): 103–111
FAN Xiaochao, YANG Liang, LIN Hongfei, et al. Irony recognition based on multiple semantic fusion[J]. Journal of Chinese information processing, 2021, 35(6): 103–111
[8] SCHIFANELLA R, DE JUAN P, TETREAULT J, et al. Detecting sarcasm in multimodal social platforms[EB/OL]. (2016-08-08)[2021-01-01]. http://arxiv.org/abs/1608.02289.pdf
[9] SHARMA D K, SINGH B, AGARWAL S, et al. Sarcasm detection over social media platforms using hybrid auto-encoder-based model[J]. Electronics, 2022, 11(18): 2844.
[10] SANGWAN S, AKHTAR M S, BEHERA P, et al. I didn’t mean what I wrote! exploring multimodality for sarcasm detection[C]//2020 International Joint Conference on Neural Networks. Glasgow: IEEE, 2020: 1-8.
[11] CAI Yitao, CAI Huiyu, WAN Xiaojun. Multi-modal sarcasm detection in twitter with hierarchical fusion model[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2019: 2506-2515.
[12] PAN Hongliang, LIN Zheng, FU Peng, et al. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection[C]//Findings of the Association for Computational Linguistics: EMNLP 2020. Online. Stroudsburg: Association for Computational Linguistics, 2020: 1383-1392.
[13] YAO Fanglong, SUN Xian, YU Hongfeng, et al. Mimicking the brain’s cognition of sarcasm from multidisciplines for twitter sarcasm detection[J]. IEEE transactions on neural networks and learning systems, 2023, 34(1): 228–242.
[14] GUPTA S, SHAH A, SHAH M, et al. FiLMing multimodal sarcasm detection with attention[C]//International Conference on Neural Information Processing. Cham: Springer, 2021: 178-186.
[15] 张继东, 蒋丽萍. 基于多模态深度学习的旅游评论反讽识别研究[J]. 情报理论与实践, 2022, 45(7): 158–164
ZHANG Jidong, JIANG Liping. Research on irony recognition of travel reviews based on multi-modal deep learning[J]. Information studies:theory & application, 2022, 45(7): 158–164
[16] 刘洋, 马莉莉, 张雯, 等. 基于跨模态深度学习的旅游评论反讽识别[J]. 数据分析与知识发现, 2022, 6(12): 23–31
LIU Yang, MA Lili, ZHANG Wen, et al. Detecting sarcasm from travel reviews based on cross-modal deep learning[J]. Data analysis and knowledge discovery, 2022, 6(12): 23–31
[17] 刘美萍. 重大突发事件网络舆情协同治理机制构建研究[J]. 求实, 2022(5): 64–76,111
LIU Meiping. Research on mechanism construction of collaborative governance of online public opinion on major emergencies[J]. Truth seeking, 2022(5): 64–76,111
[18] MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. (2013-01-16)[2021-01-01]. http://arxiv.org/abs/1301.3781.pdf.
[19] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1746–1751.
[20] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[21] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Advances in Neural Information Processing Systems 30. Long Beach, USA, 2017: 5998-6008.
[22] WU Suyan, SU Entong, LEI Binyang, et al. TextCNN-based text classification for E-government[C]//2019 6th International Conference on Information Science and Control Engineering. Shanghai: IEEE, 2020: 929-934.
[23] SHARFUDDIN A A, TIHAMI N M, ISLAM S M. A deep recurrent neural network with BiLSTM model for sentiment classification[C]//2018 International Conference on Bangla Speech and Language Processing . Sylhet: IEEE, 2018: 1-4.
[24] DEVLIN J, CHANG MINGWEI , LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: [s. n. ], 2019: 4171–4186.
[25] 陆晓蕾, 倪斌. 基于预训练语言模型的BERT-CNN多层级专利分类研究[J]. 中文信息学报, 2021, 35(11): 70–79
LU Xiaolei, NI Bin. BERT-CNN: a hierarchical patent classifier based on pre-trained language model[J]. Journal of Chinese information processing, 2021, 35(11): 70–79
[26] LI Zhengguang, LIN Hongfei, SHEN Chen, et al. Cross 2 Self-attentive bidirectional recurrent neural network with BERT for biomedical semantic text similarity[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine. Seoul: IEEE, 2021: 1051-1054.
[27] KAUR T, GANDHI T K. Automated brain image classification based on VGG-16 and transfer learning[C]//2019 International Conference on Information Technology. Bhubaneswar: IEEE, 2020: 94-98.
- 相似文献/References:
-
[1]李汝亮,杜军平.多Agent的旅游突发事件预警系统[J].智能系统学报,2009,4(5):441.[doi:10.3969/j.issn.1673-4785.2009.05.009]
LI Ru-liang,DU Jun-ping.A tourist emergency alert system based on multi-Agents[J].CAAI Transactions on Intelligent Systems,2009,4():441.[doi:10.3969/j.issn.1673-4785.2009.05.009]
[2]黄立威,李德毅.社交媒体中的信息推荐[J].智能系统学报,2012,7(1):1.
HUANG Liwei,LI Deyi.A review of information recommendation in social media[J].CAAI Transactions on Intelligent Systems,2012,7():1.
[3]李凌霄,李绍滋,曹冬林.基于多情绪源关联模型的中文微博情感分析[J].智能系统学报,2016,11(4):546.[doi:10.11992/tis.201605019]
LI Lingxiao,LI Shaozi,CAO Donglin.Emotional multi-source correlation model for chinese micro-blog sentiment analysis[J].CAAI Transactions on Intelligent Systems,2016,11():546.[doi:10.11992/tis.201605019]
[4]许灵毓,钟义信,陈志成.社交机器人对社会舆论的影响因素研究[J].智能系统学报,2024,19(1):122.[doi:10.11992/tis.202305044]
XU Lingyu,ZHONG Yixin,CHEN Zhicheng.Research on the influence factors of social robots on social opinions[J].CAAI Transactions on Intelligent Systems,2024,19():122.[doi:10.11992/tis.202305044]
备注/Memo
收稿日期:2022-12-08。
基金项目:国家自然科学基金项目(72174079)；江苏省“青蓝工程”优秀教学团队(2022-29).
作者简介:胡文彬，副教授，博士，中国计算机学会会员，江苏省人工智能学会会员,主要研究方向为社会网络隐私保护、智能信息处理。作为主要成员主持、参与完成省、市级项目5项，获省级教学成果奖1项，省级教育教学与研究成果奖1项。参与撰写专著1项，发表学术论文近20篇。E-mail：hwb1008@163.com;陈龙，硕士研究生，主要研究方向为自然语言处理、舆情分析。E-mail:956779521@qq.com;黄贤波，硕士研究生，主要研究方向为情感分析、舆情管控。E-mail：764157719@qq.com
通讯作者:胡文彬. E-mail：hwb1008@163.com
更新日期/Last Update:
1900-01-01