[1]HU Wenbin,CHEN Long,HUANG Xianbo,et al.A multimodal Chinese sarcasm detection model for emergencies based on cross attention[J].CAAI Transactions on Intelligent Systems,2024,19(2):392-400.[doi:10.11992/tis.202212011]
Copy
A multimodal Chinese sarcasm detection model for emergencies based on cross attention
CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume:
19
Number of periods:
2024 2
Page number:
392-400
Column:
学术论文—自然语言处理与理解
Public date:
2024-03-05
- Title:
- A multimodal Chinese sarcasm detection model for emergencies based on cross attention
- Keywords:
- emergency; social media; multimodal comment; Chinese sarcasm detection; Chinese sarcasm dataset; cross-attention mechanism; attention mechanism; sentiment analysis
- CLC:
- TP391
- DOI:
- 10.11992/tis.202212011
- Abstract:
- Internet users often use sarcasm when discussing emergencies on social media, which complicates emotional analysis. In addition, there is a lack of research on multimodal comments, particularly those in Chinese, and their use of sarcasm on social media platforms. Therefore, it is necessary to delve deeper into sarcasm detection in multimodal Chinese content, specifically within images and text. To address this need, we propose a multimodal Chinese sarcasm detection model called the fuse cross-attention model (FCAM). This model incorporates a cross-attention mechanism to identify inconsistencies between modes. The text convolutional neural network (TextCNN) is used to extract basic features of Chinese text, while the deep residential network (ResNet) is used to extract image features. The cross-attention mechanism is used to obtain attention features from the text and image layers. The residual method is employed to establish a connection between the basic text features and the text layer’s attention features, as well as a link between the image features and the image layer’s attention features. These two feature representations are fused using the attention mechanism, resulting in the sarcasm classification results through the classification layer. We have constructed a multimodal Chinese sarcasm data set based on Weibo comment data related to the COVID-19 pandemic in a specific region. Experimental testing on this data set confirms that FCAM holds certain advantages over the benchmark model.
- References:
-
[1] ZHANG M, ZHANG Y, FU G. Tweet sarcasm detection using deep neural network[C]// International Conference on Computational Linguistics. Osaka: The COLING 2016 Organizing Committee, 2016: 2449-2460.
[2] GHOSH A, VEALE D T. Fracking sarcasm using neural network[C]//Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. Stroudsburg: Association for Computational Linguistics, 2016: 161-169.
[3] KHOTIJAH S, TIRTAWANGSA J, SURYANI A A. Using LSTM for context based approach of sarcasm detection in twitter[C]//Proceedings of the 11th International Conference on Advances in Information Technology. New York: ACM, 2020: 1-7.
[4] LOU Chenwei, LIANG Bin, GUI Lin, et al. Affective dependency graph for sarcasm detection[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2021: 1844-1849.
[5] 孙晓, 何家劲, 任福继. 基于多特征融合的混合神经网络模型讽刺语用判别[J]. 中文信息学报, 2016, 30(6): 215–223
SUN Xiao, HE Jiajin, REN Fuji. Pragmatic analysis of irony based on hybrid neural network model with multi-feature[J]. Journal of Chinese information processing, 2016, 30(6): 215–223
[6] 卢欣, 李旸, 王素格. 融合语言特征的卷积神经网络的反讽识别方法[J]. 中文信息学报, 2019, 33(5): 31–38
LU Xin, LI Yang, WANG Suge. Linguistic features enhanced convolutional neural networks for irony recognition[J]. Journal of Chinese information processing, 2019, 33(5): 31–38
[7] 樊小超, 杨亮, 林鸿飞, 等. 基于多语义融合的反讽识别[J]. 中文信息学报, 2021, 35(6): 103–111
FAN Xiaochao, YANG Liang, LIN Hongfei, et al. Irony recognition based on multiple semantic fusion[J]. Journal of Chinese information processing, 2021, 35(6): 103–111
[8] SCHIFANELLA R, DE JUAN P, TETREAULT J, et al. Detecting sarcasm in multimodal social platforms[EB/OL]. (2016-08-08)[2021-01-01]. http://arxiv.org/abs/1608.02289.pdf
[9] SHARMA D K, SINGH B, AGARWAL S, et al. Sarcasm detection over social media platforms using hybrid auto-encoder-based model[J]. Electronics, 2022, 11(18): 2844.
[10] SANGWAN S, AKHTAR M S, BEHERA P, et al. I didn’t mean what I wrote! exploring multimodality for sarcasm detection[C]//2020 International Joint Conference on Neural Networks. Glasgow: IEEE, 2020: 1-8.
[11] CAI Yitao, CAI Huiyu, WAN Xiaojun. Multi-modal sarcasm detection in twitter with hierarchical fusion model[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2019: 2506-2515.
[12] PAN Hongliang, LIN Zheng, FU Peng, et al. Modeling intra and inter-modality incongruity for multi-modal sarcasm detection[C]//Findings of the Association for Computational Linguistics: EMNLP 2020. Online. Stroudsburg: Association for Computational Linguistics, 2020: 1383-1392.
[13] YAO Fanglong, SUN Xian, YU Hongfeng, et al. Mimicking the brain’s cognition of sarcasm from multidisciplines for twitter sarcasm detection[J]. IEEE transactions on neural networks and learning systems, 2023, 34(1): 228–242.
[14] GUPTA S, SHAH A, SHAH M, et al. FiLMing multimodal sarcasm detection with attention[C]//International Conference on Neural Information Processing. Cham: Springer, 2021: 178-186.
[15] 张继东, 蒋丽萍. 基于多模态深度学习的旅游评论反讽识别研究[J]. 情报理论与实践, 2022, 45(7): 158–164
ZHANG Jidong, JIANG Liping. Research on irony recognition of travel reviews based on multi-modal deep learning[J]. Information studies:theory & application, 2022, 45(7): 158–164
[16] 刘洋, 马莉莉, 张雯, 等. 基于跨模态深度学习的旅游评论反讽识别[J]. 数据分析与知识发现, 2022, 6(12): 23–31
LIU Yang, MA Lili, ZHANG Wen, et al. Detecting sarcasm from travel reviews based on cross-modal deep learning[J]. Data analysis and knowledge discovery, 2022, 6(12): 23–31
[17] 刘美萍. 重大突发事件网络舆情协同治理机制构建研究[J]. 求实, 2022(5): 64–76,111
LIU Meiping. Research on mechanism construction of collaborative governance of online public opinion on major emergencies[J]. Truth seeking, 2022(5): 64–76,111
[18] MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. (2013-01-16)[2021-01-01]. http://arxiv.org/abs/1301.3781.pdf.
[19] KIM Y. Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1746–1751.
[20] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[21] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Advances in Neural Information Processing Systems 30. Long Beach, USA, 2017: 5998-6008.
[22] WU Suyan, SU Entong, LEI Binyang, et al. TextCNN-based text classification for E-government[C]//2019 6th International Conference on Information Science and Control Engineering. Shanghai: IEEE, 2020: 929-934.
[23] SHARFUDDIN A A, TIHAMI N M, ISLAM S M. A deep recurrent neural network with BiLSTM model for sentiment classification[C]//2018 International Conference on Bangla Speech and Language Processing . Sylhet: IEEE, 2018: 1-4.
[24] DEVLIN J, CHANG MINGWEI , LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: [s. n. ], 2019: 4171–4186.
[25] 陆晓蕾, 倪斌. 基于预训练语言模型的BERT-CNN多层级专利分类研究[J]. 中文信息学报, 2021, 35(11): 70–79
LU Xiaolei, NI Bin. BERT-CNN: a hierarchical patent classifier based on pre-trained language model[J]. Journal of Chinese information processing, 2021, 35(11): 70–79
[26] LI Zhengguang, LIN Hongfei, SHEN Chen, et al. Cross 2 Self-attentive bidirectional recurrent neural network with BERT for biomedical semantic text similarity[C]//2020 IEEE International Conference on Bioinformatics and Biomedicine. Seoul: IEEE, 2021: 1051-1054.
[27] KAUR T, GANDHI T K. Automated brain image classification based on VGG-16 and transfer learning[C]//2019 International Conference on Information Technology. Bhubaneswar: IEEE, 2020: 94-98.
- Similar References:
Memo
-
Last Update:
1900-01-01