[1]康博,钱艺,文益民.基于抽象关系场景图的图像情感识别[J].智能系统学报,2024,19(2):335-343.[doi:10.11992/tis.202303009]
KANG Bo,QIAN Yi,WEN Yimin.Image sentiment recognition based on the abstract relational scene graph network[J].CAAI Transactions on Intelligent Systems,2024,19(2):335-343.[doi:10.11992/tis.202303009]
点击复制
KANG Bo,QIAN Yi,WEN Yimin.Image sentiment recognition based on the abstract relational scene graph network[J].CAAI Transactions on Intelligent Systems,2024,19(2):335-343.[doi:10.11992/tis.202303009]
基于抽象关系场景图的图像情感识别
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第2期
页码:
335-343
栏目:
学术论文—机器感知与模式识别
出版日期:
2024-03-05
- Title:
- Image sentiment recognition based on the abstract relational scene graph network
- Keywords:
- image sentiment recognition; abstract relationship; scene graph; graph convolutional network; attention mechanism; convolutional neural network; visual sentiment analysis; deep learning
- 分类号:
- TP391
- 文献标志码:
- 2023-11-20
- 摘要:
- 图像情感识别是通过分析视觉刺激来预测人类情感的抽象过程。现有方法大多缺乏对对象间关系以及对象与场景间相互作用的关注，并且对象间复杂多样的关系难以得到充分利用，进而导致难以正确对图像情感进行预测。为解决上述问题，提出一种基于抽象关系场景图的图像情感识别方法。首先，构建对象和属性检测器来提取图像中对象及其属性的特征。其次，使用对象特征推理对象间的亲密度和抽象关系特征，进而构建抽象关系场景图。再次，提出抽象关系图卷积网络来推理抽象关系场景图。最后，设计渐进式注意力机制对多个对象特征进行融合，以得到图像的整体对象特征。在FI、EmotionRoI和Twitter I公开数据集上的试验结果表明，该方法的分类准确率优于现有方法。
- Abstract:
- Image sentiment recognition is an abstract process of forecasting human emotions by analysis of various visual stimuli. Most of the earlier literature does not focus on the relationships among objects and the interactions between objects and scenes, and the complex and diverse relationships among objects are difficult to fully exploit, resulting in difficulty in correctly forecasting image sentiment. To deal with this problem, we develop an abstract relational scene graph network for image sentiment recognition. First, an object and attribute detector is generated to extract object features and their corresponding attribute features from images. Second, the affinities and abstract relationship features among objects are inferred through object features, and then the abstract relational scene graph is generated. Moreover, an abstract relational graph convolutional network is developed for reasoning the abstract relational scene graph. Last, a progressive attention mechanism is designed to fuse multiple object features to acquire the overall object feature of the image. Application on three public datasets, FI, EmotionRoI, and Twitter I, demonstrates that the classification accuracy of the proposed method is better than that of the existing methods.
- 参考文献/References:
-
[1] ZHAO Sicheng, GAO Yue, DING Guiguang, et al. Real-time multimedia social event detection in microblog[J]. IEEE transactions on cybernetics, 2018, 48(11): 3218–3231.
[2] 吴佩谕, 黄远水. 旅游照片的符号属性对旅游意向的影响研究: 以微信朋友圈旅游照片为例[J]. 资源开发与市场, 2019, 35(7): 993–1000
WU Peiyu, HUANG Yuanshui. Study on influence of symbolic attributes of travel photos on travel intention—taking travel photos of WeChat friends circle as an example[J]. Resource development & market, 2019, 35(7): 993–1000
[3] ZHAO Sicheng, YAO Xingxu, YANG Jufeng, et al. Affective image content analysis: two decades review and new perspectives[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(10): 6729–6751.
[4] 赵思成, 姚鸿勋. 图像情感计算综述[J]. 智能计算机与应用, 2017, 7(1): 1–5
ZHAO Sicheng, YAO Hongxun. A survey of image emotion computing[J]. Intelligent computer and applications, 2017, 7(1): 1–5
[5] 王仁武, 孟现茹. 图片情感分析研究综述[J]. 图书情报知识, 2020(3): 119–127
WANG Renwu, MENG Xianru. Review of image sentiment analysis[J]. Documentation, information & knowledge, 2020(3): 119–127
[6] 姚鸿勋, 邓伟洪, 刘洪海, 等. 情感计算与理解研究发展概述[J]. 中国图象图形学报, 2022, 27(6): 2008–2035
YAO Hongxun, DENG Weihong, LIU Honghai, et al. An overview of research development of affective computing and understanding[J]. Journal of image and graphics, 2022, 27(6): 2008–2035
[7] PANG Lei, ZHU Shiai, NGO C W. Deep multimodal learning for affective analysis and retrieval[J]. IEEE transactions on multimedia, 2015, 17(11): 2008–2020.
[8] GUO Longteng, LIU Jing, YAO Peng, et al. MSCap: multi-style image captioning with unpaired stylized text[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 4199-4208.
[9] ZHAO Wentian, WU Xinxiao, ZHANG Xiaoxun. MemCap: memorizing style knowledge for image captioning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 12984-12992.
[10] FENG Junlong, ZHAO Jianping. Improving stylized caption compatibility with image content by integrating region context[J]. Neural computing and applications, 2022, 34(6): 4151–4163.
[11] LI Zuhe, FAN Yangyu, JIANG Bin, et al. A survey on sentiment analysis and opinion mining for social multimedia[J]. Multimedia tools and applications, 2019, 78(6): 6939–6967.
[12] MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM, 2010: 83-92.
[13] ZHAO Sicheng, GAO Yue, JIANG Xiaolei, et al. Exploring principles-of-art features for image emotion recognition[C]//Proceedings of the 22nd ACM iNternational Conference on Multimedia. New York: ACM, 2014: 47-56.
[14] BORTH D, JI Rongrong, CHEN Tao, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York: ACM, 2013: 223-232.
[15] CHEN Tao, YU F X, CHEN Jiawei, et al. Object-based visual sentiment concept analysis and application[C]//Proceedings of the 22 nd ACM International Conference on Multimedia. New York: ACM, 2014: 367-376.
[16] RAO Tianrong, XU Min, LIU Huiying, et al. Multi-scale blocks based image emotion classification using multiple instance learning[C]//2016 IEEE International Conference on Image Processing. Phoenix: IEEE, 2016: 634-638.
[17] CHEN Tao, BORTH D, DARRELL T, et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[EB/OL]. (2014-10-30)[2022-01-01]. https://arxiv.org/abs/1410.8586.pdf.
[18] YOU Quanzeng, LUO Jiebo, JIN Hailin, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Austin: AAAI, 2015: 381-388.
[19] RAO Tianrong, LI Xiaoxu, XU Min. Learning multi-level deep representations for image emotion classification[J]. Neural processing letters, 2020, 51(3): 2043–2061.
[20] ZHANG Hao, XU Dan, LUO Gaifang, et al. Learning multi-level representations for affective image recognition[J]. Neural computing and applications, 2022, 34(16): 14107–14120.
[21] 李志义, 许洪凯, 段斌. 基于深度学习CNN模型的图像情感特征抽取研究[J]. 图书情报工作, 2019, 63(11): 96–107
LI Zhiyi, XU Hongkai, DUAN Bin. Research on image emotion feature extraction based on deep learning CNN model[J]. Library and information service, 2019, 63(11): 96–107
[22] 蔡国永, 贺歆灏, 储阳阳. 基于空间注意力和卷积神经网络的视觉情感分析[J]. 山东大学学报(工学版), 2020, 50(4): 8–13
CAI Guoyong, HE Xinhao, CHU Yangyang. Visual sentiment analysis based on spatial attention mechanism and convolutional neural network[J]. Journal of Shandong University (engineering science edition), 2020, 50(4): 8–13
[23] 白茹意, 郭小英, 贾春花. 基于特征融合的小样本抽象画图像情感预测[J]. 计算机应用, 2020, 40(8): 2207–2213
BAI Ruyi, GUO Xiaoying, JIA Chunhua. Sentiment prediction of small sample abstract painting image based on feature fusion[J]. Journal of computer applications, 2020, 40(8): 2207–2213
[24] 蔡国永, 储阳阳. 基于双注意力多层特征融合的视觉情感分析[J]. 计算机工程, 2021, 47(9): 227–234
CAI Guoyong, CHU Yangyang. Visual sentiment analysis based on multi-level features fusion of dual attention[J]. Computer engineering, 2021, 47(9): 227–234
[25] YANG Jufeng, SHE Dongyu, SUN Ming, et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE transactions on multimedia, 2018, 20(9): 2513–2525.
[26] XIONG Haitao, LIU Qing, SONG Shaoyi, et al. Region-based convolutional neural network using group sparse regularization for image sentiment classification[J]. EURASIP journal on image and video processing, 2019, 2019(1): 1–9.
[27] YANG Jingyuan, GAO Xinbo, LI Leida, et al. SOLVER: scene-object interrelated visual emotion reasoning network[J]. IEEE transactions on image processing, 2021, 30: 8686–8701.
[28] ZHAO Sicheng, DING Guiguang, GAO Yue, et al. Discrete probability distribution prediction of image emotions with shared sparse learning[J]. IEEE transactions on affective computing, 2020, 11(4): 574–587.
[29] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(6): 1137–1149.
[30] ANDERSON P, HE Xiaodong, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6077-6086.
[31] LI Kunpeng, ZHANG Yulun, LI Kai, et al. Visual semantic reasoning for image-text matching[C]//IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2020: 4653-4661.
[32] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-11-03)[2022-01-01]. https://arxiv.org/abs/1609.02907.pdf.
[33] YOU Quanzeng, LUO Jiebo, JIN Hailin, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[J]. Proceedings of the AAAI conference on artificial intelligence, 2016, 30(1): 308–314.
[34] PENG Kuanchuan, SADOVNIK A, GALLAGHER A, et al. Where do emotions come from? Predicting the emotion stimuli map[C]//2016 IEEE International Conference on Image Processing. Phoenix: IEEE, 2016: 614-618.
[35] KRISHNA R, ZHU Yuke, GROTH O, et al. Visual genome: connecting language and vision using crowdsourced dense image annotations[J]. International journal of computer vision, 2017, 123(1): 32–73.
[36] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009: 248-255.
[37] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-12-23)[2022-01-01]. https://arxiv.org/abs/1409.1556.pdf.
备注/Memo
收稿日期:2023-03-04。
基金项目:国家自然科学基金项目（62366011）；广西重点研发计划项目(桂科AB21220023)；广西图像图形与智能处理重点实验室项目(GIIP 2306)
作者简介:康博，硕士研究生，主要研究方向为计算机视觉和视觉情感分析。E-mail：1981480003@qq.com;钱艺，博士研究生，主要研究方向为计算机视觉与零样本学习。E-mail：qyizos@163.com;文益民，教授，博士生导师，博士，中国计算机学会 杰出会员，主要研究方向为机器学习、推荐系统和大数据分析。主持国家自然科学基金项目3项，发表学术论文50余篇。E-mail：ymwen@guet.edu.cn
通讯作者:文益民. E-mail：ymwen@guet.edu.cn
更新日期/Last Update:
1900-01-01