[1]KANG Bo,QIAN Yi,WEN Yimin.Image sentiment recognition based on the abstract relational scene graph network[J].CAAI Transactions on Intelligent Systems,2024,19(2):335-343.[doi:10.11992/tis.202303009]
Copy

Image sentiment recognition based on the abstract relational scene graph network

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 2 Page number: 335-343 Column: 学术论文—机器感知与模式识别 Public date: 2024-03-05
Title:
Image sentiment recognition based on the abstract relational scene graph network
Author(s):
KANG Bo;  QIAN Yi;  WEN Yimin
Guangxi Key Laboratory of Image and Graphic Intelligent Processing, Guilin University of Electronic Technology, Guilin 541004, China
Keywords:
image sentiment recognition; abstract relationship; scene graph; graph convolutional network; attention mechanism; convolutional neural network; visual sentiment analysis; deep learning
CLC:
TP391
DOI:
10.11992/tis.202303009
Abstract:
Image sentiment recognition is an abstract process of forecasting human emotions by analysis of various visual stimuli. Most of the earlier literature does not focus on the relationships among objects and the interactions between objects and scenes, and the complex and diverse relationships among objects are difficult to fully exploit, resulting in difficulty in correctly forecasting image sentiment. To deal with this problem, we develop an abstract relational scene graph network for image sentiment recognition. First, an object and attribute detector is generated to extract object features and their corresponding attribute features from images. Second, the affinities and abstract relationship features among objects are inferred through object features, and then the abstract relational scene graph is generated. Moreover, an abstract relational graph convolutional network is developed for reasoning the abstract relational scene graph. Last, a progressive attention mechanism is designed to fuse multiple object features to acquire the overall object feature of the image. Application on three public datasets, FI, EmotionRoI, and Twitter I, demonstrates that the classification accuracy of the proposed method is better than that of the existing methods.
References:
[1] ZHAO Sicheng, GAO Yue, DING Guiguang, et al. Real-time multimedia social event detection in microblog[J]. IEEE transactions on cybernetics, 2018, 48(11): 3218–3231.
[2] 吴佩谕, 黄远水. 旅游照片的符号属性对旅游意向的影响研究: 以微信朋友圈旅游照片为例[J]. 资源开发与市场, 2019, 35(7): 993–1000
WU Peiyu, HUANG Yuanshui. Study on influence of symbolic attributes of travel photos on travel intention—taking travel photos of WeChat friends circle as an example[J]. Resource development & market, 2019, 35(7): 993–1000
[3] ZHAO Sicheng, YAO Xingxu, YANG Jufeng, et al. Affective image content analysis: two decades review and new perspectives[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(10): 6729–6751.
[4] 赵思成, 姚鸿勋. 图像情感计算综述[J]. 智能计算机与应用, 2017, 7(1): 1–5
ZHAO Sicheng, YAO Hongxun. A survey of image emotion computing[J]. Intelligent computer and applications, 2017, 7(1): 1–5
[5] 王仁武, 孟现茹. 图片情感分析研究综述[J]. 图书情报知识, 2020(3): 119–127
WANG Renwu, MENG Xianru. Review of image sentiment analysis[J]. Documentation, information & knowledge, 2020(3): 119–127
[6] 姚鸿勋, 邓伟洪, 刘洪海, 等. 情感计算与理解研究发展概述[J]. 中国图象图形学报, 2022, 27(6): 2008–2035
YAO Hongxun, DENG Weihong, LIU Honghai, et al. An overview of research development of affective computing and understanding[J]. Journal of image and graphics, 2022, 27(6): 2008–2035
[7] PANG Lei, ZHU Shiai, NGO C W. Deep multimodal learning for affective analysis and retrieval[J]. IEEE transactions on multimedia, 2015, 17(11): 2008–2020.
[8] GUO Longteng, LIU Jing, YAO Peng, et al. MSCap: multi-style image captioning with unpaired stylized text[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 4199-4208.
[9] ZHAO Wentian, WU Xinxiao, ZHANG Xiaoxun. MemCap: memorizing style knowledge for image captioning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 12984-12992.
[10] FENG Junlong, ZHAO Jianping. Improving stylized caption compatibility with image content by integrating region context[J]. Neural computing and applications, 2022, 34(6): 4151–4163.
[11] LI Zuhe, FAN Yangyu, JIANG Bin, et al. A survey on sentiment analysis and opinion mining for social multimedia[J]. Multimedia tools and applications, 2019, 78(6): 6939–6967.
[12] MACHAJDIK J, HANBURY A. Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM International Conference on Multimedia. New York: ACM, 2010: 83-92.
[13] ZHAO Sicheng, GAO Yue, JIANG Xiaolei, et al. Exploring principles-of-art features for image emotion recognition[C]//Proceedings of the 22nd ACM iNternational Conference on Multimedia. New York: ACM, 2014: 47-56.
[14] BORTH D, JI Rongrong, CHEN Tao, et al. Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York: ACM, 2013: 223-232.
[15] CHEN Tao, YU F X, CHEN Jiawei, et al. Object-based visual sentiment concept analysis and application[C]//Proceedings of the 22 nd ACM International Conference on Multimedia. New York: ACM, 2014: 367-376.
[16] RAO Tianrong, XU Min, LIU Huiying, et al. Multi-scale blocks based image emotion classification using multiple instance learning[C]//2016 IEEE International Conference on Image Processing. Phoenix: IEEE, 2016: 634-638.
[17] CHEN Tao, BORTH D, DARRELL T, et al. DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks[EB/OL]. (2014-10-30)[2022-01-01]. https://arxiv.org/abs/1410.8586.pdf.
[18] YOU Quanzeng, LUO Jiebo, JIN Hailin, et al. Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. Austin: AAAI, 2015: 381-388.
[19] RAO Tianrong, LI Xiaoxu, XU Min. Learning multi-level deep representations for image emotion classification[J]. Neural processing letters, 2020, 51(3): 2043–2061.
[20] ZHANG Hao, XU Dan, LUO Gaifang, et al. Learning multi-level representations for affective image recognition[J]. Neural computing and applications, 2022, 34(16): 14107–14120.
[21] 李志义, 许洪凯, 段斌. 基于深度学习CNN模型的图像情感特征抽取研究[J]. 图书情报工作, 2019, 63(11): 96–107
LI Zhiyi, XU Hongkai, DUAN Bin. Research on image emotion feature extraction based on deep learning CNN model[J]. Library and information service, 2019, 63(11): 96–107
[22] 蔡国永, 贺歆灏, 储阳阳. 基于空间注意力和卷积神经网络的视觉情感分析[J]. 山东大学学报(工学版), 2020, 50(4): 8–13
CAI Guoyong, HE Xinhao, CHU Yangyang. Visual sentiment analysis based on spatial attention mechanism and convolutional neural network[J]. Journal of Shandong University (engineering science edition), 2020, 50(4): 8–13
[23] 白茹意, 郭小英, 贾春花. 基于特征融合的小样本抽象画图像情感预测[J]. 计算机应用, 2020, 40(8): 2207–2213
BAI Ruyi, GUO Xiaoying, JIA Chunhua. Sentiment prediction of small sample abstract painting image based on feature fusion[J]. Journal of computer applications, 2020, 40(8): 2207–2213
[24] 蔡国永, 储阳阳. 基于双注意力多层特征融合的视觉情感分析[J]. 计算机工程, 2021, 47(9): 227–234
CAI Guoyong, CHU Yangyang. Visual sentiment analysis based on multi-level features fusion of dual attention[J]. Computer engineering, 2021, 47(9): 227–234
[25] YANG Jufeng, SHE Dongyu, SUN Ming, et al. Visual sentiment prediction based on automatic discovery of affective regions[J]. IEEE transactions on multimedia, 2018, 20(9): 2513–2525.
[26] XIONG Haitao, LIU Qing, SONG Shaoyi, et al. Region-based convolutional neural network using group sparse regularization for image sentiment classification[J]. EURASIP journal on image and video processing, 2019, 2019(1): 1–9.
[27] YANG Jingyuan, GAO Xinbo, LI Leida, et al. SOLVER: scene-object interrelated visual emotion reasoning network[J]. IEEE transactions on image processing, 2021, 30: 8686–8701.
[28] ZHAO Sicheng, DING Guiguang, GAO Yue, et al. Discrete probability distribution prediction of image emotions with shared sparse learning[J]. IEEE transactions on affective computing, 2020, 11(4): 574–587.
[29] REN Shaoqing, HE Kaiming, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(6): 1137–1149.
[30] ANDERSON P, HE Xiaodong, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6077-6086.
[31] LI Kunpeng, ZHANG Yulun, LI Kai, et al. Visual semantic reasoning for image-text matching[C]//IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2020: 4653-4661.
[32] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. (2016-11-03)[2022-01-01]. https://arxiv.org/abs/1609.02907.pdf.
[33] YOU Quanzeng, LUO Jiebo, JIN Hailin, et al. Building a large scale dataset for image emotion recognition: the fine print and the benchmark[J]. Proceedings of the AAAI conference on artificial intelligence, 2016, 30(1): 308–314.
[34] PENG Kuanchuan, SADOVNIK A, GALLAGHER A, et al. Where do emotions come from? Predicting the emotion stimuli map[C]//2016 IEEE International Conference on Image Processing. Phoenix: IEEE, 2016: 614-618.
[35] KRISHNA R, ZHU Yuke, GROTH O, et al. Visual genome: connecting language and vision using crowdsourced dense image annotations[J]. International journal of computer vision, 2017, 123(1): 32–73.
[36] DENG Jia, DONG Wei, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009: 248-255.
[37] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2014-12-23)[2022-01-01]. https://arxiv.org/abs/1409.1556.pdf.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems