[1]代金利,曹江涛,姬晓飞.交互关系超图卷积模型的双人交互行为识别[J].智能系统学报,2024,19(2):316-324.[doi:10.11992/tis.202208001]
DAI Jinli,CAO Jiangtao,JI Xiaofei.Two-person interaction recognition based on the interactive relationship hypergraph convolution network model[J].CAAI Transactions on Intelligent Systems,2024,19(2):316-324.[doi:10.11992/tis.202208001]
点击复制
DAI Jinli,CAO Jiangtao,JI Xiaofei.Two-person interaction recognition based on the interactive relationship hypergraph convolution network model[J].CAAI Transactions on Intelligent Systems,2024,19(2):316-324.[doi:10.11992/tis.202208001]
交互关系超图卷积模型的双人交互行为识别
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第2期
页码:
316-324
栏目:
学术论文—机器感知与模式识别
出版日期:
2024-03-05
- Title:
- Two-person interaction recognition based on the interactive relationship hypergraph convolution network model
- Keywords:
- two-person interaction; behavior recognition; skeleton node data; deep learning; ST-GCN; hypergraph; graph structure; neural networks
- 分类号:
- TP183
- 文献标志码:
- 2023-11-13
- 摘要:
- 为提高学校、商场等公共场所的安全性，实现对监控视频中的偷窃、抢劫和打架斗殴等异常双人交互行为的自动识别，针对现有基于关节点数据的行为识别方法在图的创建中忽略了2个人之间的交互信息，且忽略了单人非自然连接关节点间的交互关系的问题，提出一种基于交互关系超图卷积模型用于双人交互行为的建模与识别。首先针对每一帧的关节点数据构建对应的单人超图以及双人交互关系图，其中超图同时使多个非自然连接节点信息互通，交互关系图强调节点间交互强度。将以上构建的图模型送入时空图卷积对空间和时间信息分别建模，最后通过SoftMax分类器得到识别结果。该算法框架的优势是在图的构建过程中加强考虑双人的交互关系、非自然连接点间结构关系以及四肢灵活的运动特征。在NTU数据集上的测试表明，该算法得到了97.36%的正确识别率，该网络模型提高了双人交互行为特征的表征能力，取得了比现有模型更好的识别效果。
- Abstract:
- To enhance the security of schools, shopping malls, and other public places, it is important to achieve automatic identification of abnormal two-person interaction behaviors, such as stealing, robbing, fighting, and assaulting, in surveillance videos. However, the current behavior recognition method based on joint data in graph creation neglects the two-person interaction information as well as the interaction relationship between the single unnatural connection joints. To address this issue, a two-person interaction behavior recognition model based on the interactive relationship hypergraph convolution network is proposed to model and identify human interactions. First, the corresponding single hypergraph and two-person interaction graph are created for the joint-point data of each frame, where the hypergraph makes the information of multiple unnaturally connected nodes interchangeable at the same time, and the interaction graph emphasizes the interaction strength between nodes. The above-constructed graph models are fed into the spatiotemporal graph convolution to model the spatial and temporal information separately, and lastly, the recognition results are acquired by the SoftMax classifier. The benefits of the proposed algorithm framework are that the interactive relationship between two persons, the structural relationship between unnatural connections, and the flexible motion characteristics of limbs are regarded in the graph construction process. Tests on the NTU data set demonstrate that the algorithm attains a correct recognition rate of 97.36%. The findings indicate that the network model enhances the ability to represent the characteristics of two-person interaction and has better recognition performance than the current models.
- 参考文献/References:
-
[1] 吴联世, 夏利民, 罗大庸. 人的交互行为识别与理解研究综述[J]. 计算机应用与软件, 2011, 28(11): 60–63
WU Lianshi, XIA Limin, LUO Dayong. Survey on human interactive behaviour recognition and comprehension[J]. Computer applications and software, 2011, 28(11): 60–63
[2] WANG Pichao, LI Wanqing, OGUNBONA P, et al. RGB-D-based human motion recognition with deep learning: a survey[J]. Computer vision and image understanding, 2018, 171: 118–139.
[3] BARADEL F, WOLF C, MILLE J, et al. Glimpse clouds: human activity recognition from unstructured feature points[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 469-478.
[4] YUN K, HONORIO J, CHATTOPADHYAY D, et al. Two-person interaction detection using body-pose features and multiple instance learning[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Providence: IEEE, 2012: 28-35.
[5] 姬晓飞, 谢旋, 任艳. 深度学习的双人交互行为识别与预测算法研究[J]. 智能系统学报, 2020, 15(3): 484–490
JI Xiaofei, XIE Xuan, REN Yan. Research on two-person interaction Recognition and Prediction Algorithm based on Deep Learning[J]. CAAI transactions on intelligent systems, 2020, 15(3): 484–490
[6] HUYNH-THE T, BANOS O, LE B V, et al. PAM-based flexible generative topic model for 3D interactive activity recognition[C]//2015 International Conference on Advanced Technologies for Communications. Ho Chi Minh City: IEEE, 2016: 117-122.
[7] ZAREMBA W, SUTSKEVER I, VINYALS O. Recurrent neural network regularization[EB/OL]. (2014-09-18)[2022-01-01]. https://arxiv.org/abs/1409.2329.pdf.
[8] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324.
[9] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL].(2017-09-09)[2022-01-01]. https://arxiv.org/abs/1607.07043.pdf.
[10] LIU Jun, SHAHROUDY A, XU Dong, et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[EB/OL]. (2016-07-24)[2022-01-01]. https://arxiv.org/abs/1607.07043.pdf.
[11] CHOUTAS V, WEINZAEPFEL P, REVAUD J, et al. PoTion: pose MoTion representation for action recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7024-7033.
[12] YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL]. (2018-01-25)[2022-01-01]. https://arxiv.org/abs/1801.07455.
[13] CHENG Ke, ZHANG Yifan, HE Xiangyu, et al. Skeleton-based action recognition with shift graph convolutional network[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 180-189.
[14] SONG Yifan, ZHANG Zhang, SHAN Caifeng, et al. Richly activated graph convolutional network for robust skeleton-based action recognition[J]. IEEE transactions on circuits and systems for video technology, 2021, 31(5): 1915–1925.
[15] 刘云, 薛盼盼, 李辉, 等. 基于深度学习的关节点行为识别综述[J]. 电子与信息学报, 2021, 43(6): 1789–1802
LIU Yun, XUE Panpan, LI Hui, et al. A review of action recognition using joints based on deep learning[J]. Journal of electronics & information technology, 2021, 43(6): 1789–1802
[16] LI Maosen, CHEN Siheng, CHEN Xu, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 3590-3598.
[17] 成科扬, 吴金霞, 王文杉, 等. 融合时空图卷积的多人交互行为识别[J]. 中国图象图形学报, 2021, 26(7): 1681–1691
CHENG Keyang, WU Jinxia, WANG Wenshan, et al. Multi-person interaction action recognition based on spatio-temporal graph convolution[J]. Journal of image and graphics, 2021, 26(7): 1681–1691
[18] WU Jianchao, WANG Limin, WANG Li, et al. Learning actor relation graphs for group activity recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 9956-9966.
[19] CHEN Yuxin, ZHANG Ziqi, YUAN Chunfeng, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 13339-13348.
[20] LEE J, LEE M, LEE D, et al. Hierarchically decomposed graph convolutional networks for skeleton-based action recognition[EB/OL]. (2022-12-25)[2023-01-01]. https://arxiv.org/abs/2208.10741.pdf.
[21] ZHOU D, HUANG J, SCH?LKOPF B. Learning with hypergraphs: clustering, classification, and embedding[C]//International on Neural Information Processing Systems. Vancouver MIT Press, 2006: 1601-1608.
[22] ZHOU Dengyong, HUANG Jiayuan, SCH?LKOPF B. Learning from labeled and unlabeled data on a directed graph[C]//Proceedings of the 22nd International Conference on Machine Learning. New York: ACM, 2005: 1036-1043.
[23] YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: a new method of training graph convolutional networks on hypergraphs[EB/OL]. (2018-09-07)[2022-01-01]. https://arxiv.org/abs/1809.02589.pdf.
[24] SHAHROUDY A, LIU Jun, NG T T, et al. NTU RGB D: a large scale dataset for 3D human activity analysis[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1010-1019.
[25] KE Qiuhong, AN Senjian, BENNAMOUN M, et al. SkeletonNet: mining deep part features for 3-D action recognition[J]. IEEE signal processing letters, 2017, 24(6): 731–735.
- 相似文献/References:
-
[1]梅雪,胡石,许松松,等.基于多尺度特征的双层隐马尔可夫模型及其在行为识别中的应用[J].智能系统学报,2012,7(6):512.
MEI Xue,HU Shi,XU Songsong,et al.Multi scale feature based double layer HMM and its application in behavior recognition[J].CAAI Transactions on Intelligent Systems,2012,7():512.
[2]姬晓飞,谢旋,任艳.深度学习的双人交互行为识别与预测算法研究[J].智能系统学报,2020,15(3):484.[doi:10.11992/tis.201812029]
JI Xiaofei,XIE Xuan,REN Yan.Human interaction recognition and prediction algorithm based on deep learning[J].CAAI Transactions on Intelligent Systems,2020,15():484.[doi:10.11992/tis.201812029]
[3]刘董经典,孟雪纯,张紫欣,等.一种基于2D时空信息提取的行为识别算法[J].智能系统学报,2020,15(5):900.[doi:10.11992/tis.201906054]
LIU Dongjingdian,MENG Xuechun,ZHANG Zixin,et al.A behavioral recognition algorithm based on 2D spatiotemporal information extraction[J].CAAI Transactions on Intelligent Systems,2020,15():900.[doi:10.11992/tis.201906054]
[4]申天啸,韩怡园,韩冰,等.基于人类视觉皮层双通道模型的驾驶员眼动行为识别[J].智能系统学报,2022,17(1):41.[doi:10.11992/tis.202106051]
SHEN Tianxiao,HAN Yiyuan,HAN Bing,et al.Recognition of driver’s eye movement based on the human visual cortex two-stream model[J].CAAI Transactions on Intelligent Systems,2022,17():41.[doi:10.11992/tis.202106051]
备注/Memo
收稿日期:2022-08-01。
基金项目:国家自然科学基金项目(61673199)；辽宁省科技公益研究基金项目(2016002006).
作者简介:代金利，硕士研究生，主要研究方向为计算机视觉、图像处理和模式识别。E-mail：daijinli19980904@163.com;曹江涛，教授，博士，主要研究方向为智能方法及其应用、视频分析与处理。主持国家自然科学基金项目1 项、辽宁省自然科学基金项目 1 项。参与编著英文专著 2 部，发表学术论文 50 余篇。E-mail：jtcao@lnpu.edu.cn;姬晓飞，副教授，博士，主要研究方向为视频分析与处理、模式识别理论。主持国家自然科学基金项目1项、辽宁省自然科学基金项目1项。参与编著英文专著2部，发表学术论文40余篇。E-mail：jixiaofei7804@126.com
通讯作者:姬晓飞. E-mail：jixiaofei7804@126.com
更新日期/Last Update:
1900-01-01