[1]DAI Jinli,CAO Jiangtao,JI Xiaofei.Two-person interaction recognition based on the interactive relationship hypergraph convolution network model[J].CAAI Transactions on Intelligent Systems,2024,19(2):316-324.[doi:10.11992/tis.202208001]
Copy

Two-person interaction recognition based on the interactive relationship hypergraph convolution network model

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 2 Page number: 316-324 Column: 学术论文—机器感知与模式识别 Public date: 2024-03-05
Title:
Two-person interaction recognition based on the interactive relationship hypergraph convolution network model
Author(s):
DAI Jinli1;  CAO Jiangtao1;  JI Xiaofei2
1. School of Information and Control Engineering, Liaoning Petrochemical University, Fushun 113001, China;
2. School of Automation, Shenyang Aerospace University, Shenyang 110136, China
Keywords:
two-person interaction; behavior recognition; skeleton node data; deep learning; ST-GCN; hypergraph; graph structure; neural networks
CLC:
TP183
DOI:
10.11992/tis.202208001
Abstract:
To enhance the security of schools, shopping malls, and other public places, it is important to achieve automatic identification of abnormal two-person interaction behaviors, such as stealing, robbing, fighting, and assaulting, in surveillance videos. However, the current behavior recognition method based on joint data in graph creation neglects the two-person interaction information as well as the interaction relationship between the single unnatural connection joints. To address this issue, a two-person interaction behavior recognition model based on the interactive relationship hypergraph convolution network is proposed to model and identify human interactions. First, the corresponding single hypergraph and two-person interaction graph are created for the joint-point data of each frame, where the hypergraph makes the information of multiple unnaturally connected nodes interchangeable at the same time, and the interaction graph emphasizes the interaction strength between nodes. The above-constructed graph models are fed into the spatiotemporal graph convolution to model the spatial and temporal information separately, and lastly, the recognition results are acquired by the SoftMax classifier. The benefits of the proposed algorithm framework are that the interactive relationship between two persons, the structural relationship between unnatural connections, and the flexible motion characteristics of limbs are regarded in the graph construction process. Tests on the NTU data set demonstrate that the algorithm attains a correct recognition rate of 97.36%. The findings indicate that the network model enhances the ability to represent the characteristics of two-person interaction and has better recognition performance than the current models.
References:
[1] 吴联世, 夏利民, 罗大庸. 人的交互行为识别与理解研究综述[J]. 计算机应用与软件, 2011, 28(11): 60–63
WU Lianshi, XIA Limin, LUO Dayong. Survey on human interactive behaviour recognition and comprehension[J]. Computer applications and software, 2011, 28(11): 60–63
[2] WANG Pichao, LI Wanqing, OGUNBONA P, et al. RGB-D-based human motion recognition with deep learning: a survey[J]. Computer vision and image understanding, 2018, 171: 118–139.
[3] BARADEL F, WOLF C, MILLE J, et al. Glimpse clouds: human activity recognition from unstructured feature points[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 469-478.
[4] YUN K, HONORIO J, CHATTOPADHYAY D, et al. Two-person interaction detection using body-pose features and multiple instance learning[C]//2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Providence: IEEE, 2012: 28-35.
[5] 姬晓飞, 谢旋, 任艳. 深度学习的双人交互行为识别与预测算法研究[J]. 智能系统学报, 2020, 15(3): 484–490
JI Xiaofei, XIE Xuan, REN Yan. Research on two-person interaction Recognition and Prediction Algorithm based on Deep Learning[J]. CAAI transactions on intelligent systems, 2020, 15(3): 484–490
[6] HUYNH-THE T, BANOS O, LE B V, et al. PAM-based flexible generative topic model for 3D interactive activity recognition[C]//2015 International Conference on Advanced Technologies for Communications. Ho Chi Minh City: IEEE, 2016: 117-122.
[7] ZAREMBA W, SUTSKEVER I, VINYALS O. Recurrent neural network regularization[EB/OL]. (2014-09-18)[2022-01-01]. https://arxiv.org/abs/1409.2329.pdf.
[8] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278–2324.
[9] KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL].(2017-09-09)[2022-01-01]. https://arxiv.org/abs/1607.07043.pdf.
[10] LIU Jun, SHAHROUDY A, XU Dong, et al. Spatio-temporal LSTM with trust gates for 3D human action recognition[EB/OL]. (2016-07-24)[2022-01-01]. https://arxiv.org/abs/1607.07043.pdf.
[11] CHOUTAS V, WEINZAEPFEL P, REVAUD J, et al. PoTion: pose MoTion representation for action recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7024-7033.
[12] YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[EB/OL]. (2018-01-25)[2022-01-01]. https://arxiv.org/abs/1801.07455.
[13] CHENG Ke, ZHANG Yifan, HE Xiangyu, et al. Skeleton-based action recognition with shift graph convolutional network[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 180-189.
[14] SONG Yifan, ZHANG Zhang, SHAN Caifeng, et al. Richly activated graph convolutional network for robust skeleton-based action recognition[J]. IEEE transactions on circuits and systems for video technology, 2021, 31(5): 1915–1925.
[15] 刘云, 薛盼盼, 李辉, 等. 基于深度学习的关节点行为识别综述[J]. 电子与信息学报, 2021, 43(6): 1789–1802
LIU Yun, XUE Panpan, LI Hui, et al. A review of action recognition using joints based on deep learning[J]. Journal of electronics & information technology, 2021, 43(6): 1789–1802
[16] LI Maosen, CHEN Siheng, CHEN Xu, et al. Actional-structural graph convolutional networks for skeleton-based action recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 3590-3598.
[17] 成科扬, 吴金霞, 王文杉, 等. 融合时空图卷积的多人交互行为识别[J]. 中国图象图形学报, 2021, 26(7): 1681–1691
CHENG Keyang, WU Jinxia, WANG Wenshan, et al. Multi-person interaction action recognition based on spatio-temporal graph convolution[J]. Journal of image and graphics, 2021, 26(7): 1681–1691
[18] WU Jianchao, WANG Limin, WANG Li, et al. Learning actor relation graphs for group activity recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 9956-9966.
[19] CHEN Yuxin, ZHANG Ziqi, YUAN Chunfeng, et al. Channel-wise topology refinement graph convolution for skeleton-based action recognition[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 13339-13348.
[20] LEE J, LEE M, LEE D, et al. Hierarchically decomposed graph convolutional networks for skeleton-based action recognition[EB/OL]. (2022-12-25)[2023-01-01]. https://arxiv.org/abs/2208.10741.pdf.
[21] ZHOU D, HUANG J, SCH?LKOPF B. Learning with hypergraphs: clustering, classification, and embedding[C]//International on Neural Information Processing Systems. Vancouver MIT Press, 2006: 1601-1608.
[22] ZHOU Dengyong, HUANG Jiayuan, SCH?LKOPF B. Learning from labeled and unlabeled data on a directed graph[C]//Proceedings of the 22nd International Conference on Machine Learning. New York: ACM, 2005: 1036-1043.
[23] YADATI N, NIMISHAKAVI M, YADAV P, et al. HyperGCN: a new method of training graph convolutional networks on hypergraphs[EB/OL]. (2018-09-07)[2022-01-01]. https://arxiv.org/abs/1809.02589.pdf.
[24] SHAHROUDY A, LIU Jun, NG T T, et al. NTU RGB D: a large scale dataset for 3D human activity analysis[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 1010-1019.
[25] KE Qiuhong, AN Senjian, BENNAMOUN M, et al. SkeletonNet: mining deep part features for 3-D action recognition[J]. IEEE signal processing letters, 2017, 24(6): 731–735.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems