[1]ZHONG Qiubo,ZHENG Caiming,PIAO Songhao.Research on skeleton-based action recognition with spatiotemporal fusion and human–robot interaction[J].CAAI Transactions on Intelligent Systems,2020,15(3):601-608.[doi:10.11992/tis.202006029]
Copy

Research on skeleton-based action recognition with spatiotemporal fusion and human–robot interaction

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 15 Number of periods: 2020 3 Page number: 601-608 Column: 人工智能院长论坛 Public date: 2020-05-05
Title:
Research on skeleton-based action recognition with spatiotemporal fusion and human–robot interaction
Author(s):
ZHONG Qiubo1; 2;  ZHENG Caiming1;  PIAO Songhao3
1. Robotics Institute, Ningbo University of Technology, Ningbo 315211, China;
2. State Key Laboratory of Robotics and System, Harbin Institute of Technology, Harbin 150001, China;
3. School of Computer Science and Technology, Harbin Institute of Technology, Harbin 150001, China
Keywords:
action recognition; temporal and spatial relationships; posture motion; spatiotemporal fusion; graph convolution network; temporal attention; adaptive feature enhancement; human–robot interaction
CLC:
TP312
DOI:
10.11992/tis.202006029
Abstract:
Temporal dynamics of postures over time is crucial for sequence-based action recognition. Human actions can be represented by corresponding motions of an articulated skeleton. Skeleton-based action recognition algorithm is used for studying motions of a body. Skeleton-based action recognition uses many methods, and research shows that most of them extract spatial and motion information separately from a skeleton structure and then combine them for further processing. However, this process is not able to efficiently deliver human motion features with complex temporal and spatial relationships. We propose a novel posture motion-based, spatiotemporal fused graph convolution network for skeleton-based action recognition. First, we define a local posture motion-based time attention module, which is used to constrain the disturbance information in temporal domain and learn the representation of motion posture features. Then, we design a posture motion-based, spatiotemporal fusion module. This module fuses spatial motion and temporal attitude features and adaptively enhances the skeleton joint features. Extensive experiments have been performed and the results verified the effectiveness of our proposed method. The proposed method has competitive performance, and it is concluded that the human–robot interaction system based on action recognition is superior to the speech interaction system in real-time and with respect to accuracy.
References:
[1] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal, Canada, 2014: 568-576.
[2] BAGAUTDINOV T, ALAHI A, FLEURET F, et al. Social scene understanding: end-to-end multi-person action localization and collective activity recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA, 2017: 3425-3434.
[3] WANG Heng, SCHMID C. Action recognition with improved trajectories[C]//Proceedings of the IEEE International Conference on Computer Vision. Sydney, Australia, 2013: 3551-3558.
[4] CAO Zhe, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA, 2017: 1302-1310.
[5] CHEN Yilun, WANG Zhicheng, PENG Yuxiang, et al. Cascaded pyramid network for multi-person pose estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 7103-7112.
[6] 龚冬颖, 黄敏, 张洪博, 等. RGBD人体行为识别中的自适应特征选择方法[J]. 智能系统学报, 2017, 12(1): 1-7
GONG Dongying, HUANG Min, ZHANG Hongbo, et al. Adaptive feature selection method for action recognition of human body in RGBD data[J]. CAAI transactions on intelligent systems, 2017, 12(1): 1-7
[7] 姬晓飞, 王昌汇, 王扬扬. 分层结构的双人交互行为识别方法[J]. 智能系统学报, 2015, 10(6): 893-900
JI Xiaofei, WANG Changhui, WANG Yangyang. Human interaction behavior-recognition method based on hierarchical structure[J]. CAAI transactions on intelligent systems, 2015, 10(6): 893-900
[8] 庄伟源, 成运, 林贤明, 等. 关键肢体角度直方图的行为识别[J]. 智能系统学报, 2015, 10(1): 20-26
ZHUANG Weiyuan, CHENG Yun, LIN Xianming, et al. Action recognition based on the angle histogram of key parts[J]. CAAI transactions on intelligent systems, 2015, 10(1): 20-26
[9] 徐志通, 骆炎民, 柳培忠. 联合加权重构轨迹与直方图熵的异常行为检测[J]. 智能系统学报, 2018, 13(6): 1015-1026
XU Zhitong, LUO Yanmin, LIU Peizhong. Abnormal behavior detection of joint weighted reconstruction trajectory and histogram entropy[J]. CAAI transactions on intelligent systems, 2018, 13(6): 1015-1026
[10] 吴云鹏, 赵晨阳, 时增林, 等. 基于流密度的多重交互集体行为识别算法[J]. 计算机学报, 2017, 40(11): 2519-2532
WU Yunpeng, ZHAO Chenyang, SHI Zenglin, et al. A flow density based algorithm for detecting coherent motion with multiple interaction[J]. Chinese journal of computers, 2017, 40(11): 2519-2532
[11] 陈婷婷, 阮秋琦, 安高云. 视频中人体行为的慢特征提取算法[J]. 智能系统学报, 2015, 10(3): 381-386
CHEN Tingting, RUAN Qiuqi, AN Gaoyun. Slow feature extraction algorithm of human actions in video[J]. CAAI transactions on intelligent systems, 2015, 10(3): 381-386
[12] 丁重阳, 刘凯, 李光, 等. 基于时空权重姿态运动特征的人体骨架行为识别研究[J]. 计算机学报, 2020, 43(1): 29-40
DING Chongyang, LIU Kai, LI Guang, et al. Spatio-temporal weighted posture motion features for human skeleton action recognition research[J]. Chinese journal of computers, 2020, 43(1): 29-40
[13] 莫宏伟, 汪海波. 基于Faster R-CNN的人体行为检测研究[J]. 智能系统学报, 2018, 13(6): 967-973
MO Hongwei, WANG Haibo. Research on human behavior detection based on Faster R-CNN[J]. CAAI transactions on intelligent systems, 2018, 13(6): 967-973
[14] 姬晓飞, 谢旋, 任艳. 深度学习的双人交互行为识别与预测算法研究[J]. 智能系统学报, DOI: 10.11992/tis. 201812029.
JI Xiaofei, XIE Xuan, Ren Yan. Human interaction recognition and prediction algorithm based on Deep Learning [J]. CAAI transactions on intelligent systems, DOI: 10.11992/tis. 201812029.
[15] 谢昭, 周义, 吴克伟, 等. 基于时空关注度LSTM的行为识别[J/OL]. 计算机学报: (2019-12-17) http://kns.cnki.net/kcms/detail/11.1826.TP.20191227.1658.002.html.
XIE Zhao, ZHOU Yi, WU Kewei, et al. Activity recognition based on spatial-temporal attention LSTM[J/OL] Chinese journal of computers: (2019-12-17) http://kns.cnki.net/kcms/detail/11.1826.TP.20191227.1658.002.html.
[16] 王传旭, 胡小悦, 孟唯佳, 等. 基于多流架构与长短时记忆网络的组群行为识别方法研究[J]. 电子学报, 2020, 48(4): 800-807
WANG Chuanxu, HU Xiaoyue, MENG Weijia, et al. Research on group behavior recognition method based on multi-stream architecture and long short-term memory network[J]. Acta electronica sinica, 2020, 48(4): 800-807
[17] 郑兴华, 孙喜庆, 吕嘉欣, 等. 基于深度学习和智能规划的行为识别[J]. 电子学报, 2019, 47(8): 1661-1668
ZHENG Xinghua, SUN Xiqing, LU Jiaxin, et al. Action recognition based on deep learning and artificial intelligence planning[J]. Acta electronica sinica, 2019, 47(8): 1661-1668
[18] 张冰冰,葛疏雨,王旗龙,等.基于多阶信息融合的行为识别方法研究[J/OL]. 自动化学报, [2020-06-17] DOI: 10.16383/j.aas.c180265.
ZHANG Bingbing, GE Shuyu, WANG Qilong, et al. Multi-order Information Fusion Method for Human Action Recognition[J/OL]. ACTA automatica sinica, [2020-06-17] DOI: 10.16383/j.aas.c180265.
[19] LIU Jun, SHAHROUDY A, XU Dong, et al. Spatio-temporal LSTM with trust gates for 3d human action recognition[C]//Proceedings of the 14th European Conference on Computer Vision. Amsterdam, The Netherlands, 2016: 816-833.
[20] LI Chao, ZHONG Qiaoyong, XIE Di, et al. Skeleton-based action recognition with convolutional neural networks[C]//Proceedings of 2017 IEEE International Conference on Multimedia and Expo Workshops. Hong Kong, China, 2017: 597-600.
[21] YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. New Orleans, USA, 2018: 7444-7452.
[22] SHI L, ZHANG Y, CHENG J, et al. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks[J/OL]. [2020-06-01] https://arxiv.org/abs/1912.06971, 2019.
[23] LIU Ziyu, ZHANG Hongwen, CHEN Zhenghao, et al. Disentangling and unifying graph convolutions for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA, 2020: 143-152.
[24] PENG W, HONG X, CHEN H, et al. Learning Graph Convolutional Network for Skeleton-based Human Action Recognition by Neural Searching[C]//Proceedings of Thirty-Fourth AAAI Conference on Artificial Intelligence. New York, USA, 2020: 2669-2676.
[25] OBINATA Y, YAMAMOTO T. Temporal extension module for skeleton-based action recognition[J/OL]. [2020-03-19] http://arxiv.org/abs/2003.08951.
[26] SHI L, ZHANG Y, CHENG J, et al. Two-stream adaptive graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Los Angeles, USA, 2019: 12026-12035.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems