[1]TIAN Feng,WEI Ningbin,LIU Fang,et al.Oilfield operation behavior recognition based on spatio-temporal and action adaptive fusion network[J].CAAI Transactions on Intelligent Systems,2024,19(6):1407-1418.[doi:10.11992/tis.202309021]
Copy

Oilfield operation behavior recognition based on spatio-temporal and action adaptive fusion network

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 6 Page number: 1407-1418 Column: 学术论文—机器感知与模式识别 Public date: 2024-12-05
Title:
Oilfield operation behavior recognition based on spatio-temporal and action adaptive fusion network
Author(s):
TIAN Feng;  WEI Ningbin;  LIU Fang;  HAN Yuxiang;  ZHAO Ling;  ZHANG Sirui;  MA Guibao
School of Computer and Information Technology, Northeast Petroleum University, Daqing 163318, China
Keywords:
behavior recognition; ResNet50; attention mechanism; oilfield operation; feature fusion; spatio-temporal attention; action attention; complex scenes
CLC:
TP391
DOI:
10.11992/tis.202309021
Abstract:
A spatiotemporal and action adaptive fusion network is proposed for personnel behavior recognition in oilfield operation sites to address the problems of false positives and negatives caused by the complex environment of oilfield operations interfering with behavior recognition algorithms. First, the videos are processed on the constructed network using a sparse sampling strategy, and features on the feature extraction network are then extracted. The core modules of the network include spatiotemporal attention, action reinforcement, and adaptive feature fusion modules. The spatiotemporal attention module redistributes the spatiotemporal importance of features, establishing temporal correlations between different frames. The action reinforcement module weakens the background and enhances human body movements, allowing the model to focus on human actions. The feature fusion module adaptively combines the parallel features after reinforcement. Finally, behavior classification is achieved through fully connected layers and a SoftMax layer. The model is compared with classic networks on public and self-built oilfield datasets to verify the effectiveness of the proposed network. The Top-1 accuracy on the UCF101 dataset shows a 3.33% improvement over SlowOnly, the Slow branch of the SlowFast model, and a 1.61% improvement over the temporal shift module (TSM). On the HMDB51 dataset, the Top-1 accuracy improves by 8.56% and 1.83% compared to SlowOnly and TSM, respectively. Additionally, when evaluated on the self-built oilfield dataset, the proposed model shows a notable improvement in accuracy over the temporal segment network, TSM, and SlowOnly. This result validates the effectiveness of the spatiotemporal and action adaptive fusion network in oilfield operations and confirms its suitability for behavior recognition tasks in such environments.
References:
[1] 富倩. 人体行为识别研究[J]. 信息与电脑(理论版), 2017(24): 146-147.
FU Qian. Analysis of human behavior recognition[J]. China computer & communication（theoretical edition）, 2017(24): 146-147.
[2] 梁绪, 李文新, 张航宁. 人体行为识别方法研究综述[J]. 计算机应用研究, 2022, 39(3): 651-660.
LIANG Xu, LI Wenxin, ZHANG Hangning. Review of research on human action recognition methods[J]. Application research of computers, 2022, 39(3): 651-660.
[3] TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 4489-4497.
[4] CARREIRA J, ZISSERMAN A. Quo vadis, action recognition? A new model and the kinetics dataset[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 4724-4733.
[5] TRAN D, WANG Heng, TORRESANI L, et al. A closer look at spatiotemporal convolutions for action recognition[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6450-6459.
[6] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal: MIT Press, 2014: 568-576.
[7] WANG Limin, XIONG Yuanjun, WANG Zhe, et al. Temporal segment networks: towards good practices for deep action recognition[M]//Lecture Notes in Computer Science. Cham: Springer International Publishing, 2016: 20-36.
[8] LIN Ji, GAN Chuang, HAN Song. TSM: temporal shift module for efficient video understanding[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 7082-7092.
[9] FEICHTENHOFER C, FAN Haoqi, MALIK J, et al. SlowFast networks for video recognition[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 6201-6210.
[10] YAN Sijie, XIONG Yuanjun, LIN Dahua. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. New Orleans: AAAI, 2018: 7444-7452.
[11] DUAN Haodong, ZHAO Yue, CHEN Kai, et al. Revisiting skeleton-based action recognition[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 2959-2968.
[12] 田枫, 孙晓悦, 刘芳, 等. 基于图卷积的作业行为实时检测方法[J]. 计算机工程与设计, 2022, 43(10): 2944-2952.
TIAN Feng, SUN Xiaoyue, LIU Fang, et al. Real time detection method of work behavior based on graph convolution[J]. Computer engineering and design, 2022, 43(10): 2944-2952.
[13] 陆昱翔, 徐冠华, 唐波. 基于视觉Transformer时空自注意力的工人行为识别[J]. 浙江大学学报(工学版), 2023, 57(3): 446-454.
LU Yuxiang, XU Guanhua, TANG Bo. Worker behavior recognition based on temporal and spatial self-attention of vision Transformer[J]. Journal of Zhejiang university (engineering science edition), 2023, 57(3): 446-454.
[14] 饶天荣, 潘涛, 徐会军. 基于交叉注意力机制的煤矿井下不安全行为识别[J]. 工矿自动化, 2022, 48(10): 48-54.
RAO Tianrong, PAN Tao, XU Huijun. Unsafe action recognition in underground coal mine based on cross-attention mechanism[J]. Industry and mine automation, 2022, 48(10): 48-54.
[15] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: transformers for image recognition at scale[EB/OL]. (2020-10-22)[2022-03-24]. http://arxiv.org/abs/2010.11929.
[16] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[17] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
[18] KUEHNE H, JHUANG H, GARROTE E, et al. HMDB: a large video database for human motion recognition[C]//2011 International Conference on Computer Vision. Barcelona: IEEE, 2011: 2556-2563.
[19] SOOMRO K, ZAMIR A R, SHAH M. UCF101: a dataset of 101 human actions classes from videos in the wild[EB/OL]. (2012-12-03)[2021-11-05]. http://arxiv.org/abs/1212.0402.
[20] KAY W, CARREIRA J, SIMONYAN K, et al. The kinetics human action video dataset[EB/OL]. (2017-05-19)[2021-11-05]. http://arxiv.org/abs/1705.06950.
[21] FISHER R A. The use of multiple measurements in taxonomic problems[J]. Annals of eugenics, 1936, 7(2): 179-188.
[22] GAMMULLE H, DENMAN S, SRIDHARAN S, et al. Two stream LSTM: a deep fusion framework for human action recognition[C]//2017 IEEE Winter Conference on Applications of Computer Vision. Santa Rosa: IEEE, 2017: 177-186.
[23] JI Shuiwang, XU Wei, YANG Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(1): 221-231.
[24] XIE Saining, GIRSHICK R, DOLLáR P, et al. Aggregated residual transformations for deep neural networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5987-5995.
[25] SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[J]. International journal of computer vision, 2020, 128(2): 336-359.
Similar References:

Memo

-

Last Update: 2024-11-05

Copyright © CAAI Transactions on Intelligent Systems