[1]YAN He,LIU Lingkun,HUANG Junbin,et al.Video summarization model based on the multiscale attention mechanism and bidirectional gated recurrent network[J].CAAI Transactions on Intelligent Systems,2024,19(2):446-454.[doi:10.11992/tis.202209048]
Copy

Video summarization model based on the multiscale attention mechanism and bidirectional gated recurrent network

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 2 Page number: 446-454 Column: 学术论文—人工智能基础 Public date: 2024-03-05
Title:
Video summarization model based on the multiscale attention mechanism and bidirectional gated recurrent network
Author(s):
YAN He;  LIU Lingkun;  HUANG Junbin;  ZHANG Ye;  DUAN Siyu
Liangjiang College of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China
Keywords:
video summary; self-attention mechanism; importance score; long-range dependence; computer vision; BiGRU; nonmaximum suppression (NMS); kernel temporal segmentation (KTS)
CLC:
TP391.41
DOI:
10.11992/tis.202209048
Abstract:
In the video summary task, the variance of global attention value distribution on long distance video sequences is large, the importance score of generating key frames is large, and the semantic coherence of fragments is poor due to the lack of long-range dependence on the boundary values of time series nodes. Herein, by improving the attention module, segmented local self-attention and global self-attention mechanisms are merged to acquire the key features of local and global video sequences and lower the variance of attention values. Concurrently, the bidirectional gated recurrent neural network (BiGRU) is introduced in parallel, the output is input into the enhanced classification regression module, and afterward, the results are additively fused. Lastly, nonmaximum suppression and kernel temporal segmentation methods are applied to filter fragments and segment them into high-quality representative shots. The final summary is created by the knapsack combinatorial optimization algorithm. The video summary model LG-RU, which integrates the multiscale attention mechanism and BiGRU, is developed and compared with TvSum and SumMe’s standard and enhanced data sets. It is demonstrated that the model has a higher F-score, which verifies that this model can complete the video summary robustly while preserving high accuracy.
References:
[1] 冀中, 江俊杰. 基于解码器注意力机制的视频摘要[J]. 天津大学学报(自然科学与工程技术版), 2018, 51(10): 1023–1030
JI Zhong, JIANG Junjie. Video summarization based on decoder attention mechanism[J]. Journal of Tianjin University (science and technology edition), 2018, 51(10): 1023–1030
[2] ELHAMIFAR E, VIDAL R. Sparse subspace clustering: algorithm, theory, and applications[J]. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(11): 2765–2781.
[3] ELHAMIFAR E, SAPIRO G, SASTRY S S. Dissimilarity-based sparse subset selection[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 38(11): 2182–2197.
[4] ELHAMIFAR E, DE PAOLIS KALUZA M C. Subset selection and summarization in sequential data[M]. [S.l.]: Advances in Neural Information Processing Systems, 2017, 30.
[5] ZHU Wencheng, LU Jiwen, LI Jiahao, et al. DSNet: a flexible detect-to-summarize network for video summarization[J]. IEEE transactions on image processing, 2021, 30: 948–962.
[6] ZHANG Ke, CHAO Weilun, SHA Fei, et al. Video summarization with long short-term memory[M]//Computer Vision-ECCV 2016. Cham: Springer International Publishing, 2016: 766-782.
[7] ZHOU Kaiyang, QIAO Yu, XIANG Tao. Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward[C]// Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018: 7582-7589.
[8] MA Yufei, LU Lie, ZHANG Hongjiang, et al. A user attention model for video summarization[C]//Proceedings of the tenth ACM International Conference on Multimedia. New York: ACM, 2002: 533-542.
[9] JIANG Peng, QIN Xiaolin. Keyframe-based video summary using visual attention clues[J]. IEEE MultiMedia, 2010, 17(2): 64–73.
[10] EJAZ N, MEHMOOD I, BAIK S W. Efficient visual attention based framework for extracting key frames from videos[J]. Signal processing:image communication, 2013, 28(1): 34–44.
[11] ZHU Wencheng, LU Jiwen, HAN Yucheng, et al. Learning multiscale hierarchical attention for video summarization[J]. Pattern recognition, 2022, 122: 108312.
[12] 李依依, 王继龙. 自注意力机制的视频摘要模型[J]. 计算机辅助设计与图形学学报, 2020, 32(4): 652–659
LI Yiyi, WANG Jilong. Self-attention based video summarization[J]. Journal of computer-aided design & computer graphics, 2020, 32(4): 652–659
[13] JI Zhong, XIONG Kailin, PANG Yanwei, et al. Video summarization with attention-based encoder–decoder networks[J]. IEEE transactions on circuits and systems for video technology, 2020, 30(6): 1709–1717.
[14] FAJTL J, SOKEH H S, ARGYRIOU V, et al. Summarizing videos with attention[M]//Computer Vision-ACCV 2018 Workshops. Cham: Springer International Publishing, 2019: 39-54.
[15] MAHASSENI B, LAM M, TODOROVIC S. Unsupervised video summarization with adversarial LSTM networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2982-2991.
[16] LEI Jie, LUAN Qiao, SONG Xinhui, et al. Action parsing-driven video summarization based on reinforcement learning[J]. IEEE transactions on circuits and systems for video technology, 2019, 29(7): 2126–2137.
[17] 李雷霆, 武光利, 郭振洲. 自注意力机制和随机森林回归的视频摘要生成[J]. 计算机工程与应用, 2022, 58(4): 198–205
LI Leiting, WU Guangli, GUO Zhenzhou. Video summarization generation based on self-attention mechanism and random forest regression[J]. Computer engineering and applications, 2022, 58(4): 198–205
[18] ZHOU Juanping, LU Lu. Wide and deep learning for video summarization via attention mechanism and independently recurrent neural network[C]//2020 Data Compression Conference. Snowbird: IEEE, 2020: 407.
[19] 王鈃润, 聂秀山, 杨帆, 等. 基于排序学习的视频摘要[J]. 智能系统学报, 2018, 13(6): 921–927
WANG Xingrun, NIE Xiushan, YANG Fan, et al. Video summarization based on learning to rank[J]. CAAI transactions on intelligent systems, 2018, 13(6): 921–927
[20] XU Huijuan, DAS A, SAENKO K. Two-stream region convolutional 3D network for temporal activity detection[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 41(10): 2319–2332.
[21] POTAPOV D, DOUZE M, HARCHAOUI Z, et al. Category-specific video summarization[M]//Computer Vision-ECCV 2014. Cham: Springer International Publishing, 2014: 540-555.
[22] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C] //Proceedings of the 31st International Conference on Advances in Neural Information Processing Systems. New York: Curran Associates, Inc, 2017: 5998-6008.
[23] GYGLI M, GRABNER H, RIEMENSCHNEIDER H, et al. Creating summaries from user videos[M]//Computer Vision-ECCV 2014. Cham: Springer International Publishing, 2014: 505-520.
[24] SONG Yale, VALLMITJANA J, STENT A, et al. TVSum: Summarizing web videos using titles[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 5179-5187.
[25] DE AVILA S E F, LOPES A P B, DA LUZ A, et al. VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method[J]. Pattern recognition letters, 2011, 32(1): 56–68.
[26] 王坤阳, 高伟, 滕国伟. 基于门控多头注意力机制的视频摘要[J]. 工业控制计算机, 2022, 35(12): 120–122
WANG Kunyang, GAO Wei, TENG Guowei. Video summarization based on gated multi-head attention mechanism[J]. Industrial control computer, 2022, 35(12): 120–122
[27] 梅锋, 周娟平, 陆璐. 结合局部奖励机制的视频摘要技术研究[J]. 计算机工程与应用, 2021, 57(11): 211–218
MEI Feng, ZHOU Juanping, LU Lu. Research on video summarization technology combining local reward mechanism[J]. Computer engineering and applications, 2021, 57(11): 211–218
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems