[1]LIU Shuang,CHEN Jing.Optical flow estimation network combining convolution and axial attention[J].CAAI Transactions on Intelligent Systems,2024,19(3):575-583.[doi:10.11992/tis.202210029]
Copy

Optical flow estimation network combining convolution and axial attention

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 19 Number of periods: 2024 3 Page number: 575-583 Column: 学术论文—机器感知与模式识别 Public date: 2024-05-05
Title:
Optical flow estimation network combining convolution and axial attention
Author(s):
LIU Shuang1;  CHEN Jing1; 2
1. School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi 214122, China;
2. Jiangsu Provincial Engineering Laboratory of Pattern Recognition and Computing Intelligence, Jiangnan University, Wuxi 214122, China
Keywords:
optical flow estimation; iterations; convolutional neural networks; axial attention mechanism; gated recurrent unit network; deep learning; time optimization; edge computing platform
CLC:
TP391
DOI:
10.11992/tis.202210029
Abstract:
Existing optical flow estimation networks often utilize correlation cost volume and gated recurrent unit (GRU) to realize iterative optimization for improved accuracy. However, this approach incurs high computational volume and limits deployment performance on edge computing platforms. To realize a lightweight optical flow estimation method, the local constraint and local dilation (LC-LD) module is introduced. This approach combines convolution and primary axis attention to replace self-attention. A low computational volume enables the module to realize attentions with different important degrees for peripheral areas of each matching feature point, generate an accurate correlation cost volume, further reduce the iterations, and achieve lightweight features. In addition, the shuffling convex optimization upsampling method is proposed. This technique combines group convolution, shuffle operation, and convex optimization upsampling, further increasing the precision and reducing the number of parameters. Experimental results show that the proposed method achieves significant improvements in running efficiency while maintaining high accuracy and great potential for application.
References:
[1] 李志慧, 胡永利, 赵永华, 等. 基于车载的运动行人区域估计方法[J]. 吉林大学学报(工学版), 2018, 48(3): 694–703
LI Zhihui, HU Yongli, ZHAO Yonghua, et al. Locating moving pedestrian from running vehicle[J]. Journal of Jilin University (engineering and technology edition), 2018, 48(3): 694–703
[2] 陈戈, 董明明. 基于特征点检测与光流法的运动目标跟踪算法[J]. 电子测量技术, 2017, 40(12): 214–219
CHEN Ge, DONG Mingming. Moving object tracking algorithm based on feature point detection and optical flow[J]. Electronic measurement technology, 2017, 40(12): 214–219
[3] SOUHILA K, KARIM A. Optical flow based robot obstacle avoidance[J]. International journal of advanced robotic systems, 2007, 4(1): 2.
[4] 李秀智, 杨爱林, 秦宝岭, 等. 基于光流反馈的单目视觉三维重建[J]. 光学学报, 2015, 35(5): 515001
LI Xiuzhi, YANG Ailin, QIN Baoling, et al. Monocular camera three dimensional reconstruction based on optical flow feedback[J]. Acta optica sinica, 2015, 35(5): 515001
[5] HORN B K P, SCHUNCK B G. Determining optical flow[J]. Artificial intelligence, 1981, 17(4): 185–203.
[6] BROX T, MALIK J. Large displacement optical flow: descriptor matching in variational motion estimation[J]. IEEE transactions on pattern analysis and machine intelligence, 2011, 33(3): 500–513.
[7] ZACH C, POCK T, BISCHOF H. A duality based approach for realtime TV-L 1 optical flow[C]//Joint Pattern Recognition Symposium. Berlin: Springer, 2007: 214-223.
[8] KROEGER T, TIMOFTE R, DAI Dengxin, et al. Fast optical flow using dense inverse search[C]//European Conference on Computer Vision. Cham: Springer, 2016: 471-488.
[9] DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet: learning optical flow with convolutional networks[C]//2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2015: 2758-2766.
[10] ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0: evolution of optical flow estimation with deep networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1647-1655.
[11] RANJAN A, BLACK M J. Optical flow estimation using a spatial pyramid network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2720-2729.
[12] SUN Deqing, YANG Xiaodong, LIU Mingyu, et al. PWC-net: CNNs for optical flow using pyramid, warping, and cost volume[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 8934-8943.
[13] HUI T W, TANG Xiaoou, LOY C C. LiteFlowNet: a lightweight convolutional neural network for optical flow estimation[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 8981-8989.
[14] KONG Lingtong, SHEN Chunhua, YANG Jie. FastFlowNet: a lightweight network for fast optical flow estimation[C]//2021 IEEE International Conference on Robotics and Automation. Xi’an: IEEE, 2021: 10310-10316.
[15] TEED Z, DENG Jia. RAFT: recurrent all-pairs field transforms for optical flow[C]//European Conference on Computer Vision. Cham: Springer, 2020: 402-419.
[16] CHO K, VAN MERRIENBOER B, BAHDANAU D, et al. On the properties of neural machine translation: encoder-decoder approaches[EB/OL]. (2014-09-03)[2022-10-24]. http://arxiv.org/abs/1409.1259.
[17] JIANG Shihao, CAMPBELL D, LU Yao, et al. Learning to estimate hidden motions with global motion aggregation[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 9752-9761.
[18] JIANG Shihao, LU Yao, LI Hongdong, et al. Learning optical flow from a few matches[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 16587-16595.
[19] JAEGLE A, BORGEAUD S, ALAYRAC J B, et al. Perceiver IO: a general architecture for structured inputs & outputs[EB/OL]. (2021-07-30)[2022-10-24]. http://arxiv.org/abs/2107.14795.
[20] XU Haofei, ZHANG Jing, CAI Jianfei, et al. Unifying flow, stereo and depth estimation[EB/OL]. (2022-11-10)[2023-10-24]. http://arxiv.org/abs/2211.05783.
[21] SUI Xiuchao, LI Shaohua, GENG Xue, et al. CRAFT: cross-attentional flow transformer for robust optical flow[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17581-17590.
[22] HUANG Zhaoyang, SHI Xiaoyu, ZHANG Chao, et al. FlowFormer: a transformer architecture for optical flow[EB/OL]. (2022-03-30)[2022-10-24]. http://arxiv.org/abs/2203.16194.
[23] ZHAO Shiyu, ZHAO Long, ZHANG Zhixing, et al. Global matching with overlapping attention for optical flow estimation[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 17571-17580.
[24] 周海赟, 项学智, 翟明亮, 等. 结合注意力机制的深度学习光流网络[J]. 计算机科学与探索, 2020, 14(11): 1920–1929
ZHOU Haiyun, XIANG Xuezhi, ZHAI Mingliang, et al. Deep optical flow learning networks combined with attention mechanism[J]. Journal of frontiers of computer science and technology, 2020, 14(11): 1920–1929
[25] LI Yehao, YAO Ting, PAN Yingwei, et al. Contextual transformer networks for visual recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2023, 45(2): 1489–1500.
[26] ZHANG Xiangyu, ZHOU Xinyu, LIN Mengxiao, et al. ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 6848-6856.
[27] MAYER N, ILG E, H?USSER P, et al. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 4040-4048.
[28] BUTLER D J, WULFF J, STANLEY G B, et al. A naturalistic open source movie for optical flow evaluation[C]//European Conference on Computer Vision. Berlin: Springer, 2012: 611-625.
[29] MENZE M, GEIGER A. Object scene flow for autonomous vehicles[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston: IEEE, 2015: 3061-3070.
[30] LIU Ze, LIN Yutong, CAO Yue, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 9992-10002.
[31] HUANG Zilong, WANG Xinggang, HUANG Lichao, et al. CCNet: criss-cross attention for semantic segmentation[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 603-612.
[32] HUI T W, TANG Xiaoou, LOY C C. A lightweight optical flow CNN-revisiting data fidelity and regularization[J]. IEEE transactions on pattern analysis and machine intelligence, 2021, 43(8): 2555–2569.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems