[1]LIU Zhi,LI Guangdeng,LI Shiyu,et al.Modality-shared gated two-stream Transformer for visibleinfrared person re-identification[J].CAAI Transactions on Intelligent Systems,2026,21(4):979-987.[doi:10.11992/tis.202511021]
Copy

Modality-shared gated two-stream Transformer for visibleinfrared person re-identification

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 21 Number of periods: 2026 4 Page number: 979-987 Column: 学术论文—机器感知与模式识别 Public date: 2026-07-05
Title:
Modality-shared gated two-stream Transformer for visibleinfrared person re-identification
Author(s):
LIU Zhi1;  LI Guangdeng1;  LI Shiyu1;  WANG Wei2;  ZHANG Xiaochuan1; 3
1. Liangjiang School of Artificial Intelligence, Chongqing University of Technology, Chongqing 401135, China;
2. Strategic Assessment and Consulting Center, Academy of Military Sciences, Beijing 100091, China;
3. School of Software, Chongqing Institute of Engineering, Chongqing 400056, China
Keywords:
visible-infrared person re-identification; multi-modal learning; vision Transformer; contrastive language–image pre-training; modality-shared features extraction; modality-shared gated
CLC:
TP391
DOI:
10.11992/tis.202511021
Abstract:
Visible-infrared person re-identification (VI-ReID) aims to match person images of the same identity across different modalities, representing a highly challenging task in the field of computer vision. In real-world scenarios, poor illumination often leads to recognition difficulties, severely limiting the practicality of person re-identification systems. Although traditional methods based on convolutional neural networks have been widely investigated, they are limited by local receptive fields and down-sampling operations, which tend to result in the loss of key modality information. To address these issues, this paper proposes a vision Transformer-based VI-ReID framework, termed modality-shared gating two-stream Transformer (MgtFormer). First, aiming at the modality discrepancy between infrared and RGB images, a grayscale image augmentation strategy is adopted to convert RGB images into grayscale ones to minimize the modality gap. Second, a modality-specific embedding module is designed to guide the ViT to extract modality-shared features more effectively. Subsequently, a modality-shared module is utilized to fully mine modality-invariant features from RGB, grayscale, and infrared images. Finally, a modality-shared gating layer is introduced to finely screen the features, retaining highly discriminative ones for subsequent re-identification tasks. MgtFormer is capable of comprehensively extracting multi-modal information and effectively preserving modality-invariant features. Extensive experiments on the SYSU-MM01 and RegDB datasets demonstrate that the proposed method outperforms current state-of-the-art techniques.
References:
[1] ZHENG Liang, YANG Yi, HAUPTMANN A G. Person re-identification: past, present and future[EB/OL]. (2016-10-10)[2025-11-15]. https://arxiv.org/abs/1610.02984.
[2] SUN Yifan, ZHENG Liang, YANG Yi, et al. Beyond part models: person retrieval with refined part pooling (and a strong convolutional baseline)[C]//Computer Vision–ECCV 2018. Cham: Springer International Publishing, 2018: 501-518.
[3] ZHENG Feng, DENG Cheng, SUN Xing, et al. Pyramidal person re-IDentification via multi-loss dynamic training[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2020: 8506-8514.
[4] JIN Xin, LAN Cuiling, ZENG Wenjun, et al. Style normalization and restitution for generalizable person re-identification[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 3140-3149.
[5] SUN Yifan, CHENG Changmao, ZHANG Yuhan, et al. Circle loss: a unified perspective of pair similarity optimization[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 6397-6406.
[6] NGUYEN B X, NGUYEN B D, DO T, et al. Graph-based person signature for person re-identifications[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Nashville: IEEE, 2021: 3487-3496.
[7] HE Shuting, LUO Hao, WANG Pichao, et al. TransReID: transformer-based object re-identification[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14993-15002.
[8] REN Min, HE Lingxiao, LIAO Xingyu, et al. Learning instance-level spatial-temporal patterns for person re-identification[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 14910-14919.
[9] LI Dengjie, CHEN Siyu, ZHONG Yujie, et al. DiP: learning discriminative implicit parts for person re-identification[EB/OL]. (2022-12-24)[2025-11-15]. https://arxiv.org/abs/2212.13906.
[10] LI Siyuan, SUN Li, LI Qingli. CLIP-ReID: exploiting vision-language model for image re-identification without concrete text labels[J]. Proceedings of the AAAI conference on artificial intelligence, 2023, 37(1): 1405-1413
[11] SOMERS V, DE VLEESCHOUWER C, ALAHI A. Body part-based representation learning for occluded person re-identification[C]//2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2023: 1613-1623.
[12] TIAN Xudong, ZHANG Zhizhong, WANG Cong, et al. Variational distillation for multi-view learning[EB/OL]. (2022-06-20)[2025-11-15]. https://arxiv.org/abs/2206.09548.
[13] JOSI A, ALEHDAGHI M, CRUZ R M O, et al. Multimodal data augmentation for visual-infrared person ReID with corrupted data[C]//2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. Waikoloa: IEEE, 2023: 1-10.
[14] ALEHDAGHI M, JOSI A, CRUZ R M O, et al. Visible-infrared person re-identification using privileged intermediate information[C]//Computer Vision–ECCV 2022 Workshops. Cham: Springer Nature Switzerland, 2023: 720-737.
[15] FENG Jiawei, WU Ancong, ZHENG Weishi. Shape-erased feature learning for visible-infrared person re-identification[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 22752-22761.
[16] YE Mang, WANG Zheng, LAN Xiangyuan, et al. Visible thermal person re-identification via dual-constrained top-ranking[C]//Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence Main track. Stockholm: IJCAI, 2018: 1092-1099.
[17] HAO Yi, WANG Nannan, LI Jie, et al. HSME: hypersphere manifold embedding for visible thermal person re-identification[C]//Proceedings of the AAAI conference on artificial intelligence, Honolulu: ACM, 2019: 8385-8392.
[18] LU Yan, WU Yue, LIU Bin, et al. Cross-modality person re-identification with shared-specific feature transfer[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 13376-13386.
[19] LUO Hao, WANG Pichao, XU Yi, et al. Self-supervised pre-training for Transformer-based person re-identification[EB/OL]. (2021-11-23)[2025-11-15]. https://arxiv.org/abs/2111.12084.
[20] JIA Mengxi, CHENG Xinhua, LU Shijian, et al. Learning disentangled representation implicitly via Transformer for occluded person re-identification[J]. IEEE transactions on multimedia, 2023, 25: 1294-1305
[21] DOU Shuguang, ZHAO Cairong, JIANG Xinyang, et al. Human co-parsing guided alignment for occluded person re-identification[J]. IEEE transactions on image processing, 2023, 32: 458-470
[22] TAN Lei, XIA Jiaer, LIU Wenfeng, et al. Occluded person re-identification via saliency-guided patch transfer[C]//Proceedings of the 2024 AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2024: 5070-5078.
[23] XIA Jiaer, TAN Lei, DAI Pingyang, et al. Attention disturbance and dual-path constraint network for occluded person re-identification[C]//Proceedings of the 2024 AAAI Conference on Artificial Intelligence. Vancouver: AAAI, 2024: 6198-6206.
[24] GAO Liying, JIAO Bingliang, LONG Yuzhou, et al. Contrastive pedestrian attentive and correlation learning network for occluded person re-identification[J]. IEEE transactions on circuits and systems for video technology, 2024, 34(9): 8862-8880
[25] WANG Tao, LIU Mengyuan, LIU Hong, et al. Feature completion transformer for occluded person re-identification[J]. IEEE transactions on multimedia, 2024, 26: 8529-8542
[26] LI Yanping, LIU Yizhang, ZHANG Hongyun, et al. Occlusion-aware transformer with second-order attention for person re-identification[J]. IEEE transactions on image processing, 2024, 33: 3200-3211
[27] YE Mang, RUAN Weijian, DU Bo, et al. Channel augmented joint learning for visible-infrared recognition[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 13547-13556.
[28] WU Ancong, ZHENG Weishi, YU Hongxing, et al. RGB-infrared cross-modality person re-identification[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 5390-5399.
[29] 王晋溪, 鲁鸣鸣. 基于场景图知识的文本到图像行人重识别[J]. 模式识别与人工智能, 2024, 37(11): 947-959 WANG Jinxi, LU Mingming. Scene graph knowledge based text-to-image person re-identification[J]. Pattern recognition and artificial intelligence, 2024, 37(11): 947-959
[30] 石瑞鑫, 智敏, 殷雁君. 多模态行人重识别研究综述[J]. 计算机应用研究, 2025, 42(7): 1921-1929 SHI Ruixin, ZHI Min, YIN Yanjun. Review of multimodal pedestrian re-identification[J]. Application research of computers, 2025, 42(7): 1921-1929
[31] 李俊峰, 楼琼, 钱亚冠, 等. 基于像素对齐和特征对齐的跨模态行人重识别[J]. 浙江科技学院学报, 2022(3): 251-260 LI Junfeng, LOU Qiong, QIAN Yaguan, et al. Cross-modality person re-identification based onpixel alignment and feature alignment[J]. Journal of Zhejiang University of Science and Technology, 2022(3): 251-260
[32] 冯展祥, 赖剑煌, 袁藏, 等. 走向通用行人重识别: 预训练大模型技术在行人重识别的应用综述[J]. 中国图象图形学报, 2025, 30(6): 1638-1660 FENG Zhanxiang, LAI Jianhuang, YUAN Zang, et al. Advancing universal person reidentification: a survey on the applications of large-scale, pretraining models for identifying individuals[J]. Journal of image and graphics, 2025, 30(6): 1638-1660
[33] 孙锐, 杜云, 陈龙, 等. 隐式多尺度对齐与交互的文本-图像行人重识别方法[J]. 软件学报, 2025, 36(10): 4846-4863 SUN Rui, DU Yun, CHEN Long, et al. Implicit multi-scale alignment and interaction for text-image person re-identification method[J]. Journal of software, 2025, 36(10): 4846-4863
[34] 金昌胜, 王海瑞. 基于关系挖掘的跨模态行人重识别[J]. 空军工程大学学报, 2024, 25(1): 106-114 JIN Changsheng, WANG Hairui. A cross-modal person re-identification based on relationship mining[J]. Journal of Air Force Engineering University, 2024, 25(1): 106-114
[35] NGUYEN D T, HONG H G, KIM K W, et al. Person recognition system based on a combination of body images from visible light and thermal cameras[J]. Sensors, 2017, 17(3): 605
[36] YE Mang, LAN Xiangyuan, WANG Zheng, et al. Bi-directional center-constrained top-ranking for visible thermal person re-identification[J]. IEEE transactions on information forensics and security, 2020, 15: 407-419
[37] CHOI S, LEE S, KIM Y, et al. Hi-CMD: hierarchical cross-modality disentanglement for visible-infrared person re-identification[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 10254-10263.
[38] WANG Pingyu, ZHAO Zhicheng, SU Fei, et al. Deep multi-patch matching network for visible thermal person re-identification[J]. IEEE transactions on multimedia, 2021, 23: 1474-1488
[39] PARK H, LEE S, LEE J, et al. Learning by aligning: visible-infrared person re-identification using cross-modal correspondences[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2022: 12026-12035.
[40] CHEN Y, WAN Lin, LI Zhihang, et al. Neural feature search for RGB-infrared person re-identification[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 587-597.
[41] YE Mang, SHEN Jianbing, LIN Gaojie, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(6): 2872-2893
[42] WANG Pingyu, SU Fei, ZHAO Zhicheng, et al. Deep hard modality alignment for visible thermal person re-identification[J]. Pattern recognition letters, 2020, 133: 195-201
[43] YE Mang, CHEN Cuiqun, SHEN Jianbing, et al. Dynamic tri-level relation mining with attentive graph for visible infrared re-identification[J]. IEEE transactions on information forensics and security, 2022, 17: 386-398
[44] ZHAO Jiaqi, WANG Hanzheng, ZHOU Yong, et al. Spatial-channel enhanced transformer for visible-infrared person re-identification[J]. IEEE transactions on multimedia, 2023, 25: 3668-3680
[45] WANG Guanan, ZHANG Tianzhu, CHENG Jian, et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 3622-3631.
[46] WANG Guanan, ZHANG Tianzhu, YANG Yang, et al. Cross-modality paired-images generation for RGB-infrared person re-identification[C]//The AAAI 2020 proceedings include all papers presented at the 34th AAAI Conference. New York: AAAI, 2020: 12144-12151.
[47] WANG Xiaogang, DORETTO G, SEBASTIAN T, et al. Shape and appearance context modeling[C]//2007 IEEE 11th International Conference on Computer Vision. Rio de Janeiro: IEEE, 2007: 1-8.
[48] ZHENG Liang, SHEN Liyue, TIAN Lu, et al. Scalable person re-identification: a benchmark[C]//2015 IEEE International Conference on Computer Vision. Santiago: IEEE, 2016: 1116-1124.
[49] CHENG De, HUANG Xiaojian, WANG Nannan, et al. Unsupervised visible-infrared person ReID by collaborative learning with neighbor-guided label refinement[C]//Proceedings of the 31st ACM International Conference on Multimedia. Ottawa: ACM, 2023: 7085-7093.
[50] ZHANG Xu, LIU Yinghui, GUO Liangchen, et al. Zero-shot infrared domain adaptation for pedestrian re-identification via deep learning[J]. Electronics, 2025, 14(14): 2784
[51] 张红颖, 樊世钰, 罗谦, 等. 结合视觉文本匹配和图嵌入的可见光-红外行人重识别[J]. 电子与信息学报, 2024, 46(9): 3662-3671 ZHANG Hongying, FAN Shiyu, LUO Qian, et al. Visible-infrared pedestrian recognition combined with visual text matching and graph embedding[J]. Journal of electronics & information technology, 2024, 46(9): 3662-3671
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems