[1]YANG Yuyu,YANG Xiao,PAN Zaiyu,et al.Domain adaptive semantic segmentation based on prototype-guided and adaptive feature fusion[J].CAAI Transactions on Intelligent Systems,2025,20(1):150-161.[doi:10.11992/tis.202403010]
Copy

Domain adaptive semantic segmentation based on prototype-guided and adaptive feature fusion

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 20 Number of periods: 2025 1 Page number: 150-161 Column: 学术论文—智能系统 Public date: 2025-01-05
Title:
Domain adaptive semantic segmentation based on prototype-guided and adaptive feature fusion
Author(s):
YANG Yuyu;  YANG Xiao;  PAN Zaiyu;  WANG Jun
School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China
Keywords:
deep learning; unsupervised learning; domain adaptation; semantic segmentation; attention mechanism; self-training learning; self-adaptive; transfer learning; prototype guidance
CLC:
TP301
DOI:
10.11992/tis.202403010
Abstract:
Unsupervised domain adaptation techniques are of significant importance to reducing the data annotation workload for computer vision tasks, particularly in pixel-level semantic segmentation. However, challenges such as the dispersed feature distribution and class imbalance in the target domain, such as blurred class boundaries and insufficient samples for certain categories, pose challenges to this technology. To address these challenges, this paper proposes a prototype-guided adaptive feature fusion model. It incorporates a dual attention network guided by prototypes to fuse spatial and channel attention features, enhancing class-wise compactness. Furthermore, this paper introduces an adaptive feature fusion module that flexibly adjusts the importance of each feature, enabling the network to capture more class-discriminative features across different spatial locations and channels, thereby further enhancing the performance of semantic segmentation. Experimental results on two challenging synthetic-to-real benchmarks of GTA5-to-Cityscape and SYNTHIA-to-Cityscape demonstrate the effectiveness of our method, showcasing the model’s capability to handle complex scenes and imbalanced data.
References:
[1] 景庄伟, 管海燕, 彭代峰, 等. 基于深度神经网络的图像语义分割研究综述[J]. 计算机工程, 2020, 46(10): 1-17.
JING Zhuangwei, GUAN Haiyan, PENG Daifeng, et al. Survey of research in image semantic segmentation based on deep neural network[J]. Computer engineering, 2020, 46(10): 1-17.
[2] 计梦予, 袭肖明, 于治楼. 基于深度学习的语义分割方法综述[J]. 信息技术与信息化, 2017(10): 137-140.
JI Mengyu, XI Xiaoming, YU Zhilou. A review of semantic segmentation based on deep learning[J]. Information technology and informatization, 2017(10): 137-140.
[3] SHELHAMER E, LONG J, DARRELL T. Fully convolutional networks for semantic segmentation[C]//IEEE Transactions on Pattern Analysis and Machine Intelligence. Boston: IEEE, 2017: 640-651.
[4] 范苍宁, 刘鹏, 肖婷, 等. 深度域适应综述: 一般情况与复杂情况[J]. 自动化学报, 2021, 47(3): 515-548.
FAN Cangning, LIU Peng, XIAO Ting, et al. A review of deep domain adaptation: general situation and complex situation[J]. Acta automatica sinica, 2021, 47(3): 515-548.
[5] 高德鹏. 基于跨域正则化模型的域适应方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
GAO Depeng. Research on domain adaptation method based on cross-domain regularization model[D]. Harbin: Harbin Institute of Technology, 2020.
[6] 王格格, 郭涛, 余游, 等. 基于生成对抗网络的无监督域适应分类模型[J]. 电子学报, 2020, 48(6): 1190-1197.
WANG Gege, GUO Tao, YU You, et al. Unsupervised domain adaptation classification model based on generative adversarial network[J]. Acta electronica sinica, 2020, 48(6): 1190-1197.
[7] BOUSMALIS K, SILBERMAN N, DOHAN D, et al. Unsupervised pixel-level domain adaptation with generative adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 95-104.
[8] ZHOU Wei, WANG Yukang, CHU Jiajia, et al. Affinity space adaptation for semantic segmentation across domains[J]. IEEE transactions on image processing, 2021, 30: 2549-2561.
[9] 高子航, 刘兆英, 张婷, 等. 基于对抗域适应的红外舰船目标分割[J]. 数据采集与处理, 2023, 38(3): 598-607.
GAO Zihang, LIU Zhaoying, ZHANG Ting, et al. Infrared ship target segmentation based on adversarial domain adaptation[J]. Journal of data acquisition and processing, 2023, 38(3): 598-607.
[10] 张桂梅, 鲁飞飞, 龙邦耀, 等. 结合自集成和对抗学习的域自适应城市场景语义分割[J]. 模式识别与人工智能, 2021, 34(1): 58-67.
ZHANG Guimei, LU Feifei, LONG Bangyao, et al. Domain adaptation semantic segmentation for urban scene combining self-ensembling and adversarial learning[J]. Pattern recognition and artificial intelligence, 2021, 34(1): 58-67.
[11] ZHAO Yihao, WU Ruihai, DONG Hao. Unpaired image-to-image translation using adversarial consistency loss[M]//Lecture Notes in Computer Science. Cham: Springer International Publishing, 2020: 800-815.
[12] 李美丽, 杨传颖, 石宝. 基于语义分割的图像风格迁移技术研究[J]. 计算机工程与应用, 2020, 56(24): 207-213.
LI Meili, YANG Chuanying, SHI Bao. Research on image style transfer technology based on semantic segmentation[J]. Computer engineering and applications, 2020, 56(24): 207-213.
[13] 吕佳, 李婷婷. 半监督自训练方法综述[J]. 重庆师范大学学报(自然科学版), 2021, 38(5): 98-106.
LYU Jia, LI Tingting. A summary of semi-supervised self-training methods[J]. Journal of Chongqing normal university (natural science edition), 2021, 38(5): 98-106.
[14] 张勋晖, 周勇, 赵佳琦, 等. 基于熵增强的无监督域适应遥感图像语义分割[J]. 计算机应用研究, 2021, 38(9): 2852-2856.
ZHANG Xunhui, ZHOU Yong, ZHAO Jiaqi, et al. Entropy enhanced unsupervised domain adaptive remote sensing image semantic segmentation[J]. Application research of computers, 2021, 38(9): 2852-2856.
[15] ZHANG Pan, ZHANG Bo, ZHANG Ting, et al. Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 12409-12419.
[16] ZHAO Hengshuang, SHI Jianping, QI Xiaojuan, et al. Pyramid scene parsing network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 6230-6239.
[17] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE transactions on pattern analysis and machine intelligence, 2018, 40(4): 834-848.
[18] YANG Zhen, PENG Xiaobao, YIN Zhijian, et al. Deeplab_v3_plus-net for image semantic segmentation with channel compression[C]//2020 IEEE 20th International Conference on Communication Technology. Nanning: IEEE, 2020: 1320-1324.
[19] LIU Ze, LIN Yutong, CAO Yue, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//2021 IEEE/CVF International Conference on Computer Vision. Montreal: IEEE, 2021: 9992-10002.
[20] XIE Enze, WANG Wenhai, YU Zhiding, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[J]. Advances in neural information processing systems, 2021, 34: 12077.
[21] LIU Ze, HU Han, LIN Yutong, et al. Swin transformer V2: scaling up capacity and resolution[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 11999-12009.
[22] SMOLA A J, GRETTON A, BORGWARDT K. Maximum mean discrepancy[C]//2006 ICONIP 13th International Conference on Neural Information Processing. HongKong: Springer International Publishing, 2006: 3-6.
[23] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial networks[EB/OL]. (2014-06-10)[2024-03-05]. https://arxiv.org/abs/1406.2661v1.
[24] HOFFMAN J, WANG Dequan, YU F, et al. FCNs in the wild: pixel-level adversarial and constraint-based adaptation[EB/OL]. (2016-12-08)[2024-02-15]. https://doi.org/10.48550/acxiv.
[25] ZHU Junyan, PARK T, ISOLA P, et al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision. Venice: IEEE, 2017: 2242-2251.
[26] TSAI Y H, HUNG W C, SCHULTER S, et al. Learning to adapt structured output space for semantic segmentation[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7472-7481.
[27] VU T H, JAIN H, BUCHER M, et al. ADVENT: adversarial entropy minimization for domain adaptation in semantic segmentation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019: 2512-2521.
[28] JIANG Zhengkai, LI Yuxi, YANG Ceyuan, et al. Prototypical contrast adaptation for Domain adaptive semantic segmentation[M]//Lecture Notes in Computer Science. Cham: Springer Nature Switzerland, 2022: 36-54.
[29] HOYER L, DAI Dengxin, WANG Haoran, et al. MIC: masked image consistency for context-enhanced domain adaptation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 11721-11732.
[30] CHEN Mu, ZHENG Zhedong, YANG Yi, et al. PiPa: pixel- and patch-wise self-supervised learning for domain adaptative semantic segmentation[C]//Proceedings of the 31st ACM International Conference on Multimedia. Ottawa: ACM, 2023: 1905-1914.
[31] WANG Yisen, MA Xingjun, CHEN Zaiyi, et al. Symmetric cross entropy for robust learning with noisy labels[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 322-330.
[32] ZOU Yang, YU Zhiding, LIU Xiaofeng, et al. Confidence regularized self-training[C]//2019 IEEE/CVF International Conference on Computer Vision. Seoul: IEEE, 2019: 5981-5990.
[33] RICHTER S R, VINEET V, ROTH S, et al. Playing for data: ground truth from computer games[M]//Lecture Notes in Computer Science. Cham: Springer International Publishing, 2016: 102-118.
[34] ROS G, SELLART L, MATERZYNSKA J, et al. The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vega: IEEE, 2016: 3234-3243.
[35] CORDTS M, OMRAN M, RAMOS S, et al. The cityscapes dataset for semantic urban scene understanding[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 3213-3223.
[36] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778.
[37] YANG Yanchao, SOATTO S. FDA: fourier domain adaptation for semantic segmentation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 4084-4094.
[38] KANG Guoliang, WEI Yunchao, YANG Yi, et al. Pixel-level cycle association: a new perspective for domain adaptive semantic segmentation[J]. Advances in neural information processing systems, 2020, 33: 3569.
[39] IQBAL J, ALI M. MLSL: multi-level self-supervised learning for domain adaptation with spatially independent and semantically consistent labeling[C]//2020 IEEE Winter Conference on Applications of Computer Vision. Snowmass: IEEE, 2020: 1853-1862.
[40] LUO Yawei, LIU Ping, ZHENG Liang, et al. Category-level adversarial adaptation for semantic segmentation using purified features[J]. IEEE transactions on pattern analysis and machine intelligence, 2022, 44(8): 3940-3956.
[41] TOLDO M, MICHIELI U, ZANUTTIGH P. Unsupervised domain adaptation in semantic segmentation via orthogonal and clustered embeddings[C]//2021 IEEE Winter Conference on Applications of Computer Vision. Waikoloa: IEEE, 2021: 1357-1367.
[42] MELAS-KYRIAZI L, MANRAI A K. PixMatch: unsupervised domain adaptation via pixelwise consistency training[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021: 12430-12440.
[43] IQBAL J, RAWAL H, HAFIZ R, et al. Distribution regularized self-supervised learning for domain adaptation of semantic segmentation[J]. Image and vision computing, 2022, 124: 104504.
[44] CAO Yihong, ZHANG Hui, LU Xiao, et al. Adaptive refining-aggregation-separation framework for unsupervised domain adaptation semantic segmentation[J]. IEEE transactions on circuits and systems for video technology, 2023, 33(8): 3822-3832.
[45] GUO Yaqian, WANG Xin, LI Ce, et al. Domain adaptive semantic segmentation by optimal transport[J]. Fundamental research, 2024, 4(5): 981-991.
[46] ZHANG Yuhang, TIAN Shishun, LIAO Muxin, et al. A hybrid domain learning framework for unsupervised semantic segmentation[J]. Neurocomputing, 2023, 516: 133-145.
[47] CHUNG I, YOO J, KWAK N. Exploiting inter-pixel correlations in unsupervised domain adaptation for semantic segmentation[C]//2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops. Waikoloa: IEEE, 2023: 12-21.
[48] LI Jing, ZHOU Kang, QIAN Shenhan, et al. Feature re-representation and reliable pseudo label retraining for cross-domain semantic segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2024, 46(3): 1682-1694.
[49] VAN DER MAATEN L, HINTON G. Visualizing data using t-SNE[J]. Journal of machine learning research, 2008, 9(11): 01301.
Similar References:

Memo

-

Last Update: 2025-01-05

Copyright © CAAI Transactions on Intelligent Systems