基于原型引导与自适应特征融合的域适应语义分割

PDF下载HTML

Title:

Domain adaptive semantic segmentation based on prototype-guided and adaptive feature fusion

作者:

杨宇宇, 杨霄, 潘在宇, 王军

中国矿业大学 信息与控制工程学院, 江苏 徐州 221116

Author(s):

YANG Yuyu, YANG Xiao, PAN Zaiyu, WANG Jun

School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China

关键词:

深度学习; 无监督学习; 域适应; 语义分割; 注意力机制; 自训练学习; 自适应; 迁移学习; 原型引导

Keywords:

deep learning; unsupervised learning; domain adaptation; semantic segmentation; attention mechanism; self-training learning; self-adaptive; transfer learning; prototype guidance

分类号:

TP301

DOI:

10.11992/tis.202403010

摘要:

无监督域自适应技术对于减少计算机视觉任务中的数据标注工作量具有重要意义，尤其在像素级的语义分割中。然而，目标域的特征分布离散和类别不平衡问题，如模糊的类边界和某些类别的样本过少，对无监督域自适应技术构成了挑战。针对上述挑战，本文提出了一种原型引导的自适应特征融合模型。其中，通过引入原型引导的双重注意力网络融合空间和通道注意力特征，增强类内紧凑性。此外，本文提出自适应特征融合模块，灵活调整各特征的重要性，使网络能够在不同的空间位置和通道上捕捉到更加具有类别区分性的特征，进一步提升语义分割性能。在两个具有挑战性的合成–真实基准GTA5-to-Cityscape和SYNTHIA-to-Cityscape上的实验结果证明了本文方法的有效性，展现出模型对复杂场景和不平衡数据的处理应对能力。

Abstract:

Unsupervised domain adaptation techniques are of significant importance to reducing the data annotation workload for computer vision tasks, particularly in pixel-level semantic segmentation. However, challenges such as the dispersed feature distribution and class imbalance in the target domain, such as blurred class boundaries and insufficient samples for certain categories, pose challenges to this technology. To address these challenges, this paper proposes a prototype-guided adaptive feature fusion model. It incorporates a dual attention network guided by prototypes to fuse spatial and channel attention features, enhancing class-wise compactness. Furthermore, this paper introduces an adaptive feature fusion module that flexibly adjusts the importance of each feature, enabling the network to capture more class-discriminative features across different spatial locations and channels, thereby further enhancing the performance of semantic segmentation. Experimental results on two challenging synthetic-to-real benchmarks of GTA5-to-Cityscape and SYNTHIA-to-Cityscape demonstrate the effectiveness of our method, showcasing the model’s capability to handle complex scenes and imbalanced data.