基于多教师自适应知识蒸馏的TSK模糊分类器

PDF下载HTML

Title:

TSK fuzzy classifier based on multi-teacher adaptive knowledge distillation

作者:

张雄涛^1,2, 陈天宇^1,2, 赵康^1,2, 李水苗^2,3, 申情^1,2

1. 湖州师范学院 信息工程学院, 浙江 湖州 313000;
2. 浙江省现代农业资源智慧管理与应用研究重点实验室, 浙江 湖州 313000;
3. 湖州师范学院 信息技术中心, 浙江 湖州 313000

Author(s):

ZHANG Xiongtao^1,2, CHEN Tianyu^1,2, ZHAO Kang^1,2, LI Shuimiao^2,3, SHEN Qing^1,2

1. School of Information Engineering, Huzhou University, Huzhou 313000, China;
2. Zhejiang Province Key Laboratory of Smart Management and Application of Modern Agricultural Resources, Huzhou 313000, China;
3. Information Technology Center, Huzhou University, Huzhou 313000, China

关键词:

TSK模糊分类器; 知识蒸馏; 多教师网络; 自适应权重分配; 隐藏知识; 模糊系统; 不同视角; 深度学习

Keywords:

TSK fuzzy classifier; knowledge distillation; multiple teacher networks; adaptive allocation of weights; dark knowledge; fuzzy system; different perspectives; deep learning

分类号:

TP181

DOI:

10.11992/tis.202410028

摘要:

目前层次型或深度模糊系统性能优异，但是模型复杂度较高；而基于蒸馏学习的轻量型TSK(Takagi-Sugeno-Kang)模糊分类器主要以单教师知识蒸馏为主，若教师模型表现不佳，则会影响蒸馏效果和模型的整体性能；此外，传统的多教师蒸馏通常使用无标签策略分配教师模型输出的权重，容易使低质量教师误导学生。对此，本文提出了一种基于多教师自适应知识蒸馏的TSK模糊分类器(TSK fuzzy classifier based on multi-teacher adaptive knowledge distillation, TSK-MTAKD)，以多个具有不同神经表达能力的深度神经网络为教师模型，利用本文提出的多教师知识蒸馏框架从多个深度学习模型中提取隐藏知识，并传递给具有强大不确定处理能力的TSK模糊系统。同时设计自适应权重分配器，将教师模型的输出与真实标签做交叉熵处理，更接近真实值的输出将被赋予更高权重，提高了模型的鲁棒性与隐藏知识的有效性。在13个UCI数据集上的实验结果充分验证了TSK-MTAKD的优势。

Abstract:

Currently, hierarchical and deep fuzzy systems demonstrate excellent performance, but they often suffer from high model complexity. Lightweight Takagi-Sugeno-Kang (TSK) fuzzy classifiers based on distillation learning typically rely on single-teacher knowledge distillation. However, if the teacher model underperforms, then the distillation effect and the overall model performance can be compromised. Furthermore, traditional multiteacher distillation approaches often assign weights to teacher model outputs using label-free strategies, which may allow low-quality teachers to mislead the student model. Aiming to address these issues, this paper introduces a TSK fuzzy classifier based on multiteacher adaptive knowledge distillation (TSK-MTAKD). The method employs multiple deep neural networks, each with different neural expression capabilities, as teacher models. The proposed distillation framework extracts dark knowledge from these models and transfers it to a TSK fuzzy system, leveraging its strong capability to handle uncertainty. Additionally, an adaptive weight allocator is introduced, which performs cross-entropy calculations between the output of the teacher model and the true label. Outputs that are closer to the true label are assigned higher weights, thereby improving model robustness and the quality of dark knowledge. Experimental results on 13 UCI benchmark datasets validate the advantages of the TSK-MTAKD approach.