[1]陈涛,谢在鹏,屈志昊.基于动态阈值增强原型网络的联邦半监督学习模型[J].智能系统学报,2024,19(3):534-545.[doi:10.11992/tis.202311015]
CHEN Tao,XIE Zaipeng,QU Zhihao.Federated semi-supervised learning model based on dynamic threshold enhanced prototype network[J].CAAI Transactions on Intelligent Systems,2024,19(3):534-545.[doi:10.11992/tis.202311015]
点击复制
CHEN Tao,XIE Zaipeng,QU Zhihao.Federated semi-supervised learning model based on dynamic threshold enhanced prototype network[J].CAAI Transactions on Intelligent Systems,2024,19(3):534-545.[doi:10.11992/tis.202311015]
基于动态阈值增强原型网络的联邦半监督学习模型
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
19
期数:
2024年第3期
页码:
534-545
栏目:
学术论文—机器学习
出版日期:
2024-05-05
- Title:
- Federated semi-supervised learning model based on dynamic threshold enhanced prototype network
- Keywords:
- federated learning; semi-supervised learning; knowledge sharing; prototypical network; pseudo label; dynamic threshold; unlabeled data; heterogeneous data
- 分类号:
- TP181
- 文献标志码:
- 2024-04-30
- 摘要:
- 目前，联邦半监督学习面临着有效利用训练过程中大量无标签数据的挑战。尽管通过轻量级的原型网络实现客户端之间的知识共享可以缓解伪标签质量问题，但仍然有瓶颈。本文提出一种新的动态阈值增强下的原型网络联邦半监督学习算法。通过引入课程伪标签技术，其核心是对不同类别样本的学习状态动态调整阈值，使模型能够学习高质量的样本，显著提高模型的预测性能。实验结果表明，本算法在多个数据集上均取得优异的测试性能。在CIFAR-10数据集上，本算法相对于同类算法至少提高3%的测试精度。此外在SVHN和STL-10数据集上也有1%~7%的领先优势。值得注意的是，本算法在处理异质性和同质性数据时表现出色，且对于不同比例的有标签和无标签数据都具有良好的适应性。本算法不仅提高测试精度，而且未带来额外的通信开销和计算成本。这些结果表明本算法在联邦半监督学习领域具有巨大潜力，并为实际应用提供了一个性能卓越且高效的解决方案。
- Abstract:
- Currently, federated semi-supervised learning (FSSL) faces the challenge of making effective use of a large amount of unlabeled data during training. Although knowledge sharing between clients through a lightweight prototyping network can alleviate pseudo-label quality issues, there are still bottlenecks. In this paper, we propose a federated semi-supervised learning model based on dynamic threshold enhanced prototype network. By introducing Curriculum Pseudo labeling, the core is to dynamically adjust the threshold of the learning state of different classes of samples, so that the model can learn high-quality samples and significantly improve the prediction performance of the model. Experimental results show that our proposal has achieved excellent test performance on multiple datasets. On the CIFAR-10 dataset, our proposal improves the test accuracy by at least 3% compared with similar algorithms. In addition, there is a 1%~7% lead on SVHN and STL-10 datasets. It is worth noting that our proposal performs well in handling heterogeneous and homogeneous data, and has good adaptability to different proportions of labeled and unlabeled data. Our proposal can improve the test accuracy. What’s more, it does not add additional communication overhead and computational cost. These results suggest that our proposal has great potential in the field of federated semi-supervised learning, and provides a high-performance and high-efficiency solution for practical applications.
- 参考文献/References:
-
[1] SOHN K, BERTHELOT D, LI Chunliang, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[EB/OL]. (2020–01–21)[2023–11–13]. http://arxiv.org/abs/2001.07685.
[2] MALAVIYA S, SHUKLA M, KORAT P, et al. FedFAME: a data augmentation free framework based on model contrastive learning for federated semi-supervised learning[C]//Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing. Tallinn: ACM, 2023: 1114–1121.
[3] FENG Siwei, LI Boyang, YU Han, et al. Semi-supervised federated heterogeneous transfer learning[J]. Knowledge-based systems, 2022, 252: 109384.
[4] PEI Xinjun, DENG Xiaoheng, TIAN Shengwei, et al. A knowledge transfer-based semi-supervised federated learning for IoT malware detection[J]. IEEE transactions on dependable and secure computing, 2023, 20(3): 2127–2143.
[5] WEN Tingjie, ZHAO Shengjie, ZHANG Rongqing. Federated semi-supervised learning through a combination of self and cross model ensembling[C]//2022 International Joint Conference on Neural Networks. Padua: IEEE, 2022: 1–8.
[6] ITAHARA S, NISHIO T, KODA Y, et al. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-IID private data[J]. IEEE transactions on mobile computing, 2023, 22(1): 191–205.
[7] KIM W, PARK K, SOHN K, et al. Federated semi-supervised learning with prototypical networks[EB/OL]. (2022–05–27)[2023–11–13]. http://arxiv.org/abs/2205.13921.
[8] GAO Liang, FU Huazhu, LI Li, et al. FedDC: federated learning with non-IID data via local drift decoupling and correction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 10102–10111.
[9] 窦勇敢, 袁晓彤. 基于隐式随机梯度下降优化的联邦学习[J]. 智能系统学报, 2022, 17(3): 488–495
DOU Yonggan, YUAN Xiaotong. Federated learning with implicit stochastic gradient descent optimization[J]. CAAI transactions on intelligent systems, 2022, 17(3): 488–495
[10] 谭作文, 张连福. 机器学习隐私保护研究综述[J]. 软件学报, 2020, 31(7): 2127–2156
TAN Zuowen, ZHANG Lianfu. Survey on privacy preserving techniques for machine learning[J]. Journal of software, 2020, 31(7): 2127–2156
[11] JEONG W, YOON J, YANG E, et al. Federated semi-supervised learning with inter-client consistency[C]//International Conference on Learning Representations. Virtual Only: ICLR, 2021: 901–914.
[12] WU Yaqiang, LI Yifei, ZHAO Tianzhe, et al. Improved prototypical network for active few-shot learning[J]. Pattern recognition letters, 2023, 172(C): 188–194.
[13] WANG Ruiqi, ZHANG Xuyao, LIU Chenglin. Meta-prototypical learning for domain-agnostic few-shot recognition[J]. IEEE transactions on neural networks and learning systems, 2022, 33(11): 6990–6996.
[14] SEHANOBISH A, KANNAN K, ABRAHAM N, et al. Meta-learning pathologies from radiology reports using variance aware prototypical networks[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. Abu Dhabi: Association for Computational Linguistics, 2022: 332–347.
[15] XU Xin, DU Junping, XUE Zhe. Multi-level self-adaptive prototypical networks for few-shot node classification on attributed networks[J]. Neural computing and applications, 2023, 35(12): 9131–9144.
[16] PHAPHUANGWITTAYAKUL A, YING Fangli, GUO Yi, et al. Adaptive adversarial prototyping network for few-shot prototypical translation[J]. Journal of visual communication and image representation, 2023, 94: 103845.
[17] TAN Yue, LONG Guodong, LIU Lu, et al. FedProto: federated prototype learning across heterogeneous clients[J]. Proceedings of the AAAI conference on artificial intelligence, 2022, 36(8): 8432–8440.
[18] BERTHELOT D, CARLINI N, GOODFELLOW I, et al. MixMatch: a holistic approach to semi-supervised learning[EB/OL]. (2019–05–06)[2023–11–13]. http://arxiv.org/abs/1905.02249.
[19] LI Xiaorun, CAO Zeyu, ZHAO Liaoying, et al. ALPN: active-learning-based prototypical network for few-shot hyperspectral imagery classification[J]. IEEE geoscience and remote sensing letters, 2022, 19: 5508305.
[20] CASCANTE-BONILLA P, TAN Fuwen, QI Yanjun, et al. Curriculum labeling: revisiting pseudo-labeling for semi-supervised learning[J]. Proceedings of the AAAI conference on artificial intelligence, 2021, 35(8): 6912–6920.
[21] BERTHELOT D, CARLINI N, CUBUK E D, et al. ReMixMatch: semi-supervised learning with distribution alignment and augmentation anchoring[EB/OL]. (2019–11–21) [2023–11–13]. http://arxiv.org/abs/1911.09785.
[22] HAN Yue, LIU Yuhong, JIN Zhigang. Sentiment analysis via semi-supervised learning: a model based on dynamic threshold and multi-classifiers[J]. Neural computing and applications, 2020, 32(9): 5117–5129.
[23] KARIM N, MITHUN N C, RAJVANSHI A, et al. C-SFDA: a curriculum learning aided self-training framework for efficient source free domain adaptation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 24120–24131.
[24] ZHANG Bowen, WANG Yidong, HOU Wenxin, et al. FlexMatch: boosting semi-supervised learning with curriculum pseudo labeling[EB/OL]. (2021–10–15)[2023–11–13]. http://arxiv.org/abs/2110.08263.
[25] ARAZO E, ORTEGO D, ALBERT P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning[C]//2020 International Joint Conference on Neural Networks. Glasgow: IEEE, 2020: 1–8.
[26] 王耀力, 刘晓慧, 李斌, 等. 流形嵌入的选择性伪标记与小样本数据迁移[J]. 西北工业大学学报, 2021, 39(5): 1122–1129
WANG Yaoli, LIU Xiaohui, LI Bin, et al. The manifold embedded selective pseudo-labeling algorithm and transfer learning of small sample dataset[J]. Journal of northwestern polytechnical university, 2021, 39(5): 1122–1129
[27] 张英俊, 李牛牛, 谢斌红, 等. 课程学习指导下的半监督目标检测框架[J/OL]. 计算机应用, (2023–12–20)[2024–04–26]. http://www.joca.cn/CN/10.11772/j.issn.1001-9081.2023081062.
ZHANG Yingjun, LI Niuniu, XIE Binhong, et al. Semi-supervised object detection framework guided by curriculum learning [J/OL]. Journal of computer applications, (2023–12–20)[2024–04–26]. http://www.joca.cn/CN/10.11772/j.issn.1001-9081.2023081062.
[28] XIE Qizhe, DAI Zihang, HOVY E, et al. Unsupervised data augmentation for consistency training[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020: 6256–6268.
[29] FRANTAR E, ALISTARH D. SPDY: accurate pruning with speedup guarantees[EB/OL]. (2022–01–31) [2023–11–13]. http://arxiv.org/abs/2201.13096.
[30] LIU Ren, BIAN Fengmiao, ZHANG Xiaoqun. Binary quantized network training with sharpness-aware minimization[J]. Journal of scientific computing, 2022, 94(1): 16.
[31] MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. (2016–02–17)[2023–11–13]. http://arxiv.org/abs/1602.05629.
[32] LI Tian, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[EB/OL]. (2018–12–14)[2023–11–13]. http://arxiv.org/abs/1812.06127.
[33] LIANG Xiaoxiao, LIN Yiqun, FU Huazhu, et al. RSCFed: random sampling consensus federated semi-supervised learning[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 10144–10153.
[34] ZHANG Zhengming, YANG Yaoqing, YAO Zhewei, et al. Improving semi-supervised federated learning by reducing the gradient diversity of models[C]//2021 IEEE International Conference on Big Data. Orlando: IEEE, 2021: 1214–1225.
[35] COATES A, NG A, LEE H. An analysis of single-layer networks in unsupervised feature learning[C]//Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings. Fort Lauderdale: PMLR, 2011: 215–223.
[36] 杨寒雨, 赵晓永, 王磊. 数据归一化方法综述[J]. 计算机工程与应用, 2023, 59(3): 13–22
YANG Hanyu, ZHAO Xiaoyong, WANG Lei. Review of data normalization methods[J]. Computer engineering and applications, 2023, 59(3): 13–22
- 相似文献/References:
-
[1]李建元,周脚根,关佶红,等.谱图聚类算法研究进展[J].智能系统学报,2011,6(5):405.
LI Jianyuan,ZHOU Jiaogen,GUAN Jihong,et al.A survey of clustering algorithms based on spectra of graphs[J].CAAI Transactions on Intelligent Systems,2011,6():405.
[2]刘杨磊,梁吉业,高嘉伟,等.基于Tri-training的半监督多标记学习算法[J].智能系统学报,2013,8(5):439.[doi:10.3969/j.issn.1673-4785.201305033]
LIU Yanglei,LIANG Jiye,GAO Jiawei,et al.Semi-supervised multi-label learning algorithm based on Tri-training[J].CAAI Transactions on Intelligent Systems,2013,8():439.[doi:10.3969/j.issn.1673-4785.201305033]
[3]蒋新华,高晟,廖律超,等.半监督SVM分类算法的交通视频车辆检测方法[J].智能系统学报,2015,10(5):690.[doi:10.11992/tis.201406044]
JIANG Xinhua,GAO Sheng,LIAO Ljuchao,et al.Traffic video vehicle detection based on semi-supervised SVM classification algorithm[J].CAAI Transactions on Intelligent Systems,2015,10():690.[doi:10.11992/tis.201406044]
[4]张钢,谢晓珊,黄英,等.面向大数据流的半监督在线多核学习算法[J].智能系统学报,2014,9(3):355.[doi:10.3969/j.issn.1673-4785.201403067]
ZHANG Gang,XIE Xiaoshan,HUANG Ying,et al.An online multi-kernel learning algorithm for big data[J].CAAI Transactions on Intelligent Systems,2014,9():355.[doi:10.3969/j.issn.1673-4785.201403067]
[5]赵晓晓,周治平.结合稀疏表示与约束传递的半监督谱聚类算法[J].智能系统学报,2018,13(5):855.[doi:10.11992/tis.201703013]
ZHAO Xiaoxiao,ZHOU Zhiping.A semi-supervised spectral clustering algorithm combined with sparse representation and constraint propagation[J].CAAI Transactions on Intelligent Systems,2018,13():855.[doi:10.11992/tis.201703013]
[6]严菲,王晓栋.鲁棒的半监督多标签特征选择方法[J].智能系统学报,2019,14(4):812.[doi:10.11992/tis.201809017]
YAN Fei,WANG Xiaodong.A robust, semi-supervised, and multi-label feature selection method[J].CAAI Transactions on Intelligent Systems,2019,14():812.[doi:10.11992/tis.201809017]
[7]程龙,钱文彬,王映龙,等.弱标记不完备决策系统的增量式属性约简算法[J].智能系统学报,2020,15(6):1079.[doi:10.11992/tis.202001017]
CHENG Long,QIAN Wenbin,WANG Yinglong,et al.An incremental attribute reduction algorithm for incomplete decision system with weak labeling[J].CAAI Transactions on Intelligent Systems,2020,15():1079.[doi:10.11992/tis.202001017]
[8]王健宗,肖京,朱星华,等.联邦推荐系统的协同过滤冷启动解决方法[J].智能系统学报,2021,16(1):178.[doi:10.11992/tis.202009032]
WANG Jianzong,XIAO Jing,ZHU Xinghua,et al.Cold starts in collaborative filtering for federated recommender systems[J].CAAI Transactions on Intelligent Systems,2021,16():178.[doi:10.11992/tis.202009032]
[9]窦勇敢,袁晓彤.基于隐式随机梯度下降优化的联邦学习[J].智能系统学报,2022,17(3):488.[doi:10.11992/tis.202106029]
DOU Yonggan,YUAN Xiaotong.Federated learning with implicit stochastic gradient descent optimization[J].CAAI Transactions on Intelligent Systems,2022,17():488.[doi:10.11992/tis.202106029]
[10]刘威,王薪予,刘光伟,等.融合关系特征的半监督图像分类方法研究[J].智能系统学报,2022,17(5):886.[doi:10.11992/tis.202109022]
LIU Wei,WANG Xinyu,LIU Guangwei,et al.Semi-supervised image classification method fused with relational features[J].CAAI Transactions on Intelligent Systems,2022,17():886.[doi:10.11992/tis.202109022]
备注/Memo
收稿日期:2023-11-13。
基金项目:水灾害防御全国重点实验室“一带一路”水资源与可持续发展科技基金项目（2021490811）；国家自然科学基金青年项目（62102131）；江苏省自然科学基金青年项目（BK20210361）.
作者简介:陈涛，硕士研究生，主要研究方向为分布式机器学习、联邦学习。E-mail：1033296297@qq.com;谢在鹏，副教授，博士，主要研究方向为分布式机器学习，可持续计算理论及应用。获发明专利授权15项，发表学术论文30余篇。E-mail：zaipengxie@hhu.edu.cn;屈志昊，副教授，博士，主要研究方向为边缘计算、边缘智能、联邦学习。主持国家自然科学基金青年基金、江苏省青年基金等项目5项。发表学术论文20余篇。E-mail：quzhihao@hhu.edu.cn
通讯作者:谢在鹏. E-mail：zaipengxie@hhu.edu.cn
更新日期/Last Update:
1900-01-01