[1]CHEN Tao,XIE Zaipeng,QU Zhihao.Federated semi-supervised learning model based on dynamic threshold enhanced prototype network[J].CAAI Transactions on Intelligent Systems,2024,19(3):534-545.[doi:10.11992/tis.202311015]
Copy
Federated semi-supervised learning model based on dynamic threshold enhanced prototype network
CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume:
19
Number of periods:
2024 3
Page number:
534-545
Column:
学术论文—机器学习
Public date:
2024-05-05
- Title:
- Federated semi-supervised learning model based on dynamic threshold enhanced prototype network
- Keywords:
- federated learning; semi-supervised learning; knowledge sharing; prototypical network; pseudo label; dynamic threshold; unlabeled data; heterogeneous data
- CLC:
- TP181
- DOI:
- 10.11992/tis.202311015
- Abstract:
- Currently, federated semi-supervised learning (FSSL) faces the challenge of making effective use of a large amount of unlabeled data during training. Although knowledge sharing between clients through a lightweight prototyping network can alleviate pseudo-label quality issues, there are still bottlenecks. In this paper, we propose a federated semi-supervised learning model based on dynamic threshold enhanced prototype network. By introducing Curriculum Pseudo labeling, the core is to dynamically adjust the threshold of the learning state of different classes of samples, so that the model can learn high-quality samples and significantly improve the prediction performance of the model. Experimental results show that our proposal has achieved excellent test performance on multiple datasets. On the CIFAR-10 dataset, our proposal improves the test accuracy by at least 3% compared with similar algorithms. In addition, there is a 1%~7% lead on SVHN and STL-10 datasets. It is worth noting that our proposal performs well in handling heterogeneous and homogeneous data, and has good adaptability to different proportions of labeled and unlabeled data. Our proposal can improve the test accuracy. What’s more, it does not add additional communication overhead and computational cost. These results suggest that our proposal has great potential in the field of federated semi-supervised learning, and provides a high-performance and high-efficiency solution for practical applications.
- References:
-
[1] SOHN K, BERTHELOT D, LI Chunliang, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[EB/OL]. (2020–01–21)[2023–11–13]. http://arxiv.org/abs/2001.07685.
[2] MALAVIYA S, SHUKLA M, KORAT P, et al. FedFAME: a data augmentation free framework based on model contrastive learning for federated semi-supervised learning[C]//Proceedings of the 38th ACM/SIGAPP Symposium on Applied Computing. Tallinn: ACM, 2023: 1114–1121.
[3] FENG Siwei, LI Boyang, YU Han, et al. Semi-supervised federated heterogeneous transfer learning[J]. Knowledge-based systems, 2022, 252: 109384.
[4] PEI Xinjun, DENG Xiaoheng, TIAN Shengwei, et al. A knowledge transfer-based semi-supervised federated learning for IoT malware detection[J]. IEEE transactions on dependable and secure computing, 2023, 20(3): 2127–2143.
[5] WEN Tingjie, ZHAO Shengjie, ZHANG Rongqing. Federated semi-supervised learning through a combination of self and cross model ensembling[C]//2022 International Joint Conference on Neural Networks. Padua: IEEE, 2022: 1–8.
[6] ITAHARA S, NISHIO T, KODA Y, et al. Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-IID private data[J]. IEEE transactions on mobile computing, 2023, 22(1): 191–205.
[7] KIM W, PARK K, SOHN K, et al. Federated semi-supervised learning with prototypical networks[EB/OL]. (2022–05–27)[2023–11–13]. http://arxiv.org/abs/2205.13921.
[8] GAO Liang, FU Huazhu, LI Li, et al. FedDC: federated learning with non-IID data via local drift decoupling and correction[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 10102–10111.
[9] 窦勇敢, 袁晓彤. 基于隐式随机梯度下降优化的联邦学习[J]. 智能系统学报, 2022, 17(3): 488–495
DOU Yonggan, YUAN Xiaotong. Federated learning with implicit stochastic gradient descent optimization[J]. CAAI transactions on intelligent systems, 2022, 17(3): 488–495
[10] 谭作文, 张连福. 机器学习隐私保护研究综述[J]. 软件学报, 2020, 31(7): 2127–2156
TAN Zuowen, ZHANG Lianfu. Survey on privacy preserving techniques for machine learning[J]. Journal of software, 2020, 31(7): 2127–2156
[11] JEONG W, YOON J, YANG E, et al. Federated semi-supervised learning with inter-client consistency[C]//International Conference on Learning Representations. Virtual Only: ICLR, 2021: 901–914.
[12] WU Yaqiang, LI Yifei, ZHAO Tianzhe, et al. Improved prototypical network for active few-shot learning[J]. Pattern recognition letters, 2023, 172(C): 188–194.
[13] WANG Ruiqi, ZHANG Xuyao, LIU Chenglin. Meta-prototypical learning for domain-agnostic few-shot recognition[J]. IEEE transactions on neural networks and learning systems, 2022, 33(11): 6990–6996.
[14] SEHANOBISH A, KANNAN K, ABRAHAM N, et al. Meta-learning pathologies from radiology reports using variance aware prototypical networks[C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. Abu Dhabi: Association for Computational Linguistics, 2022: 332–347.
[15] XU Xin, DU Junping, XUE Zhe. Multi-level self-adaptive prototypical networks for few-shot node classification on attributed networks[J]. Neural computing and applications, 2023, 35(12): 9131–9144.
[16] PHAPHUANGWITTAYAKUL A, YING Fangli, GUO Yi, et al. Adaptive adversarial prototyping network for few-shot prototypical translation[J]. Journal of visual communication and image representation, 2023, 94: 103845.
[17] TAN Yue, LONG Guodong, LIU Lu, et al. FedProto: federated prototype learning across heterogeneous clients[J]. Proceedings of the AAAI conference on artificial intelligence, 2022, 36(8): 8432–8440.
[18] BERTHELOT D, CARLINI N, GOODFELLOW I, et al. MixMatch: a holistic approach to semi-supervised learning[EB/OL]. (2019–05–06)[2023–11–13]. http://arxiv.org/abs/1905.02249.
[19] LI Xiaorun, CAO Zeyu, ZHAO Liaoying, et al. ALPN: active-learning-based prototypical network for few-shot hyperspectral imagery classification[J]. IEEE geoscience and remote sensing letters, 2022, 19: 5508305.
[20] CASCANTE-BONILLA P, TAN Fuwen, QI Yanjun, et al. Curriculum labeling: revisiting pseudo-labeling for semi-supervised learning[J]. Proceedings of the AAAI conference on artificial intelligence, 2021, 35(8): 6912–6920.
[21] BERTHELOT D, CARLINI N, CUBUK E D, et al. ReMixMatch: semi-supervised learning with distribution alignment and augmentation anchoring[EB/OL]. (2019–11–21) [2023–11–13]. http://arxiv.org/abs/1911.09785.
[22] HAN Yue, LIU Yuhong, JIN Zhigang. Sentiment analysis via semi-supervised learning: a model based on dynamic threshold and multi-classifiers[J]. Neural computing and applications, 2020, 32(9): 5117–5129.
[23] KARIM N, MITHUN N C, RAJVANSHI A, et al. C-SFDA: a curriculum learning aided self-training framework for efficient source free domain adaptation[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver: IEEE, 2023: 24120–24131.
[24] ZHANG Bowen, WANG Yidong, HOU Wenxin, et al. FlexMatch: boosting semi-supervised learning with curriculum pseudo labeling[EB/OL]. (2021–10–15)[2023–11–13]. http://arxiv.org/abs/2110.08263.
[25] ARAZO E, ORTEGO D, ALBERT P, et al. Pseudo-labeling and confirmation bias in deep semi-supervised learning[C]//2020 International Joint Conference on Neural Networks. Glasgow: IEEE, 2020: 1–8.
[26] 王耀力, 刘晓慧, 李斌, 等. 流形嵌入的选择性伪标记与小样本数据迁移[J]. 西北工业大学学报, 2021, 39(5): 1122–1129
WANG Yaoli, LIU Xiaohui, LI Bin, et al. The manifold embedded selective pseudo-labeling algorithm and transfer learning of small sample dataset[J]. Journal of northwestern polytechnical university, 2021, 39(5): 1122–1129
[27] 张英俊, 李牛牛, 谢斌红, 等. 课程学习指导下的半监督目标检测框架[J/OL]. 计算机应用, (2023–12–20)[2024–04–26]. http://www.joca.cn/CN/10.11772/j.issn.1001-9081.2023081062.
ZHANG Yingjun, LI Niuniu, XIE Binhong, et al. Semi-supervised object detection framework guided by curriculum learning [J/OL]. Journal of computer applications, (2023–12–20)[2024–04–26]. http://www.joca.cn/CN/10.11772/j.issn.1001-9081.2023081062.
[28] XIE Qizhe, DAI Zihang, HOVY E, et al. Unsupervised data augmentation for consistency training[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver: ACM, 2020: 6256–6268.
[29] FRANTAR E, ALISTARH D. SPDY: accurate pruning with speedup guarantees[EB/OL]. (2022–01–31) [2023–11–13]. http://arxiv.org/abs/2201.13096.
[30] LIU Ren, BIAN Fengmiao, ZHANG Xiaoqun. Binary quantized network training with sharpness-aware minimization[J]. Journal of scientific computing, 2022, 94(1): 16.
[31] MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[EB/OL]. (2016–02–17)[2023–11–13]. http://arxiv.org/abs/1602.05629.
[32] LI Tian, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[EB/OL]. (2018–12–14)[2023–11–13]. http://arxiv.org/abs/1812.06127.
[33] LIANG Xiaoxiao, LIN Yiqun, FU Huazhu, et al. RSCFed: random sampling consensus federated semi-supervised learning[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022: 10144–10153.
[34] ZHANG Zhengming, YANG Yaoqing, YAO Zhewei, et al. Improving semi-supervised federated learning by reducing the gradient diversity of models[C]//2021 IEEE International Conference on Big Data. Orlando: IEEE, 2021: 1214–1225.
[35] COATES A, NG A, LEE H. An analysis of single-layer networks in unsupervised feature learning[C]//Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings. Fort Lauderdale: PMLR, 2011: 215–223.
[36] 杨寒雨, 赵晓永, 王磊. 数据归一化方法综述[J]. 计算机工程与应用, 2023, 59(3): 13–22
YANG Hanyu, ZHAO Xiaoyong, WANG Lei. Review of data normalization methods[J]. Computer engineering and applications, 2023, 59(3): 13–22
- Similar References:
Memo
-
Last Update:
1900-01-01