[1]WANG Chunyi,WANG Xue.Multimodal collaborative learning method for defect recognition in power substation equipment[J].CAAI Transactions on Intelligent Systems,2026,21(4):1004-1012.[doi:10.11992/tis.202509009]
Copy

Multimodal collaborative learning method for defect recognition in power substation equipment

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 21 Number of periods: 2026 4 Page number: 1004-1012 Column: 学术论文—智能系统 Public date: 2026-07-05
Title:
Multimodal collaborative learning method for defect recognition in power substation equipment
Author(s):
WANG Chunyi1;  WANG Xue2
1. STATE GRID Shanghai Pudong Electric Power Supply Company, Shanghai 200122, China;
2. School of Computer Science, Northwestern Polytechnical University, Xi’an 710072, China
Keywords:
infrared image; ultraviolet image; large language model; multimodal modeling; collaborative learning; attention mechanism; power substation equipment; defect recognition
CLC:
TP391.4
DOI:
10.11992/tis.202509009
Abstract:
To address the poor robustness of fused feature learning for multispectral images in substation equipment defect recognition, this study proposes a multimodal feature learning method that integrates semantic enhancement and modal feature decoupling. This method first introduces DeepSeek, a large-scale model with semantic understanding capabilities, to analyze and supplement the semantic annotations of infrared and ultraviolet images, thereby enriching the semantic information in the annotations. Modality-shared and modality-specific feature extraction modules are then designed. These modules employ shared and modality-specific query vectors to model common and distinct features across the two modalities, enhancing multimodal feature representation. Furthermore, feature alignment and decoupling losses are introduced to strengthen trans-modal shared feature consistency and suppress interference from redundant information. Extensive experimental results demonstrate that the proposed method outperforms existing mainstream approaches in electrical equipment defect recognition. Ablation studies and visualization results further validate its effectiveness in multimodal collaborative learning. This conclusion provides a reference for research and application of multimodal fusion representation learning methods.
References:
[1] 王成山, 李鹏. 分布式发电、微网与智能配电网的发展与挑战[J]. 电力系统自动化, 2010, 34(2): 10-14, 23 WANG Chengshan, LI Peng. Development and challenges of distributed generation the micro-grid and smart distribution system[J]. Automation of electric power systems., 2010, 34(2): 10-14, 23
[2] 张智刚, 康重庆. 碳中和目标下构建新型电力系统的挑战与展望[J]. 中国电机工程学报, 2022, 42(8): 2806-2819 ZHANG Zhigang, KANG Chongqing. Challenges and prospects for constructing the new-type power system towards a carbon neutrality future[J]. Proceedings of the CSEE, 2022, 42(8): 2806-2819
[3] 程其云, 李峰, 罗澍忻, 等. 新型电力系统规划方法框架及关键支撑技术[J]. 电网技术, 2025, 49(6): 2219-2231 CHENG Qiyun, LI Feng, LUO Shuxin, et al. Research on the planning methodology framework and key supporting technologies for new power systems[J]. Power system technology, 2025, 49(6): 2219-2231
[4] 秦潘昊, 陈威宇, 胡秦然, 等. 新型电力系统设备状态监测与故障诊断传感芯片关键技术与展望[J]. 电力系统自动化, 2024, 48(6): 83-95 QIN Panhao, CHEN Wejyu, HU Qjnran, et al. Key technologies and prospect of equipment condition monitoring and diagnosis sensor chips for new power systems[J]. Automation of electric power systems, 2024, 48(6): 83-95
[5] 范澜珊, 刘云鹏, 刘一瑾, 等. 面向电力设备红外–可见光图像配准的自适应监督重训算法[J]. 高电压技术, 2025, 51(4): 1785-1800 FAN Lanshan, LIU Yunpeng, LIU Yijin. Adaptive algorithm with supervision and retraining for power equipment infrared and visible images registration[J]. High voltage engineering, 2025, 51(4): 1785-1800
[6] 赵振兵, 冯烁, 席悦, 等. 大模型时代: 电力视觉技术新起点[J]. 高电压技术, 2024, 50(5): 1813-1825 ZHAO Zhenbing, FENG Shuo, XI Yue, et al. The era of large models: a new starting point for electric power vision technology[J]. High voltage engineering, 2024, 50(5): 1813-1825
[7] 刘传洋, 吴一全. 基于红外图像的电力设备识别及发热故障诊断方法研究进展[J]. 中国电机工程学报, 2025, 45(6): 2171-2196 LIU Chuanyang, WU Yiquan. Research progress of power equipment identification and thermal fault diagnosis based on infrared images[J]. Proceedings of the CSEE, 2025, 45(6): 2171-2196
[8] 刘云鹏, 李泳霖, 裴少通, 等. 基于紫外光辐射照度特征的污秽瓷绝缘子绝缘状态评估方法[J]. 高电压技术, 2023, 49(4): 1622-1631 LIU Yunpeng, LI Yonglin, PEI Shaotong, et al. Evaluation of the insulation state of contaminated porcelain insulators based on characteristics of ultraviolet irradiance[J]. High voltage engineering, 2023, 49(4): 1622-1631
[9] 宋鹏, 白晓晨, 林晓焕, 等. 电力传输线异常放电紫外光检测定位仿真与模拟验证[J]. 光学精密工程, 2022, 30(10): 1160-1169 SONG Peng, BAI Xiaochen, LIN Xiaohuan, et al. Simulation and verification of ultraviolet detection and location of abnormal discharge in power transmission line[J]. Optics and precision engineering, 2022, 30(10): 1160-1169
[10] 律方成, 牛雷雷, 王胜辉, 等. 基于紫外成像和改进YOLOv3的瓷悬式绝缘子放电严重程度评估[J]. 高电压术, 2021, 47(2): 377-386 L? Fangcheng, NIU Leilei, WANG Shenghui, et al. Discharge severity assessment of porcelain suspension insulators based on UV images and improved YOLOv3[J]. High voltage engineering, 2021, 47(2): 377-386
[11] 黄志鸿, 杜瑞, 张辉. 面向复杂电力环境场景理解的可见光和红外图像特征级融合方法[J]. 智能系统学报, 2025, 20(3): 631-640 HUANG Zhihong, DU Rui, ZHANG Hui. Feature-level fusion method of visible and infrared images for scene understanding in complex power environments[J]. CAAI transactions on intelligent systems, 2025, 20(3): 631-640
[12] 陶岩, 张辉, 黄志鸿, 等. 面向配电网典型部件的热故障精准判别方法[J]. 智能系统学报, 2025, 20(2): 506-515 TAO Yan, ZHANG Hui, HUANG Zhihong, et al. Accurate identification of thermal faults for typical components of distribution networks[J]. CAAI transactions on intelligent systems, 2025, 20(2): 506-515
[13] 张建良, 张晓洁, 麻坚, 等. 面向新型电力系统的故障诊断技术研究进展(二): 机器学习和大语言模型技术[J]. 实验技术与管理, 2025, 42(12): 54-70 ZHANG Jianlian, ZHANG Xiaojie, MA Jian, et al. A review of fault diagnosis technology for the new type power systems(part II): machine learning and large language modeling techniques[J]. Experimental technology and management, 2025, 42(12): 54-70
[14] 赵俊华, 文福拴, 黄建伟, 等. 基于大语言模型的电力系统通用人工智能展望: 理论与应用[J]. 电力系统自动化, 2024, 48(6): 13-28 ZHAO Junhua, WEN Fushuan, HUANG Jianwei, et al. Prospect of artificial general intelligence for power systems based on large language model: theory and applications[J]. Automation of electric power systems, 2024, 48(6): 13-28
[15] 丁剑涛. 面向电力设备缺陷数据的多模态知识图谱研究[D]. 西安: 长安大学, 2022. DING Jiantao. Research on multimodal knowledge graph for power equipment defect data[D]. Xi’an: Chang’an University, 2022.
[16] 赵欣洋, 刘志远, 赵宇嘉, 等. 基于异源特征融合的电力设备分割与异常发热检测[J]. 国外电子测量技术, 2024, 43(11): 177-186 ZHAO Xinyang, LIU Zhiyuan, ZHAO Yujia, et al. Segmentation and anomalous heat detection of power equipment based on heterogeneous feature fusion[J]. Foreign electronic measurement technology, 2024, 43(11): 177-186
[17] 单其尚. 从情感分析到变压器故障诊断的多模态融合方法迁移应用[D]. 南京: 南京邮电大学, 2023. SHAN Qishang. Multimodal fusion method migration application from sentiment analysis to transformer fault diagnosis[D]. Nanjing: Nanjing University of Posts and Telecommunications, 2023.
[18] 李冰, 杜喜英, 王玉莹, 等. 基于改进YOLOv8n的变电设备红外图像实例分割算法[J]. 电子测量技术, 2024, 47(10): 151-159 LI Bing, DU Xiying, WANG Yuying, et al. Instance segmentation algorithm for infrared images of substation equipment based on improved YOLOv8n[J]. Electronic measurement technology, 2024, 47(10): 151-159
[19] 陈梦实. 基于语言模型的电网输变电设备缺陷语料库系统的设计与实现[D]. 北京: 北京邮电大学, 2024. CHEN Mengshi. Design and implementation of defect corpus system for power grid equipment[D]. Beijing : Beijing University of Posts and Telecommunications, 2024.
[20] TOUVRON H, LAVRIL T, IZACARD G, et al. LLaMa: open and efficient foundation language models[EB/OL]. (2023-02-27)[2025-05-20]. https://arxiv.org/abs/2302.13971.
[21] 杨文昊. 基于多模态信息融合的故障源定位方法研究[D]. 绵阳: 西南科技大学, 2024. YANG Wenhao. Research on fault source localization method based on multimodal information fusion[D]. Mianyang: Southwest University of Science and Technology, 2024.
[22] 姚一杨, 杜泽星, 周果清, 等. 基于提示学习的电力设备故障多谱段融合识别方法[J]. 西北工业大学学报, 2025, 43(2): 410-417 YAO Yiyang, DU Zexing, ZHOU Guoqing, et al. Multi-spectral fusion power equipment fault recognition based on prompt learning[J]. Journal of Northwestern Polytechnical University, 2025, 43(2): 410-417
[23] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the 38th International Conference on Machine Learning. Online: IMLS, 2021.
[24] LIU Haotian, LI Chunyuan, LI Yuheng, et al. Improved baselines with visual instruction tuning[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024.
[25] HAN Zeyu, GAO Chao, LIU Jinyang, et al. Parameter-efficient fine-tuning for large models: a comprehensive survey[EB/OL]. (2024-03-21)[2025-05-20]. https://arxiv.org/abs/2403.14608.
[26] ZHU Lianghui, LIAO Bencheng, ZHANG Qian, et al. Vision mamba: efficient visual representation learning with bidirectional state space model[C]//Proceedings of the 41st International Conference on Machine Learning. Vienna: IMLS, 2024.
[27] REN Tianhe, CHEN Yihao, JIANG Qing, et al. DINO-X: a unified vision model for open-world object detection and understanding[EB/OL]. (2024-11-21)[2025-05-22]. https://arxiv.org/abs/2411.14347.
[28] CHEN Qifeng, CHEN Yingcong, FANG Cheng, et al. HAWK: learning to understand open-world video anomalies[C]//Advances in Neural Information Processing Systems 37. Vancouver: NeurIPS, 2024.
[29] TIAN Xinyu, ZOU Shu, YANG Zhaoyuan, et al. ArGue: attribute-guided prompt tuning for vision-language models[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle: IEEE, 2024.
[30] LI Zheng, SONG Yibing, CHENG Mingming, et al. Advancing textual prompt learning with anchored attributes[C]//2025 IEEE/CVF International Conference on Computer Vision. Honolulu: IEEE, 2025.
[31] BAHNG H, JAHANIAN A, SANKARANARAYANAN S, et al. Exploring visual prompts for adapting large-scale models[EB/OL]. (2022-03-31)[2025-04-20]. https://arxiv.org/abs/2203.17274.
[32] ACHIAM J, ADLER S, AGARWAL S, et al. GPT-4 technical report[EB/OL]. (2023-03-15)[2025-04-20]. https://arxiv.org/abs/2303.08774.
[33] LIU Aixin, FENG Bei, XUE Bing, et al. Deepseek-V3 technical report[EB/OL]. (2024-12-27)[2025-04-20]. https://arxiv.org/abs/2412.19437.
[34] CHEN Chen, QI Jiahao, LIU Xingyue, et al. Weakly misalignment-free adaptive feature alignment for UAVs-based multimodal object detection[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognitio. Seattle: IEEE, 2024.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems