[1]张欣培,周尧,章毅.用于胎儿超声切面识别的知识蒸馏方法[J].智能系统学报,2022,17(1):181-191.[doi:10.11992/tis.202105007]
ZHANG Xinpei,ZHOU Yao,ZHANG Yi.Knowledge distillation method for fetal ultrasound section identification[J].CAAI Transactions on Intelligent Systems,2022,17(1):181-191.[doi:10.11992/tis.202105007]
点击复制
ZHANG Xinpei,ZHOU Yao,ZHANG Yi.Knowledge distillation method for fetal ultrasound section identification[J].CAAI Transactions on Intelligent Systems,2022,17(1):181-191.[doi:10.11992/tis.202105007]
用于胎儿超声切面识别的知识蒸馏方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
17
期数:
2022年第1期
页码:
181-191
栏目:
人工智能院长论坛
出版日期:
2022-01-05
- Title:
- Knowledge distillation method for fetal ultrasound section identification
- Keywords:
- deep learning; convolutional neural network; residual network; prenatal diagnosis; fetal ultrasound; computer aided diagnosis; knowledge distillation; model compression
- 分类号:
- TP30
- 摘要:
- 胎儿超声切面识别是产前超声检查的主要任务之一，直接影响了产前超声检查的质量。近年来，深度神经网络方法在临床超声辅助诊断方面取得了许多进展。然而，已有研究大多应用预训练模型微调进行迁移学习，这不仅容易导致参数冗余和过拟合问题，而且限制了在实际应用中的实时分析能力。本文提出用于胎儿超声切面识别的知识蒸馏方法。第1阶段，在学生教师网络模型中采用残差网络，对二者隐藏层特征融入注意力机制，提取隐藏层关键信息，进行一次知识迁移，使学生网络获得先验权重；第2阶段，使用教师网络模型指导学生网络模型进行知识蒸馏训练，进一步从整体上提升知识迁移的性能。实验结果表明：学生网络在提升各项性能的同时，降低了模型复杂度，有利于超声设备终端的部署和实时分析能力的提升。
- Abstract:
- Fetal ultrasound section recognition is one of the main tasks of prenatal ultrasonography, which directly affects the quality of prenatal ultrasonography. In recent years, by the Deep Neural Network method, we have made great advances in clinical ultrasound-assisted diagnosis. However, most of the existing studies have applied fine-tuned pre-trained model for migration learning, which not only easily leads to parameter redundancy and overfitting problems, but also limits the real-time analysis capability in practical applications. Therefore, this paper proposes a knowledge distillation method for fetal ultrasound section recognition. In the first stage, a residual network is used in the student and teacher network model to incorporate attention mechanisms for both hidden layer features, extract key information in the hidden layer, and perform one knowledge migration so that the student network can obtain a priori weight. In the second stage, the teacher network model is used to guide the student network model to perform knowledge distillation training, so as to further improve the performance of knowledge migration in an overall manner. The experimental results show that the student network reduces the model complexity while improving various performances, which is beneficial to the deployment of ultrasound device terminals and real-time analysis capability.
- 参考文献/References:
-
[1] MARACI M A, NAPOLITANO R, PAPAGEORGHIOU A, et al. Searching for structures of interest in an ultrasound video sequence[C]//Proceedings of the 5th International Workshop on Machine Learning in Medical Imaging. Boston, USA, 2014: 133–140.
[2] PLATT J. Sequential minimal optimization: a fast algorithm for training support vector machines[J]. Advances in kernel methods-support vector learning, 1998, 7:1–22.
[3] BAUMGARTNER C F, KAMNITSAS K, MATTHEW J, et al. SonoNet: real-time detection and localisation of fetal standard scan planes in freehand ultrasound[J]. IEEE transactions on medical imaging, 2017, 36(11): 2204–2215.
[4] MARACI M A, BRIDGE C P, NAPOLITANO R, et al. A framework for analysis of linear ultrasound videos to detect fetal presentation and heartbeat[J]. Medical image analysis, 2017, 37: 22–36.
[5] LAFFERTY J D, MCCALLUM A, PEREIRA F C N. Conditional random fields: probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning. San Francisco, USA, 2001: 282–289.
[6] RYOU H, YAQUB M, CAVALLARO A, et al. Automated 3D ultrasound biometry planes extraction for first trimester fetal assessment[C]//Proceedings of the 7th International Workshop on Machine Learning in Medical Imaging. Athens, Greece, 2016: 196–204.
[7] CHENG P M, MALHI M S. Transfer learning with convolutional neural networks for classification of abdominal ultrasound images[J]. Journal of digital imaging, 2017, 30(2): 234–243.
[8] DONAHUE J. CaffeNet(GitHubPage)[EB/OL]. (2016-05-6)[2021-05-11].https://github.com/BVLC/caffe/tree/master/models/bvlc_reference_caffenet
[9] SIMONYAN K. VGG team ILSVRC-2014 model with 16 weight layers (GitHub Page)[EB/OL]. (2016-05-16)[2021-05-11]. https://gist.github.com/ksimonyan/211839e770f7b538e2d8
[10] HOWARD A G, ZHU Menglong, CHEN Bo, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL]. (2017-04-17)[2021-05-11]. https://arxiv.org/abs/1704.04861v1.
[11] SANDLER M, HOWARD A, ZHU Menglong, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 4510–4520.
[12] ANDREW A, SANDLER M, GRACE C, et al. Searching for mobileNetV3[EB/OL]. (2019-05-06)[2021-05-11].https://arxiv.org/abs/1905.02244v3.
[13] BUCILUǎ C, CARUANA R, NICULESCU-MIZIL A. Model compression[C]//Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Philadelphia, USA, 2006: 535–541.
[14] ROMERO A, BALLAS N, KAHOU S E, et al. FitNets: hints for thin deep nets[J]. (2014-12-19)[2021-05-11]. https://arxiv.org/abs/1412.6550v2.
[15] HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[EB/OL]. (2015-03-09)[2021-05-11].https://arxiv.org/abs/1503.02531.
[16] JIN Xiao, PENG Baoyun, WU Yichao, et al. Knowledge distillation via route constrained optimization[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul, Korea , 2019: 1345–1354.
[17] PHUONG M H, LAMPERT C H. Towards understanding knowledge distillation[C]//Proceedings of the 36th International Conference on Machine Learning. Long Beach, USA, 2019.
[18] JI Guangda, ZHU Zhanxing. Knowledge distillation in wide neural networks: risk bound, data efficiency and imperfect teacher[EB/OL]. (2020-10-20)[2021-05-11].https://arxiv.org/abs/2010.10090.
[19] ZHANG Ying, XIANG Tao, HOSPEDALES T M, et al. Deep mutual learning[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA, 2018: 4320–4328.
[20] MOBAHI H, FARAJTABAR M, BARTLETT P L. Self-distillation amplifies regularization in Hilbert space[EB/OL]. (2020-02-13)[2021-05-11].https://arxiv.org/abs/2002.05715v1.
[21] ZHAI Mengyao, CHEN Lei, TUNG F, et al. Lifelong GAN: continual learning for conditional image generation[C]//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul, South Korea, 2019: 2759–2768.
[22] YUAN Li, TAY F E H , LI Guilin, et al. Revisit knowledge distillation: a teacher-free framework[C]//Proceedings of the International Conference on Learning Representation. Addis Ababa, Ethiopia, 2020.
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]丁科,谭营.GPU通用计算及其在计算智能领域的应用[J].智能系统学报,2015,10(1):1.[doi:10.3969/j.issn.1673-4785.201403072]
DING Ke,TAN Ying.A review on general purpose computing on GPUs and its applications in computational intelligence[J].CAAI Transactions on Intelligent Systems,2015,10():1.[doi:10.3969/j.issn.1673-4785.201403072]
[3]殷瑞,苏松志,李绍滋.一种卷积神经网络的图像矩正则化策略[J].智能系统学报,2016,11(1):43.[doi:10.11992/tis.201509018]
YIN Rui,SU Songzhi,LI Shaozi.Convolutional neural network’s image moment regularizing strategy[J].CAAI Transactions on Intelligent Systems,2016,11():43.[doi:10.11992/tis.201509018]
[4]马晓,张番栋,封举富.基于深度学习特征的稀疏表示的人脸识别方法[J].智能系统学报,2016,11(3):279.[doi:10.11992/tis.201603026]
MA Xiao,ZHANG Fandong,FENG Jufu.Sparse representation via deep learning features based face recognition method[J].CAAI Transactions on Intelligent Systems,2016,11():279.[doi:10.11992/tis.201603026]
[5]龚震霆,陈光喜,任夏荔,等.基于卷积神经网络和哈希编码的图像检索方法[J].智能系统学报,2016,11(3):391.[doi:10.11992/tis.201603028]
GONG Zhenting,CHEN Guangxi,REN Xiali,et al.An image retrieval method based on a convolutional neural network and hash coding[J].CAAI Transactions on Intelligent Systems,2016,11():391.[doi:10.11992/tis.201603028]
[6]师亚亭,李卫军,宁欣,等.基于嘴巴状态约束的人脸特征点定位算法[J].智能系统学报,2016,11(5):578.[doi:10.11992/tis.201602006]
SHI Yating,LI Weijun,NING Xin,et al.A facial feature point locating algorithmbased on mouth-state constraints[J].CAAI Transactions on Intelligent Systems,2016,11():578.[doi:10.11992/tis.201602006]
[7]马世龙,乌尼日其其格,李小平.大数据与深度学习综述[J].智能系统学报,2016,11(6):728.[doi:10.11992/tis.201611021]
MA Shilong,WUNIRI Qiqige,LI Xiaoping.Deep learning with big data: state of the art and development[J].CAAI Transactions on Intelligent Systems,2016,11():728.[doi:10.11992/tis.201611021]
[8]王亚杰,邱虹坤,吴燕燕,等.计算机博弈的研究与发展[J].智能系统学报,2016,11(6):788.[doi:10.11992/tis.201609006]
WANG Yajie,QIU Hongkun,WU Yanyan,et al.Research and development of computer games[J].CAAI Transactions on Intelligent Systems,2016,11():788.[doi:10.11992/tis.201609006]
[9]黄心汉.A3I:21世纪科技之光[J].智能系统学报,2016,11(6):835.[doi:10.11992/tis.201605022]
HUANG Xinhan.A3I: the star of science and technology for the 21st century[J].CAAI Transactions on Intelligent Systems,2016,11():835.[doi:10.11992/tis.201605022]
[10]杨晓兰,强彦,赵涓涓,等.基于医学征象和卷积神经网络的肺结节CT图像哈希检索[J].智能系统学报,2017,12(6):857.[doi:10.11992/tis.201706035]
YANG Xiaolan,QIANG Yan,ZHAO Juanjuan,et al.Hashing retrieval for CT images of pulmonary nodules based on medical signs and convolutional neural networks[J].CAAI Transactions on Intelligent Systems,2017,12():857.[doi:10.11992/tis.201706035]
[11]刘帅师,程曦,郭文燕,等.深度学习方法研究新进展[J].智能系统学报,2016,11(5):567.[doi:10.11992/tis.201511028]
LIU Shuaishi,CHENG Xi,GUO Wenyan,et al.Progress report on new research in deep learning[J].CAAI Transactions on Intelligent Systems,2016,11():567.[doi:10.11992/tis.201511028]
[12]宋婉茹,赵晴晴,陈昌红,等.行人重识别研究综述[J].智能系统学报,2017,12(6):770.[doi:10.11992/tis.201706084]
SONG Wanru,ZHAO Qingqing,CHEN Changhong,et al.Survey on pedestrian re-identification research[J].CAAI Transactions on Intelligent Systems,2017,12():770.[doi:10.11992/tis.201706084]
[13]王科俊,赵彦东,邢向磊.深度学习在无人驾驶汽车领域应用的研究进展[J].智能系统学报,2018,13(1):55.[doi:10.11992/tis.201609029]
WANG Kejun,ZHAO Yandong,XING Xianglei.Deep learning in driverless vehicles[J].CAAI Transactions on Intelligent Systems,2018,13():55.[doi:10.11992/tis.201609029]
[14]莫凌飞,蒋红亮,李煊鹏.基于深度学习的视频预测研究综述[J].智能系统学报,2018,13(1):85.[doi:10.11992/tis.201707032]
MO Lingfei,JIANG Hongliang,LI Xuanpeng.Review of deep learning-based video prediction[J].CAAI Transactions on Intelligent Systems,2018,13():85.[doi:10.11992/tis.201707032]
[15]葛园园,许有疆,赵帅,等.自动驾驶场景下小且密集的交通标志检测[J].智能系统学报,2018,13(3):366.[doi:10.11992/tis.201706040]
GE Yuanyuan,XU Youjiang,ZHAO Shuai,et al.Detection of small and dense traffic signs in self-driving scenarios[J].CAAI Transactions on Intelligent Systems,2018,13():366.[doi:10.11992/tis.201706040]
[16]汪鸿翔,柳培忠,骆炎民,等.高斯核函数卷积神经网络跟踪算法[J].智能系统学报,2018,13(3):388.[doi:10.11992/tis.201612040]
WANG Hongxiang,LIU Peizhong,LUO Yanmin,et al.Convolutional neutral network tracking algorithm accelerated by Gaussian kernel function[J].CAAI Transactions on Intelligent Systems,2018,13():388.[doi:10.11992/tis.201612040]
[17]冯小荣,惠康华,柳振东.基于卷积特征和贝叶斯分类器的人脸识别[J].智能系统学报,2018,13(5):769.[doi:10.11992/tis.201706052]
FENG Xiaorong,HUI Kanghua,LIU Zhendong.Face recognition based on convolution feature and Bayes classifier[J].CAAI Transactions on Intelligent Systems,2018,13():769.[doi:10.11992/tis.201706052]
[18]莫宏伟,汪海波.基于Faster R-CNN的人体行为检测研究[J].智能系统学报,2018,13(6):967.[doi:10.11992/tis.201801025]
MO Hongwei,WANG Haibo.Research on human behavior detection based on Faster R-CNN[J].CAAI Transactions on Intelligent Systems,2018,13():967.[doi:10.11992/tis.201801025]
[19]胡越,罗东阳,花奎,等.关于深度学习的综述与讨论[J].智能系统学报,2019,14(1):1.[doi:10.11992/tis.201808019]
HU Yue,LUO Dongyang,HUA Kui,et al.Overview on deep learning[J].CAAI Transactions on Intelligent Systems,2019,14():1.[doi:10.11992/tis.201808019]
[20]孙必慎,石武祯,姜峰.计算视觉核心问题：自然图像先验建模研究综述[J].智能系统学报,2019,14(1):71.[doi:10.11992/tis.201804019]
SUN Bishen,SHI Wuzhen,JIANG Feng.Core problem in computer vision: survey of natural image prior models[J].CAAI Transactions on Intelligent Systems,2019,14():71.[doi:10.11992/tis.201804019]
备注/Memo
收稿日期:2021-05-11。
基金项目:国家自然科学基金项目（62006163）.
作者简介:张欣培，硕士研究生，主要研究方向为神经网络、医学图像处理;周尧，助理研究员，主要研究方向为神经网络、进化计算。参与科技部科技创新2030–“新一代人工智能”重大项目1项，主持国家自然科学基金项目1项。发表学术论文7篇。;章毅，教授，博士生导师，IEEE Fellow，主要研究方向为人工智能与智能医学。获国家自然科学二等奖、教育部自然科学一等奖、四川省科技进步一等奖。主持科技部科技创新2030–“新一代人工智能”重大项目。发表学术论文500余篇，出版英文学术 专著 3 部。
通讯作者:章毅. E-mail: zhangyi@scu.edu.cn
更新日期/Last Update:
1900-01-01