[1]刘侠,吕志伟,王波,等.联合超声甲状腺结节分割与分类的多任务方法研究[J].智能系统学报,2023,18(4):764-774.[doi:10.11992/tis.202203063]
LIU Xia,LYU Zhiwei,WANG Bo,et al.Multi-task method for segmentation and classification of thyroid nodules combined with ultrasound images[J].CAAI Transactions on Intelligent Systems,2023,18(4):764-774.[doi:10.11992/tis.202203063]
点击复制
LIU Xia,LYU Zhiwei,WANG Bo,et al.Multi-task method for segmentation and classification of thyroid nodules combined with ultrasound images[J].CAAI Transactions on Intelligent Systems,2023,18(4):764-774.[doi:10.11992/tis.202203063]
联合超声甲状腺结节分割与分类的多任务方法研究
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
18
期数:
2023年第4期
页码:
764-774
栏目:
学术论文—机器感知与模式识别
出版日期:
2023-07-15
- Title:
- Multi-task method for segmentation and classification of thyroid nodules combined with ultrasound images
- Keywords:
- deep learning; multi-task learning; ultrasound image of thyroid nodule; image segmentation; image classification; deep layer convolutional block; multiscale convolutional block attention module; residual structure
- 分类号:
- TP391
- 摘要:
- 针对超声图像中甲状腺结节多尺度、结节边缘模糊、良恶分类不平衡问题，提出一种联合超声甲状腺结节分割与分类的多任务方法。以全卷积网络作为主干共享网络，将提取到的浅层特征共享给多任务分支网络，在分割网络分支中，先加入深层卷积块，获取分割分支深层特征，再对深层特征进行上采样。本文提出一种改进卷积注意力模块的多尺度卷积注意力模块，将上采样结果与主干共享网络每个特征提取阶段经过带有多尺度卷积注意力模块跳跃连接后的特征张量进行拼接，减少结节边缘模糊问题，提高分割性能。同时将多尺度卷积注意力模块融入到分类分支中，优化分类性能。实验结果表明：本文所提多任务方法能有效提升分割和分类的精度，较单任务深度学习网络具有更优的分割与分类性能，能有效处理甲状腺结节多尺度、结节边缘模糊的问题，降低良恶分类不平衡带来的影响。
- Abstract:
- Aiming at the problems of multi-scale thyroid nodules, blurred nodule edges, and unbalanced classification of benign and malignant thyroid nodules in ultrasound images, this paper proposes a multi-task method for segmentation and classification of thyroid nodules combined with ultrasound. The fully convolutional network is used as the backbone sharing network, and the extracted shallow features are shared to the multi-task branch network. In the branch segmentation networks, deep convolution blocks are added to obtain the deep features of the segmented branches, and then the deep features are up-sampled. An improved multi-scale convolutional attention module is proposed, which combines the up-sampling results with the feature tensor of each feature extraction stage of trunk sharing network after jumping connection with multi-scale convolution attention module, so as to reduce the fuzzy problem of nodule edge blurs and improve the segmentation performance. At the same time, a multi-scale convolutional attention module is integrated into the classification branch to optimize the classification performance. The experimental results show that the multi-task method proposed in this paper can effectively improve the accuracy of segmentation and classification, having better segmentation and classification performance than single-task deep learning network. It can effectively deal with the problem of multi-scale thyroid nodules and blurred nodule edges, and reduce the impact brought by unbalanced classification of benign and malignant.
- 参考文献/References:
-
[1] RONNEBERGER O, FISCHER P, BROX T. U-net: convolutional networks for biomedical image segmentation[M]. Cham: Springer International Publishing, 2015: 234?241.
[2] HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770?778.
[3] HU Jie, SHEN Li, SUN Gang. Squeeze-and-excitation networks[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7132?7141.
[4] 吴俊霞, 强彦, 王梦南, 等. 基于条件分割对抗网络的超声甲状腺结节分割[J]. 太原理工大学学报, 2023,54(2):3920-398.
WU Junxia, QIANG Yan, WANG Mengnan, et al. Ultrasonic thyroid nodule segmentation based on segmentation adversarial network[J]. Journal of Taiyuan University of Technology, 2023,54(2):3920-398.
[5] 胡屹杉, 秦品乐, 曾建潮, 等. 基于特征融合和动态多尺度空洞卷积的超声甲状腺分割网络[J]. 计算机应用, 2021, 41(3): 891–897
HU Yishan, QIN Pinle, ZENG Jianchao, et al. Ultrasound thyroid segmentation network based on feature fusion and dynamic multi-scale dilated convolution[J]. Journal of computer applications, 2021, 41(3): 891–897
[6] 赵科甫, 张蕾. 基于卷积神经网络的甲状腺结节超声图像分割[J]. 现代计算机, 2021(15): 54–60
ZHAO Kefu, ZHANG Lei. Segmentation of thyroid nodules on ultrasound image using convolution neural network[J]. Modern computer, 2021(15): 54–60
[7] CHU Chen, ZHENG Jihui, ZHOU yong. Ultrasonic thyroid nodule detection method based on U-Net network[J]. Computer methods and programs in biomedicine, 2021, 199: 105906.
[8] OKTAY O, SCHLEMPER J, FOLGOC L L, et al. Attention U-net: learning where to look for the pancreas[EB/OL]. (2018?04?11)[2022?03?31].https://arxiv.org/abs/1804.03999.
[9] ZHOU Zongwei, RAHMAN SIDDIQUEE M M, TAJBAKHSH N, et al. UNet++: A nested U-net architecture for medical image segmentation[M]. Cham: Springer International Publishing, 2018: 3?11.
[10] CHEN L C, ZHU Yukun, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Computer Vision - ECCV 2018: 15th European Conference. New York: ACM, 2018: 833-851.
[11] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(12): 2481–2495.
[12] HUANG Gao, LIU Zhuang, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261?2269.
[13] 迟剑宁, 于晓升, 张艺菲. 融合深度网络和浅层纹理特征的甲状腺结节癌变超声图像诊断[J]. 中国图象图形学报, 2018, 23(10): 1582–1593
CHI Jianning, YU Xiaosheng, ZHANG Yifei. Thyroid nodule malignantrisk detection in ultrasound image by fusing deep and texture features[J]. Journal of image and graphics, 2018, 23(10): 1582–1593
[14] 邹奕轩, 周蕾蕾, 赵紫婷, 等. 基于卷积神经网络的甲状腺结节超声图像良恶性分类研究[J]. 中国医学装备, 2020, 17(3): 9–13
ZOU Yixuan, ZHOU Leilei, ZHAO Ziting, et al. Study on the classification of benign and malignant thyroid nodule in ultrasound image on the basis of cnns[J]. China medical equipment, 2020, 17(3): 9–13
[15] MOUSSA O, KHACHNAOUI H, GUETARI R, et al. Thyroid nodules classification and diagnosis in ultrasound images using fine-tuning deep convolutional neural network[J]. International journal of imaging systems and technology, 2020, 30(1): 185–195.
[16] WEI Xi, GAO Ming, YU Ruiguo, et al. Ensemble deep learning model for multicenter classification of thyroid nodules on ultrasound images[J]. Med sci monit, 2020, 26: e926096.
[17] XIE Jiahao, GUO Lehang, ZHAO Chongke, et al. A hybrid deep learning and handcrafted features based approach for thyroid nodule classification in ultrasound images[J]. Journal of physics:conference series, 2020, 1693(1): 012160.
[18] SHI Guohua, WANG Jiawen, QIANG yan, et al. Knowledge-guided synthetic medical image adversarial augmentation for ultrasonography thyroid nodule classification[J]. Computer methods and programs in biomedicine, 2020, 196: 105611.
[19] SHEN Xueda, OUYANG Xi, LIU Tianjiao, et al. Cascaded networks for thyroid nodule diagnosis from ultrasound images[M]. Cham: Springer International Publishing, 2021: 145?154.
[20] AMYAR A, MODZELEWSKI R, LI H, et al. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: classification and segmentation[J]. Computers in biology and medicine, 2020, 126: 104037.
[21] KIM S J, KIM H S. Multi-tasking U-net based paprika disease diagnosis[J]. Korean institute of smart media, 2020, 9(1): 16–22.
[22] CIPOLLA R, GAL Y, KENDALL A. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7482?7491.
[23] WU Yuhuan, GAO Shanghua, MEI Jie, et al. JCS: an explainable COVID-19 diagnosis system by joint classification and segmentation[J]. IEEE transactions on image processing:a publication of the IEEE signal processing society, 2021, 30: 3113–3126.
[24] 叶剑锋, 徐轲, 熊峻峰, 等. 基于注意力机制和辅助任务的语义分割算法[J]. 计算机工程, 2021, 47(9): 203–209,216
YE Jianfeng, XU Ke, XIONG Junfeng, et al. Semantic segmentation algorithm based on attention mechanism and auxiliary task[J]. Computer engineering, 2021, 47(9): 203–209,216
[25] LYU Kejie, LI Yingming, ZHANG Zhongfei. Attention-aware multi-task convolutional neural networks[J]. IEEE transactions on image processing, 2020, 29: 1867–1878.
[26] MEHTA S, MERCAN E, BARTLETT J, et al. Y-net: joint segmentation and classification for diagnosis of breast biopsy images[M]. Cham: Springer International Publishing, 2018: 893?901.
- 相似文献/References:
-
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]丁科,谭营.GPU通用计算及其在计算智能领域的应用[J].智能系统学报,2015,10(1):1.[doi:10.3969/j.issn.1673-4785.201403072]
DING Ke,TAN Ying.A review on general purpose computing on GPUs and its applications in computational intelligence[J].CAAI Transactions on Intelligent Systems,2015,10():1.[doi:10.3969/j.issn.1673-4785.201403072]
[3]马晓,张番栋,封举富.基于深度学习特征的稀疏表示的人脸识别方法[J].智能系统学报,2016,11(3):279.[doi:10.11992/tis.201603026]
MA Xiao,ZHANG Fandong,FENG Jufu.Sparse representation via deep learning features based face recognition method[J].CAAI Transactions on Intelligent Systems,2016,11():279.[doi:10.11992/tis.201603026]
[4]刘帅师,程曦,郭文燕,等.深度学习方法研究新进展[J].智能系统学报,2016,11(5):567.[doi:10.11992/tis.201511028]
LIU Shuaishi,CHENG Xi,GUO Wenyan,et al.Progress report on new research in deep learning[J].CAAI Transactions on Intelligent Systems,2016,11():567.[doi:10.11992/tis.201511028]
[5]马世龙,乌尼日其其格,李小平.大数据与深度学习综述[J].智能系统学报,2016,11(6):728.[doi:10.11992/tis.201611021]
MA Shilong,WUNIRI Qiqige,LI Xiaoping.Deep learning with big data: state of the art and development[J].CAAI Transactions on Intelligent Systems,2016,11():728.[doi:10.11992/tis.201611021]
[6]王亚杰,邱虹坤,吴燕燕,等.计算机博弈的研究与发展[J].智能系统学报,2016,11(6):788.[doi:10.11992/tis.201609006]
WANG Yajie,QIU Hongkun,WU Yanyan,et al.Research and development of computer games[J].CAAI Transactions on Intelligent Systems,2016,11():788.[doi:10.11992/tis.201609006]
[7]黄心汉.A3I:21世纪科技之光[J].智能系统学报,2016,11(6):835.[doi:10.11992/tis.201605022]
HUANG Xinhan.A3I: the star of science and technology for the 21st century[J].CAAI Transactions on Intelligent Systems,2016,11():835.[doi:10.11992/tis.201605022]
[8]宋婉茹,赵晴晴,陈昌红,等.行人重识别研究综述[J].智能系统学报,2017,12(6):770.[doi:10.11992/tis.201706084]
SONG Wanru,ZHAO Qingqing,CHEN Changhong,et al.Survey on pedestrian re-identification research[J].CAAI Transactions on Intelligent Systems,2017,12():770.[doi:10.11992/tis.201706084]
[9]杨梦铎,栾咏红,刘文军,等.基于自编码器的特征迁移算法[J].智能系统学报,2017,12(6):894.[doi:10.11992/tis.201706037]
YANG Mengduo,LUAN Yonghong,LIU Wenjun,et al.Feature transfer algorithm based on an auto-encoder[J].CAAI Transactions on Intelligent Systems,2017,12():894.[doi:10.11992/tis.201706037]
[10]王科俊,赵彦东,邢向磊.深度学习在无人驾驶汽车领域应用的研究进展[J].智能系统学报,2018,13(1):55.[doi:10.11992/tis.201609029]
WANG Kejun,ZHAO Yandong,XING Xianglei.Deep learning in driverless vehicles[J].CAAI Transactions on Intelligent Systems,2018,13():55.[doi:10.11992/tis.201609029]
备注/Memo
收稿日期:2022-03-31。
基金项目:国家自然科学基金项目(61172167)；黑龙江省青年科学基金项目(QC2017076)；广东省教育厅青年创新人才类项目(2019GKQNCX043) ；广东省教育厅普通高校特色创新项目(2019GKTSCX029)；广东省普通高校创新团队项目(2021KCXTD079).
作者简介:刘侠，教授，主要研究方向为医学图像处理、生物特征识别。主持国家自然科学基金项目2项、黑龙江省自然科学基金项目2项，授权发明专利3项。发表学术论文40余篇。;吕志伟，硕士研究生，主要研究方向为医学图像分割和机器学习;王波，副教授，主要研究方向为模式识别、医学图像分析与处理、计算机视觉。主持或参与国家自然科学基金项目、黑龙江省自然科学基金项目等5项。发表学术论文30余篇
通讯作者:王波.E-mail:hust_wb@hrbust.edu.cn
更新日期/Last Update:
1900-01-01