[1]张冀,曹艺,王亚茹,等.融合VAE和StackGAN的零样本图像分类方法[J].智能系统学报,2022,17(3):593-601.[doi:10.11992/tis.202107012]
　ZHANG Ji,CAO Yi,WANG Yaru,et al.Zero-shot image classification method combining VAE and StackGAN[J].CAAI Transactions on Intelligent Systems,2022,17(3):593-601.[doi:10.11992/tis.202107012]
点击复制

融合VAE和StackGAN的零样本图像分类方法

《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷: 17 期数: 2022年第3期 页码: 593-601 栏目: 学术论文—人工智能基础 出版日期: 2022-05-05
Title:
Zero-shot image classification method combining VAE and StackGAN
作者:
张冀1, 曹艺1, 王亚茹2, 赵文清1, 翟永杰2
1. 华北电力大学 计算机系, 河北 保定 071003;
2. 华北电力大学 自动化系, 河北 保定 071003
Author(s):
ZHANG Ji1, CAO Yi1, WANG Yaru2, ZHAO Wenqing1, ZHAI Yongjie2
1. Department of Computer, North China Electric Power University, Baoding 071003, China;
2. Department of Automation, North China Electric Power University, Baoding 071003, China
关键词:
深度学习; 零样本学习; 图像分类; 变分自编码器; 生成对抗网络; 分阶段网络; 句向量; 辅助信息
Keywords:
Deep learning; Zero-shot learning; Image classification; Variational autoencoder; Generative adversarial network; Staged network; Sentence vector; Auxiliary information
分类号:
TP18
DOI:
10.11992/tis.202107012
摘要:
零样本分类算法旨在解决样本极少甚至缺失类别情况下的分类问题。随着深度学习的发展，生成模型在零样本分类中的应用取得了一定的突破，通过生成缺失类别的图像，将零样本图像分类转化为传统的基于监督学习的图像分类问题，但生成图像的质量不稳定，如细节缺失、颜色失真等，影响图像分类准确性。为此，提出一种融合变分自编码(variational auto-encoder, VAE)和分阶段生成对抗网络(stack generative adversarial networks, StackGAN)的零样本图像分类方法，基于VAE/GAN模型引入StackGAN，用于生成缺失类别的数据，同时使用深度学习方法训练并获取各类别的句向量作为辅助信息，构建新的生成模型stc-CLS-VAEStackGAN，提高生成图像的质量，进而提高零样本图像分类准确性。在公用数据集上进行对比实验，实验结果验证了本文方法的有效性与优越性。
Abstract:
The zero-shot classification algorithm is designed to solve the classification problem in case of a few samples or even missing categories. With the development of deep learning, the application of the generation model in zero-shot classification has made a breakthrough. By generating images of missing categories, the zero-shot image classification is transformed into a traditional image classification problem based on supervised learning. However, the generated samples are unstable in quality, including missing details and color distortion, thus affecting the accuracy of image classification. To this end, the zero-shot image classification method combining variational auto-encoding (VAE) and stack generative adversarial networks (StackGAN) is proposed. Based on the VAE/GAN model, StackGAN is introduced to generate the data of missing categories. Meanwhile, the deep learning method is used to train and obtain the sentence vectors of each category as auxiliary information and build a new generation model stc–CLS–VAEStackGAN to improve the quality of generated images and subsequently improve the classification accuracy of the zero-shot images. A comparative experiment was conducted on the public dataset, and the experimental results verified the effectiveness and superiority of the method proposed herein.
参考文献/References:
[1] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60: 84–90.
[2] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553): 436–444.
[3] BIEDERMAN I. Recognition-by-components: a theory of human image understanding[J]. Psychological review, 1987, 94(2): 115–147.
[4] PALATUCCI M, POMERLEAU D, HINTON G E, et al. Zero-shot Learning with semantic output codes[C]// Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Vancouver: NIPS, 2009: 1410–1418.
[5] LAMPERT C H, NICKISCH H, HARMELING S. Learning to detect unseen object classes by between-class attribute transfer[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami: IEEE, 2009: 951–958.
[6] ROHRBACH M, STARK M, SCHIELE B. Evaluating knowledge transfer and zero-shot learning in a large-scale setting[C]// Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011: 1641–1648.
[7] HABIBIAN A, MENSINK T, SNOEK C G M. Video2vec embeddings recognize events when examples are scarce[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 39(10): 2089–2103.
[8] 冀中, 汪浩然, 于云龙, 等. 零样本图像分类综述: 十年进展[J]. 中国科学:信息科学, 2019, 49(10): 1299–1320
JI Zhong, WANG Haoran, YU Yunlong, et al. A decadal survey of zero-shot image classification[J]. Scientia sinica (informationis), 2019, 49(10): 1299–1320
[9] 张鲁宁, 左信, 刘建伟. 零样本学习研究进展[J]. 自动化学报, 2020, 46(1): 1–23
ZHANG Luning, ZUO Xin, LIU Jianwei. Research and development on zero-shot learning[J]. Acta automatica sinica, 2020, 46(1): 1–23
[10] 冀中, 孙涛, 于云龙. 一种基于直推判别字典学习的零样本分类方法[J]. 软件学报, 2017, 28(11): 2961–2970
JI Zhong, SUN Tao, YU Yunlong. Transductive discriminative dictionary learning approach for zero-shot classification[J]. Journal of software, 2017, 28(11): 2961–2970
[11] WANG Xuesong, CHEN Chen, CHENG Yuhu. Zero-shot learning by exploiting class-related and attribute-related prior knowledge[J]. IET computer vision, 2016, 10(6): 483–492.
[12] 赵鹏, 汪纯燕, 张思颖, 等. 一种基于融合重构的子空间学习的零样本图像分类方法[J]. 计算机学报, 2021, 44(2): 409–421
ZHAO Peng, WANG Chunyan, ZHANG Siying, et al. A zero-shot image classification method based on subspace learning with the fusion of reconstruction[J]. Chinese journal of computers, 2021, 44(2): 409–421
[13] DU Yujiao, XIAO Bo, XU Wenchao, et al. Destination prediction for sharing-bikes’ trips[C]//2018 International Conference on Network Infrastructure and Digital Content (IC-NIDC). Guiyang: IEEE, 2018: 198–202.
[14] 陈祥凤, 陈雯柏. 度量学习改进语义自编码零样本分类算法[J]. 北京邮电大学学报, 2018, 41(4): 69–75
CHEN Xiangfeng, CHEN Wenbai. Improving semantic autoencoder zero-shot classification algorithm by metric learning[J]. Journal of Beijing university of posts and telecommunications, 2018, 41(4): 69–75
[15] 吴晨, 袁昱纬, 王宏伟, 等. 基于词向量融合的遥感场景零样本分类算法[J]. 计算机科学, 2019, 46(12): 286–291
WU Chen, YUAN Yuwei, WANG Hongwei, et al. Word vectors fusion based remote sensing scenes zero-shot classification algorithm[J]. Computer science, 2019, 46(12): 286–291
[16] 冀中, 谢于中, 庞彦伟. 基于典型相关分析和距离度量学习的零样本学习[J]. 天津大学学报(自然科学与工程技术版), 2017, 50(8): 813–820
JI Zhong, XIE Yuzhong, PANG Yanwei. Zero-shot learning based on canonical correlation analysis and distance metric learning[J]. Journal of Tianjin University (science and technology edition), 2017, 50(8): 813–820
[17] XIAN Yongqin, LAMPERT C H, SCHIELE B, et al. Zero-shot learning-A comprehensive evaluation of the good, the bad and the ugly[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 41(9): 2251–2265.
[18] XIAN Yongqin, LORENZ T, SCHIELE B, et al. Feature generating networks for zero-shot learning[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 5542–5551.
[19] SARIYILDIZ M B, CINBIS R G. Gradient matching generative networks for zero-shot learning[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 2163–2173.
[20] MANDAL D, NARAYAN S, DWIVEDI S K, et al. Out-of-distribution detection for generalized zero-shot action recognition[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach : IEEE, 2019: 9977–9985.
[21] XIAN Yongqin, SHARMA S, SCHIELE B, et al. F-VAEGAN-D2: a feature generating framework for any-shot learning[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach : IEEE, 2019: 10267–10276.
[22] KIM H, LEE J, BYUN H. Unseen image generating domain-free networks for generalized zero-shot learning[J]. Neurocomputing, 2020, 411: 67–77.
[23] VERMA V K, BRAHMA D, RAI P. Meta-learning for generalized zero-shot learning[J]. Proceedings of the AAAI conference on artificial intelligence, 2020, 34(4): 6062–6069.
[24] MA Yuanbo, XU Xing, SHEN Fumin, et al. Similarity preserving feature generating networks for zero-shot learning[J]. Neurocomputing, 2020, 406: 333–342.
[25] LIU Huan, YAO Lina, ZHENG Qinghua, et al. Dual-stream generative adversarial networks for distributionally robust zero-shot learning[J]. Information sciences, 2020, 519: 407–422.
[26] LIU Jinlu, ZHANG Zhaocheng, YANG Gang. Cross-class generative network for zero-shot learning[J]. Information sciences, 2021, 555: 147–163.
[27] TANG C, HE Z, LI Y, ET Al. Zero-shot learning via structure-aligned generative adversarial network[J]. IEEE transactions on neural networks and learning systems, 2021(99): 1–14.
[28] LI Zhiqun, CHEN Qiong, LIU Qingfa. Augmented semantic feature based generative network for generalized zero-shot learning[J]. Neural networks:the official journal of the international neural network society, 2021, 143: 1–11.
[29] GAO RUI, HOU XINGSONG, QIN JIE, et al. Zero-VAE-GAN: generating unseen features for generalized and transductive zero-shot learning[J]. IEEE transactions on image processing, 2020, 29: 3665–3680.
[30] KINGMA D P, WELLING M. Auto-Encoding Variational Bayes[EB/OL]. (2014–05–01)[2022–03–10]https://arxiv.org/abs/1312.6114v2.
[31] LARSEN A B L, S?NDERBY S K, LAROCHELLE H, et al. Autoencoding beyond pixels using a learned similarity metric[EB/OL]. (2016–02–10)[2022–03–10]https://arxiv.org/abs/1512.09300.
[32] ZHANG Han, XU Tao, LI Hongsheng, et al. StackGAN: text to photo-realistic image synthesis with stacked generative adversarial networks[EB/OL]. (2017–08–05)[2022-03-10]https://arxiv.org/abs/1612.03242v2.
[33] Wah C, Branson S, Welinder P, et al. The Caltech-UCSD Birds-200-2011 Dataset[J]. California institute of technology, 2011.
[34] LAMPERT C H, NICKISCH H, HARMELING S. Attribute-based classification for zero-shot visual object categorization[J]. IEEE transactions on pattern analysis and machine intelligence, 2014, 36(3): 453–465.
[35] CHE TONG, LI YANRAN, JACOB A P, et al. Mode regularized generative adversarial networks[EB/OL]. (2017–03–02)[2022–03–10]https://arxiv.org/abs/1612.02136 .
[36] 魏宏喜, 张越. 基于生成对抗网络的零样本图像分类[J]. 北京航空航天大学学报, 2019, 45(12): 2345–2350
WEI Hongxi, ZHANG Yue. Zero-shot image classification based on generative adversarial network[J]. Journal of Beijing University of Aeronautics and Astronautics, 2019, 45(12): 2345–2350
相似文献/References:
[1]张媛媛,霍静,杨婉琪,等.深度信念网络的二代身份证异构人脸核实算法[J].智能系统学报,2015,10(2):193.[doi:10.3969/j.issn.1673-4785.201405060]
　ZHANG Yuanyuan,HUO Jing,YANG Wanqi,et al.A deep belief network-based heterogeneous face verification method for the second-generation identity card[J].CAAI Transactions on Intelligent Systems,2015,10():193.[doi:10.3969/j.issn.1673-4785.201405060]
[2]丁科,谭营.GPU通用计算及其在计算智能领域的应用[J].智能系统学报,2015,10(1):1.[doi:10.3969/j.issn.1673-4785.201403072]
　DING Ke,TAN Ying.A review on general purpose computing on GPUs and its applications in computational intelligence[J].CAAI Transactions on Intelligent Systems,2015,10():1.[doi:10.3969/j.issn.1673-4785.201403072]
[3]马晓,张番栋,封举富.基于深度学习特征的稀疏表示的人脸识别方法[J].智能系统学报,2016,11(3):279.[doi:10.11992/tis.201603026]
　MA Xiao,ZHANG Fandong,FENG Jufu.Sparse representation via deep learning features based face recognition method[J].CAAI Transactions on Intelligent Systems,2016,11():279.[doi:10.11992/tis.201603026]
[4]刘帅师,程曦,郭文燕,等.深度学习方法研究新进展[J].智能系统学报,2016,11(5):567.[doi:10.11992/tis.201511028]
　LIU Shuaishi,CHENG Xi,GUO Wenyan,et al.Progress report on new research in deep learning[J].CAAI Transactions on Intelligent Systems,2016,11():567.[doi:10.11992/tis.201511028]
[5]马世龙,乌尼日其其格,李小平.大数据与深度学习综述[J].智能系统学报,2016,11(6):728.[doi:10.11992/tis.201611021]
　MA Shilong,WUNIRI Qiqige,LI Xiaoping.Deep learning with big data: state of the art and development[J].CAAI Transactions on Intelligent Systems,2016,11():728.[doi:10.11992/tis.201611021]
[6]王亚杰,邱虹坤,吴燕燕,等.计算机博弈的研究与发展[J].智能系统学报,2016,11(6):788.[doi:10.11992/tis.201609006]
　WANG Yajie,QIU Hongkun,WU Yanyan,et al.Research and development of computer games[J].CAAI Transactions on Intelligent Systems,2016,11():788.[doi:10.11992/tis.201609006]
[7]黄心汉.A3I:21世纪科技之光[J].智能系统学报,2016,11(6):835.[doi:10.11992/tis.201605022]
　HUANG Xinhan.A3I: the star of science and technology for the 21st century[J].CAAI Transactions on Intelligent Systems,2016,11():835.[doi:10.11992/tis.201605022]
[8]宋婉茹,赵晴晴,陈昌红,等.行人重识别研究综述[J].智能系统学报,2017,12(6):770.[doi:10.11992/tis.201706084]
　SONG Wanru,ZHAO Qingqing,CHEN Changhong,et al.Survey on pedestrian re-identification research[J].CAAI Transactions on Intelligent Systems,2017,12():770.[doi:10.11992/tis.201706084]
[9]杨梦铎,栾咏红,刘文军,等.基于自编码器的特征迁移算法[J].智能系统学报,2017,12(6):894.[doi:10.11992/tis.201706037]
　YANG Mengduo,LUAN Yonghong,LIU Wenjun,et al.Feature transfer algorithm based on an auto-encoder[J].CAAI Transactions on Intelligent Systems,2017,12():894.[doi:10.11992/tis.201706037]
[10]王科俊,赵彦东,邢向磊.深度学习在无人驾驶汽车领域应用的研究进展[J].智能系统学报,2018,13(1):55.[doi:10.11992/tis.201609029]
　WANG Kejun,ZHAO Yandong,XING Xianglei.Deep learning in driverless vehicles[J].CAAI Transactions on Intelligent Systems,2018,13():55.[doi:10.11992/tis.201609029]
[11]翟永杰,张智柏,王亚茹.基于改进TransGAN的零样本图像识别方法[J].智能系统学报,2023,18(2):352.[doi:10.11992/tis.202111002]
　ZHAI Yongjie,ZHANG Zhibai,WANG Yaru.An image recognition method of zero -shot learning based on an improved TransGAN[J].CAAI Transactions on Intelligent Systems,2023,18():352.[doi:10.11992/tis.202111002]

备注/Memo

收稿日期:2021-07-07。
基金项目:国家自然科学基金面上项目(61773160)；河北省自然科学基金青年科学基金项目(F2021502008)；中央高校基本科研业务费专项资金面上项目(2021MS081)
作者简介:张冀，副教授，博士，主要研究方向为计算机测控、故障诊断、信息融合、图像处理、深度学习。出版规划教材2部。发表学术论文20余篇;曹艺，硕士研究生，主要研究方向为计算机视觉。;王亚茹，讲师，博士，主要研究方向为模式识别与计算机视觉、数据挖掘、电力视觉。主持河北省自然科学基金青年基金项目1项，参与国家自然科学基金面上项目2项、横向科研项目多项。发表学术论文10余篇
通讯作者:王亚茹.E-mail:wangyaru@ncepu.edu.cn

更新日期/Last Update: 1900-01-01
Copyright © 《 智能系统学报》 编辑部
地址：（150001）黑龙江省哈尔滨市南岗区南通大街145-1号楼 电话：0451- 82534001、82518134　邮箱：tis@vip.sina.com