[1]LIU Shuaishi,CHENG Xi,GUO Wenyan,et al.Progress report on new research in deep learning[J].CAAI Transactions on Intelligent Systems,2016,11(5):567-577.[doi:10.11992/tis.201511028]
Copy
Progress report on new research in deep learning
CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume:
11
Number of periods:
2016 5
Page number:
567-577
Column:
综述
Public date:
2016-11-01
- Title:
- Progress report on new research in deep learning
- Keywords:
- deep learning; convolutional neural network; deep belief networks; deep Boltzmann machine; automatic stacking encoder
- CLC:
- TP18
- DOI:
- 10.11992/tis.201511028
- Abstract:
- Deep learning has recently received widespread attention. Using a model structure, this paper gives a summarization and analysis on deep learning by describing and reviewing the structure and characteristics of different models. The paper firstly introduces the concept and significance of deep learning, and then reviews four typical models:a convolutional neural network; deep belief networks; the deep Boltzmann machine; and an automatic stacking encoder. The paper then concludes by reviewing the applications of deep learning as regards speech processing, computer vision, natural language processing, medical science, and other aspects. Finally, the existing deep learning model is summarized and future challenges discussed.
- References:
-
[1] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. Nature, 2015, 521(7553):436-444.
[2] 林妙真. 基于深度学习的人脸识别研究[D]. 大连:大连理工大学, 2013. LIN Miaozhen. Research on face recognition based on deep learning[D]. Dalian, China:Dalian University of Technology, 2013.
[3] HINTON G E, SALAKHUTDINOV R R. Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313(5786):504-507.
[4] 刘建伟, 刘媛, 罗雄麟. 深度学习研究进展[J]. 计算机应用研究, 2014, 31(7):1921-1930, 1942. LIU Jianwei, LIU Yuan, LUO Xionglin. Research and development on deep learning[J]. Application research of computers, 2014, 31(7):1921-1930, 1942.
[5] 余滨, 李绍滋, 徐素霞, 等. 深度学习:开启大数据时代的钥匙[J]. 工程研究-跨学科视野中的工程, 2014, 6(3):233-243. YU Bin, LI Shaozi, XU Suxia, et al. Deep learning:a key of stepping into the era of big data[J]. Journal of engineering studies, 2014, 6(3):233-243.
[6] 尹宝才, 王文通, 王立春. 深度学习研究综述[J]. 北京工业大学学报, 2015, 41(1):48-59. YIN Biaocai, WANG Wentong, WANG Lichun. Review of deep learning[J]. Journal of Beijing university of technology, 2015, 41(1):48-59.
[7] 张建明, 詹智财, 成科扬, 等. 深度学习的研究与发展[J]. 江苏大学学报:自然科学版, 2015, 36(2):191-200. ZHANG Jianming, ZHAN Zhicai, CHENG Keyang, et al. Review on development of deep learning[J]. Journal of Jiangsu university:natural science editions, 2015, 36(2):191-200.
[8] LECUN Y, JACKEL L D, BOTTOU L, et al. Learning algorithms for classification:a comparison on handwritten digit recognition[M]//OH J H, KWON C, CHO S. Neural Networks:The Statistical Mechanics Perspective. Singapore:World Scientific, 1995:261-276.
[9] 陈先昌. 基于卷积神经网络的深度学习算法与应用研究[D]. 杭州:浙江工商大学, 2014. CHEN Xianchang. Research on algorithm and application of deep learning based on convolutional neural network[D]. Hangzhou, China:Zhejiang Gongshang University, 2014.
[10] 李卫. 深度学习在图像识别中的研究及应用[D]. 武汉:武汉理工大学, 2014. LI Wei. The research and application of deep learning in image recognition[D]. Wuhan:Wuhan University of Technology, 2014.
[11] JI Shuiwang, XU Wei, YANG Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(1):221-231.
[12] BRUNA J, ZAREMBA W, SZLAM A, et al. Spectral networks and locally connected networks on graphs[EB/OL].Eprint Arxiv:Arxiv,2013.[2014-10-10] http://120.52.73.79/arxiv.org/pdf/1312.6203v3.pdf.
[13] FAN Haoqiang, CAO Zhimin, JIANG Yuning, et al. Learning deep face representation[EB/OL]. Eprint Arxiv:Arxiv, 2014.[2014-10-10] http://120.52.73.80/arxiv.org/pdf/1403.2802v1.pdf.
[14] 王冠皓, 徐军. 基于多级金字塔卷积神经网络的快速特征表示方法[J]. 计算机应用研究, 2015, 32(8):2492-2495. WANG Guanhao, XU Jun. Fast feature representation method based on multi-level pyramid convolution neural network[J]. Application research of computers, 2015, 32(8):2492-2495.
[15] LEE H, GROSSE R, RANGANATH R, et al. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations[C]//Proceedings of the 26th Annual International Conference on Machine Learning. New York, NY, USA, 2009:609-616.
[16] LEE H, EKANADHAM C, NG A Y. Sparse deep belief net model for visual area V2[C]//Advances in Neural Information Processing Systems 20:21st Annual Conference on Neural Information Processing Systems. Vancouver, British Columbia, Canada, 2007:873-880.
[17] 罗恒. 基于协同过滤视角的受限玻尔兹曼机研究[D]. 上海:上海交通大学, 2011. LUO Heng. Restricted Boltzmann machines:a collaborative filtering perspective[D]. Shanghai, China:Shanghai Jiao Tong University, 2011.
[18] LAROCHELLE H, BENGIO Y. Classification using discriminative restricted Boltzmann machines[C]//Proceedings of the 25th International Conference on Machine Learning. New York, NY, USA, 2008:536-543.
[19] 张春霞, 姬楠楠, 王冠伟. 受限波尔兹曼机[J]. 工程数学学报, 2015, 32(2):159-173. ZHANG Chunxia, JI Nannan, WANG Guanwei. Restricted Boltzmann machines[J]. Chinese journal of engineering mathematics, 2015, 32(2):159-173.
[20] 刘银华. LBP和深度信念网络在非限制条件下人脸识别研究[D]. 江门:五邑大学, 2014. LIU Yinhua. The research of face recognition under unconstrained condition via LBP and deep belief network[D]. Jiangmen:Wuyi University, 2014.
[21] LEE H, GROSSE R, RANGANATH R, et al. Unsupervised learning of hierarchical representations with convolutional deep belief networks[J]. Communications of the ACM, 2011, 54(10):95-103.
[22] HALKIAS X C, PARIS S, GLOTIN H. Sparse penalty in deep belief networks:using the mixed norm constraint[EB/OL].[2014-05-08]. http://arxiv.org/pdf/1301.3533.pdf.
[23] LIU Yan, ZHOU Shusen, CHEN Qingcai. Discriminative deep belief networks for visual data classification[J]. Pattern recognition, 2011, 44(10/11):2287-2296.
[24] 郑胤, 陈权崎, 章毓晋. 深度学习及其在目标和行为识别中的新进展[J]. 中国图象图形学报, 2014, 19(2):175-184. ZHENG Yin, CHEN Quanqi, ZHANG Yujin. Deep learning and its new progress in object and behavior recognition[J]. Journal of image and graphics, 2014, 19(2):175-184.
[25] VINCENT P, LAROCHELLE H, BENGIO Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th International Conference on Machine Learning. New York, NY, USA, 2008:1096-1103.
[26] BENGIO Y, LAMBLIN P, POPOVICI D, et al. Greedy layer-wise training of deep networks[C]//Advances in Neural Information Processing Systems 19:20th Annual Conference on Neural Information Processing Systems. Vancouver, British Columbia, Canada, 2006:153-160.
[27] RIFAI S, VINCENT P, MULLER X, et al. Contractive auto-encoders:explicit invariance during feature extraction[C]//Proceedings of the 28th International Conference on Machine Learning. Bellevue, WA, USA, 2011.
[28] MASCI J, MEIER U, CIRE?AN D, et al. Stacked convolutional auto-encoders for hierarchical feature extraction[C]//Proceedings of the 21st International Conference on Artificial Neural Networks, Part I. Berlin Heidelberg, Germany, 2011:52-59.
[29] 王雅思. 深度学习中的自编码器的表达能力研究[D]. 哈尔滨:哈尔滨工业大学, 2014. WANG Yasi. Representation ability research of auto-encoders in deep learning[D]. Harbin:Harbin Institute of Technology, 2014.
[30] 李远豪. 基于深度自编码器的人脸美丽吸引力预测研究[D]. 江门:五邑大学, 2014. LI Yuanhao. A study for facial beauty attractiveness prediction based on deep autoencoder[D]. Jiangmen:Wuyi University, 2014.
[31] 林洲汉. 基于自动编码机的高光谱图像特征提取及分类方法研究[D]. 哈尔滨:哈尔滨工业大学, 2014. LIN Zhouhan. Hyperspectral image feature extraction and classification based on autoencoders[D]. Harbin:Harbin Institute of Technology, 2014.
[32] 曲建岭, 杜辰飞, 邸亚洲, 等. 深度自动编码器的研究与展望[J]. 计算机与现代化, 2014(8):128-134. QU Jianling, DU Chenfei, DI Yazhou, et al. Research and prospect of deep auto-encoders[J]. Jisuanji yu xiandaihua, 2014(8):128-134.
[33] 林少飞, 盛惠兴, 李庆武. 基于堆叠稀疏自动编码器的手写数字分类[J]. 微处理机, 2015(1):47-51. LIN Shaofei, SHENG Huixing, LI Qingwu. Handwritten digital classification based on the stacked sparse autoencoders[J]. Microprocessors, 2015(1):47-51.
[34] 陈硕. 深度学习神经网络在语音识别中的应用研究[D]. 广州:华南理工大学, 2013. CHEN Shuo. Research of deep learning neural networks applications in speech recognition[D]. Guangzhou, China:South China University of Technology, 2013.
[35] 郭丽丽, 丁世飞. 深度学习研究进展[J]. 计算机科学, 2015, 42(5):28-33. GOU Lili, DING Shifei. Research progress on deep learning[J]. Computer science, 2015, 42(5):28-33.
[36] VAN DEN OORD A, DIELEMAN S, SCHRAUWEN B. Deep content-based music recommendation[M]//Advances in Neural Information Processing Systems 26:27th Annual Conference on Neural Information Processing Systems. Lake Tahoe, 2013:2643-2651.
[37] HANNUN A, CASE C, CASPER J, et al. Deep speech:scaling up end-to-end speech recognition[EB/OL]. Eprint Arxiv:Arxiv, 2014.[2014-12-19] https://arxiv.org/pdf/1412.5567v2.pdf.
[38] 余凯, 贾磊, 陈雨强. 深度学习的昨天、今天和明天[J]. 计算机研究与发展, 2013, 50(9):1799-1804. YU Kai, JIA Lei, CHEN Yuqiang. Deep learning:yesterday, today, and tomorrow[J]. Journal of computer research and development, 2013, 50(9):1799-1804.
[39] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, 2014:580-587.
[40] TAIGMAN Y, YANG Ming, RANZATO M A, et al. DeepFace:closing the gap to human-level performance in face verification[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, 2014:1701-1708.
[41] TOSHEV A, SZEGEdY C. DeepPose:human pose estimation via deep neural networks[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Columbus, OH, 2014:1653-1660.
[42] DIELEMAN S. Classifying plankton with deep neural networks[EB/OL]. (2015-03-17)[2015-05-30]. http://benanne.github.io/2015/03/17/plankton.html.
[43] DENTON E, WESTON J, PALURI M, et al. User conditional hashtag prediction for images[C]//Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA, 2015:1731-1740.
[44] LONG J, SHELHAMER E, DARRELL T Fully convolutional networks for semantic segmentation[J]. IEEE Conference on Computer Vision & Pattern Recognition. 2015, 79(10):1337-1342.
[45] SCHWARA M, SCHULZ H, BEHNKE S. RGB-D object recognition and pose estimation based on pre-trained convolutional neural network features[C]//Proceedings of the 2015IEEE International Conference on Robotics and Automation.Seattle, WA, 2015:1329-1335.
[46] SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet:A unified embedding for face recognition and clustering[C]//Computer Vision and Pattern Recognition (CVPR), Boston,USA,2015:815-823.
[47] SARFRAZ M S, STIEFELHAGEN R. Deep perceptual mapping for thermal to visible face recognition.[EB/OL] Eprint Arxiv:Arxiv,2015.[2015-12-23].http://120.52.73.80/arxiv.org/pdf/1507.02879v1.pdf.
[48] LIU Mengyi, LI Shaoxin, SHAN Shiguang, et al. Au-aware deep networks for facial expression recognition[C]//Proceedings of the 201310th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai, China, 2013:1-6.
[49] OUELLET S. Real-time emotion recognition for gaming using deep convolutional network features[EB/OL]. Eprint Arxiv:Arxiv,2014.[2014-7-16].https://arxiv.org/pdf/1408.3750v1.pdf.
[50] SONG I, KIM H J, JEON P B. Deep learning for real-time robust facial expression recognition on a smartphone[C]//Proceedings of the 2014 IEEE International Conference on Consumer Electronics. Las Vegas, NV, 2014:564-567.
[51] IJJINA E P, MOHAN C K. Facial expression recognition using kinect depth sensor and convolutional neural networks[C]//Proceedings of the 201413th International Conference on Machine Learning and Applications. Detroit, MI, 2014:392-396.
[52] BYEON Y H, KWAK K C. Facial expression recognition using 3D convolutional neural network[J]. International journal of advanced computer science and applications, 2014, 5(12):107-112.
[53] JUNG H, LEE S, PARK S, et al. Development of deep learning-based facial expression recognition system[C]//Frontiers of Computer Vision (FCV), 201521st Korea-Japan Joint Workshop on 2015:1-4.
[54] MCLAUGHLIN T, MAI L, BAYANBAT N. Emotionrecognition with deep-belief networks[EB/OL].2008 http://cs229.stanford.edu/proj2010/McLaughlinLeBayanbat-RecognizingEmotionsWithDeepBeliefNets.pdf.
[55] HE Shan, WANG Shanfei, LAN Wuwei, et al. Facial expression recognition using deep Boltzmann machine from thermal infrared images[C]//Proceedings of the 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. Geneva, 2013:239-244.
[56] LV Yadan, FENG Zhiyong, XU Chao. Facial expression recognition via deep learning[C]//Proceedings of the 2014 International Conference on Smart Computing. Hong Kong, China, 2014:303-308.
[57] JUNG H, LEE S, PARK S, et al. Deep temporal appearance-geometry network for facial expression recognition[EB/OL].Eprint Arxiv:Arxiv,2015.[2015-6-5].http://120.52.73.75/arxiv.org/pdf/1503.01532v1.pdf.
[58] KAHOU S E, BOUTHILLIER X, LAMBLIN P et al. EmoNets:Multimodal deep learning approaches for emotion recognition in video[J]. Journal on Multimodal User Interfaces, 2015, 10(2):1-13.
[59] SASHIHITHLU S, SOMAN S S. Complex sentimentanalysis using recursive autoencoders[EB/OL]. Core.ac.Uk:CiteSeerX,2013.[2015-9-30].https://core.ac.uk/display/23426251.
[60] JOHNSON R, ZHANG Tong. Effective use of word order for text categorization with convolutional neural networks[EB/OL]. Eprint Arxiv:Arxiv,2014.[2014-10-10]. http://120.52.73.79/arxiv.org/pdf/1412.1058.pdf.
[61] 李婷. 基于深度学习的盲文识别方法[J]. 计算机与现代化, 2015(6):37-40. LI Ting. A deep learing method for braille recognition[J].Jisuanji yu xiandaihua, 2015(6):37-40.
[62] KOZIOL J A, TAN E M, DAI Liping, et al. Restricted Boltzmann machines for classification of hepatocellular carcinoma[J]. Computational biology journal, 2014, 2014:418069.
[63] FAUW J D. Detecting diabetic retinopathy in eye images[EB/OL]. 2015[2015-07-28]. http://jeffreydf.github.io/diabetic-retinopathy-detection.
[64] CUN Y L, BOSER B, DENKER J S, et al. Handwritten digit recognition with a back-propagation network[C]//Advances in Neural Information Processing Systems 2. San Francisco, CA, USA, 1990:396-404.
[65] JI Shuiwang, XU Wei, YANG Ming, et al. 3D convolutional neural networks for human action recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(1):22l-231.
[66] HINTON G E, OSINDERO S, TEH Y W. A fast learning algorithm for deep belief nets[J]. Neural computation, 2006, 18(7):1527-1554.
[67] SALAKHUTDINOV R, HINTON G E. Deep Boltzmann machines[C]//Proceedings of the 12th International Conference on Artificial Intelligence and Statistics. Clearwater, Florida, USA, 2009:448-455.
[68] 杜骞. 深度学习在图像语义分类中的应用[D]. 武汉:华中师范大学, 2014. DU Qian. Application of deep learning in image semantic classification[D]. Wuhan:Central China Normal University, 2014.
[69] BENGIO Y. Learning deep architectures for AI[J]. Foundations and trendse in machine learning, 2009, 2(1):1-127.
[70] HINTON G E, SEJNOWSKI T J. Learning and relearning in Boltzmann machines[M]//Parallel Distributed Processing:Explorations in the Microstructure of Cognition. Cambridge, MA, USA:MIT Press, 1986.
[71] SMOLENSKY P. Information processing in dynamical systems:foundations of harmony theory[M]//Parallel Distributed Processing:Explorations in the Microstructure of Cognition. Cambridge, MA, USA:MIT Press, 1986.
[72] RUMELHART D E, HINTON G E, WILLIAMS R J. Learning representations by back-propagating errors[J]. Nature, 1986, 323(6088):533-536.
[73] MIKOLOV T, KARAFIáT M, BURGET L, et al. Recurrent neural network based language model[C]//Proceedings of the Interspeech 201011th Annual Conference of the International Speech Communication Association. Makuhari, Chiba, Japan, 2010:1045-1048.
[74] DENG Li, YU Dong. Deep convex net:a scalable architecture for speech pattern classification[C]//Proceedings of the 12th Annual Conference of the International Speech Communication Association. Florence, Italy, 2011:2296-2299.
[75] MIKOLOV T, KOMBRINK S, BURGET L. Extensions of recurrent neural network language model[C]//Proceedings of the 2011 IEEE International Conference on Acoustics, Speech and Signal Processing.Prague, 2011:5528-5531.
[76] LIU Shujie, YANG Nan, LI Mu, et al. A recursive recurrent neural network for statistical machine translation[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, Maryland, USA, 2014:1491-1500.
[77] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[EB/OL]. 2014.[2014-10-12].http://120.52.73.79/arxiv.org/pdf/1409.3215v3.pdf.
[78] GRAVES A, MOHAMED A R, HINTON G. Speech recognition with deep recurrent neural networks[C]//Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. Vancouver, BC, 2013:6645-6649.
[79] SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al.Intriguing properties of neural networks[EB/OL]. Eprint Arxiv:Arxiv,2014.[2014-10-12]. http://120.52.73.78/arxiv.org/pdf/1312.6199v4.pdf.
[80] NGUYEN A, YOSINSKI J, CLUNE J Deep neural networks are easily fooled:High confidence predictions for unrecognizable images[C]//Computer Vision and Pattern Recognition. IEEE, Boston,USA,2015:427-436.
[81] LIPTON Z C. (Deep learning’s deep flaws)’s deep flaws[EB/OL].[2015-07-28]. http://www.kdnuggets.com/2015/01/deep-learning-flaws-universal-machine-learning.html.
- Similar References:
Memo
-
Last Update:
1900-01-01