[1]XIN Yuxuan,YAN Zifei.Research progress of image retrieval based on hand-drawn sketches[J].CAAI Transactions on Intelligent Systems,2015,10(2):167-177.[doi:10.3969/j.issn.1673-4785.201401045]
Copy

Research progress of image retrieval based on hand-drawn sketches

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 10 Number of periods: 2015 2 Page number: 167-177 Column: 综述 Public date: 2015-04-25
Title:
Research progress of image retrieval based on hand-drawn sketches
Author(s):
XIN Yuxuan;  YAN Zifei
School of Mechatronics Engineering, Harbin Institute of Technology, Harbin 150001, China
Keywords:
hand-drawn sketches; feature interpretation; image retrieval; human-computer interaction; big data
CLC:
TP391.41
DOI:
10.3969/j.issn.1673-4785.201401045
Abstract:
This paper build a framework for the domain-independent image retrieval system based on hand-drawn sketches by researching the existing sketch-based image retrieval related technologies. The relevant technologies and their development of the involved system stages such as: hand-drawn sketch preprocessing, sketch feature interpretation, sketch matching, and image feedback were also outlined. Applications relating to sketch-based image retrieval were summarized. Finally, the coming trends of the hand-drawn sketch-based image retrieval system under the background of natural user interface, pervasive computing and big data were forecasted.
References:
[1] RUI Y, THOMAS S, CHANG S. Image retrieval: past, present and future[J]. Journal of Visual Communication and Image Representation, 1999, 10(1): 39-62.
[2] EAKINS J. Automatic image content retrieval-Are we getting anywhere[C]//Proceedings of 3rd International Conference on Electronic Library and Visual Information Research. Newcastle, United Kingdom, 1996: 123-135.
[3] VELTKAMP R C, TANASE M. Content-based image retrieval systems: a survey. UU-CS-2000-34[R]. Dept of Computing Science, Utrecht University, 2000.
[4] NIBLACK C, BARBER R, EQUITZ W, et al. The QBIC project: querying images by content using color, texture, and shape[C]//Proceedings of SPIE Electronic Imaging: Science and Technology. San Jose, Canada, 1993: 173-187.
[5] PENTLAND A, PICARD R W, SCLAROFF S. Photobook: content-based manipulation of image databases[J]. International Journal of Computer Vision, 1996, 18(3): 233-254.
[6] 孙正兴,冯桂焕,周若鸿.基于草图的人机交互技术研究进展[J]. 计算机辅助设计与图形学学报, 2005, 9(17): 1889-1899.SUN Zhengxing, FENG Guihuan, ZHOU Ruohong. Techniques for sketch-based user interface: review and research[J]. Journal of Computer Aided Design and Computer Graphics, 2005, 9(17): 1889-1899.
[7] 尹婷婷,王丹力,戴国忠. 基于多笔划的草图绘制工具[C]//第5届全国普适计算学术会议论文集.北京,中国, 2009: 308-312.YIN Tingting, WANG Danli, DAI Guozhong. A sketch drawing tool based on multiple strokes recognition[C]//Proceedings of the 5th National Conference on Pervasive Computing. Beijing, China, 2009: 308-312.
[8] SEZGIN T, STAHOVICH T, DAVIS R. Sketch based interfaces:early processing for sketch understanding[C]//Proceedings of 2001 Workshop Perceptive User interface. New York, USA, 2001: 1-8.
[9] 李雪峰,周良. 基于自适应顶点探测的手绘草图识别[C]//2008年中国高校通信类院系学术研讨会论文集. 南京, 中国, 2009:315-320.LI Xuefeng, ZHOU Liang. Sketch recognition based on adaptive vertex detection[C]//2008 Proceedings of the Symposium on Communication Colleges and Universities in China. Nanjing, China, 2009: 315-320.
[10] 汪文睿,周良. 基于层次的草图检索框架[J]. 中国制造信息化, 2006, 35(19):78-81.WANG Wenrui, ZHOU Liang. The framework for sketch searching based on hierarchy[J]. Manufacture Information Engineering of China, 2006, 35(19): 78-81.
[11] 孙正兴,徐晓刚,孙建勇,等. 支持方案设计的手绘图形输入工具[J]. 计算机辅助设计与图形学学报, 2003, 15(9): 1145-1152.SUN Zhengxing, XU Xiaogang, SUN Jianyong, et al. Sketch-based graphic input tool for conceptual design[J]. Journal of Computer Aided Design and Computer Graphics, 2003, 15(9): 1145-1152.
[12] 龚健,费广正,石民勇,等. 基于手绘草图轮廓检索的简笔绘图系统[C]//2008系统仿真技术及其应用学术会议论文集. 北京, 中国, 2008: 761-764. GONG Jian, FEI Guangzheng, SHI Minyong, et al. A painting system based on freehand sketch contour retrieval[C]//Proceedings of 2008 Symposium on System Simulation Technology and Its Applications. Beijing, China, 2008: 761-764.
[13] EITZ M, HILDEBRAND K, BOUBEKEUR T, et al. Sketch-based image retrieval: benchmark and bag-of-features descriptors[J]. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(11): 1624-1634.
[14] MARTIN D R, FOWLKES C C, MALIK J. Learning to detect natural image boundaries using local brightness, color, and texture[J]. IEEE Transactions on Pattern Analysis and Machine, 2004, 26(5): 530-549.
[15] RUBINE D. Specifying gestures by example[C]//Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA, 1991: 329-337.
[16] LONG A C, JAMES J, LANDAY A, et al. Visual similarity of pen gestures[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, USA, 2001: 360-367.
[17] HSE H, NEWTON A. Sketched symbol recognition using Zernike moments[C]//Proceedings of the 17th International Conference on Pattern Recognition.[S.l], 2004: 367-370.
[18] 谢强, 冯桂焕, 孙正兴. 基于上下文的在线草图识别方法[J]. 计算机科学, 2007, 34(3): 216-220.XIE Qiang, FENG Guihuan, SUN Zhengxing. Online sketch recognition using contexts[J]. Computer Science, 2007, 34(3): 216-220.
[19] TANDLER P, PRANTE T. Using incremental gesture recognition to provide immediate feedback while drawing pen gestures[C]//Proceedings of ACM Symposium on User Interface Software and Technology. New York, USA, 2001: 18-25.
[20] ARVO J, NOVINS K. Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes[C]//Proceedings of the ACM Symposium on User Interface Software and Technology. New York, USA, 2000: 73-80.
[21] 李俊峰, 张习文, 敖翔,等. 具有实时反馈的草图交互识别方法[J]. 计算机辅助设计与图形学学报, 2005, 17(11): 2453-2458.LI Junfeng, ZHANG Xiwen, AO Xiang, et al. An interactive sketch recognition approach with real time feedback[J]. Journal of Computer Aided Design and Computer Graphics, 2005, 17(11): 2453-2458.
[22] CAO Y, WANG C, ZHANG L, et al. Edgel index for large-scale sketch-based image search[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.[S.l], 2011: 761-768.
[23] ZHOU R, CHEN L, ZHANG L. Guess what you draw: interactive contour-based image retrieval on a million-scale database[C]//Proceedings of the 20th International Conference on Multimedia. New York, USA, 2012: 1343-1344.
[24] REVANKAR S, YEGNANARAYANA B. Machine recognition and correction of freehand geometric line sketches[C]//Proceedings of IEEE International Conference on Systems, Man, and Cybernetics. New York, USA, 1991: 87-92.
[25] SEZGIN T. Feature point detection and curve approximation for early processing of free-hand sketches[D]. Cambridge, USA: Massachusetts Institute of Technology, 2001: 3-74.
[26] YU B, CAIN S. A domain-independent system for sketch recognition[C]//Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques. New York, USA, 2003: 141-146.
[27] PAULSON B, HAMMOND T. Accurate primitive sketch recognition and beautification[C]//Proceedings of Intelligent User Interfaces. New York, USA, 2008: 1-10.
[28] 张莉莎,孙正兴,周若鸿,等. 一种基于图匹配的复杂草图识别方法[J]. 计算机科学, 2004, 31(4): 147-150. ZHANG Lisha, SUN Zhengxing, ZHOU Ruohong, et al. A method of graph-based composite sketchy graphics recognition[J]. Computer Science, 2004, 31(4):147-150.
[29] GROSS M. The electronic cocktail napkin: a computational environment for working with design diagram[J]. Design Studies, 1996, 17(1): 53-69.
[30] CHEN C, XIE S. Freehand drawing system using a fuzzy logic concept[J]. Computer-Aided Design, 1996, 28(2): 77-89.
[31] SCIASIO E, MONGIELLO M. Query by sketch and relevance feedback for content-based image retrieval over the web[J]. Journal of Visual Languages and Computing, 1999, 10(6): 565-584.
[32] CHEE S,PARK K, PARK S. Efficient use of MPEG-7 edge histogram descriptor[J]. ETRI Journal, 2002, 24(1): 35-42.
[33] 李曼舞,孙正兴,顾庆东. 基于轮廓的手绘草图检索研究[J]. 计算机应用, 2005, 25(2): 344-347.LI Manwu, SUN Zhengxing, GU Qingdong. Study on contour-based freehand sketch retrieval[J]. Computer Applications, 2005, 25(2): 344-347.
[34] 高竹红, 汤进, 罗斌. 基于结构图的手绘草图检索[J].计算机技术与发展, 2008, 18(3): 33-35.GAO Zhuhong, TANG Jin, LUO Bin. Sketch retrieval based on structural graph[J]. Computer Technology and Development, 2008, 18(3): 33-35.
[35] 吴明珠. 利用成对几何直方图改进基于轮廓的图像检索[J].电脑知识与技术, 2007, 16(1): 1140-1147.WU Mingzhu. Improvement of contour-based image retrieval using pair wise geometric histogram[J]. Computer Knowledge and Technology, 2007, 16(1): 1140-1147.
[36] 钱晶. 按例和按绘的图像检索研究[D].合肥:中国科学技术大学, 2009: 69-100.QIAN Jing. Research on example and sketch based image retrieval[D]. Hefei: University of Science and Technology of China, 2009: 69-100.
[37] 钱晶, 汪增福. 一种基于规范化形状的手绘草图检索算法[J].中国科学技术大学学报, 2010, 10(40): 1043-1053.QIAN Jing, WANG Zengfu. A method for freehand sketch retrieval based on affine adaptive skeleton[J]. Journal of University of Science and Technology of China, 2010, 10(40): 1043-1053.
[38] LUKAS T. A new framework for online sketch-based image retrieval in web environment[C]//Proceedings of the 11th International Conference on Information Sciences, Signal Processing and their Applications. Montreal, Canada, 2012: 1430-1431.
[39] KENDALL M. A new measure of rank correlation[J]. Biometrika, 1938, 6(30): 81-93.
[40] KENDALL M, GIBBONS J. Rank correlation methods[M]. London: Griffin, 1948: 1-160.
[41] 李彬,梁爽,孙正兴. 基于空间关系的手绘草图检索[J]. 计算机科学, 2005, 32(12): 227-231.LI Bin, LIANG Shuang, SUN Zhengxing. Sketch retrieval based on topological relations[J]. Computer Science, 2005, 32(12): 227-231.
[42] SOUSA P, FONSECA M. Sketch-based retrieval of drawings using topological proximity[C]//Proceedings of the 14th International Conference on Distributed Multimedia Systems. Lisboa, Portugal, 2008: 276-281.
[43] BORGEFORS G. Hierarchical chamfer matching: a parametric edge matching algorithm[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988, 10(6): 849-865.
[44] STENGER B, THAYANANTHAN A, TORR P, et al. Model-based hand tracking using a hierarchical Bayesian filter[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(9):1372-1384.
[45] ROCCHIO J. Relevance feedback in information retrieval[M].[S.l]: Prentice Hall, 1971: 313-323.
[46] 梁爽,孙正兴. 面向草图检索的相关反馈方法[J]. 计算机辅助设计与图形学学报, 2006, 18(11):1753-1757.LIANG Shuang, SUN Zhengxing. BSVM-based relevance feedback for sketch retrieval[J]. Journal of Computer Aided Design and Computer Graphics, 2006, 18(11): 1753-1757.
[47] 袁贞明,金贵朝,张佳. 基于贝叶斯网络的在线草图识别算法[J]. 计算机工程, 2010, 36(5):32-34.YUAN Zhenming, JIN Guichao, ZHANG Jia. Online sketch recognition algorithm based on Bayesian network[J]. Computer Engineering, 2010, 36(5): 32-34.
[48] 裴继红,李翠芸,龚忻. 一种新的隐马尔可夫模型及其在手绘图形识别中的应用[J]. 计算机学报, 2005, 28(10): 1745-1752.PEI Jihong, LI Cuiyun, GONG Xin. A novel hidden Markov model and its application to recognize hand-drawn graphic symbol[J]. Chinese Journal of Computers, 2005, 28(10): 1745-1752.
[49] CAO Y, WANG H, WANG C, et al. MindFinder: interactive sketch-based image search on millions of images[C]//Proceedings of the 18th International Conference on Multimedia. Santiago, USA, 2010: 1605-1608.
[50] SUN X, WANG C, SUD A, et al. MagicBrush: image search by color sketch[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York, USA, 2013: 475-476.
[51] EITZ M, HILDEBRAND K, BOUBEKEUR T, et al. PhotoSketch: a sketch based image query and compositing system[C]//ACM SIGGRAPH Talk Program. New Orleans, USA, 2009: 60.
[52] CHEN T, CHENG M, TAN P, et al. Sketch2Photo: internet image montage[J]. ACM Transactions on Graphics, 2009, 28(5): 789-790.
[53] LEI Y, CHEN Y, CHEN B, et al. Photo search by face positions and facial attributes on touch devices[C]//Proceedings of the 19th International Conference on Multimedia. New York, USA, 2011: 651-654.
[54] EITZ M, RICHTER R, BOUBEKEUR T, et al. Sketch-based shape retrieval[J]. ACM Transactions on Graphics, 2012, 31(4): 31.
[55] 肖秦琨,张楠,高嵩. 一种基于手绘图的商标检索算法[J]. 西安工业大学学报, 2012, 32(5):373-378.XIAO Qinkun, ZHANG Nan, GAO Song. A trademark retrieval algorithm based on sketch[J]. Journal of Xi’an Technological University, 2012, 32(5): 373-378.
[56] 周良, 谢强, 丁秋林. 基于图匹配的工程图纸检索[J]. 南京航空航天大学学报, 2008, 40(3): 354-359.ZHOU Liang, XIE Qiang, DING Qiulin. Engineering drawing retrieval based on graph matching[J]. Journal of Nanjing University of Aeronautics and Astronautics, 2008, 40(3): 354-359.
[57] TENCER L, RENAKOVA M, CHERIET M. Sketch-based retrieval of document illustrations and regions of interest[C]//Proceedings of 12th International Conference on Document Analysis and Recognitio. Washington, DC, USA, 2013: 728-732.
[58] 马月娜,马力克.基于边缘检测的人脸模拟画像检索[J].计算机应用与软件, 2009, 26(8): 238-240.MA Yuena, MA Like. Simulated face portrait retrieval based on edge detection[J]. Computer Applications and Software, 2009, 26(8): 238-240.
[59] WANG C, ZHANG J, YANG B, et al. Sketch 2 Cartoon: composing cartoon images by sketching[C]//Proceedings of the 19th ACM International Conference on Multimedia. New York, USA, 2011: 789-790.
[60] 陈雪锦. 手绘现实:手绘草图的真实感解释[D].合肥:中国科学技术大学, 2008: 88-90.CHEN Xuejin. Sketching reality: realistic interpretation from freehand sketching[D]. Hefei: University of Science and Technology of China, 2008: 88-90.
[61] 刘凯,孙正兴,张尧烨. 采用草绘轮廓的3维人脸建模方法[J]. 中国图像图形学报, 2011, 16(6): 1102-1110.LIU Kai, SUN Zhengxing, ZHANG Yaoye, et al. Method of 3D face modeling from freehand portrait sketch[J]. Journal of Image and Graphics, 2011, 16(6): 1102-1110.
[62] 王丹力,詹志征,戴国忠. 儿童交互式智能讲故事系统[J]. 计算机辅助设计与图形学学报, 2011, 7(23): 1186-1193.WANG Danli, ZHAN Zhizheng, DAI Guozhong. Interactive and intelligent storytelling system for children[J]. Journal of Computer Aided Design and Computer Graphics, 2011, 7(23): 1186-1193.
[63] WANG Y, YU M. Query by sketch: an asymmetric sketch-vs-image retrieval system[C]//Proceedings of the 4th International Congress on Image and Signal Processing. Shanghai, China, 2011: 1368-1372.
[64] SUN X, WANG C, XU C, et al. Indexing billions of images for sketch-based retrieval[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York, USA, 2013: 233-242.
[65] EITZ M, HAYS J, ALEXA M. How do humans sketch object[J]. ACM Transaction on Graphics, 2012, 7(31): 1-10.
[66] SUN Z, WANG C, ZHANG L, et al. Query-adaptive shape topic mining for hand-drawn sketch recognition[C]//Proceedings of the 20th ACM International Conference on Multimedia. New York, USA, 2012: 519-528.
[67] CARLOS A F, ARAJUJO A A, CRUCIANU M, et al. Sketch-finder: efficient and effective sketch-based retrieval for large image collections[C]//Proceedings of the 2013 XXVI Conference on Graphics, Patterns and Images. Arequipa, Peru, 2013: 234-241.
Similar References:

Memo

-

Last Update: 2015-06-15

Copyright © CAAI Transactions on Intelligent Systems