[1]辛雨璇,闫子飞.基于手绘草图的图像检索技术研究进展[J].智能系统学报,2015,10(2):167-177.[doi:10.3969/j.issn.1673-4785.201401045]
XIN Yuxuan,YAN Zifei.Research progress of image retrieval based on hand-drawn sketches[J].CAAI Transactions on Intelligent Systems,2015,10(2):167-177.[doi:10.3969/j.issn.1673-4785.201401045]
点击复制
XIN Yuxuan,YAN Zifei.Research progress of image retrieval based on hand-drawn sketches[J].CAAI Transactions on Intelligent Systems,2015,10(2):167-177.[doi:10.3969/j.issn.1673-4785.201401045]
基于手绘草图的图像检索技术研究进展
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
10
期数:
2015年第2期
页码:
167-177
栏目:
综述
出版日期:
2015-04-25
- Title:
- Research progress of image retrieval based on hand-drawn sketches
- Keywords:
- hand-drawn sketches; feature interpretation; image retrieval; human-computer interaction; big data
- 分类号:
- TP391.41
- 文献标志码:
- A
- 摘要:
- 通过对现阶段基于草图的图像检索相关技术的研究,尝试构建了领域无关的基于手绘草图的图像检索系统框架,并分别对手绘草图预处理、草图特征表示、草图匹配及图像反馈等系统阶段所涉及的相关技术及其发展进行梳理,进而对基于手绘草图的图像检索系统的相关应用进行总结,展望了手绘草图检索系统在自然人机交互、普适计算、大数据背景下的研究趋势。
- Abstract:
- This paper build a framework for the domain-independent image retrieval system based on hand-drawn sketches by researching the existing sketch-based image retrieval related technologies. The relevant technologies and their development of the involved system stages such as: hand-drawn sketch preprocessing, sketch feature interpretation, sketch matching, and image feedback were also outlined. Applications relating to sketch-based image retrieval were summarized. Finally, the coming trends of the hand-drawn sketch-based image retrieval system under the background of natural user interface, pervasive computing and big data were forecasted.
- 参考文献/References:
-
[1] RUI Y, THOMAS S, CHANG S. Image retrieval: past, present and future[J]. Journal of Visual Communication and Image Representation, 1999, 10(1): 39-62.
[2] EAKINS J. Automatic image content retrieval-Are we getting anywhere[C]//Proceedings of 3rd International Conference on Electronic Library and Visual Information Research. Newcastle, United Kingdom, 1996: 123-135.
[3] VELTKAMP R C, TANASE M. Content-based image retrieval systems: a survey. UU-CS-2000-34[R]. Dept of Computing Science, Utrecht University, 2000.
[4] NIBLACK C, BARBER R, EQUITZ W, et al. The QBIC project: querying images by content using color, texture, and shape[C]//Proceedings of SPIE Electronic Imaging: Science and Technology. San Jose, Canada, 1993: 173-187.
[5] PENTLAND A, PICARD R W, SCLAROFF S. Photobook: content-based manipulation of image databases[J]. International Journal of Computer Vision, 1996, 18(3): 233-254.
[6] 孙正兴,冯桂焕,周若鸿.基于草图的人机交互技术研究进展[J]. 计算机辅助设计与图形学学报, 2005, 9(17): 1889-1899.SUN Zhengxing, FENG Guihuan, ZHOU Ruohong. Techniques for sketch-based user interface: review and research[J]. Journal of Computer Aided Design and Computer Graphics, 2005, 9(17): 1889-1899.
[7] 尹婷婷,王丹力,戴国忠. 基于多笔划的草图绘制工具[C]//第5届全国普适计算学术会议论文集.北京,中国, 2009: 308-312.YIN Tingting, WANG Danli, DAI Guozhong. A sketch drawing tool based on multiple strokes recognition[C]//Proceedings of the 5th National Conference on Pervasive Computing. Beijing, China, 2009: 308-312.
[8] SEZGIN T, STAHOVICH T, DAVIS R. Sketch based interfaces:early processing for sketch understanding[C]//Proceedings of 2001 Workshop Perceptive User interface. New York, USA, 2001: 1-8.
[9] 李雪峰,周良. 基于自适应顶点探测的手绘草图识别[C]//2008年中国高校通信类院系学术研讨会论文集. 南京, 中国, 2009:315-320.LI Xuefeng, ZHOU Liang. Sketch recognition based on adaptive vertex detection[C]//2008 Proceedings of the Symposium on Communication Colleges and Universities in China. Nanjing, China, 2009: 315-320.
[10] 汪文睿,周良. 基于层次的草图检索框架[J]. 中国制造信息化, 2006, 35(19):78-81.WANG Wenrui, ZHOU Liang. The framework for sketch searching based on hierarchy[J]. Manufacture Information Engineering of China, 2006, 35(19): 78-81.
[11] 孙正兴,徐晓刚,孙建勇,等. 支持方案设计的手绘图形输入工具[J]. 计算机辅助设计与图形学学报, 2003, 15(9): 1145-1152.SUN Zhengxing, XU Xiaogang, SUN Jianyong, et al. Sketch-based graphic input tool for conceptual design[J]. Journal of Computer Aided Design and Computer Graphics, 2003, 15(9): 1145-1152.
[12] 龚健,费广正,石民勇,等. 基于手绘草图轮廓检索的简笔绘图系统[C]//2008系统仿真技术及其应用学术会议论文集. 北京, 中国, 2008: 761-764. GONG Jian, FEI Guangzheng, SHI Minyong, et al. A painting system based on freehand sketch contour retrieval[C]//Proceedings of 2008 Symposium on System Simulation Technology and Its Applications. Beijing, China, 2008: 761-764.
[13] EITZ M, HILDEBRAND K, BOUBEKEUR T, et al. Sketch-based image retrieval: benchmark and bag-of-features descriptors[J]. IEEE Transactions on Visualization and Computer Graphics, 2011, 17(11): 1624-1634.
[14] MARTIN D R, FOWLKES C C, MALIK J. Learning to detect natural image boundaries using local brightness, color, and texture[J]. IEEE Transactions on Pattern Analysis and Machine, 2004, 26(5): 530-549.
[15] RUBINE D. Specifying gestures by example[C]//Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques. New York, USA, 1991: 329-337.
[16] LONG A C, JAMES J, LANDAY A, et al. Visual similarity of pen gestures[C]//Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, USA, 2001: 360-367.
[17] HSE H, NEWTON A. Sketched symbol recognition using Zernike moments[C]//Proceedings of the 17th International Conference on Pattern Recognition.[S.l], 2004: 367-370.
[18] 谢强, 冯桂焕, 孙正兴. 基于上下文的在线草图识别方法[J]. 计算机科学, 2007, 34(3): 216-220.XIE Qiang, FENG Guihuan, SUN Zhengxing. Online sketch recognition using contexts[J]. Computer Science, 2007, 34(3): 216-220.
[19] TANDLER P, PRANTE T. Using incremental gesture recognition to provide immediate feedback while drawing pen gestures[C]//Proceedings of ACM Symposium on User Interface Software and Technology. New York, USA, 2001: 18-25.
[20] ARVO J, NOVINS K. Fluid sketches: continuous recognition and morphing of simple hand-drawn shapes[C]//Proceedings of the ACM Symposium on User Interface Software and Technology. New York, USA, 2000: 73-80.
[21] 李俊峰, 张习文, 敖翔,等. 具有实时反馈的草图交互识别方法[J]. 计算机辅助设计与图形学学报, 2005, 17(11): 2453-2458.LI Junfeng, ZHANG Xiwen, AO Xiang, et al. An interactive sketch recognition approach with real time feedback[J]. Journal of Computer Aided Design and Computer Graphics, 2005, 17(11): 2453-2458.
[22] CAO Y, WANG C, ZHANG L, et al. Edgel index for large-scale sketch-based image search[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.[S.l], 2011: 761-768.
[23] ZHOU R, CHEN L, ZHANG L. Guess what you draw: interactive contour-based image retrieval on a million-scale database[C]//Proceedings of the 20th International Conference on Multimedia. New York, USA, 2012: 1343-1344.
[24] REVANKAR S, YEGNANARAYANA B. Machine recognition and correction of freehand geometric line sketches[C]//Proceedings of IEEE International Conference on Systems, Man, and Cybernetics. New York, USA, 1991: 87-92.
[25] SEZGIN T. Feature point detection and curve approximation for early processing of free-hand sketches[D]. Cambridge, USA: Massachusetts Institute of Technology, 2001: 3-74.
[26] YU B, CAIN S. A domain-independent system for sketch recognition[C]//Proceedings of the 1st International Conference on Computer Graphics and Interactive Techniques. New York, USA, 2003: 141-146.
[27] PAULSON B, HAMMOND T. Accurate primitive sketch recognition and beautification[C]//Proceedings of Intelligent User Interfaces. New York, USA, 2008: 1-10.
[28] 张莉莎,孙正兴,周若鸿,等. 一种基于图匹配的复杂草图识别方法[J]. 计算机科学, 2004, 31(4): 147-150. ZHANG Lisha, SUN Zhengxing, ZHOU Ruohong, et al. A method of graph-based composite sketchy graphics recognition[J]. Computer Science, 2004, 31(4):147-150.
[29] GROSS M. The electronic cocktail napkin: a computational environment for working with design diagram[J]. Design Studies, 1996, 17(1): 53-69.
[30] CHEN C, XIE S. Freehand drawing system using a fuzzy logic concept[J]. Computer-Aided Design, 1996, 28(2): 77-89.
[31] SCIASIO E, MONGIELLO M. Query by sketch and relevance feedback for content-based image retrieval over the web[J]. Journal of Visual Languages and Computing, 1999, 10(6): 565-584.
[32] CHEE S,PARK K, PARK S. Efficient use of MPEG-7 edge histogram descriptor[J]. ETRI Journal, 2002, 24(1): 35-42.
[33] 李曼舞,孙正兴,顾庆东. 基于轮廓的手绘草图检索研究[J]. 计算机应用, 2005, 25(2): 344-347.LI Manwu, SUN Zhengxing, GU Qingdong. Study on contour-based freehand sketch retrieval[J]. Computer Applications, 2005, 25(2): 344-347.
[34] 高竹红, 汤进, 罗斌. 基于结构图的手绘草图检索[J].计算机技术与发展, 2008, 18(3): 33-35.GAO Zhuhong, TANG Jin, LUO Bin. Sketch retrieval based on structural graph[J]. Computer Technology and Development, 2008, 18(3): 33-35.
[35] 吴明珠. 利用成对几何直方图改进基于轮廓的图像检索[J].电脑知识与技术, 2007, 16(1): 1140-1147.WU Mingzhu. Improvement of contour-based image retrieval using pair wise geometric histogram[J]. Computer Knowledge and Technology, 2007, 16(1): 1140-1147.
[36] 钱晶. 按例和按绘的图像检索研究[D].合肥:中国科学技术大学, 2009: 69-100.QIAN Jing. Research on example and sketch based image retrieval[D]. Hefei: University of Science and Technology of China, 2009: 69-100.
[37] 钱晶, 汪增福. 一种基于规范化形状的手绘草图检索算法[J].中国科学技术大学学报, 2010, 10(40): 1043-1053.QIAN Jing, WANG Zengfu. A method for freehand sketch retrieval based on affine adaptive skeleton[J]. Journal of University of Science and Technology of China, 2010, 10(40): 1043-1053.
[38] LUKAS T. A new framework for online sketch-based image retrieval in web environment[C]//Proceedings of the 11th International Conference on Information Sciences, Signal Processing and their Applications. Montreal, Canada, 2012: 1430-1431.
[39] KENDALL M. A new measure of rank correlation[J]. Biometrika, 1938, 6(30): 81-93.
[40] KENDALL M, GIBBONS J. Rank correlation methods[M]. London: Griffin, 1948: 1-160.
[41] 李彬,梁爽,孙正兴. 基于空间关系的手绘草图检索[J]. 计算机科学, 2005, 32(12): 227-231.LI Bin, LIANG Shuang, SUN Zhengxing. Sketch retrieval based on topological relations[J]. Computer Science, 2005, 32(12): 227-231.
[42] SOUSA P, FONSECA M. Sketch-based retrieval of drawings using topological proximity[C]//Proceedings of the 14th International Conference on Distributed Multimedia Systems. Lisboa, Portugal, 2008: 276-281.
[43] BORGEFORS G. Hierarchical chamfer matching: a parametric edge matching algorithm[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1988, 10(6): 849-865.
[44] STENGER B, THAYANANTHAN A, TORR P, et al. Model-based hand tracking using a hierarchical Bayesian filter[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(9):1372-1384.
[45] ROCCHIO J. Relevance feedback in information retrieval[M].[S.l]: Prentice Hall, 1971: 313-323.
[46] 梁爽,孙正兴. 面向草图检索的相关反馈方法[J]. 计算机辅助设计与图形学学报, 2006, 18(11):1753-1757.LIANG Shuang, SUN Zhengxing. BSVM-based relevance feedback for sketch retrieval[J]. Journal of Computer Aided Design and Computer Graphics, 2006, 18(11): 1753-1757.
[47] 袁贞明,金贵朝,张佳. 基于贝叶斯网络的在线草图识别算法[J]. 计算机工程, 2010, 36(5):32-34.YUAN Zhenming, JIN Guichao, ZHANG Jia. Online sketch recognition algorithm based on Bayesian network[J]. Computer Engineering, 2010, 36(5): 32-34.
[48] 裴继红,李翠芸,龚忻. 一种新的隐马尔可夫模型及其在手绘图形识别中的应用[J]. 计算机学报, 2005, 28(10): 1745-1752.PEI Jihong, LI Cuiyun, GONG Xin. A novel hidden Markov model and its application to recognize hand-drawn graphic symbol[J]. Chinese Journal of Computers, 2005, 28(10): 1745-1752.
[49] CAO Y, WANG H, WANG C, et al. MindFinder: interactive sketch-based image search on millions of images[C]//Proceedings of the 18th International Conference on Multimedia. Santiago, USA, 2010: 1605-1608.
[50] SUN X, WANG C, SUD A, et al. MagicBrush: image search by color sketch[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York, USA, 2013: 475-476.
[51] EITZ M, HILDEBRAND K, BOUBEKEUR T, et al. PhotoSketch: a sketch based image query and compositing system[C]//ACM SIGGRAPH Talk Program. New Orleans, USA, 2009: 60.
[52] CHEN T, CHENG M, TAN P, et al. Sketch2Photo: internet image montage[J]. ACM Transactions on Graphics, 2009, 28(5): 789-790.
[53] LEI Y, CHEN Y, CHEN B, et al. Photo search by face positions and facial attributes on touch devices[C]//Proceedings of the 19th International Conference on Multimedia. New York, USA, 2011: 651-654.
[54] EITZ M, RICHTER R, BOUBEKEUR T, et al. Sketch-based shape retrieval[J]. ACM Transactions on Graphics, 2012, 31(4): 31.
[55] 肖秦琨,张楠,高嵩. 一种基于手绘图的商标检索算法[J]. 西安工业大学学报, 2012, 32(5):373-378.XIAO Qinkun, ZHANG Nan, GAO Song. A trademark retrieval algorithm based on sketch[J]. Journal of Xi’an Technological University, 2012, 32(5): 373-378.
[56] 周良, 谢强, 丁秋林. 基于图匹配的工程图纸检索[J]. 南京航空航天大学学报, 2008, 40(3): 354-359.ZHOU Liang, XIE Qiang, DING Qiulin. Engineering drawing retrieval based on graph matching[J]. Journal of Nanjing University of Aeronautics and Astronautics, 2008, 40(3): 354-359.
[57] TENCER L, RENAKOVA M, CHERIET M. Sketch-based retrieval of document illustrations and regions of interest[C]//Proceedings of 12th International Conference on Document Analysis and Recognitio. Washington, DC, USA, 2013: 728-732.
[58] 马月娜,马力克.基于边缘检测的人脸模拟画像检索[J].计算机应用与软件, 2009, 26(8): 238-240.MA Yuena, MA Like. Simulated face portrait retrieval based on edge detection[J]. Computer Applications and Software, 2009, 26(8): 238-240.
[59] WANG C, ZHANG J, YANG B, et al. Sketch 2 Cartoon: composing cartoon images by sketching[C]//Proceedings of the 19th ACM International Conference on Multimedia. New York, USA, 2011: 789-790.
[60] 陈雪锦. 手绘现实:手绘草图的真实感解释[D].合肥:中国科学技术大学, 2008: 88-90.CHEN Xuejin. Sketching reality: realistic interpretation from freehand sketching[D]. Hefei: University of Science and Technology of China, 2008: 88-90.
[61] 刘凯,孙正兴,张尧烨. 采用草绘轮廓的3维人脸建模方法[J]. 中国图像图形学报, 2011, 16(6): 1102-1110.LIU Kai, SUN Zhengxing, ZHANG Yaoye, et al. Method of 3D face modeling from freehand portrait sketch[J]. Journal of Image and Graphics, 2011, 16(6): 1102-1110.
[62] 王丹力,詹志征,戴国忠. 儿童交互式智能讲故事系统[J]. 计算机辅助设计与图形学学报, 2011, 7(23): 1186-1193.WANG Danli, ZHAN Zhizheng, DAI Guozhong. Interactive and intelligent storytelling system for children[J]. Journal of Computer Aided Design and Computer Graphics, 2011, 7(23): 1186-1193.
[63] WANG Y, YU M. Query by sketch: an asymmetric sketch-vs-image retrieval system[C]//Proceedings of the 4th International Congress on Image and Signal Processing. Shanghai, China, 2011: 1368-1372.
[64] SUN X, WANG C, XU C, et al. Indexing billions of images for sketch-based retrieval[C]//Proceedings of the 21st ACM International Conference on Multimedia. New York, USA, 2013: 233-242.
[65] EITZ M, HAYS J, ALEXA M. How do humans sketch object[J]. ACM Transaction on Graphics, 2012, 7(31): 1-10.
[66] SUN Z, WANG C, ZHANG L, et al. Query-adaptive shape topic mining for hand-drawn sketch recognition[C]//Proceedings of the 20th ACM International Conference on Multimedia. New York, USA, 2012: 519-528.
[67] CARLOS A F, ARAJUJO A A, CRUCIANU M, et al. Sketch-finder: efficient and effective sketch-based retrieval for large image collections[C]//Proceedings of the 2013 XXVI Conference on Graphics, Patterns and Images. Arequipa, Peru, 2013: 234-241.
- 相似文献/References:
-
[1]李海林,梁叶.分段聚合近似和数值导数的动态时间弯曲方法[J].智能系统学报,2016,11(2):249.[doi:10.11992/tis.201507064]
LI Hailin,LIANG Ye.Dynamic time warping based on piecewise aggregate approximation and data derivatives[J].CAAI Transactions on Intelligent Systems,2016,11():249.[doi:10.11992/tis.201507064]
[2]王丽娟,丁世飞.一种基于ELM-AE特征表示的谱聚类算法[J].智能系统学报,2021,16(3):560.[doi:10.11992/tis.202005021]
WANG Lijuan,DING Shifei.A spectral clustering algorithm based on ELM-AE feature representation[J].CAAI Transactions on Intelligent Systems,2021,16():560.[doi:10.11992/tis.202005021]
[3]杨梦茵,陈俊芬,翟俊海.非对称卷积编码器的聚类算法[J].智能系统学报,2022,17(5):900.[doi:10.11992/tis.202107021]
YANG Mengyin,CHEN Junfen,ZHAI Junhai.A clustering method based on the asymmetric convolutional autoencoder[J].CAAI Transactions on Intelligent Systems,2022,17():900.[doi:10.11992/tis.202107021]
备注/Memo
收稿日期:2014-1-22;改回日期:。
基金项目:国家自然科学基金资助项目(61102037);哈尔滨工业大学科研创新基金资助项目(HIT. NSRIF. 2015057).
作者简介:辛雨璇,女,1988年生,硕士研究生,主要研究方向为多媒体技术与人机交互技术;闫子飞,女,1980年生,讲师,博士。主要研究方向为图像处理、模式识别、非物质文化遗产的数字化保护,基于手绘草图的认知界面设计等。主持和参与多项国家自然科学基金项目;发表学术论文10余篇,其中,被SCI检索1篇,EI检索9篇。
通讯作者:闫子飞.E-mail:cszfyan@gmail.com.
更新日期/Last Update:
2015-06-15