[1]李海林,邹金串.基于分类词典的文本相似性度量方法[J].智能系统学报,2017,(04):556-562.[doi:10.11992/tis.201608010]
 LI Hailin,ZOU Jinchuan.Text similarity measure method based on classified dictionary[J].CAAI Transactions on Intelligent Systems,2017,(04):556-562.[doi:10.11992/tis.201608010]
点击复制

基于分类词典的文本相似性度量方法(/HTML)
分享到:

《智能系统学报》[ISSN:1673-4785/CN:23-1538/TP]

卷:
期数:
2017年04期
页码:
556-562
栏目:
出版日期:
2017-08-25

文章信息/Info

Title:
Text similarity measure method based on classified dictionary
作者:
李海林1 邹金串2
1. 华侨大学 信息管理系, 福建 泉州 362021;
2. 华侨大学 现代应用统计与大数据研究中心, 福建 厦门 361021
Author(s):
LI Hailin1 ZOU Jinchuan2
1. Department of Information Systems, Huaqiao University, Quanzhou 362021, China;
2. Research Center of Applied Statistics and Big Data, Huaqiao University, Xiamen 361021, China
关键词:
文本挖掘语义分析分类词典关键词提取词语编码相似性度量聚类分类
Keywords:
data miningsemantic analysisclassified dictionarykeywords extractionencodersimilarity measureclusteringclassification
分类号:
TP301
DOI:
10.11992/tis.201608010
摘要:
针对现有基于语义知识规则分析的文本相似性度量方法存在时间复杂度高的局限性,提出基于分类词典的文本相似性度量方法。利用汉语词法分析系统ICTCLAS对文本分词,运用TF×IDF方法提取文本关键词,遍历分类词典获取关键词编码,通过计算文本关键词编码的近似性来衡量原始文本之间的相似度。选取基于语义知识规则和基于统计两个类别的相似性度量方法作为对比方法,通过传统聚类与KNN分类分别对相似性度量方法进行效果验证。数值实验结果表明,新方法在聚类与分类实验中均能取得较好的实验结果,相较于其他基于语义分析的相似性度量方法还具有良好的时间效率。
Abstract:
Existing text-similarity measurement methods based on the semantic knowledge rules analysis have the limitation of high time complexity. In this paper, we propose a text-similarity measurement method based on the Classified Dictionary. First, we segmented texts using the Chinese Lexical Analysis System. Then, we extracted text keywords using the term frequency-inverse document frequency (tf*idf) method and performed keywords coding by traversing the dictionary. By calculating the coding similarity of the text keywords, we can determine the similarity of the original texts. As our two comparison methods, we selected similarity measurement methods based on semantic knowledge rules and statistics. We verified our similarity measurement results using traditional clustering algorithms and the k-nearest neighbors classification method. Our numerical results show that our proposed method can obtain relatively good results in clustering and classification experiments. In addition, compared with other semantic analysis measurement methods, this method has better time efficiency.

参考文献/References:

[1] 李海林,郭韧,万校基.基于特征矩阵的多元时间序列最小距离度量方法[J].智能系统学报, 2015, 10(3):442-447, 2015.LI Hailin, GUO Ren, WAN Xiaoji. A minimum distance measurement method for a multivariate time series based on the feature matrix[J]. CAAI transactions on intelligent systems, 2015, 10(3):442-447.
[2] XU R, WUNSCH D. Survey of clustering algorithms[J]. IEEE transactions on neural networks, 2005, 16(3):645-678.
[3] CHEN Wei, HUO Junge. Judicial determination of malicious forwarding cyber false information[J]. Journal of Chongqing university:social science edition,2017(5):103-113.
[4] 苗传江.HNC(概念层次网络理论)引导[M]. 北京:清华大学出版社,2005.
[5] PARK E K, RA D Y, JANG M G. Techniques for improving web retrieval effectiveness[J]. Information processing and management, 2005, 41(5):1207-1223.
[6] WordNet Documentation[EB/OL].[2010-10-27].http://wordnet.princeton.edu/wordnet/documentation/.
[7] RICHARDSON S D, DOLAN W B.VANDERWENDE L. MindNet:Acquiring and structuring semantic information from text[C]//Proceeding of the 17th International Conference on Computer Linguistics Volume 2.Stroudsburg:Association for Computational Linguistics, 1998:1098-1102.
[8] BAKER C F, FILLMORE C J, LOWE J B. The Berkeley framenet project[C]//Proceeding of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computer Linguistics Volume 1. Stroudsburg:Association for Computational Linguistics, 1998:86-90.
[9] 翟延东,王康平. 一种基于WordNet的短文本语义相似性算法[J]. 电子学报, 2012, 40(3):617-620.ZHAI Yandong, WANG Kangping. An algorithm for semantic similarity of short text based on WordNet[J]. Acta electronica sinica, 2012, 40(3):617-620.
[10] 梅家驹,竺一鸣,高蕴琦,等.同义词词林[M].上海:上海辞书出版社,1996.
[11] 董振东,董强. 知网简[EB/OL].http://www.keenage.com.
[12] 刘群,李素建. 基于"知网"的词汇语义相似度计算[C]//第三届汉语词汇语义学研究会论文集.台北,中国, 2002:59-76.
[13] 林丽,薛方,任仲晟. 一种改进的基于知网的词语相似度计算方法[J].计算机应用,2009, 29(1):217-220.LIN Li, XUE Fang, REN Zhongsheng. Modified word similarity computation approach based on Hownet[J]. Journal of computer applications, 2009, 29(1):217-220.
[14] 王小林,杨林,王东. 基于知网的新词语相似度算法研究[J]. 情报科学, 2015, 33(2):67-71.WANG Xiaolin, YANG Lin, WANG Dong. New word similarity algorithm research based on HowNet[J]. Information science, 2015, 33(2):67-71.
[15] 张亮,尹存燕.基于语义树的中文词语相似度计算与分析[J]. 中文信息学报, 2007, 21(3):99-105.ZHANG Liang, YIN Cunyan. Chinese word similarity computing based on semantic tree[J]. Journal of Chinese information processing, 2007, 21(3):99-105.
[16] 田久乐,赵蔚. 基于同义词词林的词语相似度计算方法[J]. 吉林大学学报:信息科学版,2010, 26(6):602-608.TIAN Jiule, ZHAO Wei. Word similarity algorithm based on Yongyici Cilin in Semantic Web adaptive learning system[J]. Journal of Jilin university:information science edition, 2010, 26(6):602-608.
[17] 徐庆,段利国.基于实体语义相似度的中文实体关系抽取[J]. 山东大学学报:工学版, 2015, 45(6):7-14.XU Qing, DUAN Liguo. Chinese entity relation extraction based on entity semantic similarity[J]. Journal of Shandong university:engineering science, 2015, 45(6):7-14.
[18] 郑红艳,张东站.基于同义词词林的文本特征选择方法[J]. 厦门大学学报:自然科学版, 2012, 5(2):200-203.ZHENG Hongyan, ZHANG Dongzhan. Atext featureselectionmethodbasedonTongYiCiCiLin[J].Journal of Xiamen University:Natural Science, 2012, 5(2):200-203.
[19] 苏新春.现代汉语分类词典[M]. 上海:商务印书馆, 2013.
[20] SALTON G. The transformation analysis and retrival of information by computer[M]. Wesley Reading Massach-uetts, 1989.
[21] FREY B J, DUECK D. Clustering by passing messages between data points[J]. Science, 2007, 315(5814):972-976.
[22] FORGY E W. Cluster analysis of multivariate data:efficiency versus interpretability of classifications[J]. Biometric, 1965, 21:768-769.
[23] 丁世飞,贾洪杰.基于自适应Nystrom采样的大数据谱聚类算法[J]. 软件学报, 2014, 25(9):2037-2049.DING Shifei, JIA Hongjie. Spectral clustering algorit-hm based on adaptive nystrom sampling for big data analysis[J]. Journal of software, 2014, 25(9):2037-2049.
[24] WU Xindong, KUMAR V, QUINLAN J R, et al. Top 10 algorit-hms in data mining[J]. Knowledge and information systems, 2008, 14(1):1-37.
[25] 搜狗实验室语料[EB/OL]. http://www.sogou.com/labs/resource/list_yuliao.php.

备注/Memo

备注/Memo:
收稿日期:2016-08-30。
基金项目:国家自然科学基金项目(61300139);福建省自然科学基金项目(2015J01581);华侨大学中青年教师科研提升计划项目(ZQN-PY220);华侨大学研究生科研创新能力培育计划项目(1511307006).
作者简介:李海林,男,1982年生,副教授,博士,主要研究方向为数据挖掘与决策支持,主持国家自然科学基金1项和省部级基金2项,发表学术论文40余篇,其中被SCI检索11篇,EI检索20余篇;邹金串,女,1993年生,硕士研究生,主要研究方向为文本挖掘。
通讯作者:邹金串,E-mail:Zou_jinchuan@163.com.
更新日期/Last Update: 2017-08-25