[1]王宇晖,杜军平,邵蓥侠.基于Transformer与技术词信息的知识产权实体识别方法[J].智能系统学报,2023,18(1):186-193.[doi:10.11992/tis.202203036]
WANG Yuhui,DU Junping,SHAO Yingxia.An intellectual property entity recognition method based on Transformer and technological word information[J].CAAI Transactions on Intelligent Systems,2023,18(1):186-193.[doi:10.11992/tis.202203036]
点击复制
WANG Yuhui,DU Junping,SHAO Yingxia.An intellectual property entity recognition method based on Transformer and technological word information[J].CAAI Transactions on Intelligent Systems,2023,18(1):186-193.[doi:10.11992/tis.202203036]
基于Transformer与技术词信息的知识产权实体识别方法
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
18
期数:
2023年第1期
页码:
186-193
栏目:
吴文俊人工智能科学技术奖论坛
出版日期:
2023-01-05
- Title:
- An intellectual property entity recognition method based on Transformer and technological word information
- Keywords:
- entity recognition named in Chinese; intellectual property; Transformer encoder; information fusion; vector representation; science and technology big data; patent; deep learning
- 分类号:
- TP391
- 摘要:
- 专利文本中包含了大量实体信息，通过命名实体识别可以从中抽取包含关键信息的知识产权实体信息，帮助研究人员更快了解专利内容。现有的命名实体提取方法难以充分利用专业词汇变化带来的词层面的语义信息。本文提出基于Transformer和技术词信息的知识产权实体提取方法，结合BERT语言方法提供精准的字向量表示，并在字向量生成过程中，加入利用字向量经迭代膨胀卷积网络提取的技术词信息，提高对知识产权实体的表征能力。最后使用引入相对位置编码的Transformer编码器，从字向量序列中学习文本的深层语义信息，并实现实体标签预测。在公开数据集和标注的专利数据集的实验结果表明，该方法提升了实体识别的准确性。
- Abstract:
- Patent text contains abundant entity information, from which the intellectual property (IP) entity information containing key information can be extracted through named entity recognition, which helps researchers understand patent content faster. For the existing named entity extraction method, the semantic information at the word level brought by a change in technical words is difficult to fully use. In this paper, the IP entity information extraction method based on Transformer and technical word information is proposed, which provides exact word vector representation based on the BERT language model. In the process of word vector generation, this method improves the representation ability of IP entities by adding the technical word information extracted by iterated dilated convolution neural network. Finally, the Transformer encoder with relative position coding is used to learn the deep semantic information of the text from the word vector sequence, realizing the prediction of the entity label. Experimental results on public and annotated patent datasets show that this method improves entity recognition accuracy.
- 参考文献/References:
-
[1] 杨佳鑫, 杜军平, 邵蓥侠, 等. 面向知识产权的科技资源画像构建方法[J]. 软件学报, 2022, 33(4): 1439–1450
YANG Jiaxin, DU Junping, SHAO Yingxia, et al. Construction method of intellectual-property-oriented scientific and technological resources portrait[J]. Journal of software, 2022, 33(4): 1439–1450
[2] WANG Yuhui, DU Junping, SHAO Yingxia, et al. A patent text classification method based on phrase-context fusion feature[C]//Proceedings of 2021 Chinese Intelligent Automation Conference. Singapore: Springer, 2022: 157-164.
[3] XU Mingying, DU Junping, XUE Zhe, et al. A scientific research topic trend prediction model based on multi-LSTM and graph convolutional network[J]. International journal of intelligent systems, 2022, 37(9): 6331–6353.
[4] KOWSARI K, JAFARI M K, MOJTABA H, et al. Text classification algorithms: A survey[J]. Information, 2019, 10(4): 150.
[5] DEVLIN J, CHANG MING-WEI, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018?10?11)[2022?05?23].https://arxiv.org/abs/1810.04805.
[6] KOU Feifei, DU Junping, HE Yijiang, et al. Social network search based on semantic analysis and learning[J]. CAAI transactions on intelligence technology, 2016, 1(4): 293–302.
[7] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017: 30.
[8] CHEN Hui, LIN Zijia, DING Guiguang, et al. GRN: gated relation network to enhance convolutional neural network for named entity recognition[J]. Proceedings of the AAAI conference on artificial intelligence, 2019, 33(1): 6236–6243.
[9] SUN Bo, DU Junping, GAO Tian. Study on the improvement of K-nearest-neighbor algorithm[C]//2009 International Conference on Artificial Intelligence and Computational Intelligence. Shanghai: IEEE, 2009: 390-393.
[10] CHEN Tianci, LUO Mengfei, FU Hao, et al. Application of NER and association rules to traditional Chinese medicine patent mining[C]//2020 International Conferences on Internet of Things and IEEE Green Computing and Communications and IEEE Cyber, Physical and Social Computing and IEEE Smart Data and IEEE Congress on Cybermatics. Rhodes: IEEE, 2020: 767?772.
[11] XUE Zhe, DU Junping, DU Dawei, et al. Deep low-rank subspace ensemble for multi-view clustering[J]. Information sciences, 2019, 482: 210–227.
[12] FANG Yuke, DENG Weihong, DU Junping, et al. Identity-aware CycleGAN for face photo-sketch synthesis and recognition[J]. Pattern recognition, 2020, 102: 107249.
[13] KRESTEL R, CHIKKAMATH R, HEWEL C, et al. A survey on deep learning for patent analysis[J]. World patent information, 2021, 65: 102035.
[14] WANG Yu, LI Yun, ZHU Ziye, et al. SC-NER: a sequence-to-sequence model with sentence classification for named entity recognition[M]//Advances in Knowledge Discovery and Data Mining. Cham: Springer International Publishing, 2019: 198?209.
[15] SAAD F, ARAS H, HACKL-SOMMER R. Improving named entity recognition for biomedical and patent data using Bi-LSTM deep neural network models[M]//Natural Language Processing and Information Systems. Cham: Springer International Publishing, 2020: 25?36.
[16] ZHAI Zenan, NGUYEN D Q, AKHONDI S A, et al. Improving chemical named entity recognition in patents with contextualized word embeddings[EB/OL]. (2019?07?05) [2022?05?23].https://arxiv.org/abs/1907.02679.
[17] ZHANG Yue, YANG Jie. Chinese NER using lattice LSTM[EB/OL]. (2018?05?05)[2022?05?23].https://arxiv.org/abs/1805.02023.
[18] YAN Xingyu, XIONG Xiaofan, CHENG Xiufeng, et al. HMM-BiMM: hidden Markov model-based word segmentation via improved bi-directional maximal matching algorithm[J]. Computers & electrical engineering, 2021, 94: 107354.
[19] ZHAO Hongke, LIU Qi, ZHU Hengshu, et al. A sequential approach to market state modeling and analysis in online P2P lending[J]. IEEE transactions on systems, man, and cybernetics:systems, 2018, 48(1): 21–33.
[20] ALZAIDY R, CARAGEA C, GILES C L. Bi-LSTM-CRF sequence labeling for keyphrase extraction from scholarly documents[C]//WWW’19: The World Wide Web Conference. New York: ACM, 2019: 2551?2557.
[21] JIN Yanliang, XIE Jinfei, GUO Weisi, et al. LSTM-CRF neural network with gated self attention for Chinese NER[J]. IEEE access, 2019, 7: 136694–136703.
[22] LI Xiaonan, YAN Hang, QIU Xipeng, et al. FLAT: Chinese NER using flat-lattice transformer[EB/OL]. (2020-04-24)[2022-05-23].https://arxiv.org/abs/2004.11795.
[23] DAI Zihang, YANG Zhilin, YANG Yiming, et al. Transforme-xl: Attentive language models beyond a fixed-length context[EB/OL]. (2019?01?09)[2022?05?23].https://arxiv.org/abs/1901.02860.
[24] YAN Hang, DENG Bocao, LI Xiaonan, et al. TENER: adapting transformer encoder for named entity recognition[EB/OL]. (2019-11-10)[2022-05-23].https://arxiv.org/abs/1911.04474.
[25] YIN Xunwei, ZHENG Shuang, WANG Quanmin. Fine-grained Chinese named entity recognition based on RoBERTa-WWM-BiLSTM-CRF model[C]//2021 6th International Conference on Image, Vision and Computing. Qingdao: IEEE, 2021: 408?413.
备注/Memo
收稿日期:2022-03-21。
基金项目:国家重点研发计划项目(2018YFB1402600);国家自然科学基金项目(61772083).
作者简介:王宇晖,硕士研究生,CCF会员,主要研究方向为自然语言处理和数据挖掘;杜军平,教授,CCF会士,主要研究方向为人工智能、机器学习和模式识别。荣获吴文俊人工智能自然科学奖二等奖;邵蓥侠,副教授,CCF高级会员,主要研究方向为大规模图分析、并行计算框架和知识图谱分析
通讯作者:杜军平.E-mail:junpingdu@126.com
更新日期/Last Update:
1900-01-01