[1]郭一楠,王斌,巩敦卫,等.实体结构与语义融合的多层注意力知识表示学习[J].智能系统学报,2023,18(3):577-588.[doi:10.11992/tis.202204026]
GUO Yinan,WANG Bin,GONG Dunwei,et al.Multi-layer attention knowledge representation learning by integrating entity structure with semantics[J].CAAI Transactions on Intelligent Systems,2023,18(3):577-588.[doi:10.11992/tis.202204026]
点击复制
GUO Yinan,WANG Bin,GONG Dunwei,et al.Multi-layer attention knowledge representation learning by integrating entity structure with semantics[J].CAAI Transactions on Intelligent Systems,2023,18(3):577-588.[doi:10.11992/tis.202204026]
实体结构与语义融合的多层注意力知识表示学习
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
18
期数:
2023年第3期
页码:
577-588
栏目:
学术论文—知识工程
出版日期:
2023-07-05
- Title:
- Multi-layer attention knowledge representation learning by integrating entity structure with semantics
- Keywords:
- knowledge representation learning; entity structure embedding; semantic information; attention mechanism; knowledge graph; knowledge reasoning; complex entity description; Transformer
- 分类号:
- TP305
- 摘要:
- 基于知识图谱的知识表示学习虽然可以获得实体的结构和关系嵌入，但是缺少对实体描述文本的语义信息利用。此外，随着知识图谱规模的增长，实体和关系的类别与数量，以及实体描述的内容和来源随之增加，实体的文本描述与三元组结构信息之间的对应关系更加难以获得。基于此，本文提出一种实体结构与语义融合的多层注意力知识表示学习方法，通过构建多层注意力机制，将实体的结构嵌入用于增强实体描述中的语义表达，再通过Transformer模型获取实体描述的语义关系，并采用关系的结构嵌入对其增强和整合，最后利用整合后的语义关系对关系嵌入集合加以丰富和整合。特别是，构建了面向实体结构与语义融合多层注意力机制的损失函数。实验结果表明，本文所提方法能有效推理包含复杂实体描述的实体之间隐藏链路关系，在三元组分类任务中具有比同类方法更准确的分类精度。
- Abstract:
- Though the structure and relationship embedding of entities can be obtained by knowledge representation learning based on the knowledge graph, there lacks of the semantic information utilization of entity description texts. In addition, with the increase of the scale of knowledge graph, the categories and quantities of entities and their relationships, as well as the contents and sources of entity descriptions increase accordingly. It is more difficult to obtain the corresponding relationship between entity text descriptions and triple structure information. Therefore, this paper presents a multi-layer attention knowledge representation learning method, which integrates entity structure with semantics. By constructing a multi-layer attention mechanism, the structural embedding of entities is used to enhance the semantic expression in entity description, and then the semantic relationship of entity description is obtained by Transformer model, and it is enhanced and integrated by structural embedding of relationships. Finally, the integrated semantic relationship is used to enrich and integrate the relationship embedding set. Specifically, a loss function is constructed for a multi-layer attention mechanism, which integrates entity structure with semantic information. The experimental results show that the method proposed in this paper can effectively infer the hidden link relationship between entities containing complex entity descriptions, and has more accurate classification accuracy than the other similar methods in triple classification tasks.
- 参考文献/References:
-
[1] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Vancouver: ACM, 2008: 1247–1250.
[2] MILLER G A. WordNet[J]. Communications of the ACM, 1995, 38(11): 39–41.
[3] SUCHANEK F M, KASNECI G, WEIKUM G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th International Conference on World Wide Web. Banff: ACM, 2007: 697–706.
[4] 徐有为, 张宏军, 程恺, 等. 知识图谱嵌入研究综述[J]. 计算机工程与应用, 2022, 58(9): 30–50
XU Youwei, ZHANG Hongjun, CHENG Kai, et al. Comprehensive survey on knowledge graph embedding[J]. Computer engineering and applications, 2022, 58(9): 30–50
[5] 张晓林. 机构知识库的发展趋势与挑战[J]. 现代图书情报技术, 2014(2): 1–7
ZHANG Xiaolin. Trends and challenges for institutional repositories[J]. New technology of library and information service, 2014(2): 1–7
[6] LEHMANN J, ISELE R, JAKOB M, et al. DBpedia–A large-scale, multilingual knowledge base extracted from Wikipedia[J]. Semantic web, 2015, 6(2): 167–195.
[7] HACHEY B, RADFORD W, NOTHMAN J, et al. Evaluating entity linking with wikipedia[J]. Artificial intelligence, 2013, 194: 130–150.
[8] 张明星, 张骁雄, 刘姗姗, 等. 利用知识图谱的推荐系统研究综述[J]. 计算机工程与应用, 2023, 59(4): 30–42
ZHANG Mingxing, ZHANG Xiaoxiong, LIU Shanshan, et al. Review of recommendation systems using knowledge graph[J]. Computer engineering and applications, 2023, 59(4): 30–42
[9] 王智悦, 于清, 王楠, 等. 基于知识图谱的智能问答研究综述[J]. 计算机工程与应用, 2020, 56(23): 1–11
WANG Zhiyue, YU Qing, WANG Nan, et al. Survey of intelligent question answering research based on knowledge graph[J]. Computer engineering and applications, 2020, 56(23): 1–11
[10] LIU Zhiyuan, SUN Maosong, LIN Yankai, et al. Knowledge representation learning: a review[J]. Journal of computer research and development, 2016, 53(2): 247–261.
[11] BORDES A, USUNIER N, GARCIA-DURáN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2. Lake Tahoe: ACM, 2013: 2787–2795.
[12] MIKOLOV T, SUTSKEVER I, CHEN Kai, et al. Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. Lake Tahoe: ACM, 2013: 3111–3119.
[13] WANG Zhen, ZHANG Jianwen, FENG Jianlin, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence. Québec City: ACM, 2014: 1112–1119.
[14] LIN Yankai, LIU Zhiyuan, SUN Maosong, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin: ACM, 2015: 2181–2187.
[15] JI Guoliang, HE Shizhu, XU Liheng, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing: Association for Computational Linguistics, 2015: 687?696.
[16] JI Guoliang, LIU Kang, HE Shizhu, et al. Knowledge graph completion with adaptive sparse transfer matrix[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix: ACM, 2016: 985–991.
[17] XIAO Han, HUANG Minlie, HAO Yu, et al. TransA: an adaptive approach for knowledge graph embedding[EB/OL]. (2015?09?18)[2022?04?22].https://arxiv.org/abs/1509.05490.
[18] HE Shizhu, LIU Kang, JI Guoliang, et al. Learning to represent knowledge graphs with Gaussian embedding[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. Melbourne: ACM, 2015: 623–632.
[19] XIAO Han, HUANG Minlie, ZHU Xiaoyan. TransG: a generative model for knowledge graph embedding[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin: Association for Computational Linguistics, 2016: 2316?2325.
[20] 饶官军, 古天龙, 常亮, 等. 基于相似性负采样的知识图谱嵌入[J]. 智能系统学报, 2020, 15(2): 218–226
RAO Guanjun, GU Tianlong, CHANG Liang, et al. Knowledge graph embedding based on similarity negative sampling[J]. CAAI transactions on intelligent systems, 2020, 15(2): 218–226
[21] SOCHER R, CHEN Danqi, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1. New York: ACM, 2013: 926–934.
[22] SANTOS C N D, XIANG Bing, ZHOU Bowen. Classifying relations by ranking with convolutional neural networks[EB/OL]. (2015?04?24)[2022?04?22].https://arxiv.org/abs/1504.06580.
[23] CHEN Yahui. Convolutional neural network for sentence classification[D]. Master’s thesis: University of Waterloo, 2015.
[24] ZHANG Dongxu, YUAN Bin, WANG Dong, et al. Joint semantic relevance learning with text data and graph knowledge[C]//Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Beijing: Association for Computational Linguistics, 2015: 32?40.
[25] LE Q V, MIKOLOV T. Distributed representations of sentences and documents[EB/OL]. (2014?05?16)[2022?04?22].https://arxiv.org/abs/1405.4053.
[26] XU JIACHENG, CHEN KAN, QIU XIPENG, et al. Knowledge graph representation with jointly structural and textual encoding[EB/OL]. (2016?11?26)[2022?04?22].https://arxiv.org/abs/1611.08661.
[27] XIE Ruobing, LIU Zhiyuan, JIA Jia, et al. Representation learning of knowledge graphs with entity descriptions[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix: ACM, 2016: 2659–2665.
[28] MIKOLOV T, CHEN KAI, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. (2013?01?16)[2022?04?22].https://arxiv.org/abs/1301.3781.
[29] WANG Zhigang, LI Juanzi. Text-enhanced representation learning for knowledge graph[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. New York: ACM, 2016: 1293–1299.
[30] WANG Peng, ZHOU Jing, LIU Yuzhang, et al. TransET: knowledge graph embedding with entity types[J]. Electronics, 2021, 10(12): 1407.
[31] HU Linmei, ZHANG Mengmei, LI Shaohua, et al. Text-graph enhanced knowledge graph representation learning[J]. Frontiers in artificial intelligence, 2021, 4: 697856.
[32] JI Zizheng, LEI Zhengchao, SHEN Tingting, et al. Joint representations of knowledge graphs and textual information via reference sentences[J]. IEICE transactions on information and systems, 2020, E103.D(6): 1362–1370.
[33] CHEN Wenrui, HONG Dongpao, ZHENG Chao. Learning knowledge graph embedding with entity descriptions based on LSTM networks[C]//2020 IEEE International Symposium on Product Compliance Engineering-Asia. Chongqing: IEEE, 2021: 1?7.
[34] ZHAO Feng, XU Tao, JIN L, et al. Convolutional network embedding of text-enhanced representation for knowledge graph completion[J]. IEEE Internet of Things journal, 2021, 8(23): 16758–16769.
[35] WU Jiajun, LI Bohan, JI Ye, et al. Text-enhanced knowledge graph representation model in hyperbolic space[C]//International Conference on Advanced Data Mining and Applications. Cham: Springer, 2022: 137?149.
[36] DO P, PHAM P. W-KG2Vec: a weighted text-enhanced meta-path-based knowledge graph embedding for similarity search[J]. Neural computing and applications, 2021, 33(23): 16533–16555.
[37] ZHENG Wenbo, YAN Lan, GOU Chao, et al. Pay attention to doctor-patient dialogues: multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis[J]. Information fusion, 2021, 75: 168–185.
[38] JIANG Tingting, WANG Hao, LUO Xiangfeng, et al. MIFAS: Multi-sourceheterogeneous information fusion with adaptive importance sampling for link prediction[J]. Expert systems, 2022, 39(4): e12888.
[39] CHEN Junfan, XU Jie, BO Manhui, et al. Augmenting embedding projection with entity descriptions for knowledge graph completion[J]. IEEE access, 2021, 9: 159955–159964.
[40] CHENG Jingwei, ZHANG Fu, YANG Zhi. Knowledge graph representation learning with multi-scale capsule-based embedding model incorporating entity descriptions[J]. IEEE acscess, 2020, 8: 203028–203038.
[41] WANG Yashen, ZHANG Huanhuan, LI Yifeng, et al. Simplified representation learning model based on parameter-sharing for knowledge graph completion[C]// China Conference on Information Retrieval. Cham: Springer, 2019: 67?78.
[42] DEVLIN J, CHANG MING-WEI, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018?10?11)[2022?04?22]. https://arxiv.org/abs/1810.04805.
[43] TOUTANOVA K, CHEN Danqi. Observed versus latent features for knowledge base and text inference[C]//Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Beijing: Association for Computational Linguistics, 2015: 57?66.
[44] BOTTOU L. Large-scale machine learning with stochastic gradient descent[C]//Proceedings of COMPSTAT’2010. Paris: SpringerLink, 2010: 177?186.
备注/Memo
收稿日期:2022-04-16。
基金项目:国家自然科学基金项目(61973305，52121003)；恒玖（徐州）智能科技有限公司资助项目（2021360001）.
作者简介:郭一楠,教授,主要研究方向为智能数据感知与分析、群智优化与控制。主持国家面上和青年项目3项、国家重点研发计划子课题1项。研究成果获高等学校科学研究优秀成果奖自然科学二等奖、江苏省科学技术二等奖3项,授权发明专利20项,发表包括TEVC、TCYB、TNNLS、TME等在内的中科院一、二区期刊论文43篇,入选ESI前1%高被引论文2篇;王斌,硕士研究生,主要研究方向为知识图谱、自然语言处理。;巩敦卫,教授,主要研究方向为智能优化与控制。主持国家重点研发计划项目1项、国家自然科学基金重点项目1项、面上和青年项目6项、国家重点基础研究发展计划子课题1项、国家重点研发计划子课题1项。研究成果获高等学校科学研究优秀成果奖自然科学二等奖、江苏省科学技术二等奖3项(均排名第1),授权发明专利26项,发表包括IEEETSE、TEVC、TCYB、TNNLS、TASE、TR和ACMTOSEM、ECJ等在内的中科院一、二区期刊论文86篇,入选ESI前1%高被引论文15篇
通讯作者:巩敦卫.E-mail:dwgong@vip.163.com
更新日期/Last Update:
1900-01-01