[1]GUO Yinan,WANG Bin,GONG Dunwei,et al.Multi-layer attention knowledge representation learning by integrating entity structure with semantics[J].CAAI Transactions on Intelligent Systems,2023,18(3):577-588.[doi:10.11992/tis.202204026]
Copy

Multi-layer attention knowledge representation learning by integrating entity structure with semantics

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 18 Number of periods: 2023 3 Page number: 577-588 Column: 学术论文—知识工程 Public date: 2023-07-05
Title:
Multi-layer attention knowledge representation learning by integrating entity structure with semantics
Author(s):
GUO Yinan1; 2;  WANG Bin1;  GONG Dunwei1;  YU Zekuan3
1. School of Information and Control Engineering, China University of Mining and Technology, Jiangsu 221116, China;
2. School of Mechatronics and Information Engineering, China University of Mining and Technology (Beijing), Beijing 100083, China;
3. Institute of Engineering and Applied Technology, Fudan University, Shanghai 200433, China
Keywords:
knowledge representation learning; entity structure embedding; semantic information; attention mechanism; knowledge graph; knowledge reasoning; complex entity description; Transformer
CLC:
TP305
DOI:
10.11992/tis.202204026
Abstract:
Though the structure and relationship embedding of entities can be obtained by knowledge representation learning based on the knowledge graph, there lacks of the semantic information utilization of entity description texts. In addition, with the increase of the scale of knowledge graph, the categories and quantities of entities and their relationships, as well as the contents and sources of entity descriptions increase accordingly. It is more difficult to obtain the corresponding relationship between entity text descriptions and triple structure information. Therefore, this paper presents a multi-layer attention knowledge representation learning method, which integrates entity structure with semantics. By constructing a multi-layer attention mechanism, the structural embedding of entities is used to enhance the semantic expression in entity description, and then the semantic relationship of entity description is obtained by Transformer model, and it is enhanced and integrated by structural embedding of relationships. Finally, the integrated semantic relationship is used to enrich and integrate the relationship embedding set. Specifically, a loss function is constructed for a multi-layer attention mechanism, which integrates entity structure with semantic information. The experimental results show that the method proposed in this paper can effectively infer the hidden link relationship between entities containing complex entity descriptions, and has more accurate classification accuracy than the other similar methods in triple classification tasks.
References:
[1] BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. Vancouver: ACM, 2008: 1247–1250.
[2] MILLER G A. WordNet[J]. Communications of the ACM, 1995, 38(11): 39–41.
[3] SUCHANEK F M, KASNECI G, WEIKUM G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th International Conference on World Wide Web. Banff: ACM, 2007: 697–706.
[4] 徐有为, 张宏军, 程恺, 等. 知识图谱嵌入研究综述[J]. 计算机工程与应用, 2022, 58(9): 30–50
XU Youwei, ZHANG Hongjun, CHENG Kai, et al. Comprehensive survey on knowledge graph embedding[J]. Computer engineering and applications, 2022, 58(9): 30–50
[5] 张晓林. 机构知识库的发展趋势与挑战[J]. 现代图书情报技术, 2014(2): 1–7
ZHANG Xiaolin. Trends and challenges for institutional repositories[J]. New technology of library and information service, 2014(2): 1–7
[6] LEHMANN J, ISELE R, JAKOB M, et al. DBpedia–A large-scale, multilingual knowledge base extracted from Wikipedia[J]. Semantic web, 2015, 6(2): 167–195.
[7] HACHEY B, RADFORD W, NOTHMAN J, et al. Evaluating entity linking with wikipedia[J]. Artificial intelligence, 2013, 194: 130–150.
[8] 张明星, 张骁雄, 刘姗姗, 等. 利用知识图谱的推荐系统研究综述[J]. 计算机工程与应用, 2023, 59(4): 30–42
ZHANG Mingxing, ZHANG Xiaoxiong, LIU Shanshan, et al. Review of recommendation systems using knowledge graph[J]. Computer engineering and applications, 2023, 59(4): 30–42
[9] 王智悦, 于清, 王楠, 等. 基于知识图谱的智能问答研究综述[J]. 计算机工程与应用, 2020, 56(23): 1–11
WANG Zhiyue, YU Qing, WANG Nan, et al. Survey of intelligent question answering research based on knowledge graph[J]. Computer engineering and applications, 2020, 56(23): 1–11
[10] LIU Zhiyuan, SUN Maosong, LIN Yankai, et al. Knowledge representation learning: a review[J]. Journal of computer research and development, 2016, 53(2): 247–261.
[11] BORDES A, USUNIER N, GARCIA-DURáN A, et al. Translating embeddings for modeling multi-relational data[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2. Lake Tahoe: ACM, 2013: 2787–2795.
[12] MIKOLOV T, SUTSKEVER I, CHEN Kai, et al. Distributed representations of words and phrases and their compositionality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2. Lake Tahoe: ACM, 2013: 3111–3119.
[13] WANG Zhen, ZHANG Jianwen, FENG Jianlin, et al. Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence. Québec City: ACM, 2014: 1112–1119.
[14] LIN Yankai, LIU Zhiyuan, SUN Maosong, et al. Learning entity and relation embeddings for knowledge graph completion[C]//Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. Austin: ACM, 2015: 2181–2187.
[15] JI Guoliang, HE Shizhu, XU Liheng, et al. Knowledge graph embedding via dynamic mapping matrix[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. Beijing: Association for Computational Linguistics, 2015: 687?696.
[16] JI Guoliang, LIU Kang, HE Shizhu, et al. Knowledge graph completion with adaptive sparse transfer matrix[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix: ACM, 2016: 985–991.
[17] XIAO Han, HUANG Minlie, HAO Yu, et al. TransA: an adaptive approach for knowledge graph embedding[EB/OL]. (2015?09?18)[2022?04?22].https://arxiv.org/abs/1509.05490.
[18] HE Shizhu, LIU Kang, JI Guoliang, et al. Learning to represent knowledge graphs with Gaussian embedding[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. Melbourne: ACM, 2015: 623–632.
[19] XIAO Han, HUANG Minlie, ZHU Xiaoyan. TransG: a generative model for knowledge graph embedding[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Berlin: Association for Computational Linguistics, 2016: 2316?2325.
[20] 饶官军, 古天龙, 常亮, 等. 基于相似性负采样的知识图谱嵌入[J]. 智能系统学报, 2020, 15(2): 218–226
RAO Guanjun, GU Tianlong, CHANG Liang, et al. Knowledge graph embedding based on similarity negative sampling[J]. CAAI transactions on intelligent systems, 2020, 15(2): 218–226
[21] SOCHER R, CHEN Danqi, MANNING C D, et al. Reasoning with neural tensor networks for knowledge base completion[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1. New York: ACM, 2013: 926–934.
[22] SANTOS C N D, XIANG Bing, ZHOU Bowen. Classifying relations by ranking with convolutional neural networks[EB/OL]. (2015?04?24)[2022?04?22].https://arxiv.org/abs/1504.06580.
[23] CHEN Yahui. Convolutional neural network for sentence classification[D]. Master’s thesis: University of Waterloo, 2015.
[24] ZHANG Dongxu, YUAN Bin, WANG Dong, et al. Joint semantic relevance learning with text data and graph knowledge[C]//Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Beijing: Association for Computational Linguistics, 2015: 32?40.
[25] LE Q V, MIKOLOV T. Distributed representations of sentences and documents[EB/OL]. (2014?05?16)[2022?04?22].https://arxiv.org/abs/1405.4053.
[26] XU JIACHENG, CHEN KAN, QIU XIPENG, et al. Knowledge graph representation with jointly structural and textual encoding[EB/OL]. (2016?11?26)[2022?04?22].https://arxiv.org/abs/1611.08661.
[27] XIE Ruobing, LIU Zhiyuan, JIA Jia, et al. Representation learning of knowledge graphs with entity descriptions[C]//Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. Phoenix: ACM, 2016: 2659–2665.
[28] MIKOLOV T, CHEN KAI, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL]. (2013?01?16)[2022?04?22].https://arxiv.org/abs/1301.3781.
[29] WANG Zhigang, LI Juanzi. Text-enhanced representation learning for knowledge graph[C]//Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. New York: ACM, 2016: 1293–1299.
[30] WANG Peng, ZHOU Jing, LIU Yuzhang, et al. TransET: knowledge graph embedding with entity types[J]. Electronics, 2021, 10(12): 1407.
[31] HU Linmei, ZHANG Mengmei, LI Shaohua, et al. Text-graph enhanced knowledge graph representation learning[J]. Frontiers in artificial intelligence, 2021, 4: 697856.
[32] JI Zizheng, LEI Zhengchao, SHEN Tingting, et al. Joint representations of knowledge graphs and textual information via reference sentences[J]. IEICE transactions on information and systems, 2020, E103.D(6): 1362–1370.
[33] CHEN Wenrui, HONG Dongpao, ZHENG Chao. Learning knowledge graph embedding with entity descriptions based on LSTM networks[C]//2020 IEEE International Symposium on Product Compliance Engineering-Asia. Chongqing: IEEE, 2021: 1?7.
[34] ZHAO Feng, XU Tao, JIN L, et al. Convolutional network embedding of text-enhanced representation for knowledge graph completion[J]. IEEE Internet of Things journal, 2021, 8(23): 16758–16769.
[35] WU Jiajun, LI Bohan, JI Ye, et al. Text-enhanced knowledge graph representation model in hyperbolic space[C]//International Conference on Advanced Data Mining and Applications. Cham: Springer, 2022: 137?149.
[36] DO P, PHAM P. W-KG2Vec: a weighted text-enhanced meta-path-based knowledge graph embedding for similarity search[J]. Neural computing and applications, 2021, 33(23): 16533–16555.
[37] ZHENG Wenbo, YAN Lan, GOU Chao, et al. Pay attention to doctor-patient dialogues: multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis[J]. Information fusion, 2021, 75: 168–185.
[38] JIANG Tingting, WANG Hao, LUO Xiangfeng, et al. MIFAS: Multi-sourceheterogeneous information fusion with adaptive importance sampling for link prediction[J]. Expert systems, 2022, 39(4): e12888.
[39] CHEN Junfan, XU Jie, BO Manhui, et al. Augmenting embedding projection with entity descriptions for knowledge graph completion[J]. IEEE access, 2021, 9: 159955–159964.
[40] CHENG Jingwei, ZHANG Fu, YANG Zhi. Knowledge graph representation learning with multi-scale capsule-based embedding model incorporating entity descriptions[J]. IEEE acscess, 2020, 8: 203028–203038.
[41] WANG Yashen, ZHANG Huanhuan, LI Yifeng, et al. Simplified representation learning model based on parameter-sharing for knowledge graph completion[C]// China Conference on Information Retrieval. Cham: Springer, 2019: 67?78.
[42] DEVLIN J, CHANG MING-WEI, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. (2018?10?11)[2022?04?22]. https://arxiv.org/abs/1810.04805.
[43] TOUTANOVA K, CHEN Danqi. Observed versus latent features for knowledge base and text inference[C]//Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality. Beijing: Association for Computational Linguistics, 2015: 57?66.
[44] BOTTOU L. Large-scale machine learning with stochastic gradient descent[C]//Proceedings of COMPSTAT’2010. Paris: SpringerLink, 2010: 177?186.
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems