[1]杜永萍,赵以梁,阎婧雅,等.基于深度学习的机器阅读理解研究综述[J].智能系统学报,2022,17(6):1074-1083.[doi:10.11992/tis.202107024]
DU Yongping,ZHAO Yiliang,YAN Jingya,et al.Survey of machine reading comprehension based on deep learning[J].CAAI Transactions on Intelligent Systems,2022,17(6):1074-1083.[doi:10.11992/tis.202107024]
点击复制
DU Yongping,ZHAO Yiliang,YAN Jingya,et al.Survey of machine reading comprehension based on deep learning[J].CAAI Transactions on Intelligent Systems,2022,17(6):1074-1083.[doi:10.11992/tis.202107024]
基于深度学习的机器阅读理解研究综述
《智能系统学报》[ISSN 1673-4785/CN 23-1538/TP] 卷:
17
期数:
2022年第6期
页码:
1074-1083
栏目:
综述
出版日期:
2022-11-05
- Title:
- Survey of machine reading comprehension based on deep learning
- Keywords:
- machine reading comprehension; natural language processing; deep learning; neural network; end-to-end model; knowledge reasoning; pretrained language model; artificial intelligence
- 分类号:
- TP391
- 文献标志码:
- 2022-10-18
- 摘要:
- 机器阅读理解任务在近年来备受关注，它赋予计算机从文本数据中获取知识和回答问题的能力。如何让机器理解自然语言是人工智能领域长期存在的挑战之一，近年来大规模高质量数据集的发布和深度学习技术的运用，使得机器阅读理解取得了快速发展。基于神经网络的端到端的模型结构，基于预训练语言模型以及推理技术的应用，其性能在大规模评测数据集上有很大提升，但距离真正的理解语言还有较大差距。本文对机器阅读理解任务的研究现状与发展趋势进行了综述，主要包括任务划分、机器阅读理解模型与相关技术的分析，特别是基于知识推理的机器阅读理解技术，总结并讨论了该领域的发展趋势。
- Abstract:
- In recent years, there has been a great deal of interest in the task of machine reading comprehension. It enables computers to learn and answer questions based on text input. One of the long-standing challenges in the field of artificial intelligence is how to make machines understand natural language. In recent years, machine reading comprehension has advanced rapidly as a result of the large-scale release of high-quality data sets and the application of deep learning technology. The use of an end-to-end model structure based on neural networks, a pre-trained language model, and reasoning technology has greatly improved their performance on large-scale evaluation data sets. However, there is still a big gap in real language understanding. This paper summarizes the research status and development trend of machine reading comprehension tasks, including division of tasks, analysis of machine reading comprehension model and related technologies, particularly machine reading comprehension technology based on knowledge reasoning, and finally discusses the development trend in this field.
- 参考文献/References:
-
[1] HILL F, BORDES A, CHOPRA S, et al. The goldilocks principle: reading children’s books with explicit memory representations[C]//Proceedings of the 4th International Conference on Learning Representations. San Juan: OpenReview, 2016: 1-13.
[2] RAJPURKAR P, ZHANG J, LOPYREV K, et al. SQuAD: 100, 000+ questions for machine comprehension of text[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin: ACL, 2016: 2383–2392.
[3] JOSHI M, CHOI E, WELD D, et al. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: ACL, 2017: 1601–1611.
[4] SEO M, KEMBHAVI A, FARHADI A, et al. Bidirectional attention flow for machine comprehension[C]//5th International Conference on Learning Representations. Toulon: OpenReview, 2017: 1-12.
[5] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017(30): 5998–6008.
[6] WANG W H, YANG N, WEI F R, et al. Gated Self-matching networks for reading comprehension and question answering[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouve: ACL, 2017: 189–198.
[7] 梁小波, 任飞亮, 刘永康, 等. N-Reader: 基于双层Self-attention的机器阅读理解模型[J]. 中文信息学报, 2018, 32(10): 130–137
LIANG Xiaobo, REN Feiliang, LIU Yongkang, et al. N-reader: machine reading comprehension model based on double layers of self-attention[J]. Journal of Chinese information processing, 2018, 32(10): 130–137
[8] HE Wei, LIU Kai, LIU Jing, et al. DuReader: a Chinese machine reading comprehension dataset from real-world applications[C]//Proceedings of the Workshop on Machine Reading for Question Answering. Melbourne: ACL, 2018: 37–46.
[9] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: ACL, 2019: 4171–4186.
[10] YANG Zhilin, DAI Zihang, YANG Yiming, et al. XLNet: generalized autoregressive pretraining for language understanding[C]//Annual Conference on Neural Information Processing Systems. Vancouver: ACM, 2019: 1-18.
[11] DING M, ZHOU C, CHEN Q, et al. Cognitive graph for multi-hop reading comprehension at scale[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: ACL, 2019: 2694–2703.
[12] YASUNAGA M, RWN H, BOSSELUT A, et al. QA-GNN: reasoning with language models and knowledge graphs for question answering[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 2021: 535–546.
[13] YAGCIOGLU S, ERDEM A, ERDEM E, et al. RecipeQA: a challenge dataset for multimodal comprehension of cooking recipes[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 1358–1368.
[14] KEMBHAVI A, SEO M, SCHWENK D, et al. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5376–5384.
[15] HERMANN K M, KO?ISK? T, GREFENSTETTE E, et al. Teaching machines to read and comprehend[C]//Advances in Neural Information Processing Systems. Montréal: MIT Press, 2015: 1693–1701.
[16] WELBL J, STENETORP P, RIEDEL S. Constructing datasets for multi-hop reading comprehension across documents[J]. Transactions of the association for computational linguistics, 2018, 6(1): 287–302.
[17] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: a question answering challenge targeting commonsense knowledge[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: NAACL, 2019: 4149–4158.
[18] TRISCHLER A, WANG T, YUAN X, et al. NewsQA: a machine comprehension dataset[C]//Proceedings of the 2nd Workshop on Representation Learning for NLP. Vancouver: ACL, 2017: 191–200.
[19] DUA D, WANG Y, DASIGI P, et al. DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: NAACL, 2019: 2368–2378.
[20] YU A W, DOHAN D, LUONG M T, et al. QANet: combining local convolution with global self-attention for reading comprehension[EB/OL]. (2018-04-23)[ 2021-07-13].https://arxiv.org/abs/1804.09541.
[21] PENNINGTON J, SOCHER R, MANNING CD. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Qatar: ACL, 2014: 1532–1543.
[22] 郑玉昆, 李丹, 范臻, 等. T-Reader: 一种基于自注意力机制的多任务深度阅读理解模型[J]. 中文信息学报, 2018, 32(11): 128–134
ZHENG Yukun, LI Dan, FAN Zheng, et al. T-Reader: a multi-task deep reading comprehension model with self-attention mechanism[J]. Journal of Chinese information processing, 2018, 32(11): 128–134
[23] DU Yongping, GUO Wenyang, ZHAO Yiliang. Hierarchical question-aware context learning with augmented data for biomedical question answering[C]//2019 IEEE International Conference on Bioinformatics and Biomedicine. San Diego: IEEE, 2019: 370–375.
[24] RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving Language Understanding by Generative Pre-training[EB/OL]. [2021-07-13].https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
[25] QIN Y, LIN Y, TAKANOBU R, et al. ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Virtual Event: ACL, 2021: 3350–3363.
[26] LIU Y, OTT M, GOYAL N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[EB/OL]. (2019-07-26)[2021-07-13].https://arxiv.org/abs/1907.11692.
[27] SUN Zijun, LI Xiaoya, SUN Xiaofei, et al. ChineseBERT: Chinese pretraining enhanced by glyph and pinyin information[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 2065–2075.
[28] ZHOU Xuhui, ZHANG Yue, CUI Leyang, et al. Evaluating commonsense in pre-trained language models[C]//The Thirty-Fourth AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 9733–9740.
[29] SHI Jihao, DING Xiao, DU Li, et al. Neural natural logic inference for interpretable question answering[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Virtual Event: ACL, 2021: 3673–3684.
[30] DALVI B, JANSEN P, TAFJORD O, et al. Explaining answers with entailment trees[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Virtual Event: ACL, 2021: 7358–7370.
[31] LIU H, SINGH P. Conceptnet—a practical commonsense reasoning tool-kit[J]. BT technology journal, 2004, 10(22): 211–226.
[32] BIZER C, LEHMANN J, KOBILAROV G, et al. DBpedia—a crystallization point for the web of data[J]. Journal of web semantics, 2019, 7(3): 154–165.
[33] TANON T P, WEIKUM G, SUCHANEK F. YAGO 4: a reason-able knowledge base[C]//European Semantic Web Conference. Heraklion: Springer, 2020: 583–596.
[34] BIAN Ning, HAN Xianpei, CHEN Bo, et al. Benchmarking knowledge-enhanced commonsense question answering via knowledge-to-text transformation[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 12574–12582.
[35] YAN Yuanmeng, LI Rumei, WANG Sirui, et al. Large-scale relation learning for question answering over knowledge bases with pre-trained language models[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 3653–3660.
[36] LIU Ye, WAN Yao, HE Lifang, et al. KG-bart: knowledge graph-augmented bart for generative commonsense reasoning[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 6418–6425.
[37] SHI Jiaxin, CAO Shulin, HOU Lei, et al. TransferNet: an effective and transparent framework for multi-hop question answering over relation graph[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 4149–4158.
[38] LI Shaobo, LI Xiaoguang, SHANG Lifeng, et al. Hopretriever: retrieve hops over wikipedia to answer complex questions[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 13279–13287.
[39] JIANG Shuoran, CHEN Qingcai, LIU Xin, et al. Multi-hop graph convolutional network with high-order chebyshev approximation for text reasoning[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 6563–6573.
[40] FENG Yanlin, CHEN Xinyue, LIN B Y, et al. Scalable multi-hop relational reasoning for knowledge-aware question answering[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Punta Cana: ACL, 2020: 1295–1309.
[41] RICHARDSON M, BURGES C J, RENSHAW E. MCTest: a challenge dataset for the open-domain machine comprehension of text[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: ACL, 2013: 193–203.
[42] LAI G, XIE Q, LIU H, et al. RACE: Large-scale reading comprehension dataset from examinations[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017: 785–794.
[43] MIHAYLOV T, CLARK P, KHOT T, et al. Can a suit of armor conduct electricity? a new dataset for open book question answering[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 2381–2391.
[44] YANG Y, YIH W, MEEK C. WikiQA: a challenge dataset for open-domain question answering[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon: ACL, 2015: 2013–2018.
[45] DUNN M, SAGUN L, HIGGINS M, et al. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine[EB/OL]. [2017–04–18](2021-07-13). https://arxiv.org/abs/1704.05179.
[46] RAJPURKAR P, JIA R, LIANG P. Know what you don’t know: unanswerable questions for squad[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 784–789.
[47] REDDY S, CHEN D, MANNING C D. CoQA: a conversational question answering challenge[J]. Transactions of the association for computational linguistics, 2019(7): 249–266.
[48] CUI Y, LIU T, CHE W, et al. A span-extraction dataset for Chinese machine reading comprehension[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: ACL, 2019: 5883–5889.
[49] YANG Z, QI P, ZHANG S, et al. HotpotQA: A Dataset for diverse, explainable multi-hop question answering[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 2369–2380.
[50] QI P, LEE H, SIDO T, et al. Answering open-domain questions of varying reasoning steps from text[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana: ACL, 2021: 3599–3614.
[51] KO?ISK? T, SCHWARZ J, BLUNSOM P, et al. The NarrativeQA reading comprehension challenge[J]. Transactions of the association for computational linguistics, 2018(6): 317–328.
[52] NGUYEN T, ROSENBERG M, SONG X, et al. MS MARCO: a human generated machine reading comprehension dataset[C]//Conference and Workshop on Neural Information Processing Systems. Barcelona: MIT Press, 2016: 1-10.
[53] KWIATKOWSKI T, PALOMAKI J, REDFIELD O, et al. Natural questions: a benchmark for question answering research[J]. Transactions of the association for computational linguistics, 2019(7): 453–466.
[54] XU Y, ZHU C, XU R, et al. Fusing context into knowledge graph for commonsense question answering[EB/OL]. (2020-12-09)[2021-07-13]. https://arxiv.org/abs/2012.04808.
[55] 顾迎捷, 桂小林, 李德福, 等. 基于神经网络的机器阅读理解综述[J]. 软件学报, 2020, 31(7): 2095–2126
GU Yingjie, GUI Xiaolin, LI Defu, et al. Survey of machine reading comprehension based on neural network[J]. Journal of software, 2020, 31(7): 2095–2126
[56] LIN C Y. ROUGE: a Package for automatic evaluation of summaries[EB/OL]. (2004-07-21)[2021-07-13]. https://aclanthology.org/W04-1013.pdf.
[57] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. Philadelphia: ACL, 2002: 311–318.
[58] 曾帅, 王帅, 袁勇, 等. 面向知识自动化的自动问答研究进展[J]. 自动化学报, 2017, 43(9): 1491–1508
ZENG Shuai, WANG Shuai, YUAN Yong, et al. Towards knowledge automation: a survey on question answering systems[J]. Acta automatica sinica, 2017, 43(9): 1491–1508
- 相似文献/References:
-
[1]李 蕾,周延泉,钟义信.基于语用的自然语言处理研究与应用初探[J].智能系统学报,2006,1(2):1.
LI Lei,ZHOU Yan-quan,ZHONG Yi-xin.Pragmatic Information Based NLP Research and Application[J].CAAI Transactions on Intelligent Systems,2006,1():1.
[2]李德毅.AI——人类社会发展的加速器[J].智能系统学报,2017,12(5):583.[doi:10.11992/tis.201710016]
LI Deyi.Artificial intelligence:an accelerator for the development of human society[J].CAAI Transactions on Intelligent Systems,2017,12():583.[doi:10.11992/tis.201710016]
[3]陈培,景丽萍.融合语义信息的矩阵分解词向量学习模型[J].智能系统学报,2017,12(5):661.[doi:10.11992/tis.201706012]
CHEN Pei,JING Liping.Word representation learning model using matrix factorization to incorporate semantic information[J].CAAI Transactions on Intelligent Systems,2017,12():661.[doi:10.11992/tis.201706012]
[4]张森,张晨,林培光,等.基于用户查询日志的网络搜索主题分析[J].智能系统学报,2017,12(5):668.[doi:10.11992/tis.201706096]
ZHANG Sen,ZHANG Chen,LIN Peiguang,et al.Web search topic analysis based on user search query logs[J].CAAI Transactions on Intelligent Systems,2017,12():668.[doi:10.11992/tis.201706096]
[5]王一成,万福成,马宁.融合多层次特征的中文语义角色标注[J].智能系统学报,2020,15(1):107.[doi:10.11992/tis.201910012]
WANG Yicheng,WAN Fucheng,MA Ning.Chinese semantic role labeling with multi-level linguistic features[J].CAAI Transactions on Intelligent Systems,2020,15():107.[doi:10.11992/tis.201910012]
[6]毛明毅,吴晨,钟义信,等.加入自注意力机制的BERT命名实体识别模型[J].智能系统学报,2020,15(4):772.[doi:10.11992/tis.202003003]
MAO Mingyi,WU Chen,ZHONG Yixin,et al.BERT named entity recognition model with self-attention mechanism[J].CAAI Transactions on Intelligent Systems,2020,15():772.[doi:10.11992/tis.202003003]
[7]张恒,何文玢,何军,等.医学知识增强的肿瘤分期多任务学习模型[J].智能系统学报,2021,16(4):739.[doi:10.11992/tis.202010005]
ZHANG Heng,HE Wenbin,HE Jun,et al.Multi-task tumor stage learning model with medical knowledge enhancement[J].CAAI Transactions on Intelligent Systems,2021,16():739.[doi:10.11992/tis.202010005]
[8]胡康,何思宇,左敏,等.基于CNN-BLSTM的化妆品违法违规行为分类模型[J].智能系统学报,2021,16(6):1151.[doi:10.11992/tis.202104001]
HU Kang,HE Siyu,ZUO Min,et al.Classification model for judging illegal and irregular behavior for cosmetics based on CNN-BLSTM[J].CAAI Transactions on Intelligent Systems,2021,16():1151.[doi:10.11992/tis.202104001]
[9]喻波,王志海,孙亚东,等.非结构化文档敏感数据识别与异常行为分析[J].智能系统学报,2021,16(5):932.[doi:10.11992/tis.202104028]
YU Bo,WANG Zhihai,SUN Yadong,et al.Unstructured document sensitive data identification and abnormal behavior analysis[J].CAAI Transactions on Intelligent Systems,2021,16():932.[doi:10.11992/tis.202104028]
[10]于润羽,杜军平,薛哲,等.面向科技学术会议的命名实体识别研究[J].智能系统学报,2022,17(1):50.[doi:10.11992/tis.202107010]
YU Runyu,DU Junping,XUE Zhe,et al.Research on named entity recognition for scientific and technological conferences[J].CAAI Transactions on Intelligent Systems,2022,17():50.[doi:10.11992/tis.202107010]
备注/Memo
收稿日期:2021-07-13。
基金项目:北京市自然科学基金项目（4212013）；国家语委信息化项目（YB135-89）.
作者简介:杜永萍，教授，主要研究方向为信息检索、信息抽取和自然语言处理。主持国家语委科研项目和北京自然科学基金项目2项。发表学术论文50余篇;赵以梁，硕士研究生，主要研究方向为自然语言处理和自动问答;阎婧雅，硕士研究生，主要研究方向为自然语言处理和自动问答
通讯作者:杜永萍.E-mail:ypdu@bjut.edu.cn
更新日期/Last Update:
1900-01-01