[1]DU Yongping,ZHAO Yiliang,YAN Jingya,et al.Survey of machine reading comprehension based on deep learning[J].CAAI Transactions on Intelligent Systems,2022,17(6):1074-1083.[doi:10.11992/tis.202107024]
Copy

Survey of machine reading comprehension based on deep learning

CAAI Transactions on Intelligent Systems[ISSN 1673-4785/CN 23-1538/TP] Volume: 17 Number of periods: 2022 6 Page number: 1074-1083 Column: 综述 Public date: 2022-11-05
Title:
Survey of machine reading comprehension based on deep learning
Author(s):
DU Yongping;  ZHAO Yiliang;  YAN Jingya;  GUO Wenyang
Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
Keywords:
machine reading comprehension; natural language processing; deep learning; neural network; end-to-end model; knowledge reasoning; pretrained language model; artificial intelligence
CLC:
TP391
DOI:
10.11992/tis.202107024
Abstract:
In recent years, there has been a great deal of interest in the task of machine reading comprehension. It enables computers to learn and answer questions based on text input. One of the long-standing challenges in the field of artificial intelligence is how to make machines understand natural language. In recent years, machine reading comprehension has advanced rapidly as a result of the large-scale release of high-quality data sets and the application of deep learning technology. The use of an end-to-end model structure based on neural networks, a pre-trained language model, and reasoning technology has greatly improved their performance on large-scale evaluation data sets. However, there is still a big gap in real language understanding. This paper summarizes the research status and development trend of machine reading comprehension tasks, including division of tasks, analysis of machine reading comprehension model and related technologies, particularly machine reading comprehension technology based on knowledge reasoning, and finally discusses the development trend in this field.
References:
[1] HILL F, BORDES A, CHOPRA S, et al. The goldilocks principle: reading children’s books with explicit memory representations[C]//Proceedings of the 4th International Conference on Learning Representations. San Juan: OpenReview, 2016: 1-13.
[2] RAJPURKAR P, ZHANG J, LOPYREV K, et al. SQuAD: 100, 000+ questions for machine comprehension of text[C]//Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Austin: ACL, 2016: 2383–2392.
[3] JOSHI M, CHOI E, WELD D, et al. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: ACL, 2017: 1601–1611.
[4] SEO M, KEMBHAVI A, FARHADI A, et al. Bidirectional attention flow for machine comprehension[C]//5th International Conference on Learning Representations. Toulon: OpenReview, 2017: 1-12.
[5] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017(30): 5998–6008.
[6] WANG W H, YANG N, WEI F R, et al. Gated Self-matching networks for reading comprehension and question answering[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouve: ACL, 2017: 189–198.
[7] 梁小波, 任飞亮, 刘永康, 等. N-Reader: 基于双层Self-attention的机器阅读理解模型[J]. 中文信息学报, 2018, 32(10): 130–137
LIANG Xiaobo, REN Feiliang, LIU Yongkang, et al. N-reader: machine reading comprehension model based on double layers of self-attention[J]. Journal of Chinese information processing, 2018, 32(10): 130–137
[8] HE Wei, LIU Kai, LIU Jing, et al. DuReader: a Chinese machine reading comprehension dataset from real-world applications[C]//Proceedings of the Workshop on Machine Reading for Question Answering. Melbourne: ACL, 2018: 37–46.
[9] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: ACL, 2019: 4171–4186.
[10] YANG Zhilin, DAI Zihang, YANG Yiming, et al. XLNet: generalized autoregressive pretraining for language understanding[C]//Annual Conference on Neural Information Processing Systems. Vancouver: ACM, 2019: 1-18.
[11] DING M, ZHOU C, CHEN Q, et al. Cognitive graph for multi-hop reading comprehension at scale[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence: ACL, 2019: 2694–2703.
[12] YASUNAGA M, RWN H, BOSSELUT A, et al. QA-GNN: reasoning with language models and knowledge graphs for question answering[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. ACL, 2021: 535–546.
[13] YAGCIOGLU S, ERDEM A, ERDEM E, et al. RecipeQA: a challenge dataset for multimodal comprehension of cooking recipes[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 1358–1368.
[14] KEMBHAVI A, SEO M, SCHWENK D, et al. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 5376–5384.
[15] HERMANN K M, KO?ISK? T, GREFENSTETTE E, et al. Teaching machines to read and comprehend[C]//Advances in Neural Information Processing Systems. Montréal: MIT Press, 2015: 1693–1701.
[16] WELBL J, STENETORP P, RIEDEL S. Constructing datasets for multi-hop reading comprehension across documents[J]. Transactions of the association for computational linguistics, 2018, 6(1): 287–302.
[17] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: a question answering challenge targeting commonsense knowledge[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: NAACL, 2019: 4149–4158.
[18] TRISCHLER A, WANG T, YUAN X, et al. NewsQA: a machine comprehension dataset[C]//Proceedings of the 2nd Workshop on Representation Learning for NLP. Vancouver: ACL, 2017: 191–200.
[19] DUA D, WANG Y, DASIGI P, et al. DROP: a reading comprehension benchmark requiring discrete reasoning over paragraphs[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis: NAACL, 2019: 2368–2378.
[20] YU A W, DOHAN D, LUONG M T, et al. QANet: combining local convolution with global self-attention for reading comprehension[EB/OL]. (2018-04-23)[ 2021-07-13].https://arxiv.org/abs/1804.09541.
[21] PENNINGTON J, SOCHER R, MANNING CD. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Qatar: ACL, 2014: 1532–1543.
[22] 郑玉昆, 李丹, 范臻, 等. T-Reader: 一种基于自注意力机制的多任务深度阅读理解模型[J]. 中文信息学报, 2018, 32(11): 128–134
ZHENG Yukun, LI Dan, FAN Zheng, et al. T-Reader: a multi-task deep reading comprehension model with self-attention mechanism[J]. Journal of Chinese information processing, 2018, 32(11): 128–134
[23] DU Yongping, GUO Wenyang, ZHAO Yiliang. Hierarchical question-aware context learning with augmented data for biomedical question answering[C]//2019 IEEE International Conference on Bioinformatics and Biomedicine. San Diego: IEEE, 2019: 370–375.
[24] RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving Language Understanding by Generative Pre-training[EB/OL]. [2021-07-13].https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf.
[25] QIN Y, LIN Y, TAKANOBU R, et al. ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Virtual Event: ACL, 2021: 3350–3363.
[26] LIU Y, OTT M, GOYAL N, et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach[EB/OL]. (2019-07-26)[2021-07-13].https://arxiv.org/abs/1907.11692.
[27] SUN Zijun, LI Xiaoya, SUN Xiaofei, et al. ChineseBERT: Chinese pretraining enhanced by glyph and pinyin information[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 2065–2075.
[28] ZHOU Xuhui, ZHANG Yue, CUI Leyang, et al. Evaluating commonsense in pre-trained language models[C]//The Thirty-Fourth AAAI Conference on Artificial Intelligence. New York: AAAI, 2020: 9733–9740.
[29] SHI Jihao, DING Xiao, DU Li, et al. Neural natural logic inference for interpretable question answering[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Virtual Event: ACL, 2021: 3673–3684.
[30] DALVI B, JANSEN P, TAFJORD O, et al. Explaining answers with entailment trees[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Virtual Event: ACL, 2021: 7358–7370.
[31] LIU H, SINGH P. Conceptnet—a practical commonsense reasoning tool-kit[J]. BT technology journal, 2004, 10(22): 211–226.
[32] BIZER C, LEHMANN J, KOBILAROV G, et al. DBpedia—a crystallization point for the web of data[J]. Journal of web semantics, 2019, 7(3): 154–165.
[33] TANON T P, WEIKUM G, SUCHANEK F. YAGO 4: a reason-able knowledge base[C]//European Semantic Web Conference. Heraklion: Springer, 2020: 583–596.
[34] BIAN Ning, HAN Xianpei, CHEN Bo, et al. Benchmarking knowledge-enhanced commonsense question answering via knowledge-to-text transformation[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 12574–12582.
[35] YAN Yuanmeng, LI Rumei, WANG Sirui, et al. Large-scale relation learning for question answering over knowledge bases with pre-trained language models[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 3653–3660.
[36] LIU Ye, WAN Yao, HE Lifang, et al. KG-bart: knowledge graph-augmented bart for generative commonsense reasoning[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 6418–6425.
[37] SHI Jiaxin, CAO Shulin, HOU Lei, et al. TransferNet: an effective and transparent framework for multi-hop question answering over relation graph[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 4149–4158.
[38] LI Shaobo, LI Xiaoguang, SHANG Lifeng, et al. Hopretriever: retrieve hops over wikipedia to answer complex questions[C]//The Thirty-Fifth AAAI Conference on Artificial Intelligence. New York: AAAI, 2021: 13279–13287.
[39] JIANG Shuoran, CHEN Qingcai, LIU Xin, et al. Multi-hop graph convolutional network with high-order chebyshev approximation for text reasoning[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. Krung Thep Maha Nakhon: ACL, 2021: 6563–6573.
[40] FENG Yanlin, CHEN Xinyue, LIN B Y, et al. Scalable multi-hop relational reasoning for knowledge-aware question answering[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Punta Cana: ACL, 2020: 1295–1309.
[41] RICHARDSON M, BURGES C J, RENSHAW E. MCTest: a challenge dataset for the open-domain machine comprehension of text[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle: ACL, 2013: 193–203.
[42] LAI G, XIE Q, LIU H, et al. RACE: Large-scale reading comprehension dataset from examinations[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017: 785–794.
[43] MIHAYLOV T, CLARK P, KHOT T, et al. Can a suit of armor conduct electricity? a new dataset for open book question answering[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 2381–2391.
[44] YANG Y, YIH W, MEEK C. WikiQA: a challenge dataset for open-domain question answering[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon: ACL, 2015: 2013–2018.
[45] DUNN M, SAGUN L, HIGGINS M, et al. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine[EB/OL]. [2017–04–18](2021-07-13). https://arxiv.org/abs/1704.05179.
[46] RAJPURKAR P, JIA R, LIANG P. Know what you don’t know: unanswerable questions for squad[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 784–789.
[47] REDDY S, CHEN D, MANNING C D. CoQA: a conversational question answering challenge[J]. Transactions of the association for computational linguistics, 2019(7): 249–266.
[48] CUI Y, LIU T, CHE W, et al. A span-extraction dataset for Chinese machine reading comprehension[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing. Hong Kong: ACL, 2019: 5883–5889.
[49] YANG Z, QI P, ZHANG S, et al. HotpotQA: A Dataset for diverse, explainable multi-hop question answering[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: ACL, 2018: 2369–2380.
[50] QI P, LEE H, SIDO T, et al. Answering open-domain questions of varying reasoning steps from text[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana: ACL, 2021: 3599–3614.
[51] KO?ISK? T, SCHWARZ J, BLUNSOM P, et al. The NarrativeQA reading comprehension challenge[J]. Transactions of the association for computational linguistics, 2018(6): 317–328.
[52] NGUYEN T, ROSENBERG M, SONG X, et al. MS MARCO: a human generated machine reading comprehension dataset[C]//Conference and Workshop on Neural Information Processing Systems. Barcelona: MIT Press, 2016: 1-10.
[53] KWIATKOWSKI T, PALOMAKI J, REDFIELD O, et al. Natural questions: a benchmark for question answering research[J]. Transactions of the association for computational linguistics, 2019(7): 453–466.
[54] XU Y, ZHU C, XU R, et al. Fusing context into knowledge graph for commonsense question answering[EB/OL]. (2020-12-09)[2021-07-13]. https://arxiv.org/abs/2012.04808.
[55] 顾迎捷, 桂小林, 李德福, 等. 基于神经网络的机器阅读理解综述[J]. 软件学报, 2020, 31(7): 2095–2126
GU Yingjie, GUI Xiaolin, LI Defu, et al. Survey of machine reading comprehension based on neural network[J]. Journal of software, 2020, 31(7): 2095–2126
[56] LIN C Y. ROUGE: a Package for automatic evaluation of summaries[EB/OL]. (2004-07-21)[2021-07-13]. https://aclanthology.org/W04-1013.pdf.
[57] PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th annual meeting of the Association for Computational Linguistics. Philadelphia: ACL, 2002: 311–318.
[58] 曾帅, 王帅, 袁勇, 等. 面向知识自动化的自动问答研究进展[J]. 自动化学报, 2017, 43(9): 1491–1508
ZENG Shuai, WANG Shuai, YUAN Yong, et al. Towards knowledge automation: a survey on question answering systems[J]. Acta automatica sinica, 2017, 43(9): 1491–1508
Similar References:

Memo

-

Last Update: 1900-01-01

Copyright © CAAI Transactions on Intelligent Systems