Abstract
The goal of complex question answering over knowledge bases (KBQA) is to find an answer entity in a knowledge graph. Recent information retrieval-based methods have focused on the topology of the knowledge graph, ignoring inconsistencies between knowledge graph embeddings and natural language embeddings, and cannot effectively utilize both implicit and explicit knowledge for reasoning. In this paper, we propose a novel model, ReLMKG, to address this challenge. This approach performs joint reasoning on a pre-trained language model and the associated knowledge graph. The complex question and textual paths are encoded by the language model, bridging the gap between the question and the knowledge graph and exploiting implicit knowledge without introducing additional unstructured text. The outputs of different layers in the language model are used as instructions to guide a graph neural network to perform message propagation and aggregation in a step-by-step manner, which utilizes the explicit knowledge contained in the structured knowledge graph. We analyse the reasoning ability of the ReLMKG model for knowledge graphs with different degrees of sparseness and evaluate the generalizability of the model. Experiments conducted on the Complex WebQuestions and WebQuestionsSP datasets demonstrate the effectiveness of our approach on KBQA tasks.











Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Data Availability
The data generated and analysed during the current study are available from the corresponding author upon reasonable request.
Code Availability
Some or all models, code that support the findings of this study are available from the corresponding author upon reasonable request.
References
Yang Z, Qi P, Zhang S, Bengio Y, Cohen W, Salakhutdinov R, Manning CD (2018) Hotpotqa: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 2369–2380
Etezadi R, Shamsfard M (2022) The state of the art in open domain complex question answering: a survey. Appl Intell, 1–21
Fang Y, Sun S, Gan Z, Pillai R, Wang S, Liu J (2020) Hierarchical graph network for multi-hop question answering. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 8823–8838
Jiang Y, Bansal M (2019) Avoiding reasoning shortcuts: adversarial evaluation, training, and model development for multi-hop qa. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 2726–2736
Cao X, Liu Y (2021) Coarse-grained decomposition and fine-grained interaction for multi-hop question answering. J Intell Inf Syst, 1–21
Zhou M, Huang M, Zhu X (2018) An interpretable reasoning network for multi-relation question answering. In: Proceedings of the 27th international conference on computational linguistics, pp 2010–2022
Cao X, Liu Y, Hu B, Zhang Y (2021) Dual-channel reasoning model for complex question answering. Complexity, 2021
Lan Y, He G, Jiang J, Jiang J, Zhao WX, Wen J-R (2021) A survey on complex knowledge base question answering: Methods, challenges and solutions. In: Proceedings of the thirtieth international joint conference on artificial intelligence, IJCAI-21
Yen A-Z, Huang H-H, Chen H-H (2021) Unanswerable question correction in question answering over personal knowledge base. In: Thirty-Fifth AAAI conference on artificial intelligence (AAAI-21)
Kapanipathi P, Abdelaziz I, Ravishankar S, Roukos S, Gray A, Astudillo RF, Chang M, Cornelio C, Dana S, Fokoue-Nkoutche A et al (2021) Leveraging abstract meaning representation for knowledge base question answering. In: Findings of the association for computational linguistics: ACL-IJCNLP 2021, pp 3884–3894
Saxena A, Chakrabarti S, Talukdar P (2021) Question answering over temporal knowledge graphs. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp 6663–6676
Auer S, Bizer C, Kobilarov G, Lehmann J, Cyganiak R, Ives Z (2007) Dbpedia: a nucleus for a web of open data. In: The semantic web, pp 722–735
Suchanek FM, Kasneci G, Weikum G (2007) Yago: a core of semantic knowledge. In: Proceedings of the 16th international conference on world wide web, pp 697–706
Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data, pp 1247–1250
Qiu Y, Wang Y, Jin X, Zhang K (2020) Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In: Proceedings of the 13th international conference on web search and data mining, pp 474–482
He G, Lan Y, Jiang J, Zhao WX, Wen J-R (2021) Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In: Proceedings of the 14th ACM international conference on web search and data mining, pp 553–561
Han J, Cheng B, Wang X (2020) Open domain question answering based on text enhanced knowledge graph with hyperedge infusion Findings of the association for computational linguistics: EMNLP 2020, pp 1475–1481
Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen WW Open domain question answering using early fusion of knowledge bases and text
Jiang D, Li G, Sun Y, Kong J, Tao B, Chen D (2019) Grip strength forecast and rehabilitative guidance based on adaptive neural fuzzy inference system using semg. Pers Ubiquit Comput, 1–10
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neur Comput 9(8):1735–1780
Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on deep learning, December 2014
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inform Process Syst, 30
Han J, Cheng B, Wang X (2021) Two-phase hypergraph based reasoning with dynamic relations for multi-hop kbqa. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 3615–3621
Shen Y, Yang M, Li Y, Wang D, Zheng H, Chen D (2021) Knowledge-based reasoning network for relation detection. IEEE Transactions on Neural Networks and Learning Systems
Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems, 26
Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI conference on artificial intelligence, vol 28
Lin Y, Liu Z, Sun M, Liu Y, Zhu X (2015) Learning entity and relation embeddings for knowledge graph completion. In: Twenty-ninth AAAI conference on artificial intelligence
Peng Y, Zhang J (2020) Lineare: simple but powerful knowledge graph embedding for link prediction. In: 2020 IEEE International conference on data mining (ICDM). IEEE, pp 422–431
Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543
Lin BY, Chen X, Chen J, Ren X (2019) Kagnet: knowledge-aware graph networks for commonsense reasoning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2829–2839
Yasunaga M, Ren H, Bosselut A, Liang P, Leskovec J (2021) Qa-gnn: reasoning with language models and knowledge graphs for question answering. In: Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pp 535–546
Molchanov D, Ashukha A, Vetrov D (2017) Variational dropout sparsifies deep neural networks. In: International Conference on Machine Learning. PMLR, pp 2498-2507
Xiong W, Yu M, Chang S, Guo X, Wang WY (2019) Improving question answering over incomplete kbs with knowledge-aware reader. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 4258–4264
Qiu Y, Zhang K, Wang Y, Jin X, Bai L, Guan S, Cheng X (2020) Hierarchical query graph generation for complex question answering over knowledge graph. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp 1285–1294
Sun H, Bedrax-Weiss T, Cohen W (2019) Pullnet: open domain question answering with iterative retrieval on knowledge bases and text. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2380–2390
Yan Y, Li R, Wang S, Zhang H, Daoguang Z, Zhang F, Wu W, Xu W (2021) Large-scale relation learning for question answering over knowledge bases with pre-trained language models. In: Proceedings of the 2021 conference on empirical methods in natural language processing, pp 3653–3660
Devlin J, Chang M -W, Lee K, Toutanova K (2019) Bert: ptraining of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp 4171–4186
Yu M, Li G, Jiang D, Jiang G, Tao B, Chen D (2019) Hand medical monitoring system based on machine learning and optimal emg feature set. Pers Ubiquit Comput, 1–17
Liu X, Jiang D, Tao B, Jiang G, Sun Y, Kong J, Tong X, Zhao G, Chen B (2021) Genetic algorithm-based trajectory optimization for digital twin robots. Frontiers in Bioengineering and Biotechnology, 9
Haveliwala TH (2003) Topic-sensitive pagerank: a context-sensitive ranking algorithm for web search. IEEE Transa Knowl Data Eng 15(4):784–796
Zhang Y, Dai H, Kozareva Z, Smola AJ, Song L (2018) Variational reasoning for question answering with knowledge graph. In: Thirty-Second AAAI conference on artificial intelligence
Xu K, Lai Y, Feng Y, Wang Z (2019) Enhancing key-value memory neural networks for knowledge based question answering. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp 2937–2947
Luo K, Lin F, Luo X, Zhu K (2018) Knowledge base question answering via encoding of complex query graphs. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 2185–2194
Chen Y, Li H, Hua Y, Qi G (2021) Formal query building with query structure prediction for complex question answering over knowledge base. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 3751–3758
Zhu S, Cheng X, Su S (2020) Knowledge-based question answering by tree-to-sequence learning. Neurocomputing 372:64–72
Sun Y, Zhang L, Cheng G, Qu Y (2020) Sparqa: skeleton-based semantic parsing for complex questions over knowledge bases. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 8952–8959
Bhutani N, Zheng X, Jagadish H (2019) Learning to answer complex questions over knowledge bases with query composition. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 739–748
Saxena A, Tripathi A, Talukdar P (2020) Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 4498–4507
Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (2016) Complex embeddings for simple link prediction. In: International Conference on Machine Learning. PMLR, pp 2071-2080
Ren H, Hu W, Leskovec J (2019) Query2box: reasoning over knowledge graphs in vector space using box embeddings. In: International conference on learning representations
Ren H, Dai H, Dai B, Chen X, Yasunaga M, Sun H, Schuurmans D, Leskovec J, Zhou D (2021) Lego: latent execution-guided reasoning for multi-hop question answering on knowledge graphs. In: International conference on machine learning. PMLR, pp 89598970
Lan Y, Jiang J (2020) Query graph generation for answering multi-hop complex questions from knowledge bases. Assoc Comput Linguis
Zhang Y, Jin L, Zhang Z, Li X, Liu Q, Wang H (2022) Sf-ann: leveraging structural features with an attention neural network for candidate fact ranking. Appl Intell 52(5):5841–5856
Zhang Q, Weng X, Zhou G, Zhang Y, Huang JX (2022) Arl: an adaptive reinforcement learning framework for complex question answering over knowledge base. Inform Process Manag 59(3):102933
Acknowledgments
This research was supported by the Fundamental Research Funds for the Central Universities (Grant number 2020YJS012).
Funding
This research was funded by the Fundamental Research Funds for the Central Universities (Grant number 2020YJS012).
Author information
Authors and Affiliations
Contributions
Xing Cao and Yun Liu designed the study and performed the experiments; Xing Cao performed the experiments, analyzed the data, and wrote the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cao, X., Liu, Y. ReLMKG: reasoning with pre-trained language models and knowledge graphs for complex question answering. Appl Intell 53, 12032–12046 (2023). https://doi.org/10.1007/s10489-022-04123-w
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10489-022-04123-w