Abstract
This paper explores the impact of context selection on the efficiency of Large Language Models (LLMs) in generating Artificial Intelligence (AI) research leaderboards, a task defined as the extraction of (Task, Dataset, Metric, Score) quadruples from scholarly articles. By framing this challenge as a text generation objective and employing instruction finetuning with the FLAN-T5 collection, we introduce a novel method that surpasses traditional Natural Language Inference (NLI) approaches in adapting to new developments without a predefined taxonomy. Through experimentation with three distinct context types of varying selectivity and length, our study demonstrates the importance of effective context selection in enhancing LLM accuracy and reducing hallucinations, providing a new pathway for the reliable and efficient generation of AI leaderboards. This contribution not only advances the state of the art in leaderboard generation but also sheds light on strategies to mitigate common challenges in LLM-based information extraction.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Altbach, P.G., De Wit, H.: Too much academic research is being published. Int. High. Educ. 96, 2–3 (2019)
Bornmann, L., Haunschild, R., Mutz, R.: Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Human. Soc. Sci. Commun. 8(1), 1–15 (2021)
Chung, H.W., et al.: Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022)
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: efficient finetuning of quantized LLMS. Adv. Neural Inf. Process. Syst. 36 (2024)
Jiang, A.Q., et al.: Mistral 7b (2023)
Kabongo, S., D’Souza, J., Auer, S.: Zero-shot entailment of leaderboards for empirical AI research. In: Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2023 (2023)
Kabongo, S., D’Souza, J., Auer, S.: Automated mining of leaderboards for empirical AI research. In: Towards Open and Trustworthy Digital Societies: 23rd International Conference on Asia-Pacific Digital Libraries, ICADL 2021, Virtual Event, 1–3 December 2021, pp. 453–470. Springer, Cham (2021)
Kardas, M., et al.: Axcell: automatic extraction of results from machine learning papers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8580–8594 (2020)
Longpre, S., et al.: The flan collection: designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688 (2023)
Ouyang, L., et al.: Training language models to follow instructions with human feedback. Adv. Neural. Inf. Process. Syst. 35, 27730–27744 (2022)
Shamsabadi, M., D’Souza, J., Auer, S.: Large language models for scientific information extraction: an empirical study for virology (2024)
Shi, F., et al.: Large language models can be easily distracted by irrelevant context. In: International Conference on Machine Learning, pp. 31210–31227. PMLR (2023)
Touvron, H., et al.: Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Instructions: Qualitative Examples
In this section, we elicit each of the instructions that were considered in this work as formulated in the FLAN 2022 Collection for the SQuAD_v2 and DROP datasets (Table 5).
ROUGE Evaluation Metrics
The ROUGE metrics are commonly used for evaluating the quality of text summarization systems. ROUGE-1 measures the overlap of unigram (single word) units between the generated summary and the reference summary. ROUGE-2 extends this to measure the overlap of bigram (two consecutive word) units. ROUGE-L calculates the longest common subsequence between the generated and reference summaries, which takes into account the order of words. ROUGE-LSum is an extension of ROUGE-L that considers multiple reference summaries by treating them as a single summary.
Additional Data Statistics and Hyperparameters
We used a context length of 2400 and based on GPU availability, a batch size of 2 and gradient_accumulation_steps of 4 were used, leading to a final batch size of 8. All experiments were run on five epochs and we used AdafactorSchedule and Adafactor optimizer with scale_parameter=True, relative_step=True, warmup_init=True, lr=1e-4.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kabongo, S., D’Souza, J., Auer, S. (2024). Effective Context Selection in LLM-Based Leaderboard Generation: An Empirical Study. In: Rapp, A., Di Caro, L., Meziane, F., Sugumaran, V. (eds) Natural Language Processing and Information Systems. NLDB 2024. Lecture Notes in Computer Science, vol 14763. Springer, Cham. https://doi.org/10.1007/978-3-031-70242-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-70242-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70241-9
Online ISBN: 978-3-031-70242-6
eBook Packages: Computer ScienceComputer Science (R0)