Abstract
For green AI, it is crucial to measure and reduce the carbon footprint emitted during the training of large language models. In NLP, performing pre-training on Transformer models requires significant computational resources. This pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. Thus, it is important that we select the correct data in the form of domain-specific data from this vast corpus to achieve optimum results aligned with our domain-specific tasks. While training on large unsupervised data is expensive, it can be optimized by performing a data selection step before pretraining. Selecting important data reduces the space overhead and the substantial amount of time required to pre-train the model while maintaining constant accuracy. We investigate the existing selection strategies and propose our own domain-adaptive data selection method - TextGram - that effectively selects essential data from large corpora. We compare and evaluate the results of finetuned models for text classification task with and without data selection. We show that the proposed strategy works better compared to other selection methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Axelrod, A., He, X., Gao, J.: Domain adaptation via pseudo in-domain data selection. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 355–362 (2011)
Bafna, P., Pramod, D., Vaidya, A.: Document clustering: TF-IDF approach. In: 2016 International Conference on Electrical, Electronics, and Optimization Techniques (ICEEOT), pp. 61–66. IEEE (2016)
Cruz Silva, C., Liu, C.H., Poncelas, A., Way, A.: Extracting in-domain training corpora for neural machine translation using data selection methods. Association for Computational Linguistics (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint: arXiv:1810.04805 (2018)
Eck, M., Vogel, S., Waibel, A.: Low cost portability for statistical machine translation based on N-gram frequency and TF-IDF. In: Proceedings of the Second International Workshop on Spoken Language Translation (2005)
Foster, G., Goutte, C., Kuhn, R.: Discriminative instance weighting for domain adaptation in statistical machine translation. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 451–459 (2010)
Gao, J., Goodman, J., Li, M., Lee, K.F.: Toward a unified approach to statistical language modeling for Chinese. ACM Trans. Asian Lang. Inf. Process. (TALIP) 1(1), 3–33 (2002)
Ladkat, A., Miyajiwala, A., Jagadale, S., Kulkarni, R., Joshi, R.: Towards simple and efficient task-adaptive pre-training for text classification. arXiv preprint: arXiv:2209.12943 (2022)
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150. Association for Computational Linguistics, Portland (2011). https://aclanthology.org/P11-1015
Mihalcea, R., Tarau, P.: TextRank: bringing order into text. In: Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 404–411 (2004)
Moore, R.C., Lewis, W.: Intelligent selection of language model training data. In: Proceedings of the ACL 2010 Conference Short Papers, pp. 220–224 (2010)
Parcollet, T., Ravanelli, M.: The energy and carbon footprint of training end-to-end speech recognizers (2021)
Roh, J., Oh, S.H., Lee, S.Y.: Unigram-normalized perplexity as a language model performance measure with different vocabulary sizes. arXiv preprint: arXiv:2011.13220 (2020)
Van Der Wees, M., Bisazza, A., Monz, C.: Dynamic data selection for neural machine translation. arXiv preprint: arXiv:1708.00712 (2017)
Yasuda, K., Zhang, R., Yamamoto, H., Sumita, E.: Method of selecting training data to build a compact and efficient translation model. In: Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II (2008)
Zellers, R., Holtzman, A., Rashkin, H., Bisk, Y., Farhadi, A., Roesner, F., Choi, Y.: Defending against neural fake news (2020)
Acknowledgments
This work was done under the L3Cube Pune mentorship program. We would like to express our gratitude towards our mentors at L3Cube for their continuous support and encouragement.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hiwarkhedkar, S. et al. (2024). TextGram: Towards a Better Domain-Adaptive Pretraining. In: Chakravarthi, B.R., et al. Speech and Language Technologies for Low-Resource Languages. SPELLL 2023. Communications in Computer and Information Science, vol 2046. Springer, Cham. https://doi.org/10.1007/978-3-031-58495-4_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-58495-4_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-58494-7
Online ISBN: 978-3-031-58495-4
eBook Packages: Computer ScienceComputer Science (R0)