Abstract
This paper presents a targeted syntactic evaluation of Transformer models for Galician-Portuguese. We defined three experiments that allow to explore how these models, trained with a masked language modeling objective, encode syntactic knowledge. To do so, we created a new dataset including test instances of number (subject-verb), gender (subject-predicative adjective), and person (subject-inflected infinitive) agreement. This dataset was used to evaluate monolingual and multilingual BERT models, controlling for various aspects such as the presence of attractors or the distance between the dependent elements. The results show that Transformer models perform competently in many cases, but they are generally confounded by the presence of attractors in long-distance dependencies. Moreover, the different behavior of monolingual models trained with the same corpora reinforces the need for a deep exploration of the network architectures and their learning process.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
Galician and Portuguese are usually considered varieties of a single language [7, 14], but the recent standardization of the former adopting a Spanish-based orthography [22] makes difficult to process it using resources and tools built for Portuguese. Thus, our division of Galician and Portuguese is based solely on their different spellings.
- 3.
As this variation hardly exists in European Portuguese, the analysis of the person feature can be easily done for this variety, and we leave this for further work. It is worth mentioning, however, that most neural language models for Portuguese are trained using large amounts of Brazilian data.
- 4.
Data available at https://github.com/crespoalfredo/PROPOR2022-gl-pt.
- 5.
Note that we also included sentences without attractors to observe their impact.
- 6.
We also evaluated the Bertinho models [26], with lower results not discussed here.
- 7.
“Vostedes” (formal pronoun in the \(2^\text {nd}\) person plural, agreeing with the \(3^\text {rd}\) person plural) is not used as it does not appear in the mBERT vocabulary.
References
Ariel, M.: The development of person agreement markers: from pronouns to higher accessibility markers. Usage-based models of language, pp. 197–260 (2000)
Baroni, M.: On the proper role of linguistically-oriented deep net analysis in linguistic theorizing (2021). arXiv preprint arXiv:2106.08694
Bock, K., Miller, C.A.: Broken agreement. Cogn. Psychol. 23(1), 45–93 (1991)
Collobert, R., Weston, J.: A unified architecture for natural language processing: Deep neural networks with multitask learning. In: Proceedings of the 25th International Conference on Machine Learning, pp. 160–167 (2008)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 4171–4186. Association for Computational Linguistics, Minneapolis, Minnesota (2019)
Edunov, S., Ott, M., Auli, M., Grangier, D.: Understanding back-translation at scale. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489–500. Association for Computational Linguistics, Brussels, Belgium (2018)
Freixeiro Mato, X.R.: Gramática da Lingua Galega IV. Gramática do texto. A Nosa Terra, Vigo (2003)
Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., Levy, R.: Neural language models as psycholinguistic subjects: representations of syntactic state. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 32–42. Association for Computational Linguistics, Minneapolis, Minnesota (2019)
Garcia, M.: Exploring the representation of word meanings in context: a case study on homonymy and synonymy. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, vol. 1, Long Papers, pp. 3625–3640. Association for Computational Linguistics, Online, August 2021
Goldberg, Y.: Assessing BERT’s Syntactic Abilities (2019). arXiv preprint arXiv:1901.05287
Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., Baroni, M.: Colorless green recurrent networks dream hierarchically. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 1195–1205. Association for Computational Linguistics, New Orleans, Louisiana (2018)
Hall Maudslay, R., Cotterell, R.: Do syntactic probes probe syntax? Experiments with jabberwocky probing. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 124–131. Association for Computational Linguistics (2021)
Hu, J., Gauthier, J., Qian, P., Wilcox, E., Levy, R.: A systematic assessment of syntactic generalization in neural language models. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1725–1744. Association for Computational Linguistics (2020)
Lindley Cintra, L.F., Cunha, C.: Nova Gramática do Português Contemporâneo. Livraria Sá da Costa, Lisbon (1984)
Linzen, T., Baroni, M.: Syntactic structure from deep learning. Ann. Rev. Linguist. 7, 195–212 (2021)
Linzen, T., Dupoux, E., Goldberg, Y.: Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Trans. Assoc. Comput. Linguist. 4, 521–535 (2016)
Lucchesi, D., Baxter, A., da Silva, J.A.A.: A concordância verbal. In: O português afro-brasileiro, pp. 331–371. SciELO Books (2009)
Marvin, R., Linzen, T.: Targeted syntactic evaluation of language models. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192–1202. Association for Computational Linguistics, Brussels, Belgium (2018)
Mueller, A., Nicolai, G., Petrou-Zeniou, P., Talmina, N., Linzen, T.: Cross-linguistic syntactic evaluation of word prediction models. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5523–5539. Association for Computational Linguistics, July 2020
Newman, B., Ang, K.S., Gong, J., Hewitt, J.: Refining targeted syntactic evaluation of language models. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3710–3723. Association for Computational Linguistics (2021)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018). https://openai.com/blog/language-unsupervised
Samartim, R.: Língua somos: A construção da ideia de língua e da identidade coletiva na Galiza (pré-)constitucional. In: Actas do IX Congreso Internacional de Estudos Galegos. Novas achegas ao estudo da cultura galega II: enfoques socio-históricos e lingüístico-literarios, pp. 27–36. Universidade da Coruña (2012)
Schnabel, T., Labutov, I., Mimno, D., Joachims, T.: Evaluation methods for unsupervised word embeddings. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 298–307. Association for Computational Linguistics, Lisbon, Portugal (2015)
Souza, F., Nogueira, R., Lotufo, R.: BERTimbau: pretrained BERT models for Brazilian Portuguese. In: Cerri, R., Prati, R.C. (eds.) BRACIS 2020. LNCS (LNAI), vol. 12319, pp. 403–417. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is All You Need (2017). arXiv preprint arXiv:1706.03762
Vilares, D., Garcia, M., Gómez-Rodríguez, C.: Bertinho: Galician BERT representations. Procesamiento del Lenguaje Natural 66, 13–26 (2021)
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. Association for Computational Linguistics (2020)
Acknowledgments
This research is funded by a Ramón y Cajal grant (RYC2019-028473-I), by the Galician Government (ERDF 2014-2020: Call ED431G 2019/04, and ED431F 2021/01), and by a summer internship of the CiTIUS Research Center.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Garcia, M., Crespo-Otero, A. (2022). A Targeted Assessment of the Syntactic Abilities of Transformer Models for Galician-Portuguese. In: Pinheiro, V., et al. Computational Processing of the Portuguese Language. PROPOR 2022. Lecture Notes in Computer Science(), vol 13208. Springer, Cham. https://doi.org/10.1007/978-3-030-98305-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-98305-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98304-8
Online ISBN: 978-3-030-98305-5
eBook Packages: Computer ScienceComputer Science (R0)