Skip to Main content Skip to Navigation
Conference papers

Transfer learning and distant supervision for multilingual Transformer models: A study on African languages

Abstract : Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and Yorùbá on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.
Document type :
Conference papers
Complete list of metadata

https://hal.inria.fr/hal-03350901
Contributor : Emmanuel Vincent Connect in order to contact the contributor
Submitted on : Tuesday, September 21, 2021 - 8:19:42 PM
Last modification on : Thursday, September 23, 2021 - 3:08:08 AM
Long-term archiving on: : Wednesday, December 22, 2021 - 7:15:17 PM

File

hedderich_EMNLP2020.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03350901, version 1

Collections

Citation

Michael A Hedderich, David I Adelani, Dawei Zhu, Jesujoba Alabi, Udia Markus, et al.. Transfer learning and distant supervision for multilingual Transformer models: A study on African languages. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Nov 2020, Punta Cana, Dominica. ⟨hal-03350901⟩

Share

Metrics

Record views

8

Files downloads

92