TRANSFORMERS FOR NAMED ENTITY RECOGNITION IN ZERO AND FEW SHOT CONTEXTS

Safarov R. A. & Davronov R. R.


Annotatsiya

Large Language Models (LLMs) have gained popularity recently due to their ability to solve numerous natural language processing tasks. This study discusses the application of transformer models such as RoBERTa and T5 to the Uzbek NER problem under zero-shot and few-shot training conditions. We use novel dataset approaches and active learning to improve the model performance with a minimal number of labeled examples. This is groundbreaking compared to previous studies that mainly focused on supervised learning with large labeled corpora. We demonstrate that even with limited labeling, remarkable results in prediction quality can be achieved. As a trigger for developments in certain areas of natural language processing for languages with limited language resource, our work makes significant contributions to these directions.


Adabiyotlar

  1. J. Li, A. Sun, J. Han, C. Li, A survey on deep learning for named entity recognition, IEEE Trans. on Knowl. and Data Eng. 34 (1) (2022) 50–70. doi:10.1109/TKDE.2020.2981314.
  2. J. C. Salinas Alvarado, K. Verspoor, T. Baldwin, Domain adaption of named entity recognition to support credit risk assessment, in: B. Hachey, K. Webster (Eds.), Proceedings of the Australasian Language Technology Association Workshop 2015, Parramatta, Australia, 2015, pp. 84–90.
  3. M. Del Rio, N. Delworth, R. Westerman, M. Huang, N. Bhandari, J. Palakapilly, Q. McNamara, J. Dong, P. Żelasko, M. Jetté, Earnings-21: A Practical Benchmark for ASR in the Wild, in: Proceedings of Interspeech 2021, 2021, pp. 3465–3469. doi:10.21437/Interspeech.2021-1915.
  4. P. H. Luz de Araujo, T. E. de Campos, R. R. R. de Oliveira, M. Stauffer, S. Couto, P. Bermejo, LeNER-Br: a dataset for named entity recognition in Brazilian legal text, in: International Conference on the Computational Processing of Portuguese (PROPOR), Lecture Notes on Computer Science (LNCS), Springer, Canela, RS, Brazil, 2018, pp. 313–323. doi:10.1007/978-3-319-99722-3\_32.
  5. B. Song, F. Li, Y. Liu, X. Zeng, Deep learning methods for biomedical named entity recognition: a survey and qualitative comparison, Briefings in Bioinformatics 22 (6) (2021) bbab282. doi:10.1093/bib/bbab282.
  6. Kuriyozov, E., Vilares, D., & Gómez-Rodríguez, C. (2024). BERTbek: A Pretrained Language Model for Uzbek. SIGUL2024 Workshop, pages 33–44
  7. Yinhan Liu, Myle Ott, Naman Goyal et. RoBERTa: A Robustly Optimized BERT Pretraining Approach, https://arxiv.org/abs/1907.11692v1
  8. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. https://arxiv.org/abs/1910.10683
  9. Adilova, F., Davronov, R., Kushmuratov, S., Safarov, R. (2022). Uztext-3Gb-BPE-Roberta (Revision 0c87494). Hugging Face. https://huggingface.co/rifkat/uztext-3Gb-BPE-Roberta.
  10. 10. Davronov, R. (2022). T5 Base Uzbek. Hugging Face. Retrieved from https://huggingface.co/rifkat/t5-base-uzbek
  11. 11. Safarov, R., Kushmuratov, S. (2024). NER dataset for Uzbek language [Dataset]. GitHub. https://github.com/rsafarov1995/NER-dataset
  12. 12. Košprdić, M., Prodanović, N., Ljajić, A., Bašaragin, B., & Milošević, N. (2024). From Zero to Hero: Harnessing Transformers for Biomedical Named Entity Recognition in Zero- and Few-shot Contexts. arXiv. https://arxiv.org/abs/2305.04928