1. Harnessing the power of hugging face's multilingual transformers: unravelling the code-mixed named entity recognition enigma
- Author
-
Shamim, Rejuwan and Shaikh, Asadullah
- Abstract
Named entity recognition (NER) in code-mixed documents, which have different languages, is hard for natural language processing. In this paper, we use hugging face's multilingual transformers to come up with a way to do code-mixed NER without any problems. Our work tries to solve the problems that come up when you try to recognise named entities in more than one language within the same text. We did thorough tests by fine-tuning the multilingual transformer model on a dataset with mixed codes. With an F1-score of 0.85, we got great results. This works better than previous methods and proves that our model can accurately find named items. We also examine at how well the model works with other language pairs and code-mixed patterns. This shows how well the model can handle different language situations. Our study helps us understand how to handle data in multiple languages, makes code-mixed NER techniques better, and shows how multilingual transformers can help break down language barriers. The research has implications for areas that need to understand more than one language, such as analysing social media, creating language-specific customer service systems, and finding information across languages. Speaking different languages can communicate more easily and effectively in these fields, which encourage inclusion.
- Published
- 2024
- Full Text
- View/download PDF