Back to Search
Start Over
BnVec: Towards the Development of Word Embedding for Bangla Language Processing
- Source :
- International Journal of Engineering & Technology. 10:95
- Publication Year :
- 2021
- Publisher :
- Science Publishing Corporation, 2021.
-
Abstract
- Progression in machine learning and statistical inference are facilitating the advancement of domains like computer vision, natural language processing (NLP), automation & robotics, and so on. Among the different persuasive improvements in NLP, word embedding is one of the most used and revolutionary techniques. In this paper, we manifest an open-source library for Bangla word extraction systems named BnVec which expects to furnish the Bangla NLP research community by the utilization of some incredible word embedding techniques. The BnVec is splitted up into two parts, the first one is the Bangla suitable defined class to embed words with access to the six most popular word embedding schemes (CountVectorizer, TF-IDF, Hash Vectorizer, Word2vec, fastText, and Glove). The other one is based on the pre-trained distributed word embedding system of Word2vec, fastText, and GloVe. The pre-trained models have been built by collecting content from the newspaper, social media, and Bangla wiki articles. The total number of tokens used to build the models exceeds 395,289,960. The paper additionally depicts the performance of these models by various hyper-parameter tuning and then analyzes the results.
- Subjects :
- Environmental Engineering
Word embedding
Computer science
business.industry
General Chemical Engineering
General Engineering
computer.software_genre
language.human_language
Bengali
Development (topology)
Hardware and Architecture
Computer Science (miscellaneous)
language
Artificial intelligence
business
computer
Natural language processing
Biotechnology
Subjects
Details
- ISSN :
- 2227524X
- Volume :
- 10
- Database :
- OpenAIRE
- Journal :
- International Journal of Engineering & Technology
- Accession number :
- edsair.doi...........1f9d0aa7365682a29534bba7866d0636