Search

Stemming is an essential step in various Natural Language Processing (NLP) applications and is used to reduce different variants of the query words to a standard form to avoid the vocabulary mismatch issue in Information Retrieval (IR) systems. Due to specific grammatical rules and complex morphological structures, finding an effective stemming algorithm in Urdu is a challenging task. Although, several stemming algorithms have been proposed for the Urdu text stemming; however, none of them extract the stem from multilevel inflected forms. In this context, according to the best of our knowledge, this is a first effort towards the proposition and evaluation of a novel Urdu Text Stemmer (UTS) that can deal with multi-level inflection forms in Urdu text. The experimental evaluation of the proposed scheme has been conducted on the text-based and word-based custom-developed corpus. The proposed stemming technique is rigorously evaluated and compared with state-of-the-art stemming algorithms. Experimental results demonstrate that UTS outperforms existing Urdu stemmers and achieves an accuracy of 94.92% and 91.8% on word corpus and text corpus, respectively. We also evaluated our proposed system in an Information Retrieval application for Urdu, using the Collection for Urdu Retrieval Evaluation (CURE) dataset. Our approach for information retrieval outperformed and improved both recall and precision metrics.

The purpose of this study is to develop a stemming method by involved several methods including morphological (with affix and pro-lexeme removal), syllable (canonical) pattern, and corpus data as a comparison of the final results of stemming. The algorithm checks a number of the string first and removes affixes, then check the syllable pattern according to the stripping result, then compares to the corpus data which determines the final stemming process. In this study, the corpus data was taken from Sundanese dictionary consists of a single word used for the root word and the extracted dataset from the online Sundanese magazine. The results showed that the stripping of affix and pro-lexeme can remove the corresponding affixes and pro-lexeme then compares words that have a syllable pattern then executes the basic words quickly and the use of corpus can improve accuracy and reduce the over-stemming problems that occur in the stemming process.

Text classification is very useful to search large amount of textual data available online by dividing it into smaller relevant units. Now a day's large amount of digital documents are available in Indian languages. Designing text classifiers in Indian languages is one of the research areas so that people can search and read required documents in their local languages. In proposed work tried to design Text classifier for Hindi text documents and tried to show how stemmer affects the performance of Hindi text classifiers. Stemming is a process to convert words in any language to its base or root words. Stemmers are used for written documents not for spoken languages. Performance of many applications such as text summarization, Information Retrieval (IR) system, text classification systems, syntactic parsing can be improved by applying stemmers. Stemmer eliminates suffix or prefix of the word and form original root word. These root words helps in the preprocessing step required in many algorithms. We applied various stemmers on Hindi text classification models. Experiments and results show that performance of the classifiers is improved by applying stemmers. [ABSTRACT FROM AUTHOR]

A phonemicization or grapheme-to-phoneme conversion (G2P) is a process of converting a word into its pronunciation. It is one of the essential components in speech synthesis, speech recognition, and natural language processing. The deep learning (DL)-based state-of-the-art G2P model generally gives low phoneme error rate (PER) as well as word error rate (WER) for high-resource languages, such as English and European, but not for low-resource languages. Therefore, some conventional machine learning (ML)-based G2P models incorporated with specific linguistic knowledge are preferable for low-resource languages. However, these models are poor for several low-resource languages because of various issues. For instance, an Indonesian G2P model works well for roots but gives a high PER for derivatives. Most errors come from the ambiguities of some roots and derivative words containing four prefixes: 〈ber〉, 〈meng〉, 〈peng〉, and 〈ter〉. In this research, an Indonesian G2P model based on n-gram combined with stemmer and phonotactic rules (NGTSP) is proposed to solve those problems. An investigation based on 5-fold cross-validation, using 50 k Indonesian words, informs that the proposed NGTSP gives a much lower PER of 0.78% than the state-of-the-art Transformer-based G2P model (1.14%). Besides, it also provides a much faster processing time.

This work is an attempt to devise a Stemmer that can remove both prefix and suffix together from a given word in English language. For a given input word, our method considers all possible internal N-grams for detection of potential stems. We frame a hypothesis where the stem length is closest to the half of the length of the input word. A standard English dictionary has been employed to identify morphologically correct N-grams in the process. We apply our techniques over a random sample of 100 English words, each possessing both prefix and suffix. We also compare our proposed Stemmer with three standard algorithms from the literature. Empirical results exhibit that our technique performs better than the rest of the stemmers.

Utmost natural language processing systems practices stemmer as a distinct module in their architecture. Specially, it is crucial for developing, machine translator, speech recognizer and search engines. In linguistic morphology, stemming is the process for reducing inflected (or sometimes derived) words to their root, stem or base form. In this article, a stemming system for Afar-af is presented. This system takes as input a word/terms and removes its affixes (suffix, prefix) rendering to a rule based algorithm. This stemmer is not adequate to describe every rule applied in Afar-af word formation. Consequently, N-gram is combined with the rule to handle cases that are not covered by rule in the hybrid approach of this stemmer. The algorithm follows the well-known Porter algorithm for the English language and it is advanced according to the grammatical rules of the Afar, language. Afar-af morphology was studied and defined in order to model the language and develop an automatic procedure for conflation. The inflectional and derivational morphologies of the language are discussed Afar-af words are very rich in morphology and requires an operative stemming algorithm, which can regulate diverse morphological arrangements that are associated with words. An evaluation of the system indicates that the algorithms accuracy works with better performance than other earlier stemming algorithms for Afar-af giving accuracy of 98.73 percent. Furthermore, Possible extensions of the planned work and advance evaluation approaches are briefly reviewed. [ABSTRACT FROM AUTHOR]

This research work is concerned with the development of a rule-based stemmer for stemming of adjectives in the Punjabi language. Stemming is a method of deriving the root word from the inflected word. The proposed Punjabi Adjective Stemmer (PAS) uses a rule-based approach for converting the inflected Punjabi adjectives to their root forms. A database containing valid root adjectives occurring in the Punjabi language has been created. This database stores 1,762 Punjabi root adjectives. When an adjective word is fed to PAS as an input, first it compares the input word with the root database to determine whether the input adjective is a root adjective or an inflected one. If the input adjective is a root adjective, then no stemming is required and the input adjective is returned as the output. Otherwise, the inflected input adjective is sent to the suffix-stripping algorithm to get the corresponding root adjective. The suffix-stripping algorithm uses a set of predefined rules. India is a linguistically rich country with 22 languages recognized officially. But the computational resources developed for these languages are very scarce. Most of the stemmers developed for Punjabi language so far concentrated on nouns and proper names. PAS is the only stemmer developed so far for specifically addressing the problem of stemming of Punjabi adjectives. PAS has an overall accuracy of 88.76%. [ABSTRACT FROM AUTHOR]

This work is an attempt to devise a Stemmer that can remove both prefix and suffix together from a given word in English language. For a given input word, our method considers all possible internal N -grams for detection of potential stems. We frame a hypothesis where the stem length is closest to the half of the length of the input word. A standard English dictionary has been employed to identify morphologically correct N -grams in the process. We apply our techniques over a random sample of 100 English words, each possessing both prefix and suffix. We also compare our proposed Stemmer with three standard algorithms from the literature. Empirical results exhibit that our technique performs better than the rest of the stemmers. [ABSTRACT FROM AUTHOR]

Stemming adalah proses yang sangat penting untuk mencari kata dasar dari sebuah kata derivatif. Inti dari proses stemming adalah menghilangkan imbuhan pada suatu kata. Stemming sangat dibutuhkan untuk proses information retrieval system. Algoritma pada proses stemming bisa berbeda-beda pada setiap bahasa di berbeda negara. Data yang digunakan adalah 176 kata dasar dalam bahasa Tetun yang merupakan bahasa asli warga negara Timor Leste. Penelitian ini bertujuan untuk merancang algoritma baru yang tepat untuk stemming bahasa Tetun. Tahap awal stemming bahasa Tetun adalah proses filterisasi untuk menghilangkan tanda baca, angka, dan kata yang tidak penting. Lalu tahap tokenisasi untuk membuat variabel yang terdiri dari satu kata. Lalu setiap kata melalui proses stemming untuk menghilangkan imbuhan awalan, akhiran, dan konfiks. Analisis dilakukan berdasarkan kasus error stemming seperti overstemming, understemming, unchanged, dan spelling exception. Hasil uji coba yang didapatkan adalah algoritma stemming bahasa Tetun menghasilkan akurasi sebesar 90.52%.

In today's world, digital content is becoming significantly abundant. Finding ways to come up with a tool that can aid with this is of fundamental importance. People are faced with what is referred to as information overload. A tool that can make a summary of a text without losing its message, coherence and cohesion is vital. We live in a digital age and that technology saves us time. This means that users can only focus on points they are interested in. This is one of the research areas in natural language processing/information retrieval which this work tries to contribute to. It tries to contextualise the tools and technologies that are developed for other languages to automatically summarise textual Xhosa news articles. The work specifically aims to develop a text summariser for textual Xhosa news articles based on extraction methods. In doing so, it examines the literature to try to understand the techniques and technologies used to analyse the contents of a written text in order to transform and synthesise it. The study also examines the phonology and morphology of the Xhosa language, and finally, designs, implements, and tests an extraction-based automatic news article for the Xhosa language. Two approaches were used to extract relevant sentences: term frequency and sentence position. The Xhosa summariser is evaluated using a test set. This study has employed both subjective and objective evaluation methods. [ABSTRACT FROM AUTHOR]

Data and information's are overloaded on the web and it is a primary problem faced by people and institutions today. Grouping out some useful information from the sentence given in the Gujarati language and remove suffix, prefix and unnecessary characters is the important process of data cleaning and make effective use of our database. These are challenging task for every Indian language due to its rich morphological variance. This paper presents a lightweight Morphological analyzer for Gujarati language using a rule based classifier. Searching a given word in the Gujarati text is called text mining in a Gujarati language but database of the Gujarati stems need to be strengthen so that searching become effective and easy. The source of text mining is the process of stemming. It is normally used in many types of applications such as Natural Language Processing (NLP), Information Retrieval (IR) and Text Mining (TM). In this paper we present a stemming algorithm. [ABSTRACT FROM AUTHOR]

Abstract In Information Retrieval systems, stemming handles the words that can occur in different morphological forms, and hence matches the terms of the documents and the queries that are related in meanings. In this article, we have proposed a cognitive inspired language-independent stemming that learns group of morphologically related words from the ambient corpus without any linguistic knowledge or human intervention and it behaves in a way the human brain works. The main idea of our proposed algorithm is to determine only those variants of the words from the ambient corpus that match the original intent of the query terms. We conducted ad-hoc retrieval experiments in a number of languages of varying morphological complexity using standard TREC, FIRE, and CLEF document collection. The results indicate that stemming improves the retrieval accuracy and the effectiveness of stemming algorithm increases with the increase in the morphological complexity of algorithm. The results also indicates that the performance of our proposed algorithm is better than the stemmers based on linguistic knowledge and other state-of-the-art statistical stemmers in almost all the languages under study. In multi-lingual setup these results are quite encouraging. [ABSTRACT FROM AUTHOR]

Rule based machine translation system embraces a set of grammar rules which are mandatory for the mapping of syntactic representations of a source language, on the target language. The system necessitates good linguistic knowledge to write rules and require of acquaintance source such as corpus and bilingual dictionary. In this paper, we have described the grammar rules intended for our English to Hindi machine translation system to translate the homoeopathic literatures, medical reports, prescription etc. The rules which have been written follow the transfer based approach for reordering of rules between two languages. The paper first discusses about our developed stemmer and its rules, further we discuss the Part of Speech tagging (PoS) rules for categorizing each word of the sentence grammatically and our developed homoeopathy corpus in English and Hindi of size 20085 and 20072 words respectively and at the last we discuss the agreement/translation rules for translating various homoeopathic sentences. [ABSTRACT FROM AUTHOR]

Suffix stripping is a problem of removing morphological suffixes from a word to get the stem. We present suffix stripping as an unconstrained optimization problem. Free from linguistic or morphological knowledge, a simple algorithm is being developed. Superiority of the algorithm over an established technique for English language is being demonstrated. [ABSTRACT FROM AUTHOR]

In linguistics, stemming is the operation of reducing words to their more general form, which is called the 'stem'. Stemming is an important step in information retrieval systems, natural language processing, and text mining. Information retrieval systems are evaluated by metrics like precision and recall and the fundamental superiority of an information retrieval system over another one is measured by them. Stemmers decrease the indexed file, increase the speed of information retrieval systems, and improve the performance of these systems by boosting precision and recall. There are few Persian stemmers and most of them work based on morphological rules. In this paper we carefully study Persian stemmers, which are classified into three main classes: structural stemmers, lookup table stemmers, and statistical stemmers. We describe the algorithms of each class carefully and present the weaknesses and strengths of each Persian stemmer. We also propose some metrics to compare and evaluate each stemmer by them. [ABSTRACT FROM AUTHOR]

A massive amount of documents are being posted online every minute. The task of document classification requires extensive background work on the content of documents, where keyword-based matching alone may not be sufficient. Much research has been carried out in several languages that has revealed significant results. However, Arabic documents still pose a great challenge due to the nature of Arabic language. Extracting roots or stems from the breakdown of multiple Arabic words and phrases are an important task that must be completed before applying text classification. The research at hand proposes an algorithm for classifying Arabic-Text documents using semantic relations between words based on an Arabic thesaurus, mainly synonyms, hyperonyms and hyponyms. The experiments conducted in this study evaluated the results using F1-Measure and compared them to results obtained via other existing methods, such as utilizing stemmers and part-of-speech taggers, where it indicated an increment of more than 12.6% for the novel method using semantic relation over other methods. Arabic-WordNet was utilized as a thesaurus for indicating possible relations to be examined. The obtained results indicate that the domain of the semantic web reveals a variety of options for enhancing text classifications, which are highly competitive with current methods. Future work will include identifying best relations to be utilized among the available 20 relations. [ABSTRACT FROM AUTHOR]