Search

The papers in the second half of this volume all testify to the way in which the study of identity has moved from categorial (yes/no) distinctions towards the delicacies of social and cultural positions in discourse. They also show that in order for such finer distinctions to take shape in analysis, attention needs to be given to details of the deployment of specific discourse and semiotic resources. None of the analyses falls into the wide-open trap of equating ‘language’ with identity; they all focus on connections between actual discourse and semiosis, and subject/speaker positions, inhabited and ascribed micro-identities, and features of the social and cultural imagination that determine available identity repertoires. They represent, thus, a stage in a paradigm shift that has been underway for a couple of decades now, in which scholars abandon homogenizing and static categorial notions of identity (as in ‘national’ or ‘ethnic’ identity) and develop performative, inter-subjective and pragmatic/metapragmatic models of identity. In sociolinguistics, the paradigm shift is one that takes us from a sociolinguistics of static and immobile languages associated with ‘communities’ to a sociolinguistics of mobile resources and speech associated with flexible networks of language participants. This process is not complete and steps in it, consequently, bear the traces of the older paradigm. Thus, Tsitsipis and Georgeakopoulou & Finnis start their analysis from within a ‘community’, rather traditionally defined. In Tsitsipis’ case, the community is defined as a ‘national minority’ of Arvanitika speakers – a definition that incorporates both a state categorisation device of ‘minorities’ and an older ethnolinguistic-identity diacritic, and both are of questionable validity in empirical terms. In the case of Georgakopoulou & Finnis, the national paradigm also emerges, diasporically this time, in defining the community as ‘Greek Cypriots in London’, and the same restrictions apply here. In both papers, there is a tension between the flexible and constantly shifting microscopic identity patterns (or ‘positioning’ patterns) that are analytically demonstrated, and the a priori and conservative categorisation of the research target groups that forms the point of departure of the research. The tension occurs whenever the ‘community’ is described as ‘changing’: Undoubtedly such changes occur, and some of these changes (e.g. the alternated or ‘confused’ use of ‘Greek’ and ‘Cypriot’ by Georgakopoulou & Finnis’ respondents) may call into question the foundations of the community. The reference point for such changes, we can see from such examples, need not be a traditional conception of national, ethnic or ethnolinguistic communities, because all of these categories are products of particular (‘modernist’) paradigms of scholarship, and features of the ‘modern’ nation-state apparatuses for distinguishing people. We see such apparatuses at work in Giaxoglou’s data, where an early (‘modernist’) philologist deploys particular orthographic tactics in the representations of

Unlike other tongues, Arabic language is characterized by its written form which is essentially consonant and may not have short vowels. One of the major functions of short vowels is to determine and facilitate the meaning of words or sentences. However, MSA texts are generally written without vowels. This fact gives rise to a great deal of morphological, semantic, and syntactic ambiguities. Thus, this ambiguity problem is not only associated with Modern Standard Arabic (MSA) but also related to Arabic dialects in general and Tunisian Dialect (TD) in particular. Compared to MSA, TD suffers from the unavailability of basic tools and linguistic resources, like sufficient amount of corpora, multilingual dictionaries, morphological and syntactic analyzers of these resources makes the processing of this language a great challenge (Masmoudi et al., 2020). Despite the numerous efforts currently underway, still some shortages persist in this field. Hence, we tried to challenge this lack by presenting our work that investigates the automatic diacritization of TD texts. In this respect, we regard the diacritization problem as a simplified phrase-based SMT (Statistical Machine Translation) task. The source language is the undiacritic text while the target language is the diacritic text. We initially go deeper into the details of TD corpus creation. This corpus is finally approved and used to build a diacritic restoration system for the TD. It is called TDTACHKIL and it can achieve a Word Error Rate (WER) of 16.7% and Diacritic Error Rate (DER) of 8.89%.

Diacritics (short vowels) are typically omitted when writing Arabic text, and readers have to reintroduce them to correctly pronounce words. There are two types of Arabic diacritics: The first are core-word diacritics (CW), which specify the lexical selection, and the second are case endings (CE), which typically appear at the end of word stems and generally specify their syntactic roles. Recovering CEs is relatively harder than recovering core-word diacritics due to inter-word dependencies, which are often distant. In this article, we use feature-rich recurrent neural network model that use a variety of linguistic and surface-level features to recover both core word diacritics and case endings. Our model surpasses all previous state-of-the-art systems with a CW error rate (CWER) of 2.9% and a CE error rate (CEER) of 3.7% for Modern Standard Arabic (MSA) and CWER of 2.2% and CEER of 2.5% for Classical Arabic (CA). When combining diacritized word cores with case endings, the resultant word error rates are 6.0% and 4.3% for MSA and CA, respectively. This highlights the effectiveness of feature engineering for such deep neural models.

In this study, stress diacritics were used to investigate the processing of stress information in lexical decision. We ran two experiments in Italian, a language in which stress position is not predictable by rule and only final stress—that is, the less common pattern—is orthographically marked with a diacritic. In Experiment 1, a lexical decision task, two factors were manipulated: the stress pattern of words—antepenultimate (nondominant) and penultimate (dominant)—and the presence/absence of the diacritics, signalling the stress position. Participants were faster to categorise stimuli as words when they bear dominant than nondominant stress. However, the advantage disappeared when the diacritic was used. In Experiment 2, a same-different verification task was used in which participants had to decide if a referent word and a target were same (carota-CAROTA, /ka’rɔta/; tavolo-TAVOLO, /’tavolo/) or different. We compared two conditions requiring a “different” response, in which referent and target with dominant and nondominant stress were congruent (caròta-CAROTA; tàvolo-TAVOLO) or incongruent (càrota-CAROTA; tavòlo-TAVOLO) with the word’s stress. For words with dominant stress, “different” responses were faster in the incongruent condition than the congruent condition. This congruency effect was not observed for words with nondominant stress pattern. Overall, the data suggest that stress information is based on lexical phonology, and the stress dominance effect has a lexical base in word recognition.

Research reveals that although the effect of diacritics) the symbols that tell the reader how to pronounce a letter) on reading and reading comprehension was examined in a series of studies, the in...

While building a text-to-speech system for the Arabic language, we found that the system synthesized speeches with many pronunciation errors. The primary source of these errors is the lack of diacritics in modern standard Arabic writing. These diacritics are small strokes that appear above or below each letter to provide pronunciation and grammatical information. We propose three deep learning models to recover Arabic text diacritics based on our work in a text-to-speech synthesis system using deep learning. The first model is a baseline model used to test how a simple deep learning model performs on the corpora. The second model is based on an encoder-decoder architecture, which resembles our text-to-speech synthesis model with many modifications to suit this problem. The last model is based on the encoder part of the text-to-speech model, which achieves state-of-the-art performances in both word error rate and diacritic error rate metrics. These models will benefit a wide range of natural language processing applications such as text-to-speech, part-of-speech tagging, and machine translation.

A Quranic optical character recognition (OCR) system based on convolutional neural network (CNN) followed by recurrent neural network (RNN) is introduced in this work. Six deep learning models are built to study the effect of different representations of the input and output, and the accuracy and performance of the models, and compare long short-term memory (LSTM) and gated recurrent unit (GRU). A new Quranic OCR dataset is developed based on the most famous printed version of the Holy Quran (Mushaf Al-Madinah), and a page and line-text image with the corresponding labels is prepared. This work's contribution is a Quranic OCR model capable of recognizing the Quranic image's diacritic text. A better performance in word recognition rate (WRR) and character recognition rate (CRR) is achieved in the experiments. The LSTM and GRU are compared in the Arabic text recognition domain. In addition, a public database is built for research purposes in Arabic text recognition that contains the diacritics and the Uthmanic script, and is large enough to be used with the deep learning models. The outcome of this work shows that the proposed system obtains an accuracy of 98% on the validation data, and a WRR of 95% and a CRR of 99% in the test dataset.

This paper presents a novel Arabic dataset that considers the characteristics of the Arabic language filling some gaps not covered by existing datasets. Conventional datasets consider Arabic in a similar way to Latin languages. These datasets either delete diacritic and supplement marks, considering them as defects, or keep them without considering the actual meaning. More than half of all Arabic characters have diacritics above or below characters. In this context, this work presents the novel Detailed Arabic Dataset (DAD) for bridging these gaps. The additional marks included in this dataset are the single dot, two dots "-", three dots "^", Hamza and two supplement marks: The bar for Tah, or Zah and the complement bar for Kaf. A special application was built to generate a dataset for Arabic online recognition and writer identification (called OFMArabicDatasetBuilder). Totally the ground truth contains 93064 entries based on sub-word and letter parts (not on words or lines as other datasets). This dataset will provide researchers with a strong tool for online Arabic language text recognition especially in the segmentation phase and writer identification. This paper also presents benchmarking results of using k-nearest neighbours machine learning with DAD.

Despite the modern boom in technology, we are still faced with the fact that people write texts without diacritics. There are two main reasons for this. The first, historical reason stems from the past when the use of diacritics was troublesome and people would write text without them. The second one is the speed - typing without diacritics is usually faster. Text without diacritics is easy to understand for people, but for some types of documents, missing diacritics can cause a problem. This is also an issue when computers process such text. In this paper, we propose an algorithm based on word n-grams (a contiguous sequence of n words) that can restore diacritics of text written in the Slovak language. We also compare and evaluate our results with other algorithms developed for Slovak text.

Introduction. The manuscript of The Sutra of the White Old Man — call no. B 228(1) — in the collection of the Institute of Oriental Manuscripts (RAS) is written in the Oirat Clear Script. The manuscript was compiled during the period when the traditional rules of graphics and spelling were strongly influenced by the spoken language. Goals. The article aims at eliciting special features of the handwriting and spelling of this particular text of the Sutra. Results. If compared to the classical rules, the changes traced in the text affected both the graphics (letters and diacritic signs) and spelling of words. The spelling patterns in the manuscript reflect (although inconsistently and irregularly) some phonetic processes that took place in the spoken language, such as the development of diphthongs into long vowels, assimilation, insertion and reduction of vowels. The sign ‘udan’ denoting long vowels is used in the text only occasionally, and reduplication of labial vowels in the same function is only preserved in monosyllabic words and final syllables. The author speculates that the ‘udan’ sign could have lost its meaning to become a decorative element of the handwriting in certain positions. The paper transliterates the text and publishes a facsimile of the manuscript.

PurposeNormalization is an important step in all the natural language processing applications that are handling social media text. The text from social media poses a different kind of problems that are not present in regular text. Recently, a considerable amount of work has been done in this direction, but mostly in the English language. People who do not speak English code mixed the text with their native language and posted text on social media using the Roman script. This kind of text further aggravates the problem of normalizing. This paper aims to discuss the concept of normalization with respect to code-mixed social media text, and a model has been proposed to normalize such text.Design/methodology/approachThe system is divided into two phases – candidate generation and most probable sentence selection. Candidate generation task is treated as machine translation task where the Roman text is treated as source language and Gurmukhi text is treated as the target language. Character-based translation system has been proposed to generate candidate tokens. Once candidates are generated, the second phase uses the beam search method for selecting the most probable sentence based on hidden Markov model.FindingsCharacter error rate (CER) and bilingual evaluation understudy (BLEU) score are reported. The proposed system has been compared with Akhar software and RB\_R2G system, which are also capable of transliterating Roman text to Gurmukhi. The performance of the system outperforms Akhar software. The CER and BLEU scores are 0.268121 and 0.6807939, respectively, for ill-formed text.Research limitations/implicationsIt was observed that the system produces dialectical variations of a word or the word with minor errors like diacritic missing. Spell checker can improve the output of the system by correcting these minor errors. Extensive experimentation is needed for optimizing language identifier, which will further help in improving the output. The language model also seeks further exploration. Inclusion of wider context, particularly from social media text, is an important area that deserves further investigation.Practical implicationsThe practical implications of this study are: (1) development of parallel dataset containing Roman and Gurmukhi text; (2) development of dataset annotated with language tag; (3) development of the normalizing system, which is first of its kind and proposes translation based solution for normalizing noisy social media text from Roman to Gurmukhi. It can be extended for any pair of scripts. (4) The proposed system can be used for better analysis of social media text. Theoretically, our study helps in better understanding of text normalization in social media context and opens the doors for further research in multilingual social media text normalization.Originality/valueExisting research work focus on normalizing monolingual text. This study contributes towards the development of a normalization system for multilingual text.

No artigo, esclareço o proveito que o educador poderá obter da transparência do sistema do português escrito para uma alfabetização excelente para a leitura, ilustrando tal transparência com a aplicação do princípio da economia e do valor do zero linguístico, para a atribuição do acento de intensidade às palavras mais frequentes do português escrito, isentando-as do acento gráfico. Sendo assim, todas as palavras com mais de duas sílabas que terminam pelas letras ‘a’, ‘e’, ‘o’, seguidas ou não de ‘s’ e por ‘em’, ‘ens’, ‘am’, sem nenhum acento gráfico (as mais frequentes do português escrito), são lidas como paroxítonas. Demonstro, igualmente, como o sistema do português escrito permite a identificação dos clíticos que terminam pelas letras ‘a’, ‘e’, ‘o’, seguidas ou não de ‘s’, em contraste com a impossibilidade de sua delimitação na cadeia da fala.

The current study examined the widely held, but un-tested, assumption that morphological decomposition can compensate for missing phonological information in reading opaque orthographies. In addition, we tested whether morphological decomposition can compensate for the phonological decoding deficits in readers with dyslexia. Hebrew provides a unique opportunity to test these questions as it has a rich Semitic morphology, and two versions of script: a transparent orthography (with diacritic marks, ‘pointed’) and an opaque orthography (without diacritic marks, ‘un-pointed’). In two experiments, one behavioral and one fMRI, skilled and dyslexic readers read aloud Hebrew nouns: half bi-morphemic (root + pattern) and half mono-morphemic (non-decomposable). Each word was presented both in the transparent orthography (pointed), and in the opaque orthography (un-pointed). While skilled readers were faster, and showed no effects of diacritics or morphology, dyslexic readers read pointed words more slowly than un-pointed words and bi-morphemic words faster than mono-morphemic words. The imaging results showed: 1) In both groups a morphological effect was found in un-pointed words, in left inferior and middle frontal gyri, associated with morpho-phonological decomposition. 2) Only readers with dyslexia showed a morphological effect in pointed words in the left occipito-temporal cortex, associated with orthographic processing. 3) Dyslexic readers also showed a positive association between morphological awareness and activation in the left occipito-temporal cortex during reading of all words, and activation in inferior frontal cortex during reading of un-pointed bi-morphemic words. Altogether, these findings suggest that in both typical and dyslexic readers morphological decomposition can compensate for the missing phonological information in an opaque orthography. The results also show that readers with dyslexia can rely on morphological decomposition to compensate for their deficits in phonological decoding. Finally, these results highlight the way in which unique language specific properties shape the neural mechanisms underlying typical and atypical reading.

Arabic text-to-speech synthesis from non-diacritized text is still a big challenge, because of unique Arabic language rules and characteristics. Indeed, the diacritic and gemination signs, which are special characters representing respectively short vowels and consonant doubling, have a major effect on accurate pronunciation of Arabic. However these signs are often not mentioned in written texts, since most of Arab readers are used to guess them from the context. To tackle this issue, this paper presents a grapheme-to-phoneme conversion system for Arabic, which constitutes the text processing module of a deep neural networks (DNN)-based Arabic TTS systems. In the case of Arabic text, this step starts with predicting the diacritic and gemination signs. In this work, this step was fully realized based on DNN. Finally, the grapheme-to-phoneme conversion of the diacritized text was achieved using the Buckwalter code. In comparison to state-of-the-art approaches, the proposed system gives a higher accuracy rate either for all phonemes or for each class, and high precision, recall and F1 score for each class of diacritic signs.

Arabic diacritics play a significant role in distinguishing words with the same orthography but different meanings, pronunciations, and syntactic functions. The presence of Arabic diacritics can be useful in many natural language processing applications, such as text-to-speech tasks, machine translation, and part-of-speech tagging. This article discusses the use of bidirectional long short-term memory neural networks with conditional random fields for Arabic diacritization. This approach requires no morphological analyzers, dictionary, or feature engineering, but rather uses a sequence-to-sequence schema. The input is a sequence of characters that constitute the sentence, and the output consists of the corresponding diacritic(s) for each character in that sentence. The performance of the proposed approach was examined using four datasets with different sizes and genres, namely, the King Abdulaziz City for Science and Technology text-to-speech (KACST TTS) dataset, the Holy Quran, Sahih Al-Bukhary, and the Penn Arabic Treebank (ATB). For training, 60% of the sentences were randomly selected from each dataset, 20% were selected for validation, and 20% were selected for testing. The trained models achieved diacritic error rates of 3.41%, 1.34%, 1.57%, and 2.13% and word error rates of 14.46%, 4.92%, 5.65%, and 8.43% on the KACST TTS, Holy Quran, Sahih Al-Bukhary, and ATB datasets, respectively. Comparison of the proposed method with those used in other studies and existing systems revealed that its results are comparable to or better than those of the state-of-the-art methods.

Diacritic restoration is a challenging problem in natural language processing (NLP). With diacritic restoration, one can text faster and easier. Diacritic restoration is also helpful in making use of diacritic-missing texts, which are normally discarded in many NLP applications. This paper deals with the diacritic restoration problem for Vietnamese text. Three state- of-the-art deep learning models including Gated Recurrent Unit, Bidirectional Long-short Term Memory and Bidirectional Gated Recurrent Unit have been examined for the problem and the last one turned out to be the best among them. Besides deep learning models, it was found in this paper that word tokenization, which is the final pre-processing step applied on the data before feeding it to deep learning models also have influences on the final accuracy. Between two examined word tokenization methods: morpheme-based tokenization and phrase-based tokenization, the former yield better results regardless of the applied deep learning models. The experimental results show that the combination of morpheme-based tokenization and Bidirectional-GRU achieve the best performance of diacritic restoration with the Bleu-score of 88.06%.

Are letters with a diacritic (e.g., a) recognized as a variant of the base letter (e.g., a), or as a separate letter identity? Two recent masked priming studies, one in French and one in Spanish, investigated this question, concluding that this depends on the language-specific linguistic function served by the diacritic. Experiment 1 tested this linguistic function hypothesis using Japanese kana, in which diacritics signal consonant voicing, and like French and unlike Spanish, provide lexical contrast. Contrary to the hypothesis, Japanese kana yielded the pattern of diacritic priming like Spanish. Specifically, for a target kana with a diacritic (e.g., (sic), /ga/), the kana prime without the diacritic (e.g., (sic), /ka/) facilitated recognition almost as much as the identity prime (e.g., (sic) = (sic)), whereas for a target kana without a diacritic, the kana prime with the diacritic produced less facilitation than the identity prime (e.g., (sic) < (sic)). We suggest that the pattern of diacritic priming has little to do with linguistic function, and instead it stems from a general property of visual object recognition. Experiment 2 tested this hypothesis using visually similar letters of the Latin alphabet that differ in the presence/absence of a visual feature (e.g., O and Q). The same asymmetry in priming was observed. These findings are consistent with the noisy channel model of letter/word recognition (Norris & Kinoshita, Psychological Review, 119, 517-545, 2012a).

The proposed article substantiates the importance of dictionary works of one dialect, which form a general-linguistic dictionary register. It is a strong factor source for advanced scientific studies in lexicology, language history, comparative research, etc. The fragment of the dictionary presented in the article is based on the author's records in Rosokhi village of Starosambir district of Lviv region in July 2010. The dialect is interesting, because, first of all, it belongs to the transitional dialects: boyko dialect, Sian dialect, naddnistriannia dialect, which meet together on the western territory of Lviv region suburbs. The objective of such dictionary is to present the vocabulary of the dialect researched as fully as possible. In order to provide a dictionary for the widest range of interested readers, simplified text transcription and diacritic accent are applied. It is noted that there are words, set phrases, phraseologisms and paremias units included in the dictionary register of Rosokhy village. This vocabulary is known to the literary language (denoted by local features in pronunciation, words stressing, grammar or has another meaning different from a standard language) and narrow-sided.

Words in Arabic consist of letters and short vowel symbols called diacritics inscribed atop regular letters. Changing diacritics may change the syntax and semantics of a word; turning it into another. This results in difficulties when comparing words based solely on string matching. Typically, Arabic NLP applications resort to morphological analysis to battle ambiguity originating from this and other challenges. In this article, we introduce three alternative algorithms to compare two words with possibly different diacritics. We propose the Subsume knowledge-based algorithm, the Imply rule-based algorithm, and the Alike machine-learning-based algorithm. We evaluated the soundness, completeness, and accuracy of the algorithms against a large dataset of 86,886 word pairs. Our evaluation shows that the accuracy of Subsume (100%), Imply (99.32%), and Alike (99.53%). Although accurate, Subsume was able to judge only 75% of the data. Both Subsume and Imply are sound, while Alike is not. We demonstrate the utility of the algorithms using a real-life use case -- in lemma disambiguation and in linking hundreds of Arabic dictionaries.

Social media platforms such as Twitter have grown at a tremendous pace in recent years and have become an important source of data providing information countless field. This situation was of interest to researchers and many studies on machine learning and natural language processing was conducted on social media data. However, the language is used in social media contains a very high amount of noisy data than the formal writing language. In this article, we present a study on diacritic restoration which is one of the important difficulties of social media text normalization in order to reduce the noise problem. Diacritic is a set of marks used to change the sound values of letters and is used on many languages besides Turkish. We suggest a 3-step model for this study to overcome the top of the diacritic restoration problem. In the first step, a candidate word generator produces possible word forms, in the second step the language validator chooses the correct word forms and at the final Word2vec is used to create vector representations of the words and make the most appropriate word choice by using cosine similarities. The proposed method was tested on both the 2 ad-hoc created datasets and the real dataset. Studies on small ad-hoc created dataset and real dataset provided a relative error reduction of 37.8% with an average performance of 94.5%. In addition, tests on more than 6 M words on large ad-hoc created dataset yielded a serious performance with an error rate of 3.9%. Furthermore, the proposed method was tested on the binary classification problem consisting of highway traffic data in order to evaluate the effects on classification performance, and a 3.1% increase in classification performance was achieved. Keywords: Text mining, Diacritics restoration, Twitter, Tweet normalization

Arabic dialects differ substantially from modern standard arabic and each other in terms of phonology, morphology, lexical choice and syntax. This makes the identification of dialects from speeches a very difficult task. In this paper, we introduce a speech recognition system that automatically identifies the gender of speaker, the emphatic letter pronounced and also the diacritic of these emphatic letters given a sample of author’s speeches. Firstly we examined the performance of the single case classifier hidden markov models (HMM) applied to the samples of our data corpus. Then we evaluated our proposed approach KNN-DT which is a hybridization of two classifiers namely decision trees (DT) and k-nearest neighbors (KNN). Both models are singularly applied directly to the data corpus to recognize the emphatic letter of the sound and to the diacritic and the gender of the speaker. This hybridization proved quite interesting; it improved the speech recognition accuracy by more than 10% compared to state-of-the-art approaches.

The development and availability of high quality corpus for many African languages is still hampered by dearth of appropriate software tools and devices. To be able to rapidly create large quantities of high quality corpus of majority of African and Nigerian languages, a diacritic tool is required. The presentation of texts of natural languages without diacritic marks presents significant problems to both human and computational processing systems due to partial or total loss of the accompanying grammatical, syntactic and or semantic information. This paper investigated the effect of diacritic quality of a small-sized training corpus on the classification accuracy of some simple and commonly used machine learning algorithms for diacritic restoration tasks following the character-based approach. The classification accuracy of eight of the diacritic-bearing characters of Yoruba language of Nigeria were investigated. The results show that the completeness and correctness of diacritics has a significant effect on the performance of the algorithms; decision tree algorithm produced the overall best accuracy response of 3.22 % to the data quality improvement. The observations from the learning behaviours of the algorithms suggests that a 100,000 words corpus is adequate to train a decision tree model for automatic diacritic restoration for Yoruba language but insufficient to obtain a state-of-the art results for the LDA, LOGREG and SVM algorithms.

Potentialized by Natural Language Processing (NLP) technology, we can build a chatbot or an AI Agent to automatically address the need to automatically get credible and timely information, especially in the fight against epidemics. However, Vietnamese understanding is still a big challenge for NLP. This paper introduces an AI Agent using the Attention algorithm and Albert model to implement the question/answering task in the Covid-19 field for the Vietnamese language. In the end, we also built two other modules, one for Vietnamese diacritic auto-correction and another for updating Covid-19 statistics (using RASA framework), to deploy a Covid-19 chatbot application on mobile devices.

Gujarati is the language used for everyday communication in the state of Gujarat, India. The Gujarati language is also officially recognized by the constitution and the government of India. Gujarati script is based on the Devanagari script. An idiom is an expression, phrase, or word that has a different meaning from the literal meaning of the words in it. Idioms represent the cultural heritage of Gujarati language. Idioms are used in Gujarati language for effective communication and convey of an accurate message. No Machine Translation System does the accurate translation of Gujarati idioms to English or any other language. Different idiom phrases can be generated by adding diacritic(s) as well as suffix to the root or base form of the idiom. Many forms of single idiom make automatic idiom identification as well as machine translation more challenging. This paper focuses on the design and implementation of diacritics and suffix-based rules for dynamic phrase generation and detection of idioms of Gujarati language. This implementation helps in identifying Gujarati idiom present in any possible form in the Gujarati text. The obtained results with the execution of 7050 different Gujarati idiom phrases yield an accuracy of 99.73%. The results are encouraging enough to make the proposed implementation useful for Natural Language processing tasks related to Gujarati language idioms.

Automatic Arabic diacritization is one of the most important and challenging problems in Arabic natural language processing (NLP). Recurrent neural networks (RNNs) have proved recently to achieve state-of-the-art results for sequence transcription problems in general, and Arabic diacritization in specific. In this work, we investigate the effect of varying the size of the training corpus on the accuracy of diacritization. We produce a cleaned corpus of approximately 550k sequences extracted from the full dataset of Tashkeela and use subsets of this corpus in our training experiments. Our base model is a deep bidirectional long short-term memory (BiLSTM) RNN that transcribes undiacritized sequences of Arabic letters with fully diacritized sequences. Our experiments show that error rates improve as the size of training corpus increases. Our best performing model achieves average diacritic and word error rates of 1.45% and 3.89%, respectively. When compared with state-of-the-art diacritization systems, we reduce the word error rate by 12% over the best published results.

Objective: We recently showed that genderize.io is not a sufficiently powerful gender detection tool due to a large number of nonclassifications. In the present study, we aimed to assess whether the accuracy of inference by genderize.io can be improved by manipulating the first names in the database.Methods: We used a database containing the first names, surnames, and gender of 6,131 physicians practicing in a multicultural country (Switzerland). We uploaded the original CSV file (file #1), the file obtained after removing all diacritic marks, such as accents and cedilla (file #2), and the file obtained after removing all diacritic marks and retaining only the first term of the compound first names (file #3). For each file, we computed three performance metrics: proportion of misclassifications (errorCodedWithoutNA), proportion of nonclassifications (naCoded), and proportion of misclassifications and nonclassifications (errorCoded).Results: naCoded, which was high for file #1 (16.4%), was reduced after data manipulation (file #2: 11.7%, file #3: 0.4%). As the increase in the number of misclassifications was small, the overall performance of genderize.io (i.e., errorCoded) improved, especially for file #3 (file #1: 17.7%, file #2: 13.0%, and file #3: 2.3%).Conclusions: A relatively simple manipulation of the data improved the accuracy of gender inference by genderize.io. We recommend using genderize.io only with files that were modified in this way.

Abugida refers to a phonogram writing system where each syllable is represented using a single consonant or typographic ligature, along with a default vowel or optional diacritic(s) to denote other vowels. However, texting in these languages has some unique challenges in spite of the advent of devices with soft keyboard supporting custom key layouts. The number of characters in these languages is large enough to require characters to be spread over multiple views in the layout. Having to switch between views many times to type a single word hinders the natural thought process. This prevents popular usage of native keyboard layouts. On the other hand, supporting romanized scripts (native words transcribed using Latin characters) with language model based suggestions is also set back by the lack of uniform romanization rules. To this end, we propose a disambiguation algorithm and showcase its usefulness in two novel mutually non-exclusive input methods for languages natively using the abugida writing system: (a) disambiguation of ambiguous input for abugida scripts, and (b) disambiguation of word variants in romanized scripts. We benchmark these approaches using public datasets, and show an improvement in typing speed by 19.49%, 25.13%, and 14.89%, in Hindi, Bengali, and Thai, respectively, using Ambiguous Input, owing to the human ease of locating keys combined with the efficiency of our inference method. Our Word Variant Disambiguation (WDA) maps valid variants of romanized words, previously treated as Out-of-Vocab, to a vocabulary of 100k words with high accuracy, leading to an increase in Error Correction F1 score by 10.03% and Next Word Prediction (NWP) by 62.50% on average., Comment: Published in 2021 IEEE 15th International Conference on Semantic Computing (ICSC)

Diacritic characters can be considered as a unique set of characters providing us with adequate and significant clue in identifying a given language with considerably high accuracy. Diacritics, though associated with phonetics often serve as a distinguishing feature for many languages especially the ones with a Latin script. In this proposed work, we aim to identify language of text in images using the presence of diacritic characters in order to improve Optical Character Recognition (OCR) performance in any given automated environment. We showcase our work across 13 Latin languages encompassing 85 diacritic characters. We use an architecture similar to Squeezedet for object detection of diacritic characters followed by a shallow network to finally identify the language. OCR systems when accompanied with identified language parameter tends to produce better results than sole deployment of OCR systems. The discussed work apart from guaranteeing an improvement in OCR results also takes on-device (mobile phone) constraints into consideration in terms of model size and inference time.

Today, chatbots or conversational agents are increasingly being used in various fields. In recent years, governments, organizations, and businesses have invested in dialogue systems to improve the engagement of their users. A smart chatbot is a chatbot that can understand the user’s intents. To achieve this goal, the developer always needs to improve the ability to classify the user’s intents. However, the user’s raw text usually contains spelling errors, abbreviations, or slang that the chatbot cannot understand. Therefore, processing the input text is also an urgent task of natural language understanding (NLU). In the Vietnamese dialogue system, there are always cases when users enter questions without signs. These questions greatly affect the understanding of the user’s intents. In this paper, an improvement in the intent classification for a Vietnamese chatbot is presented. We propose the Encoder-Decoder Bidirectional Long Short-Term Memory (BiLSTM) model for the diacritic restoration. On our evaluation dataset, this approach has an accuracy of 99.12% by using the pre-trained language model for word embedding. Then, the proposed model is applied as a custom component in the NLU pipeline. Using the proposed approach for constructing a model of intent classification is considered, comparison with basic models has shown the best results for the proposed model.

Sinhala script is categorized as a segmental writing system and therefore consonant-vowel sequences are identified as a unit. Depending on the vowel and the consonant combination, the diacritic can attach above, below, following or preceding the consonant. The identification of characters with the modifier symbols is important in recognizing generally written Sinhala text and has not been addressed sufficiently in the existing research. Sinhala handwriting recognition is much difficult compared to the other popular languages due to the complexity of the shapes of the characters and the presence of the modifier symbols. This paper discusses on a projection profile, distance profile, partial distance profile and contour based approach to identify characters along with character modifiers in Sinhala script. The proposed method has given an average recognition rate of 75% for 283 character modifier combinations. The proposed solution can be used to identify Sinhala handwritten text with a proper segmentation mechanism.

Speech sound disorders affect 10% of preschool and school-age children, adversely affecting their communication, academic performance, and interaction level. Effective pronunciation training requires prolonged supervised practice and interaction. Unfortunately, many children have limited or no access to a speech-language pathologist. Computer-assisted pronunciation training has the potential for being a highly effective teaching aid; however, to-date such systems remain incapable of identifying pronunciation errors with sufficient accuracy. We propose a system that combines a multi-target architecture with weighted finite-state transducers to first segment and then analyze an utterance in terms of its phonological features. We analyze a corpus of 90 children aged 4-7 and find differences between the typically developing and the speech disordered groups.

Arabic is a broadly utilized alphabetic composition framework on the planet, and it has 28 essential letters. The letters in order was first used to compose messages in Arabic, most prominently the Qur'an the holy book of Islam. However, Arabic language has diacritics in the word or letters which are not something extra or discretionary to the language, rather they are a vital piece of it. By changing some diacritics may change both the syntax and semantics of a word by turning a word into another. However, the current researches address the foreground image and consider the diacritics as noises or secondary images. Thus, it is not suitable for Arabic handwritten. The diacritics will be removed from the image and this will lead to losing some good features. Furthermore, to extract the diacritics, the region-based segmentation technique is used. The image will be measured based on the region properties by first finding the connected component in binary image, and then we will determine the best area range measurement in that region for each image. The proposed technique region based has been tested in nine different images with different handwritten style, and successfully extracted secondary foreground images (diacritics) for each image.

In many languages like Arabic, diacritics are used to specify pronunciations as well as meanings. Such diacritics are often omitted in written text, increasing the number of possible pronunciations and meanings for a word. This results in a more ambiguous text making computational processing on such text more difficult. Diacritic restoration is the task of restoring missing diacritics in the written text. Most state-of-the-art diacritic restoration models are built on character level information which helps generalize the model to unseen data, but presumably lose useful information at the word level. Thus, to compensate for this loss, we investigate the use of multi-task learning to jointly optimize diacritic restoration with related NLP problems namely word segmentation, part-of-speech tagging, and syntactic diacritization. We use Arabic as a case study since it has sufficient data resources for tasks that we consider in our joint modeling. Our joint models significantly outperform the baselines and are comparable to the state-of-the-art models that are more complex relying on morphological analyzers and/or a lot more data (e.g. dialectal data).

Keyword spotting is an alternative methodology for document indexing based on spotting words directly on images without the use of a character recognition system. In this paper, an overview of recent techniques and available databases for keyword spotting is presented focusing on the specific characteristics of old Greek machine-printed and handwritten documents. These documents are import treasures of cultural heritage and a valuable source of information for scholars. Indexing of such content is a very challenging task considering the additional problem of having many character classes as well as a variety of different diacritic mark combinations that may appear above or below Greek characters. All steps of a keyword spotting system are highlighted, namely, preprocessing for image binarization, enhancement and segmentation, feature representation, matching and word retrieval, in order to report the efficiency of current keyword spotting approaches when applied to old Greek documents.

Speech synthesis is the artificial production of human speech. A typical text-to-speech system converts a language text into a waveform. There exist many English TTS systems that produce mature, natural, and human-like speech synthesizers. In contrast, other languages, including Arabic, have not been considered until recently. Existing Arabic speech synthesis solutions are slow, of low quality, and the naturalness of synthesized speech is inferior to the English synthesizers. They also lack essential speech key factors such as intonation, stress, and rhythm. Different works were proposed to solve those issues, including the use of concatenative methods such as unit selection or parametric methods. However, they required a lot of laborious work and domain expertise. Another reason for such poor performance of Arabic speech synthesizers is the lack of speech corpora, unlike English that has many publicly available corpora (LjSpeech, https://keithito.com/LJ-Speech-Dataset/., Blizzard 2012, http://www.cstr.ed.ac.uk/projects/blizzard/2012/phase_one/.) and audiobooks. This work describes how to generate high quality, natural, and human-like Arabic speech using an end-to-end neural deep network architecture. This work uses just \(\langle \)text, audio\(\rangle \) pairs with a relatively small amount of recorded audio samples with a total of 2.41 h. It illustrates how to use English character embedding despite using diacritic Arabic characters as input and how to preprocess these audio samples to achieve the best results.

The purpose of this article is to describe and analyze one of the manuscripts of the collection of reports of denunciation of the Metropolitan Curia of Sao Paulo. Given its historical and linguistic importance, we will address the main points that refer to the philology description of the document. Considering the different linguistic aspects present in the manuscript, the use of the acute diacritic will be analyzed, trying to connect to the function that is given to it by the grammarians of the time and its applicability.

Restoring syntactic diacritics is an essential task for Arabic text-to-speech systems. Syntactic diacritic mark is defined as the last diacritic mark of the stem of a whitespace delimited word. It is usually assigned based on the syntax of the Arabic language. In this paper, we formulate the problem as a tagging problem and propose the use of long short-term memory (LSTM) networks to assign the syntactic diacritics for a sentence of Arabic words. These LSTM networks were augmented with sparse direct connections between the input and output layers of the tagger (i.e., maximum entropy (MaxEnt) connections). On the Arabic tree bank task, this hybrid LSTM/MaxEnt approach achieves competitive results to the state-of-the-art systems.

W artykule zaprezentowano w skrócie najstarsze etapy rozwoju czeskiej i polskiej grafii, ponieważ omawiane tutaj słowiańskie języki oddziaływały na siebie od momentu ich powstawania. Jak wynika z artykułu, w obydwu badanych systemach językowych można wyodrębnić analogiczne okresy funkcjonowania grafii prostej (pravopis jednoduchý), złożonej (pravopis spřeťkový) i diakrytycznej (diakritický pravopis). W efekcie różnorakich zmian na czeskim gruncie językowym powstała grafia diakrytyczna (wyjątek stanowi dziś "ch"), w polszczyźnie natomiast – jak wiadomo – zaczął funkcjonować połączony system grafii złożonej i diakrytycznej. The article presents the oldest stages of the development of the Czech and Polish graphics. The Slavic languages discussed here interacted with each other from the moment they were created. As can be seen from the article, in both studied language systems it is possible to distinguish analogous periods of simple graphic (pravopis jednoduchý), composed (pravopis spřeťkový) and diacritic (diakritický pravopis). As a result of various changes on the Czech linguistic ground diacritic graphics were created (the exception is today "ch"). In Polish – as you know – started to function the combined system of complex and diacritical graphics.

Lexical ambiguity, a challenging phenomenon in all natural languages, is particularly prevalent for languages with diacritics that tend to be omitted in writing, such as Arabic. Omitting diacritics leads to an increase in the number of homographs: different words with the same spelling. Diacritic restoration could theoretically help disambiguate these words, but in practice, the increase in overall sparsity leads to performance degradation in NLP applications. In this paper, we propose approaches for automatically marking a subset of words for diacritic restoration, which leads to selective homograph disambiguation. Compared to full or no diacritic restoration, these approaches yield selectively-diacritized datasets that balance sparsity and lexical disambiguation. We evaluate the various selection strategies extrinsically on several downstream applications: neural machine translation, part-of-speech tagging, and semantic textual similarity. Our experiments on Arabic show promising results, where our devised strategies on selective diacritization lead to a more balanced and consistent performance in downstream applications., accepted in WANLP 2019