Search

The proliferation of hoaxes on the Internet and toxic messages (with offensive or very negative content and high dissemination rates) constitute a current problem on the road to truthful and respectful information. Due to the enormous influence that social media have in the generation of opinion and as a channel of information for society, the efforts that various companies, organisations and institutions are making to detect and counteract the high volume of disinformation circulating on the networks are important. This project deals with the implementation of a proof of concept of a system for analysing the fairness of messages published through social media, built on the basis of various methods and algorithms from human language technologies. These methods and algorithms are the result of research that the participating groups have been working on for the last few years and are promising solutions for the determination of different levels of quality of publications in two fundamental aspects: their veracity and their toxicity. To address the proof of concept, activities aimed at the definition and integration of these technologies and their evaluation by stakeholders are proposed. This will make it possible to establish the responsiveness of these technologies to the needs of society and industry, as well as their viability to work towards higher levels of technological maturity.

[Abstract]: Recent advances in neural networks based language representation made it possible for pretrained language models to outperform previous models in many downstream natural language processing (NLP) tasks. These pretrained language models have also shown that if large enough, they exhibit good few-shot abilities, which is especially beneficial for low-resource scenarios. In this respect, although there are some large-scale multilingual pretrained language models available, language-specific pretrained models have demonstrated to be more accurate for monolingual evaluation setups. In this work, we present BERTbek - pretrained language models based on the BERT (Bidirectional Encoder Representations from Transformers) architecture for the low-resource Uzbek language. We also provide a comprehensive evaluation of the models on a number of NLP tasks: sentiment analysis, multi-label topic classification, and named entity recognition, comparing the models with various machine learning methods as well as multilingual BERT (mBERT). Experimental results indicate that our models outperform mBERT and other task-specific baseline models in all three tasks. Additionally, we also show the impact of training data size and quality on the downstream performance of BERT models, by training three different models with different text sources and corpus sizes.

The Grand Duchy of Luxembourg is a small country in Western Europe, which, despite its size, is an important global financial centre. Due to its highly multilingual population, and the fact that one of its national languages - Luxembourgish - is regarded as a low-resource language, this country lends itself naturally to a wide variety of interesting research opportunities in the domain of Natural Language Processing (NLP). This thesis discusses and addresses challenges with regard to domain-specific and language-specific NLP, using the unique linguistic situation in Luxembourg as an elaborate case study. We focus on three main topics: (I) NLP challenges present in the financial domain, specifically handling personal names in sensitive documents, (II) NLP challenges related to multilingualism, and (III) NLP challenges for low-resource languages with Luxembourgish as the language of interest. With regard to NLP challenges in the financial domain, we address the challenge of finding and anonymising names in documents. Firstly, an empirical study on the usefulness of Transformer-based deep learning models is presented on the task of Fine-Grained Named Entity Recognition. This empirical study was conducted for a wide array of domains, including the financial domain. We show that Transformer-based models, and in particular BERT models, yield the best performance for this task. We furthermore show that the performance is also strongly dependent on the domain itself, regardless of the choice of model. The automatic detection of names in text documents in turn facilitates the anonymisation of these documents. However, anonymisation can distort data and have a negative effect on models that are built on that data. We investigate the impact of anonymisation of personal names on the performance of deep learning models trained on a large number of NLP tasks. Based on our experiments, we establish which anonymisation strategy should be used to guarantee accurate NLP models. Regar

The Grand Duchy of Luxembourg is a small country in Western Europe, which, despite its size, is an important global financial centre. Due to its highly multilingual population, and the fact that one of its national languages - Luxembourgish - is regarded as a low-resource language, this country lends itself naturally to a wide variety of interesting research opportunities in the domain of Natural Language Processing (NLP). This thesis discusses and addresses challenges with regard to domain-specific and language-specific NLP, using the unique linguistic situation in Luxembourg as an elaborate case study. We focus on three main topics: (I) NLP challenges present in the financial domain, specifically handling personal names in sensitive documents, (II) NLP challenges related to multilingualism, and (III) NLP challenges for low-resource languages with Luxembourgish as the language of interest. With regard to NLP challenges in the financial domain, we address the challenge of finding and anonymising names in documents. Firstly, an empirical study on the usefulness of Transformer-based deep learning models is presented on the task of Fine-Grained Named Entity Recognition. This empirical study was conducted for a wide array of domains, including the financial domain. We show that Transformer-based models, and in particular BERT models, yield the best performance for this task. We furthermore show that the performance is also strongly dependent on the domain itself, regardless of the choice of model. The automatic detection of names in text documents in turn facilitates the anonymisation of these documents. However, anonymisation can distort data and have a negative effect on models that are built on that data. We investigate the impact of anonymisation of personal names on the performance of deep learning models trained on a large number of NLP tasks. Based on our experiments, we establish which anonymisation strategy should be used to guarantee accurate NLP models. Regar

Natural language modeling is used to predict or generate the next word or character of modern languages. Furthermore, statistical character-based language models have been found useful in authorship attribution analyses by studying the linguistic proximity of excerpts unknown to the model. In prior work, we modeled Homeric language and provided empirical findings regarding the authorship nature of the 48 Iliad and Odyssey books. Following this line of work, and considering the current philological views and trends, we break down the two poems further into smaller portions. By employing language modeling we identify outlying passages, indicating reduced linguistic affinity with the main body of the two works and, by extension, potentially different authorship. Our results show that some of the passages isolated as outliers by the language models were also identified as such by human researchers. We further test our methodology and models on texts of similar language and genre created by other authors, namely Hesiod’s “Theogony” and “Work and Days”.

Software Engineering practices are changing in an age of artificial intelligence. While the core activities of design, develop, maintain, test and evaluate remain, the methods used in these activities are evolving. The prevalence of generative programming models has the potential to reconstitute the duties of a software engineer. Widely adopted models like Copilot and Bard are IDE-based pair-programming assistants that create code from virtually any input: contextual code, natural language, specifications, input output pairs, etc. The way developers interact with these models will redefine some core ideas of software engineering. These models empower virtually anyone, of varying coding proficiency, to create software. Models with the capacity to code will surely manage to inherit software design and analysis capabilities [186], albeit for now, with specific training or prompting.Naturally, one wonders how language modeling, or more specifically the modeling of source code and its features, will impact developers. Researchers often conjecture on the varying degree of influence these methods will have, but certainly, these tools will support developers in new and existing tasks: code completion, bug and vulnerability detection, code summarization, type annotation, and more are already prominent use cases. One can envision a world where software developers delegate portions of their work to machine learning pipelines, such as unit testing and vulnerability testing of their code; how much of that code they actually write is up for debate as well. Developers will likely automate portions of their work flow but simultaneously gain new tasks and responsibilities. These tasks might include passing automatic code reviews that detect code smells, place code comments automatically, and detect refactorings; maybe using models from [56], [133], [59]. These capabilities come from modeling source code and its features directly by distilling down meaningful representations for the

Transformer-based language models have led to substantial progress in natural language processing. However, these models struggle to model long dependency, because the amount of computation in the attention layer grows quadratically with the length of the input. In this paper, I propose a special cache, namely the selective cache, which stores the selected key-value pairs from the previous context. When the model processes long documents step by step, at each step the model is fed with a slice of the document, and the key-value pairs in the attention layer are saved into the cache. The proposed cache differs from previous caches because it selects those key-value pairs that are likely to be more important. By selection, the model makes better use of the cache so that in limited cache size, a longer history context can be stored. I designed three kinds of selection methods. One is based on eye-tracking data. If the key-value pairs correspond to tokens that are long-fixated by humans, they are selected. To enable the application to more tasks, I use neural networks to predict human fixation duration. The other two selecting methods are to train additional small neural networks, namely the selector and the replacer. They are integrated into the language model and can learn from the language modeling task. I utilized the method from neural network pruning, which is based on stochastic gates and a differentiable relaxation of the L0 norm. By learning to prune some key-value pairs, the selector can be used to select the pairs that are needed by the language model. On the other hand, the replacer learns to trade off between adding new items or keeping the old ones. I demonstrate that the proposed cache mechanism improves the language modeling performance across different datasets. With the same amount of stored key-value pairs (cache size), the proposed cache outperforms XL-cache (Transformer XL) by a considerable margin. What’s more, through various qualitative and quanti

Transformer-based language models have led to substantial progress in natural language processing. However, these models struggle to model long dependency, because the amount of computation in the attention layer grows quadratically with the length of the input. In this paper, I propose a special cache, namely the selective cache, which stores the selected key-value pairs from the previous context. When the model processes long documents step by step, at each step the model is fed with a slice of the document, and the key-value pairs in the attention layer are saved into the cache. The proposed cache differs from previous caches because it selects those key-value pairs that are likely to be more important. By selection, the model makes better use of the cache so that in limited cache size, a longer history context can be stored. I designed three kinds of selection methods. One is based on eye-tracking data. If the key-value pairs correspond to tokens that are long-fixated by humans, they are selected. To enable the application to more tasks, I use neural networks to predict human fixation duration. The other two selecting methods are to train additional small neural networks, namely the selector and the replacer. They are integrated into the language model and can learn from the language modeling task. I utilized the method from neural network pruning, which is based on stochastic gates and a differentiable relaxation of the L0 norm. By learning to prune some key-value pairs, the selector can be used to select the pairs that are needed by the language model. On the other hand, the replacer learns to trade off between adding new items or keeping the old ones. I demonstrate that the proposed cache mechanism improves the language modeling performance across different datasets. With the same amount of stored key-value pairs (cache size), the proposed cache outperforms XL-cache (Transformer XL) by a considerable margin. What’s more, through various qualitative and quanti

As music video streaming occupies a significant market share in how people consume music, gaining an understanding of user behavioural patterns becomes increasingly crucial. This understanding can enable better music video streaming experiences by tailoring them towards more personalized and user-centric designs. Though prior works have emphasized user behaviour during solely listening to music, understanding user actions/clicks while consuming music videos remains largely unexplored. Given the unique experience offered by the combination of audio and visual elements, there is a need for focused research in this area. Therefore this study attempts to bridge this research gap by collecting and analysing a large dataset of streaming sessions from a music video streaming company - XITE. In total, we analyzed 1.8 million sessions from approximately 270,000 unique users. The behaviour exhibited during those sessions is interpreted as a language and modelled using the Language Model - Doc2Vec. This facilitated the conversion of session action sequences into embeddings. Our findings suggest that music video streaming sessions exhibit cohesive user interaction patterns, which can be grouped into distinct clusters, thereby enabling the detection of distinct behavioural patterns across user sessions. Furthermore, previous studies have indicated that user interactions with multimedia streaming platforms can be influenced by the context in which content is consumed. Extending these findings, our analysis of behavioural clusters revealed that certain user behaviours while consuming music videos are associated with specific music video genres and temporal factors. For instance, we discovered that passive sessions predominantly commence around 10 am, while sessions requiring more active engagement typically start in the evening. The insights derived from this study are valuable for improving user-centric design in music video streaming platforms and providin, Computer Science

With careful manipulation, malicious agents can reverse engineer private information encoded in pre-trained language models. Security concerns motivate the development of quantum pre-training. In this work, we propose a highly portable quantum language model (PQLM) that can easily transmit information to downstream tasks on classical machines. The framework consists of a cloud PQLM built with random Variational Quantum Classifiers (VQC) and local models for downstream applications. We demonstrate the ad hoc portability of the quantum model by extracting only the word embeddings and effectively applying them to downstream tasks on classical machines. Our PQLM exhibits comparable performance to its classical counterpart on both intrinsic evaluation (loss, perplexity) and extrinsic evaluation (multilingual sentiment analysis accuracy) metrics. We also perform ablation studies on the factors affecting PQLM performance to analyze model stability. Our work establishes a theoretical foundation for a portable quantum pre-trained language model that could be trained on private data and made available for public use with privacy protection guarantees. © 2023 IEEE.

The current largest open-source generic automatic speech recognition (ASR) system for Dutch, Kaldi NL, does not include a domain-specific healthcare jargon in the lexicon. Commercial alternatives (e.g., Google ASR system) are also not suitable for this purpose, not only because of the lexicon issue, but they do not safeguard privacy of sensitive data sufficiently and reliably. These reasons motivate that just a small amount of medical staff employs speech technology in the Netherlands. This paper proposes an innovative ASR training method developed within the Homo Medicinalis (HoMed) project. On the semantic level it specifically targets automatic transcription of doctor-patient consultation recordings with a focus on the use of medicines. In the first stage of HoMed, the Kaldi NL language model (LM) is fine-tuned with lists of Dutch medical terms and transcriptions of Dutch online healthcare news bulletins. Despite the acoustic challenges and linguistic complexity of the domain, we reduced the word error rate (WER) by 5.2%. The proposed method could be employed for ASR domain adaptation to other domains with sensitive and special category data. These promising results allow us to apply this methodology on highly sensitive audiovisual recordings of patient consultations at the Netherlands Institute for Health Services Research (Nivel).

[EN] Although Long-Short Term Memory (LSTM) networks and deep Transformers are now extensively used in offline ASR, it is unclear how best offline systems can be adapted to work with them under the streaming setup. After gaining considerable experience on this regard in recent years, in this paper we show how an optimized, low-latency streaming decoder can be built in which bidirectional LSTM acoustic models, together with general interpolated language models, can be nicely integrated with minimal performance degradation. In brief, our streaming decoder consists of a one-pass, real-time search engine relying on a limited-duration window sliding over time and a number of ad hoc acoustic and language model pruning techniques. Extensive empirical assessment is provided on truly streaming tasks derived from the well-known LibriSpeech and TED talks datasets, as well as from TV shows on a main Spanish broadcasting station.

[EN] Although Long-Short Term Memory (LSTM) networks and deep Transformers are now extensively used in offline ASR, it is unclear how best offline systems can be adapted to work with them under the streaming setup. After gaining considerable experience on this regard in recent years, in this paper we show how an optimized, low-latency streaming decoder can be built in which bidirectional LSTM acoustic models, together with general interpolated language models, can be nicely integrated with minimal performance degradation. In brief, our streaming decoder consists of a one-pass, real-time search engine relying on a limited-duration window sliding over time and a number of ad hoc acoustic and language model pruning techniques. Extensive empirical assessment is provided on truly streaming tasks derived from the well-known LibriSpeech and TED talks datasets, as well as from TV shows on a main Spanish broadcasting station.

We explored the effect of augmenting a standard language model’s architecture (BERT) with a structural component based on the Abstract Syntax Trees (ASTs) of the source code. We created a universal abstract syntax tree structure that can be applied to multiple languages to enable the model to work in a multilingual setting. We adapted the general graph transformer architecture to function as the structural component of the transformer. Furthermore, we extended the Embeddings from Language Models (ELMo) style embeddings to work in a multilingual setting when working with incomplete source code. The final results showed that the multilingual setting was beneficial to achieving higher quality embeddings for the embedding model, however, monolingual models performed better in most cases for the transformer model. The addition of ASTs resulted in increased performance in the best performing models on all languages, while also reducing the need for a pre-training task to achieve the best performance. The largest increase in performance for a Java model compared to its baseline counterpart was 3.0% on average on the test set, the largest increase in performance for a Julia model compared to its baseline counterpart was 1.1% on average on the test set, and the largest increase in performance of a CPP model compared to its baseline counterpart was 5.7% on average on the test set., Computer Science

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Grammatical Error Correction refers to the task of automatically correcting errors in written text, typically with respect to texts written by learners of a second language. The work in this thesis implements and evaluates two methods to Grammatical Error Correction for Swedish. In addition, the proposed methods are compared to an existing, rule-based system. Previous research on GEC for the Swedish language is limited and has not yet utilized the potential of neural networks. The first method implemented in this work is based on a neural machine translation approach, training a Transformer model to translate erroneous text into a corrected version. A parallel dataset containing artificially generated errors is created to train the model. The second method utilizes a Swedish version of the pre-trained language model BERT to estimate the likelihood of potential corrections in an erroneous text. Employing the SweLL gold corpus consisting of essays written by learners of Swedish, the proposed methods are evaluated using GLEU and through a manual evaluation based on the types of errors and their corresponding corrections found in the essays. The results show that the two methods correct approximately the same amount of errors, while differing in terms of which error types that are best handled. Specifically, the translation approach has a wider coverage of error types and is superior for syntactical and punctuation errors. In contrast, the language model approach yields consistently higher recall and outperforms the translation approach with regards to lexical and morphological errors. To improve the results, future work could investigate the effect of increased model size and amount of training data, as well as the potential in combining the two methods.

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Data imbalance in datasets is a common issue where the number of instances in one or more categories far exceeds the others, so is the case with the educational domain. Collecting feedback on a course on a large scale and the lack of publicly available datasets in this domain limits models' performance, especially for deep neural network based models which are data hungry. A model trained on such an imbalanced dataset would naturally favor the majority class. However, the minority class could be critical for decision-making in prediction systems, and therefore it is usually desirable to train a model with equally high class-level accuracy. This paper addresses the data imbalance issue for the sentiment analysis of users' opinions task on two educational feedback datasets utilizing synthetic text generation deep learning models. Two state-of-the-art text generation GAN models namely CatGAN and SentiGAN, are employed for synthesizing text used to balance the highly imbalanced datasets in this study. Particular emphasis is given to the diversity of synthetically generated samples for populating minority classes. Experimental results on highly imbalanced datasets show significant improvement in models' performance on CR23K and CR100K after balancing with synthetic data for the sentiment classification task.

Grammatical Error Correction refers to the task of automatically correcting errors in written text, typically with respect to texts written by learners of a second language. The work in this thesis implements and evaluates two methods to Grammatical Error Correction for Swedish. In addition, the proposed methods are compared to an existing, rule-based system. Previous research on GEC for the Swedish language is limited and has not yet utilized the potential of neural networks. The first method implemented in this work is based on a neural machine translation approach, training a Transformer model to translate erroneous text into a corrected version. A parallel dataset containing artificially generated errors is created to train the model. The second method utilizes a Swedish version of the pre-trained language model BERT to estimate the likelihood of potential corrections in an erroneous text. Employing the SweLL gold corpus consisting of essays written by learners of Swedish, the proposed methods are evaluated using GLEU and through a manual evaluation based on the types of errors and their corresponding corrections found in the essays. The results show that the two methods correct approximately the same amount of errors, while differing in terms of which error types that are best handled. Specifically, the translation approach has a wider coverage of error types and is superior for syntactical and punctuation errors. In contrast, the language model approach yields consistently higher recall and outperforms the translation approach with regards to lexical and morphological errors. To improve the results, future work could investigate the effect of increased model size and amount of training data, as well as the potential in combining the two methods.

Online dating has become the most popular method of finding potential romantic partners. At the core of these platforms, there is a reciprocal recommender system which recommends users to other users on the platform. Breeze is an example of such a dating app, serving its users potential romantic partners every day in the hopes of sending them on a date. Current approaches to matchmaking depend exclusively on interaction data and static features such as age, height and location. There is more information present on the user profiles, however, in the form of bios and answers to open questions. Given the state-of-the-art performance of pre-trained language models in a multitude of general language understanding tasks, these free-text profile sections present a source of untapped potential. Our work combines the potential of these free-text profile sections and the state-of-the-art performance of recent language models to create a new approach for matchmaking, or reciprocal recommender systems in general. In this work, the goal is to find out how textual data from user profiles can be leveraged to make good suggestions in a reciprocal recommender system, and how language models can be used for this. Furthermore, this work also explores whether the cold-start problem can be alleviated using this approach. We also perform user research to find out how Breeze users make their decisions on suggestions served by the platform. We introduce LoveBERT, a model to serve text-based recommendations, harnessing the power of several language models fine-tuned on different free-text user profile sections. Our results show that while LoveBERT is better at predicting unidirectional likes than traditional recommender system approaches, it does not outperform them when considering bidirectional matches. Furthermore, we show that while LoveBERT is not able to circumvent the cold-start problem, it is more robust, losing less performance than traditional techniques, Computer Science

We explored the effect of augmenting a standard language model’s architecture (BERT) with a structural component based on the Abstract Syntax Trees (ASTs) of the source code. We created a universal abstract syntax tree structure that can be applied to multiple languages to enable the model to work in a multilingual setting. We adapted the general graph transformer architecture to function as the structural component of the transformer. Furthermore, we extended the Embeddings from Language Models (ELMo) style embeddings to work in a multilingual setting when working with incomplete source code. The final results showed that the multilingual setting was beneficial to achieving higher quality embeddings for the embedding model, however, monolingual models performed better in most cases for the transformer model. The addition of ASTs resulted in increased performance in the best performing models on all languages, while also reducing the need for a pre-training task to achieve the best performance. The largest increase in performance for a Java model compared to its baseline counterpart was 3.0% on average on the test set, the largest increase in performance for a Julia model compared to its baseline counterpart was 1.1% on average on the test set, and the largest increase in performance of a CPP model compared to its baseline counterpart was 5.7% on average on the test set., Computer Science

[Abstract]: This paper presents a monolingual BERT model for Galician. We follow the recent trend that shows that it is feasible to build robust monolingual BERT models even for relatively low-resource languages, while performing better than the well-known official multilingual BERT (mBERT). More particularly, we release two monolingual Galician BERT models, built using 6 and 12 transformer layers, respectively; trained with limited resources (∼45 million tokens on a single GPU of 24GB). We then provide an exhaustive evaluation on a number of tasks such as POS-tagging, dependency parsing and named entity recognition. For this purpose, all these tasks are cast in a pure sequence labeling setup in order to run BERT without the need to include any additional layers on top of it (we only use an output classification layer to map the contextualized representations into the predicted label). The experiments show that our models, especially the 12-layer one, outperform the results of mBERT in most tasks., [Resumen]: Este artículo presenta un modelo BERT monolingüe para el gallego. Nos basamos en la tendencia actual que ha demostrado que es posible crear modelos BERT monolingües robustos incluso para aquellos idiomas para los que hay una relativa escasez de recursos, funcionando éstos mejor que el modelo BERT multilingüe oficial (mBERT). Concretamente, liberamos dos modelos monolingües para el gallego, creados con 6 y 12 capas de transformers, respectivamente, y entrenados con una limitada cantidad de recursos (~45 millones de palabras sobre una única GPU de 24GB.) Para evaluarlos realizamos un conjunto exhaustivo de experimentos en tareas como análisis morfosintáctico, análisis sintáctico de dependencias o reconocimiento de entidades. Para ello, abordamos estas tareas como etiquetado de secuencias, con el objetivo de ejecutar los modelos BERT sin la necesidad de incluir ninguna capa adicional (únicamente se a~nade la capa de salida encargada de transformar las representaciones contextualizadas en la etiqueta predicha). Los experimentos muestran que nuestros modelos, especialmente el de 12 capas, mejoran los resultados de mBERT en la mayor parte de las tareas.

Software developers frequently reuse source code from repositories as it saves development time and effort. Code clones (similar code fragments) accumulated in these repositories represent often repeated functionalities and are candidates for reuse in an exploratory or rapid development. To facilitate code clone reuse, we previously presented DeepClone, a novel deep learning approach for modeling code clones along with non-cloned code to predict the next set of tokens (possibly a complete clone method body) based on the code written so far. The probabilistic nature of language modeling, however, can lead to code output with minor syntax or logic errors. To resolve this, we propose a novel approach called Clone-Advisor. We apply an information retrieval technique on top of DeepClone output to recommend real clone methods closely matching the predicted clone method, thus improving the original output by DeepClone. In this paper we have discussed and refined our previous work on DeepClone in much more detail. Moreover, we have quantitatively evaluated the performance and effectiveness of Clone-Advisor in clone method recommendation.

As society's mutual problems grow, there is an increasing demand for understanding the intra- and inter-cultural differences. Rising polarization within national borders and stranded dialogues between nations on mutual problems, daily reach the headlines. It is argued that current models of opinion only scratch the surface of actual human opinion formation with the traditional value based exact approach. Developments in the domain of natural language processing highlight the overlap between real world cultural biases and biased language models. Although within their own domain these biases are seen as problematic, it is argued that it is exactly this associative prejudice that can be used to model human like opinion formation. This idea is emphasized by an conceptual approach on opinion, belief and knowledge, in which only the probability of an association distinguishes between 'objective' and 'subjective'. Through this concept, and by using identified significant cognitive tendencies, a framework is developed for inferring an opinion from text. Using this framework, an opinion model based on a language model is proposed. The contribution of this thesis is two-fold. Firstly, the proposed model provides evidence that agent perception through language is a significant model element, in which the bias in a language model can possibly be exploited to approach cultural perception. Secondly, as the bridge between the two domains has not yet been build, the findings of the results as well as the research process give direction for much needed future research., Mechanical Engineering | Systems and Control

This paper presents a monolingual BERT model for Galician. We follow the recent trend that shows that it is feasible to build robust monolingual BERT models even for relatively low-resource languages, while performing better than the well-known official multilingual BERT (mBERT). More particularly, we release two monolingual Galician BERT models, built using 6 and 12 transformer layers, respectively; trained with limited resources (~45 million tokens on a single GPU of 24GB). We then provide an exhaustive evaluation on a number of tasks such as POS-tagging, dependency parsing and named entity recognition. For this purpose, all these tasks are cast in a pure sequence labeling setup in order to run BERT without the need to include any additional layers on top of it (we only use an output classification layer to map the contextualized representations into the predicted label). The experiments show that our models, especially the 12-layer one, outperform the results of mBERT in most tasks., Este artículo presenta un modelo BERT monolingüe para el gallego. Nos basamos en la tendencia actual que ha demostrado que es posible crear modelos BERT monolingües robustos incluso para aquellos idiomas para los que hay una relativa escasez de recursos, funcionando éstos mejor que el modelo BERT multilingüe oficial (mBERT). Concretamente, liberamos dos modelos monolingües para el gallego, creados con 6 y 12 capas de transformers, respectivamente, y entrenados con una limitada cantidad de recursos (~45 millones de palabras sobre una única GPU de 24GB.) Para evaluarlos realizamos un conjunto exhaustivo de experimentos en tareas como análisis morfosintáctico, análisis sintáctico de dependencias o reconocimiento de entidades. Para ello, abordamos estas tareas como etiquetado de secuencias, con el objetivo de ejecutar los modelos BERT sin la necesidad de incluir ninguna capa adicional (únicamente se añade la capa de salida encargada de transformar las representaciones contextualizadas en la etiqueta predicha). Los experimentos muestran que nuestros modelos, especialmente el de 12 capas, mejoran los resultados de mBERT en la mayor parte de las tareas.

Predictive text can speed up authoring of everyday tasks, such as writing an SMS or a URL. When deployed in a clinical setting, it can enable practitioners to compile diagnostic text reports in a speedier manner, hence allowing them to be more time-efficient when examining patients. The language used by medical practitioners when authoring clinical reports is, however, far from common, not only between practitioners but also between medical units. In this paper, we demonstrate this clinical language variation, by showing that a model trained on texts written by some physicians may not work for predicting the text of others. We use a dataset created out of the clinical notes of 17 caregivers to show that language models trained on the notes of each caregiver outperform the ones trained with texts from several ones.

This paper presents a monolingual BERT model for Galician. We follow the recent trend that shows that it is feasible to build robust monolingual BERT models even for relatively low-resource languages, while performing better than the well-known official multilingual BERT (mBERT). More particularly, we release two monolingual Galician BERT models, built using 6 and 12 transformer layers, respectively; trained with limited resources (~45 million tokens on a single GPU of 24GB). We then provide an exhaustive evaluation on a number of tasks such as POS-tagging, dependency parsing and named entity recognition. For this purpose, all these tasks are cast in a pure sequence labeling setup in order to run BERT without the need to include any additional layers on top of it (we only use an output classification layer to map the contextualized representations into the predicted label). The experiments show that our models, especially the 12-layer one, outperform the results of mBERT in most tasks., Este artículo presenta un modelo BERT monolingüe para el gallego. Nos basamos en la tendencia actual que ha demostrado que es posible crear modelos BERT monolingües robustos incluso para aquellos idiomas para los que hay una relativa escasez de recursos, funcionando éstos mejor que el modelo BERT multilingüe oficial (mBERT). Concretamente, liberamos dos modelos monolingües para el gallego, creados con 6 y 12 capas de transformers, respectivamente, y entrenados con una limitada cantidad de recursos (~45 millones de palabras sobre una única GPU de 24GB.) Para evaluarlos realizamos un conjunto exhaustivo de experimentos en tareas como análisis morfosintáctico, análisis sintáctico de dependencias o reconocimiento de entidades. Para ello, abordamos estas tareas como etiquetado de secuencias, con el objetivo de ejecutar los modelos BERT sin la necesidad de incluir ninguna capa adicional (únicamente se añade la capa de salida encargada de transformar las representaciones contextualizadas en la etiqueta predicha). Los experimentos muestran que nuestros modelos, especialmente el de 12 capas, mejoran los resultados de mBERT en la mayor parte de las tareas.

Software developers frequently reuse source code from repositories as it saves development time and effort. Code clones (similar code fragments) accumulated in these repositories represent often repeated functionalities and are candidates for reuse in an exploratory or rapid development. To facilitate code clone reuse, we previously presented DeepClone, a novel deep learning approach for modeling code clones along with non-cloned code to predict the next set of tokens (possibly a complete clone method body) based on the code written so far. The probabilistic nature of language modeling, however, can lead to code output with minor syntax or logic errors. To resolve this, we propose a novel approach called Clone-Advisor. We apply an information retrieval technique on top of DeepClone output to recommend real clone methods closely matching the predicted clone method, thus improving the original output by DeepClone. In this paper we have discussed and refined our previous work on DeepClone in much more detail. Moreover, we have quantitatively evaluated the performance and effectiveness of Clone-Advisor in clone method recommendation.

As society's mutual problems grow, there is an increasing demand for understanding the intra- and inter-cultural differences. Rising polarization within national borders and stranded dialogues between nations on mutual problems, daily reach the headlines. It is argued that current models of opinion only scratch the surface of actual human opinion formation with the traditional value based exact approach. Developments in the domain of natural language processing highlight the overlap between real world cultural biases and biased language models. Although within their own domain these biases are seen as problematic, it is argued that it is exactly this associative prejudice that can be used to model human like opinion formation. This idea is emphasized by an conceptual approach on opinion, belief and knowledge, in which only the probability of an association distinguishes between 'objective' and 'subjective'. Through this concept, and by using identified significant cognitive tendencies, a framework is developed for inferring an opinion from text. Using this framework, an opinion model based on a language model is proposed. The contribution of this thesis is two-fold. Firstly, the proposed model provides evidence that agent perception through language is a significant model element, in which the bias in a language model can possibly be exploited to approach cultural perception. Secondly, as the bridge between the two domains has not yet been build, the findings of the results as well as the research process give direction for much needed future research., Mechanical Engineering | Systems and Control

Software developers frequently reuse source code from repositories as it saves development time and effort. Code clones (similar code fragments) accumulated in these repositories represent often repeated functionalities and are candidates for reuse in an exploratory or rapid development. To facilitate code clone reuse, we previously presented DeepClone, a novel deep learning approach for modeling code clones along with non-cloned code to predict the next set of tokens (possibly a complete clone method body) based on the code written so far. The probabilistic nature of language modeling, however, can lead to code output with minor syntax or logic errors. To resolve this, we propose a novel approach called Clone-Advisor. We apply an information retrieval technique on top of DeepClone output to recommend real clone methods closely matching the predicted clone method, thus improving the original output by DeepClone. In this paper we have discussed and refined our previous work on DeepClone in much more detail. Moreover, we have quantitatively evaluated the performance and effectiveness of Clone-Advisor in clone method recommendation. Subjects Data Mining and Machine Learning, Software Engineering

Human reading behavior is tuned to the statistics of natural lan-guage: the time it takes human subjects to read a word can bepredicted from estimates of the word’s probability in context.However, it remains an open question what computational ar-chitecture best characterizes the expectations deployed in realtime by humans that determine the behavioral signatures ofreading. Here we test over two dozen models, independentlymanipulating computational architecture and training datasetsize, on how well their next-word expectations predict humanreading time behavior on naturalistic text corpora. Consistentwith previous work, we find that across model architecturesand training dataset sizes the relationship between word log-probability and reading time is (near-)linear. We next evalu-ate how features of these models determine their psychometricpredictive power, or ability to predict human reading behav-ior. In general, the better a model’s next-word expectations(as measured by the traditional language modeling perplexityobjective), the better its psychometric predictive power. How-ever, we find nontrivial differences in psychometric predictivepower across model architectures. For any given perplexity,deep Transformer models and n-gram models generally showsuperior psychometric predictive power over LSTM or struc-turally supervised neural models, especially for eye movementdata. Finally, we compare models’ psychometric predictivepower to the depth of their syntactic knowledge, as measuredby a battery of syntactic generalization tests developed usingmethods from controlled psycholinguistic experiments. Onceperplexity is controlled for, we find no significant relationshipbetween syntactic knowledge and predictive power. These re-sults suggest that, at least for the present state of natural lan-guage technology, different approaches may be required to bestmodel human real-time language comprehension behavior innaturalistic reading versus behavior for controlled linguisticmate

[Resumen] El proyecto descrito en este documento tiene como objetivo construir un comentario de texto personalizado de cada usuario de un Sistema de Recomendación (RS, por sus siglas en inglés) hacia un producto o servicio que este no haya utilizado, basándose en el total de comentarios textuales de los distintos usuarios utilizados en el RS. El fin de un RS es sugerir a los usuarios nuevos productos o servicios que puede que no conozcan aún, basados en las preferencias de usuarios con características similares. Desde un punto de vista económico, estos RS son de vital importancia, ya que muchas de las mayores compañías del mundo por capitalización de mercado (como Google, Amazon o Facebook) están substancialmente basadas en plataformas que recomiendan productos a sus usuarios. Habitualmente, los usuarios de un producto o servicio dejan patente su opinión mediante un sistema de puntuación numérico, un comentario en lenguaje natural y, opcionalmente, una serie de fotos. Este proyecto se centra en utilizar los comentarios como forma de evaluar la opinión del usuario y extraer información sobre sus gustos para poder ofrecer recomendaciones más personalizadas. En concreto, el objetivo de este proyecto será evaluar las opiniones de usuarios de TripAdvisor sobre hoteles para tratar de predecir la reacción de un usuario frente a un hotel determinado que este no haya visitado aún. Para ello, primeramente será necesario obtener un conjunto de datos con opiniones reales de usuarios de TripAdvisor, para posteriormente preprocesar los comentarios para poder trabajar con ellos (eliminación de palabras vacías, puntuación, etc.). Seguidamente se obtendrá una representación densa del lenguaje para poder interpretar los comentarios y se usará una aproximación probabilística junto con Deep Learning para poder predecir las reseñas de usuario/hotel. Por último se implementarán distintas métricas para evaluar los comentarios obtenidos., [Abstract] The project described in this document has the objective of building a personalized textual comment for each user in a Recommender System (RS) to a product or service that the user hasn’t used in the past, based on every users’ textual comments used in the RS. The end goal of a RS is to suggest new products or services to users that they may not know about, based on the preferences of users with a similar profile. From an economic point of view, RS are of vital importance, since many of the world’s biggest companies by revenue (like Google, Amazon or Facebook) are substantially based on platforms that recommend products to their users. Users of a product or service usually show their opinion by means of numeric-based ratings, a natural language comment and, optionally, a series of photos. This project is centered around the idea of exploiting the comments as a mean of evaluating user opinions and extracting information about their preferences with the end goal of making personalized decisions. Specifically, the objective of this project will be to evaluate TripAdvisor’s user’s opinions on hotels to try and predict the reaction of a user when presented with an hotel he/she has not visited yet. To bring about this project, it will be necessary to obtain a dataset with real reviews of TripAdvisor’s users, later preprocess the comments (removal of stop-words, punctuation etc.). Next, a dense representation of the language of the comments will be obtained so as to be able to interpret them and a probabilistic approach will be used in conjunction with Deep Learning to predict the reviews. Lastly, different metrics will be implemented to assess the quality of the obtained reviews.

Dynamic Topic Modeling (DTM) is the ultimate solution for extracting topics from short texts generated in Online Social Networks (OSNs) like Twitter. A DTM solution is required to be scalable and to be able to account for sparsity in short texts and dynamicity of topics. Current solutions combine probabilistic mixture models like Dirichlet Multinomial or PitmanYor Process with approximate inference approaches like Gibbs Sampling and Stochastic Variational Inference to, respectively, account for dynamicity and scalability in DTM. However, these solutions rely on weak probabilistic language models, which do not account for sparsity in short texts. In addition, their inference is based on iterative optimization algorithms, which have scalability issues when it comes to DTM. We present GDTM, a single-pass graph-based DTM algorithm, to solve the problem. GDTM combines a context-rich and incremental feature representation model, called Random Indexing (RI), with a novel online graph partitioning algorithm to address scalability and dynamicity. In addition, GDTM uses a rich language modeling approach based on the Skip-gram technique to account for sparsity. We run multiple experiments over a large-scale Twitter dataset to analyze the accuracy and scalability of GDTM and compare the results with four state-of-the-art approaches. The results show that GDTM outperforms the best approach by 11% on accuracy and performs by an order of magnitude faster while creating 4 times better topic quality over standard evaluation metrics., QC 20191115.QC 20210909QC 20211001

A language model can be used to predict the next word during authoring, to correct spelling or to accelerate writing (e.g., in sms or emails). Language models, however, have only been applied in a very small scale to assist physicians during authoring (e.g., discharge summaries or radiology reports). But along with the assistance to the physician, computer-based systems which expedite the patient's exit also assist in decreasing the hospital infections. We employed statistical and neural language modeling to predict the next word of a clinical text and assess all the models in terms of accuracy and keystroke discount in two datasets with radiology reports. We show that a neural language model can achieve as high as 51.3% accuracy in radiology reports (one out of two words predicted correctly). We also show that even when the models are employed only for frequent words, the physician can save valuable time.

Объектом исследования являются нейросетевые лингвистические модели. Цель работы: разработка алгоритма для генерации текста на основе нейросетевой модели. В ходе данной работы рассмотрена одна из задач обработки естественного языка - лингвистическое моделирование, произведен обзор архитектуры различных нейросетевых моделей, разработан алгоритм для генерации текста, произведено сравнение данного алгоритма с использованием различных нейросетевых моделей. Данная разработка позволит автоматизировать процесс создания различного текстового контента для интернет ресурсов., The object of study are neural network language modeling. Purpose of work: development an algorithm for generating text based on a neural network model. In the course of this work, one of the tasks of natural language processing, language modeling, was reviewed, the architecture of various neural network models was reviewed, a text generation algorithm was developed, and a comparison was made with this algorithm using various neural network models. This development will automate the process of creating various text content for Internet resources.

This article investigates the evaluation of a word prediction system in an Augmentative and Alternative Communication (AAC) software for disabled people. In addition to having a reduced mobility, these users have an altered use of speech that must be compensated by a technological aid offering input methods adapted to their capabilities. To improve their communication speed, different prediction and language modeling techniques are used. We present the parameterization of statistical predictors. Their configuration in French is evaluated by a simulator and tested by a disabled person. The results show that a language model built from a large literary corpus saves more than one keystroke out of two, the performance of these systems varying according to several parameters.

[Resumen] El proyecto descrito en este documento tiene como objetivo construir un comentario de texto personalizado de cada usuario de un Sistema de Recomendación (RS, por sus siglas en inglés) hacia un producto o servicio que este no haya utilizado, basándose en el total de comentarios textuales de los distintos usuarios utilizados en el RS. El fin de un RS es sugerir a los usuarios nuevos productos o servicios que puede que no conozcan aún, basados en las preferencias de usuarios con características similares. Desde un punto de vista económico, estos RS son de vital importancia, ya que muchas de las mayores compañías del mundo por capitalización de mercado (como Google, Amazon o Facebook) están substancialmente basadas en plataformas que recomiendan productos a sus usuarios. Habitualmente, los usuarios de un producto o servicio dejan patente su opinión mediante un sistema de puntuación numérico, un comentario en lenguaje natural y, opcionalmente, una serie de fotos. Este proyecto se centra en utilizar los comentarios como forma de evaluar la opinión del usuario y extraer información sobre sus gustos para poder ofrecer recomendaciones más personalizadas. En concreto, el objetivo de este proyecto será evaluar las opiniones de usuarios de TripAdvisor sobre hoteles para tratar de predecir la reacción de un usuario frente a un hotel determinado que este no haya visitado aún. Para ello, primeramente será necesario obtener un conjunto de datos con opiniones reales de usuarios de TripAdvisor, para posteriormente preprocesar los comentarios para poder trabajar con ellos (eliminación de palabras vacías, puntuación, etc.). Seguidamente se obtendrá una representación densa del lenguaje para poder interpretar los comentarios y se usará una aproximación probabilística junto con Deep Learning para poder predecir las reseñas de usuario/hotel. Por último se implementarán distintas métricas para evaluar los comentarios obtenidos., [Abstract] The project described in this document has the objective of building a personalized textual comment for each user in a Recommender System (RS) to a product or service that the user hasn’t used in the past, based on every users’ textual comments used in the RS. The end goal of a RS is to suggest new products or services to users that they may not know about, based on the preferences of users with a similar profile. From an economic point of view, RS are of vital importance, since many of the world’s biggest companies by revenue (like Google, Amazon or Facebook) are substantially based on platforms that recommend products to their users. Users of a product or service usually show their opinion by means of numeric-based ratings, a natural language comment and, optionally, a series of photos. This project is centered around the idea of exploiting the comments as a mean of evaluating user opinions and extracting information about their preferences with the end goal of making personalized decisions. Specifically, the objective of this project will be to evaluate TripAdvisor’s user’s opinions on hotels to try and predict the reaction of a user when presented with an hotel he/she has not visited yet. To bring about this project, it will be necessary to obtain a dataset with real reviews of TripAdvisor’s users, later preprocess the comments (removal of stop-words, punctuation etc.). Next, a dense representation of the language of the comments will be obtained so as to be able to interpret them and a probabilistic approach will be used in conjunction with Deep Learning to predict the reviews. Lastly, different metrics will be implemented to assess the quality of the obtained reviews.

The Simple Recurrent Network (SRN) has a long tradition incognitive models of language processing. More recently, gatedrecurrent networks have been proposed that often outperformthe SRN on natural language processing tasks. Here, we in-vestigate whether two types of gated networks perform betteras cognitive models of sentence reading than SRNs, beyondtheir advantage as language models. This will reveal whetherthe filtering mechanism implemented in gated networks corre-sponds to an aspect of human sentence processing. We traina series of language models differing only in the cell types oftheir recurrent layers. We then compute word surprisal valuesfor stimuli used in self-paced reading, eye-tracking, and elec-troencephalography experiments, and quantify the surprisalvalues’ fit to experimental measures that indicate human sen-tence reading effort. While the gated networks provide betterlanguage models, they do not outperform their SRN counter-part as cognitive models when language model quality is equalacross network types. Our results suggest that the differentarchitectures are equally valid as models of human sentenceprocessing.

Automatsko prepoznavanje govora je tehnologija koja računarima omogućava pretvaranje izgovorenih reči u tekst. Ona se može primeniti u mnogim savremenim sistemima koji uključuju komunikaciju između čoveka i mašine. U ovoj disertaciji detaljno je opisana jedna od dve glavne komponente sistema za prepoznavanje govora, a to je jezički model, koji specificira rečnik sistema, kao i pravila prema kojim se pojedinačne reči mogu povezati u rečenicu. Srpski jezik spada u grupu visoko inflektivnih i morfološki bogatih jezika, što znači da koristi veći broj različitih završetaka reči za izražavanje željene gramatičke, sintaksičke ili semantičke funkcije date reči. Ovakvo ponašanje često dovodi do velikog broja grešaka sistema za prepoznavanje govora kod kojih zbog dobrog akustičkog poklapanja prepoznavač pogodi osnovni oblik reči, ali pogreši njen završetak. Taj završetak može da označava drugu morfološku kategoriju, na primer, padež, rod ili broj. U radu je predstavljen novi alat za modelovanje jezika, koji uz identitet reči u modelu može da koristi dodatna leksička i morfološka obeležja reči, čime je testirana hipoteza da te dodatne informacije mogu pomoći u prevazilaženju značajnog broja grešaka prepoznavača koje su posledica inflektivnosti srpskog jezika., Automatic speech recognition is a technology that allows computers to convert spoken words into text. It can be applied in various areas which involve communication between humans and machines. This thesis primarily deals with one of two main components of speech recognition systems - the language model, that specifies the vocabulary of the system, as well as the rules by which individual words can be linked into sentences. The Serbian language belongs to a group of highly inflective and morphologically rich languages, which means that it uses a number of different word endings to express the desired grammatical, syntactic, or semantic function of the given word. Such behavior often leads to a significant number of errors in speech recognition systems where due to good acoustic matching the recognizer correctly guesses the basic form of the word, but an error occurs in the word ending. This word ending may indicate a different morphological category, for example, word case, grammatical gender, or grammatical number. The thesis presents a new language modeling tool which, along with the word identity, can also model additional lexical and morphological features of the word, thus testing the hypothesis that this additional information can help overcome a significant number of recognition errors that result from the high inflectivity of the Serbian language.

[EN] Handwritten Text Recognition is a important requirement in order to make visible the contents of the myriads of historical documents residing in public and private archives and libraries world wide. Automatic Handwritten Text Recognition (HTR) is a challenging problem that requires a careful combination of several advanced Pattern Recognition techniques, including but not limited to Image Processing, Document Image Analysis, Feature Extraction, Neural Network approaches and Language Modeling. The progress of this kind of systems is strongly bound by the availability of adequate benchmarking datasets, software tools and reproducible results achieved using the corresponding tools and datasets. Based on English and German historical documents proposed in recent open competitions at ICDAR and ICFHR conferences between 2014 and 2017, this paper introduces four HTR benchmarks in order of increasing complexity from several points of view. For each benchmark, a specific system is proposed which overcomes results published so far under comparable conditions. Therefore, this paper establishes new state of the art baseline systems and results which aim at becoming new challenges that would hopefully drive further improvement of HTR technologies. Both the datasets and the software tools used to implement the baseline systems are made freely accessible for research purposes. (C) 2019 Elsevier Ltd. All rights reserved.

[EN] Handwritten Text Recognition is a important requirement in order to make visible the contents of the myriads of historical documents residing in public and private archives and libraries world wide. Automatic Handwritten Text Recognition (HTR) is a challenging problem that requires a careful combination of several advanced Pattern Recognition techniques, including but not limited to Image Processing, Document Image Analysis, Feature Extraction, Neural Network approaches and Language Modeling. The progress of this kind of systems is strongly bound by the availability of adequate benchmarking datasets, software tools and reproducible results achieved using the corresponding tools and datasets. Based on English and German historical documents proposed in recent open competitions at ICDAR and ICFHR conferences between 2014 and 2017, this paper introduces four HTR benchmarks in order of increasing complexity from several points of view. For each benchmark, a specific system is proposed which overcomes results published so far under comparable conditions. Therefore, this paper establishes new state of the art baseline systems and results which aim at becoming new challenges that would hopefully drive further improvement of HTR technologies. Both the datasets and the software tools used to implement the baseline systems are made freely accessible for research purposes. (C) 2019 Elsevier Ltd. All rights reserved.

[EN] Handwritten Text Recognition is a important requirement in order to make visible the contents of the myriads of historical documents residing in public and private archives and libraries world wide. Automatic Handwritten Text Recognition (HTR) is a challenging problem that requires a careful combination of several advanced Pattern Recognition techniques, including but not limited to Image Processing, Document Image Analysis, Feature Extraction, Neural Network approaches and Language Modeling. The progress of this kind of systems is strongly bound by the availability of adequate benchmarking datasets, software tools and reproducible results achieved using the corresponding tools and datasets. Based on English and German historical documents proposed in recent open competitions at ICDAR and ICFHR conferences between 2014 and 2017, this paper introduces four HTR benchmarks in order of increasing complexity from several points of view. For each benchmark, a specific system is proposed which overcomes results published so far under comparable conditions. Therefore, this paper establishes new state of the art baseline systems and results which aim at becoming new challenges that would hopefully drive further improvement of HTR technologies. Both the datasets and the software tools used to implement the baseline systems are made freely accessible for research purposes. (C) 2019 Elsevier Ltd. All rights reserved.

Dialogue speech widely exists in scenarios such as chitchat, meeting and customer service. General-purpose speech recognition systems usually neglect the topic information in the context of dialogue speech, which has great potential for improving the performance of speech recognition. In this paper, we propose a transfer learning mechanism to conduct topic-aware recognition for dialogue speech. We first propose a new probabilistic topic model named Dialogue Speech Topic Model (DSTM) that is specialized for modeling the context of dialogue speech. We further propose a novel transfer learning mechanism for DSTM to significantly reduce its training cost while preserving its effectiveness for accurate topic inference. The experiment results demonstrate that proposed techniques in language model adaptation effectively improve the performance of the state-of-the-art Automatic Speech Recognition (ASR) system. Copyright © 2019 ISCA

Automatsko prepoznavanje govora je tehnologija koja računarima omogućava pretvaranje izgovorenih reči u tekst. Ona se može primeniti u mnogim savremenim sistemima koji uključuju komunikaciju između čoveka i mašine. U ovoj disertaciji detaljno je opisana jedna od dve glavne komponente sistema za prepoznavanje govora, a to je jezički model, koji specificira rečnik sistema, kao i pravila prema kojim se pojedinačne reči mogu povezati u rečenicu. Srpski jezik spada u grupu visoko inflektivnih i morfološki bogatih jezika, što znači da koristi veći broj različitih završetaka reči za izražavanje željene gramatičke, sintaksičke ili semantičke funkcije date reči. Ovakvo ponašanje često dovodi do velikog broja grešaka sistema za prepoznavanje govora kod kojih zbog dobrog akustičkog poklapanja prepoznavač pogodi osnovni oblik reči, ali pogreši njen završetak. Taj završetak može da označava drugu morfološku kategoriju, na primer, padež, rod ili broj. U radu je predstavljen novi alat za modelovanje jezika, koji uz identitet reči u modelu može da koristi dodatna leksička i morfološka obeležja reči, čime je testirana hipoteza da te dodatne informacije mogu pomoći u prevazilaženju značajnog broja grešaka prepoznavača koje su posledica inflektivnosti srpskog jezika., Automatic speech recognition is a technology that allows computers to convert spoken words into text. It can be applied in various areas which involve communication between humans and machines. This thesis primarily deals with one of two main components of speech recognition systems - the language model, that specifies the vocabulary of the system, as well as the rules by which individual words can be linked into sentences. The Serbian language belongs to a group of highly inflective and morphologically rich languages, which means that it uses a number of different word endings to express the desired grammatical, syntactic, or semantic function of the given word. Such behavior often leads to a significant number of errors in speech recognition systems where due to good acoustic matching the recognizer correctly guesses the basic form of the word, but an error occurs in the word ending. This word ending may indicate a different morphological category, for example, word case, grammatical gender, or grammatical number. The thesis presents a new language modeling tool which, along with the word identity, can also model additional lexical and morphological features of the word, thus testing the hypothesis that this additional information can help overcome a significant number of recognition errors that result from the high inflectivity of the Serbian language.

Dialogue speech widely exists in scenarios such as chitchat, meeting and customer service. General-purpose speech recognition systems usually neglect the topic information in the context of dialogue speech, which has great potential for improving the performance of speech recognition. In this paper, we propose a transfer learning mechanism to conduct topic-aware recognition for dialogue speech. We first propose a new probabilistic topic model named Dialogue Speech Topic Model (DSTM) that is specialized for modeling the context of dialogue speech. We further propose a novel transfer learning mechanism for DSTM to significantly reduce its training cost while preserving its effectiveness for accurate topic inference. The experiment results demonstrate that proposed techniques in language model adaptation effectively improve the performance of the state-of-the-art Automatic Speech Recognition (ASR) system. Copyright © 2019 ISCA

Dialogue speech widely exists in scenarios such as chitchat, meeting and customer service. General-purpose speech recognition systems usually neglect the topic information in the context of dialogue speech, which has great potential for improving the performance of speech recognition. In this paper, we propose a transfer learning mechanism to conduct topic-aware recognition for dialogue speech. We first propose a new probabilistic topic model named Dialogue Speech Topic Model (DSTM) that is specialized for modeling the context of dialogue speech. We further propose a novel transfer learning mechanism for DSTM to significantly reduce its training cost while preserving its effectiveness for accurate topic inference. The experiment results demonstrate that proposed techniques in language model adaptation effectively improve the performance of the state-of-the-art Automatic Speech Recognition (ASR) system. Copyright © 2019 ISCA

[EN] Neural Network Language Models (NNLMs) are a successful approach to Natural Language Processing tasks, such as Machine Translation. We introduce in this work a Statistical Machine Translation (SMT) system which fully integrates NNLMs in the decoding stage, breaking the traditional approach based on n-best list rescoring. The neural net models (both language models (LMs) and translation models) are fully coupled in the decoding stage, allowing to more strongly influence the translation quality. Computational issues were solved by using a novel idea based on memorization and smoothing of the softmax constants to avoid their computation, which introduces a trade-off between LM quality and computational cost. These ideas were studied in a machine translation task with different combinations of neural networks used both as translation models and as target LMs, comparing phrase-based and N-gram-based systems, showing that the integrated approach seems more promising for N-gram-based systems, even with nonfull-quality NNLMs.