Search

This paper presents the multidisciplinary work carried out in the RTVE-UGR Chair within the IVERES project, whose main objective is the development of a tool for journalists to verify the veracity of the audios that reach the newsrooms. In the current context, voice synthesis has both beneficial and detrimental applications, with audio deepfakes being a significant concern in the world of journalism due to their ability to mislead and misinform. This is a multifaceted problem that can only be tackled adopting a multidisciplinary perspective. In this article, we describe the approach we adopted within the RTVE-UGR Chair to successfully address the challenges derived from audio deepfakes involving a team with different backgrounds and a specific methodology of iterative co-creation. As a result, we present several outcomes including the compilation and generation of audio datasets, the development and deployment of several audio fake detection models, and the development of a web audio verification tool addressed to journalists. As a conclusion, we highlight the importance of this systematic collaborative work in the fight against misinformation and the future potential of audio verification technologies in various applications.

The ChatSubs dataset [5] contains dialogue data in Spanish and three of Spain's co-official languages (Catalan, Basque, and Galician). It has been obtained from OpenSubtitles, from which we have gathered the movie subtitles in our languages of interest and processed them to generate clearly segmented dialogues and their turns. The data processing code is publicly accessible. The result is 206.706 JSON files with more than 20 million dialogues and 96 million turns, which represents one of the biggest dialogue corpus available, as other similar datasets in better resourced languages do not reach 500k dialogues or present less defined conversations. Thus, the ChatSubs dataset is an ideal resource for research teams that are interested in training dialogue models in Spanish, Catalan, Basque, and Galician.

This paper describes a novel qualitative dataset regarding coffee pests based on the ancestral knowledge of coffee farmers in the Department of Cauca, Colombia. The dataset has been obtained from a survey applied to coffee growers with 432 records and 41 variables collected weekly from September 2020 to August 2021. The qualitative dataset includes climatic conditions, productive activities, external conditions, and coffee bio-aggressors. This dataset allows researchers to find patterns for coffee crop protection through the ancestral knowledge not detected by real-time agricultural sensors. As far as we are concerned, there are no datasets like the one presented in this paper with similar characteristics of qualitative value that express the empirical knowledge of coffee farmers used to detect triggers of causal behaviors of pests and diseases in coffee crops.

In recent years, transformer-based models have played a significant role in advancing language modeling for natural language processing. However, they require substantial amounts of data and there is a shortage of high-quality non-English corpora. Some recent initiatives have introduced multilingual datasets obtained through web crawling. However, there are notable limitations in the results for some languages, including Spanish. These datasets are either smaller compared to other languages or suffer from lower quality due to insufficient cleaning and deduplication. In this paper, we present esCorpius-m, a multilingual corpus extracted from around 1 petabyte of Common Crawl data. It is the most extensive corpus for some languages with such a level of high-quality content extraction, cleanliness, and deduplication. Our data curation process involves an efficient cleaning pipeline and various deduplication methods that maintain the integrity of document and paragraph boundaries. We also ensure compliance with EU regulations by retaining both the source web page URL and the WARC shared origin URL.

We present an educational data analytics case study aimed at the early detection of potential dropout in Computer Engineering studies in Cuba. We have employed institutional data of 456 students and performed several experiments for predicting their permanency into three (promotion, repetition, and dropout) or two classes (promoting, not promoting). We have also tested a combination of classification features for training and testing decision trees and neural networks; including information obtained at the time of enrollment, after the first semester and after the first academic year. Our results show a considerable accuracy using all features (96.71%). Using only the features available at the time of enrolment and after the first semester we obtain very positive results (68.86% and 93.85% accuracy respectively) with a high recall of non-promoting students. Thus, it is possible to obtain an early assessment of the risk of dropout that can help defining prevention policies.

Crop disease management in smart agriculture involves applying and using new technologies to reduce the impact of diseases on the quality of products. Coffee rust is a disease that factors such as poor agronomic management activities and climate conditions may favor. Therefore, it is crucial to identify the relationships between these factors and this disease to learn how to face its consequences and build intelligent systems to provide appropriate management or help farmers and experts make decisions accordingly. Nevertheless, there are no studies in the literature that propose ontologies to model these factors and coffee rust. This paper presents a new ontology called RustOnt to help experts more accurately model data, expressions, and samples related to coffee rust and apply it whilst taking into account the geographical location where the ontology is adopted. Consequently, this ontology is crucial for coffee rust monitoring and management by means of smart agriculture systems. RustOnt was successfully evaluated considering quality criteria such as clarity, consistency, modularity, and competence against a set of initial requirements for which it was built.

Intent recognition is a key component of any task-oriented conversational system. The intent recognizer can be used first to classify the user’s utterance into one of several predefined classes (intents) that help to understand the user’s current goal. Then, the most adequate response can be provided accordingly. Intent recognizers also often appear as a form of joint models for performing the natural language understanding and dialog management tasks together as a single process, thus simplifying the set of problems that a conversational system must solve. This happens to be especially true for frequently asked question (FAQ) conversational systems. In this work, we first present an exploratory analysis in which different deep learning (DL) models for intent detection and classification were evaluated. In particular, we experimentally compare and analyze conventional recurrent neural networks (RNN) and state-of-the-art transformer models. Our experiments confirmed that best performance is achieved by using transformers. Specifically, best performance was achieved by fine-tuning the so-called BETO model (a Spanish pretrained bidirectional encoder representations from transformers (BERT) model from the Universidad de Chile) in our intent detection task. Then, as the main contribution of the paper, we analyze the effect of inserting unseen domain words to extend the vocabulary of the model as part of the fine-tuning or domain-adaptation process. Particularly, a very simple word frequency cut-off strategy is experimentally shown to be a suitable method for driving the vocabulary learning decisions over unseen words. The results of our analysis show that the proposed method helps to effectively extend the original vocabulary of the pretrained models. We validated our approach with a selection of the corpus acquired with the Hispabot-Covid19 system obtaining satisfactory results.

Emotion recognition is attracting the attention of the research community due to its multiple applications in different fields, such as medicine or autonomous driving. In this paper, we proposed an automatic emotion recognizer system that consisted of a speech emotion recognizer (SER) and a facial emotion recognizer (FER). For the SER, we evaluated a pre-trained xlsr-Wav2Vec2.0 transformer using two transfer-learning techniques: embedding extraction and fine-tuning. The best accuracy results were achieved when we fine-tuned the whole model by appending a multilayer perceptron on top of it, confirming that the training was more robust when it did not start from scratch and the previous knowledge of the network was similar to the task to adapt. Regarding the facial emotion recognizer, we extracted the Action Units of the videos and compared the performance between employing static models against sequential models. Results showed that sequential models beat static models by a narrow difference. Error analysis reported that the visual systems could improve with a detector of high-emotional load frames, which opened a new line of research to discover new ways to learn from videos. Finally, combining these two modalities with a late fusion strategy, we achieved 86.70% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. Results demonstrated that these modalities carried relevant information to detect users’ emotional state and their combination allowed to improve the final system performance.

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

Speech Technologies and Language Processing have made possible the development of a number of new applications which are based on conversational interfaces. In this paper, we describe two approaches to bridge the gap between the academic and industrial perspectives in order to develop conversational interfaces using an academic paradigm for dialog management while employing the industrial standards. The advances in these technologies have made possible to extend the initial applications of conversational interfaces from only spoken interaction (for instance, by means of VoiceXML-based systems) to multimodal services by means of mobile devices (for instance, using the facilities provided by the Android OS). Our proposal has been evaluated with the successful development of different spoken and multimodal conversational interfaces.

Research in techniques to simulate users has a long history within the fields of language processing, speech technologies and conversational interfaces. In this paper, we describe a technique to develop heterogeneous user models that are able to interact with this kind of interfaces. By means of simulated users, it is possible not only to automatically evaluate the overall operation of a conversational interface, but also to assess the impact of the user responses on the decisions that are selected by the system. The selection of the user responses by the simulated user are based on a statistical model that considers the complete history of the interaction to carry out this selection. We describe this technique and its practical application to measure the influence of the most important user's features characteristics that affect the interaction of the simulated user with the a conversational interface.

The emergence of smart devices and recent advances in spoken language technology are currently extending the use of conversational interfaces and spoken interaction to perform many tasks. The dialog management task of a conversational interface consists of selecting the next system response considering the user's actions, the dialog history, and the results of accessing the data repositories. In this paper we describe a dialog management technique adapted to multi-task conversational systems. In our proposal, specialized dialog models are used to deal with each specific subtask of dialog objective for which the dialog system has been designed. The practical application of the proposed technique to develop a dialog system acting as a customer support service shows that the use of these specialized dialog models increases the quality and number of successful interactions with the system in comparison with developing a single dialog model.

In this paper, we assess the applicability of user simulation techniques to generate dialogs which are similar to real human-machine spoken interactions.To do so, we present the results of the comparison between three corpora acquired by means of different techniques. The first corpus was acquired with real users.A statistical user simulation technique has been applied to the same task to acquire the second corpus. In this technique, the next user answer is selected by means of a classification process that takes into account the previous dialog history, the lexical information in the clause, and the subtask of the dialog to which it contributes. Finally, a dialog simulation technique has been developed for the acquisition of the third corpus. This technique uses a random selection of the user and system turns, defining stop conditions for automatically deciding if the simulated dialog is successful or not. We use several evaluation measures proposed in previous research to compare between our three acquired corpora, and then discuss the similarities and differences with regard to these measures.

Conversational agents have become a strong alternative to enhance educational systems with intelligent communicative capabilities, provide motivation and engagement, and increment significant learning and helping in the acquisition of meta-cognitive skills. In this paper, we present Geranium, a multimodal conversational agent that helps children to appreciate and protect their environment. The system, which integrates an interactive chatbot, has been developed by means of a modular and scalable framework that eases building pedagogic conversational agents that can interact with the students using speech and natural language.

In this paper we present a novel framework for the integration of visual sensor networks and speech-based interfaces. Our proposal follows the standard reference architecture in fusion systems (JDL), and combines different techniques related to Artificial Intelligence, Natural Language Processing and User Modeling to provide an enhanced interaction with their users. Firstly, the framework integrates a Cooperative Surveillance Multi-Agent System (CS-MAS), which includes several types of autonomous agents working in a coalition to track and make inferences on the positions of the targets. Secondly, enhanced conversational agents facilitate human-computer interaction by means of speech interaction. Thirdly, a statistical methodology allows modeling the user conversational behavior, which is learned from an initial corpus and improved with the knowledge acquired from the successive interactions. A technique is proposed to facilitate the multimodal fusion of these information sources and consider the result for the decision of the next system action.

Los dispositivos móviles inteligentes han fomentado en los últimos años nuevos escenarios de interacción que requieren interfaces persona-máquina más sofisticados. Los principales desarrolladores de sistemas operativos para estos dispositivos proporcionan actualmente APIs (Application Programming Interfaces) para que los desarrolladores puedan implementar sus propias aplicaciones, incluyendo diferentes soluciones para el desarrollo de interfaces gráficas, el control de sensores y proveer interacción oral. A pesar de la utilidad de este tipo de recursos, no se han definido aún estrategias para desarrollar interfaces multimodales que aprovechen las posibilidades que ofrecen estos dispositivos para identificar y adaptarse a las necesidades de los usuarios. De este modo, las aplicaciones suelen desarrollarse ad-hoc y la interfaz oral posibilita únicamente la comunicación mediante comandos simples. En este artículo proponemos la aplicación práctica de agentes conversacionales multimodales sensibles al contexto para proporcionar servicios bibliotecarios avanzados que incorporen dinámicamente necesidades específicas del usuario y sus preferencias, así como las características específicas del entorno en el que se produce interacción, con el fin de mejorar y personalizar el servicio que se proporciona mediante un dispositivo móvil con acceso a Internet. Nuestra propuesta integra funcionalidades de APIs de Android en una arquitectura modular que enfatiza la gestión de la interacción y la sensibilidad al contexto para crear aplicaciones robustas, de fácil actualización y adaptadas al usuario.

Animated characters are beginning to be used as pedagogical tools, as they have the power to capture students' attention and foster their motivation for discovery and learning. However, in order for them to be widely employed and accepted as a learning resource, they must be easy to use and friendly. In this paper we present an architecture that facilitates building interactive pedagogical chatbots that can interact with students in natural language. Our proposal provides a modular and scalable framework to develop such systems efficiently. Additionally, we present Geranium , a system that helps children to appreciate and protect their environment with an interactive chatbot developed following our scheme.