Search

Proceedings of: 14th Conference of the Spanish Association for Artificial Intelligence, CAEPIA 2011, La Laguna, Spain, November 7-11, 2011 Maximizing accessibility is not always the main objective in the design of web applications, specially if it is concerned with facilitating access for disabled people. In this paper we present the VoiceApp multimodal dialog system, which enables to access and browse Internet by means of speech. The system consists of several modules that provide different user experiences on the web. Voice Dictionary allows the multimodal access to the Wikipedia encyclopedia, Voice Pronunciations has been developed to facilitate the learning of new languages by means of games with words and images, whereas Voice Browser provides a fast and effective multimodal interface to the Google web search engine. All the applications in the system can be accessed multimodally using traditional graphic user interfaces such as keyboard and mouse, and/or by means of voice commands. Thus, the results are accessible also for motorhandicapped and visually impaired users and are easier to access by any user in small hand-held devices where graphical interfaces are in some cases difficult to employ. Research funded by projects CICYT TIN 2008-06742-C02-02/TSI, CICYT TEC 2008-06732-C02-02/TEC, CAM CONTEXTS (S2009/TIC-1485), and DPS 2008-07029-C02-02. Publicado

The main objective of multimodal conversational agents is to provide a more engaged and participative communication by allowing users to employ more than one input methodologies and providing output channels that are different to exclusively using voice. This chapter presents a detailed study on the benefits, disadvantages, and implications of incorporating multimodal interaction in conversational agents. Initially, it focuses on implementation techniques. Next, it explains the fusion and fission of multimodal information and focuses on the core module of these agents: the dialogue manager. Later on, the chapter addresses architectures, tools to develop some typical components of the agents, and evaluation methodologies. As a case of study, it describes the multimodal conversational agent in which we are working at the moment to provide assistance to professors and students in some of their daily activities in an academic centre, for example, a University’s Faculty.

Proceedings of: 2nd International Symposium on Ambient Intelligence (ISAmI 2011). Salamanca, 06-08 April 2011. In this paper, we describe a multi-agent system developed for teaching support and student's self-learning. The main objective of the EducAgent platform is the creation of an innovative virtual space following the principles of the European Higher Education Area to make subjects and e-learning initiatives to become a more flexible, participatory and attractive space. One of the most important characteristics of the developed platform is to facilitate a more natural interaction between the system and students by means of conversational agents. We describe the main features of the EducAgent platform and its application in the new European Computer Science Degree at the Carlos III University of Madrid. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, CAMCONTEXTS (S2009/TIC-1485), and DPS2008-07029-C02-02. Publicado

Proceedings of: International Symposium on Distributed Computing and Artificial Intelligence (DCAI 2011). Salamanca, 06-08 April 2011. In this paper, we present a technique for developing user simulators which are able to interact and evaluate conversational agents. Our technique is based on a statistical model that is automatically learned from a dialog corpus. This model is used by the user simulator to provide the following answer taking into account the complete history of the interaction. The main objective of our proposal is not only to evaluate the conversational agent, but also to improve this agent by employing the simulated dialogs to learn a better dialog model. We have applied this technique to design and evaluate a conversational agent which provides academic information in a multi-agent system. The results of the evaluation show that the conversational agent reduces the time needed to fulfill to complete the the dialogs, thereby allowing the conversational agent to tackle new situations and generate new coherent answers for the situations already present in an initial model. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02- 02/TEC, CAM CONTEXTS (S2009/TIC-1485), and DPS2008-07029-C02-02. Publicado

Proceedings of: 9th International Conference on Practical Applications of Agents and Multiagent Systems (PAAMS 11). Salamanca, 6-8 April, 2011 In this paper we present a methodology to develop commercial conversational agents that avoids the effort of manually defining the dialog strategy for the dialog management module. Our corpus-based methodology is based on selecting the next system answer by means of a classification process in which the complete dialog history is considered. This way, system developers can employ standards like VoiceXML to simply define system prompts and the associated grammars to recognize the users responses to the prompt, and the statistical dialog model automatically selects the next system prompt.We have applied this methodology for the development of an academic conversational agent. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC 2008-06732-C02-02/TEC, CAM CONTEXTS (S2009/TIC-1485), and DPS2008-07029- C02-02. Publicado

Proceeding of: 5th KES International Conference, KES-AMSTA 2011, Manchester, UK, June 29 – July 1, 2011 Much effort has been spent on suggesting and implementing new architectures of MAS to specific domains. Often each new architecture is not even compared to any existing architectures in order to evaluate their potential benefits. The evaluation of Multi-Agent Systems (MAS) is a complex problem and it does not have a single form. The present work follows the research line of considering the agent interaction as the main evaluation criteria, the most important characteristic of any complex software as autonomous agents according to [9]. So, in this paper, we have suggested an assignment of evaluation values to Agents interaction in an specific MAS architecture for providing context services by means of conversational agents. This evaluation is mainly based on the relevance of the messages content brought by an interaction. For dependant nature of the relevance of the messages, the valuation has to be adhoc, but our paper provides an example of how interesting is this alternative in order to evaluate any MAS architecture theoretically. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732- C02-02/TEC, SINPROB, CAM MADRINET S-0505/TIC/0255, and DPS2008-07029- C02-02.

Proceedings of: 6th International Conference on Soft Computing Models in Industrial and Environmental Applications (SOCO 2011). Salamanca, April 6-8, 2011 In this paper we present a proposal for the development of conversational agents that, on the one hand, takes into account the benefits of using standards like VoiceXML, whilst on the other, includes a module with a soft computing model that avoids the effort of manually defining the dialog strategy. This module is trained using a labeled dialog corpus, and selects the next system response considering a classification process based on neural networks that takes into account the dialog history. Thus, system developers only need to define a set of VoiceXML files, each including a system prompt and the associated grammar to recognize the users responses to the prompt. We have applied this technique to develop a conversational agent in VoiceXML that provides railway information in Spanish. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02- 02/TEC, CAM CONTEXTS (S2009/TIC-1485), and DPS2008-07029-C02-02. Publicado

This paper proposes a new technique to develop more robust spoken dialogue systems, which aims to repair incorrect semantic representations obtained by the systems due to ASR errors. To do so, it relies on a training stage that takes into account previous system misunderstandings for each dialogue state. Experiments have been carried out employing two systems (Saplen and Viajero) previously developed in our lab, which employ a prompt-independent language model and several prompt-dependent language models for ASR. The results, obtained for a corpus of 20,000 simulated dialogues, show that the technique enhances system performance for both kinds of language model, especially for the prompt-independent language model.

In this paper we present a proposal to develop commercial dialog systems that avoids the effort of manually defining the dialog strategy for the dialog manager and also takes into account the benefits of using standards like VoiceXML. In our proposal the dialog manager is trained using a labeled dialog corpus, and selects the next system response considering a classification process based on neural networks that takes into account the complete dialog history. Thus, system developers only need to define a set of VoiceXML files, each including a system prompt and the associated grammar to recognize user responses. The statistical dialog model automatically selects the next system prompt.We have applied this technique to develop a dialog system in VoiceXML that provides railway information in Spanish.

Proceedings of: 9th International Conference on Practical Applications of Agents and Multiagent Systems (PAAMS 11). Salamanca, 6-8 April, 2011 In this paper, we present an agent-based dialog simulation technique for learning new dialog strategies and evaluate conversational agents. Using this technique the effort necessary to acquire data required to train the dialog model and then explore new dialog strategies is considerably reduced. A set of measures has also been defined to evaluate the dialog strategy that is automatically learned and compare different dialog corpora. We have applied this technique to explore the space of possible dialog strategies and evaluate the dialogs acquired for a conversational agent that collects monitored data from patients suffering from diabetes. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02- 02/TEC, CAM CONTEXTS (S2009/TIC-1485), and DPS2008-07029-C02-02. Publicado

This paper presents a new toolkit for the evaluation of spoken dialogue systems in Ambient Intelligence domains. It describes the design and implementation of the toolkit, which consists in several tools that can be employed to process corpora of recorded spoken interaction and automatically obtain statistical measures in order to assess and improve these systems. The toolkit has been applied to evaluate the Mayordomo multimodal dialogue system developed in our laboratory, which enables automatic control of home appliances. This evaluation was useful to detect problems in the dialogue system suggesting which modules must be improved.

Continuous advances in the field of spoken dialogue systems make the processe of design, implementation and evaluation of these systems more and more com-s plex. To solve problems emerging from this complexity, a technique which has attracted increasing interest during the last decades is based on the automatic generation of dialogues between the system and a user simulator, which is another system that represents human interactions with the dialogue system. This chapter describes the main methodologies and techniques developed to create user simulators, and presents a discussion of their main characteristics and the benefits that they provide for the development, improvement and assessment of this kind of systems. Additionally, we propose a user simulation technique to test the performance of spoken dialogue systems. The technique is based on a novel approach to simulating different levels of user cooperativeness, which allows carrying out a more detailed system assessment. In the experiments we have evaluated a spoken dialogue system designed for the fast food domain. Theevaluation has focused on the performance of the speech recogniser, semantic analyser and dialogue manager of this system. The results show that the technique provides relevant information to obtain a solid evaluation of the system, enabling us to find problems in these modules which cannot be observed taking into account just one cooperativeness level.

In this paper, we present an approach to spoken dialog management based on the use of a Stochastic Finite-State Transducer estimated from a dialog corpus. The states of the Stochastic Finite-State Transducer represent the dialog states, the input alphabet includes all the possible user utterances, without considering specific values, and the set of system answers constitutes the output alphabet. Then, a dialog describes a path in the transducer model from the initial state to the final one. An automatic dialog generation technique was used in order to generate the dialog corpus from which the transducer parameters are estimated. Our proposal for dialog management has been evaluated in a sport facilities booking task.

Proceedings of: Forth Internationl Wokshop on User-Centric Technologies and Applications (CONTEXTS 2010), Valencia, September 7-10, 2010 In human-human interaction, a great deal of information is conveyed without explicit communication. This context information characterizes the situation of the different entities involved in the communication process (users, place, environment and computational objects). In this paper, we present an agent-based architecture that incorporates this valuable information to provide the most adapted service to the user. One of the main characteristics of our proposal is the incorporation of conversational agents handling different domains and adapted taking into account the different users requirements and preferences by means of a context manager. This way, we ensure a natural communication between the user and the system to provide a personalized service. The implementation of our proposed architecture to develop and evaluate a context-aware railway information system is also described. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, SINPROB, CAM MADRINET S-0505/TIC/0255, and DPS2008-07029-C02-02. Publicado

This paper presents a multimodal dialogue system called Mayordomo which aims at easing the interaction with home appliances using speech and a graphical interface within an Ambient Intelligence environment. We present the methods employed for implementing the system describing the design of the user-system interactions as well as additional features such as the management of user profiles to restrict the access to domestic appliances and customize the recognition grammars and the generated responses.

This paper proposes a new technique to implicitly correct some ASR errors made by spoken dialogue systems, which is implemented at two levels: statistical and linguistic. The goal of the former level is to employ for the correction knowledge extracted from the analysis of a training corpus comprised of utterances and their corresponding ASR results. The outcome of the analysis is a set of syntactic-semantic models and a set of lexical models, which are optimally selected during the correction. The goal of the correction at the linguistic level is to repair errors not detected during the statistical level which affects the semantics of the sentences. Experiments carried out with a previouslydeveloped spoken dialogue system for the fast food domain indicate that the technique allows enhancing word accuracy, spoken language understanding and task completion by 8.5%, 16.54% and 44.17% absolute, respectively. Ministerio de Ciencia y Tecnología TIN2007-64718 HADA

This paper proposes a new technique to enhance the performance of spoken dialogue systems employing a method that automatically corrects semantic frames which are incorrectly generated by the semantic analyser of these systems. Experiments have been carried out using two spoken dialogue systems previously developed in our lab: Saplen and Viajero, which employ prompt-dependent and prompt-independent language models for speech recognition. The results obtained from 10,000 simulated dialogues show that the technique improves the performance of the two systems for both kinds of language modelling, especially for the prompt-independent language model. Using this type of model the Saplen system increased sentence understanding by 19.54%, task completion by 26.25%, word accuracy by 7.53%, and implicit recovery of speech recognition errors by 20.30%, whereas for the Viajero system these figures increased by 14.93%, 18.06%, 6.98% and 15.63%, respectively.

Proceedings of: Forth International Workshop on User-Centric Technologies and applications (CONTEXTS 2010). Valencia, 07-10 September , 2010. The current interest and extension of social networking are rapidly introducing a large number of applications that originate new communication and interaction forms among their users. Social networks and virtual worlds, thus represent a perfect environment for interacting with applications that use multimodal information and are able to adapt to the specific characteristics and preferences of each user. As an example of this application, in this paper we present an example of the integration of conversational agents in social networks, describing the development of a conversational avatar that provides academic information in the virtual world of Second Life. For its implementation techniques from Speech Technologies and Natural Language Processing have been used to allow a more natural interaction with the system using voice. Funded by projects CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02-02/TEC, SINPROB, CAM MADRINET S-0505/TIC/0255, and DPS2008-07029-C02-02. Publicado

In this paper, we present a technique for learning new dialog strategies by using a statistical dialog manager that is trained from a dialog corpus. A dialog simulation technique has been developed to acquire data required to train the dialog model and then explore new dialog strategies. A set of measures has also been defined to evaluate the dialog strategy that is automatically learned. We have applied this technique to explore the space of possible dialog strategies for a dialog system that collects monitored data from patients suffering from diabetes.

Proceedings of: 8th International Conference on Practical Applications of Agents and Multiagent Systems, Salamanca, Spain, April 26-28, 2010. In this paper, we present a architecture for the development of conversational agents that provide a personalized service to the user. The different agents included in our architecture facilitate an adapted service by taking into account context information and users specific requirements and preferences. This functionality is achieved by means of the introduction of a context manager and the definition of user profiles. We describe the main characteristics of our architecture and its application to develop and evaluate an information system for an academic domain. CICYT TIN2008-06742-C02-02/TSI, CICYT TEC2008-06732-C02- 02/TEC, SINPROB, CAM MADRINET S-0505/TIC/0255 and DPS2008-07029-C02-02. Publicado

We are interested in the problem of understanding human conversation structure in the context of human-machine and human-human interaction. We present a statistical methodology for detecting the structure of spoken dialogs based on a generative model learned using decision trees. To evaluate our approach we have used the LUNA corpora, collected from real users engaged in problem solving tasks. The results of the evaluation show that automatic segmentation of spoken dialogs is very effective not only with models built using separately human-machine dialogs or human-human dialogs, but it is also possible to infer the task-related structure of human-human dialogs with a model learned using only human-machine dialogs. Index Terms: Domain Knowledge Acquisition, Spoken Dialog Systems, Dialog Structure Annotation.

In this paper, we test the applicability of a stochastic user simulation technique to generate dialogs which are similar to real human-machine spoken interactions. To do so, we present a comparison between two corpora employing a comprehensive set of evaluation measures. The first corpus was acquired from real interactions of users with a spoken dialog system, whereas the second was generated by means of the simulation technique, which decides the next user answer taking into account the previous user turns, the last system answer and the objective of the dialog.

In this paper, we present a context-aware architecture to facilitate adaptable web services. In our approach, users interact with the system by means of a speech-based engine that allows using spontaneous speech to access the different services. These services are personalized by taking into account context information. The Appear platform is used for gathering context of use data and building a model based on the information needs of end-users. A practical web application of our approach for a railway information domain is also described.

In this paper, we present a statistical methodology for dialog management. This methodology is based on a classification procedure that considers all of the previous history of the dialog and the result of the queries to the module that controls the application. The classification procedure is divided into two phases to generate the next system turn. We summarize the work that has been carried out to apply this methodology for developing a dialog system for booking sports facilities. A dialog simulation technique has been used to acquire a dialog corpus for this task and to develop a dialog manager following the proposed methodology. The final model has been evaluated using a corpus recorded for the task with real users.

In this paper, we present a statistical approach for the development of a dialog manager and for learning optimal dialog strategies. This methodology is based on a classification procedure that considers all of the previous history of the dialog to select the next system answer. To evaluate the performance of the dialog system, the statistical approach for dialog management has been extended to model the user behavior. The statistical user simulator has been used for the evaluation and improvement of the dialog strategy. Both the user model and the system model are automatically learned from a training corpus that is labeled in terms of dialog acts. New measures have been defined to evaluate the performance of the dialog system. Using these measures, we evaluate both the quality of the simulated dialogs and the improvement of the new dialog strategy that is obtained with the interaction of the two modules. This methodology has been applied to develop a dialog manager within the framework of the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish. We propose the use of corpus-based methodologies to develop the main modules in the dialog system.

In this paper, we present a statistical approach for dialog management within the framework of two different domains. The dialog model, that is automatically learned from a data corpus, is based on the use of a classification process to generate the next system answer. A neural network classifier is used for the selection process. This methodology has been applied in a spoken dialog system that provides railway information. The definition of an extended methodology that takes into account new system functionalities and its application for developing a dialog system for booking sports facilities is also described.

Since the design and acquisition of a new dialog corpus is a complex task, new methods to facilitate this task are necessary. In this paper, we present a methodology to make use of our previous work within the framework of dialog systems in order to acquire a dialog corpus for a new domain. The main idea is the simulation of recognition and understanding errors in the acquisition of the new dialog corpus. This simulation is based on the analysis of such errors in a previously acquired corpus and the definition of a correspondence table among the concepts and attributes of both tasks. This correspondence table is based on the similarity of semantic meaning and frequencies. Finally, the application of this methodology is illustrated in some examples.

In this paper, we present a statistical approach for the automatic generation of dialogs by means of a user simulator. This technique can be used to generate dialogs with reduced effort, facilitating the evaluation and improvement of spoken dialog systems. In our approach for user simulation, the user answer is selected taking into account the history of the dialog and the last system turn, as well as the objective(s) set for the dialog. The user model is automatically learned from a training corpus that is labeled in terms of dialog acts. This methodology has been evaluated within the framework of the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish.

This paper presents a Passage Retrieval-based approach for the development of spoken Question Answering systems. Question Answering can be seen as a particular aspect of Information Retrieval where the user's information need are not satisfied by means of a document, but by a portion of text. Currently, the performance of typical Question Answering systems is rather poor, if compared with other Information Retrieval tasks, however, it is predictable that these system will be used in combination with some speech interface when high precision will be achieved. The most important evaluation competitions for Question Answering, such as TREC and CLEF, still do not have a spoken Question Answering track, therefore this work presents a novel approach to this task in order to study the influence of recognition errors over Question Answering systems.

In this article, we present an approach to the development of a stochastic dialog manager. The model used by this dialog manager to generate its turns takes into account both the last turns of the user and system, and the information supplied by the user throughout the dialog. As the space of situations that can be presented in the dialogs is too large, some techniques for reducing this space have been proposed. This system has been developed in the DIHANA project, whose goal is the design and development of a dialog system to access a railway information system using spontaneous speech in Spanish. A training corpus of 900 dialogs, that was acquired through the Wizard of Oz, was used to learn the models. An evaluation of the dialog manager is also presented

Recent reports indicate the multiple benefits of adopting conversational systems for healthcare, providing automation of management tasks in overburdened healthcare systems, improved patient experiences, and better efficiency and productivity rates. In this paper, we describe a conversational system aimed at promoting healthy lifestyle habits related to nutrition and physical exercise. The system has been developed using Google’s Dialogflow platform and the combination of different APIs and data repositories in the cloud. It has been integrated in the Facebook Messenger instant messaging platform. The results of the preliminary assessment of the system through a subjective questionnaire show the high degree of satisfaction of the users with the functionalities provided and the help offered to allow fulfilling the predefined objectives.

This preprint follows ACM policy: “Authors who publish with ACM have the freedom to post peer-reviewed pre-print versions of their papers to personal websites and institutional repositories. They can add a single-click link to their final published papers, and re-use any portion of their published work with the inclusion of a citation and DOI link. Authors can also post on any repository legally mandated by the agency funding the research on which the work is based, and on any non-commercial repository or aggregation that does not duplicate ACM tables of contents/substantially duplicate an ACM-copyrighted volume or issue” (https://authors.acm.org/author-resources/author-rights.) 2, e-Coaches are promising intelligent systems that aims at supporting human everyday life, dispatching advice through different interfaces, such as apps, conversational interfaces and augmented reality interfaces. This workshop aims at exploring how e-coaches might benefit from spatially and timemultiplexed interfaces and from different communication modalities (e.g., text, visual, audio, etc.) according to the context of the interaction., The NESTORE, SAAM, CAPTAIN, HOLOBALANCE, EMPATHIC, MENHIR, vCare projects are supported by the European Commission under the Horizon 2020 programmes SC1-PM-15-2017 H2020-MSCA-RISE-2018, and H2020-1.3.3, respectively through the project grants N.769643, 769661, 769830, 769574, 769872, 823907, 769807. The authors want to thank their respective Consortia. The opinions expressed in this paper are those of the authors and are not necessarily those of the project partners or the European Commission.

This is a pre-print version of the chapter: Zoraida Callejas, David Griol. 2021. Conversational agents for mental healthand wellbeing. Accepted for the book: Lopez-Soto, T. (Ed.). (2021). Dialog Systems: A Perspective from Language,Logic and Computation. Springer International Publishing.https://doi.org/10.1007/978-3-030-61438-6(https://www.springer.com/gp/book/9783030614379)This preprint follows Springer Self-archiving policy for non-open access books and chapters (https://www.springer.com/gp/open-access/publication-policies/self-archiving-policy): “authors may deposit a portion ofthe pre-submission version of their manuscript (preprint) in a recognised preprint server (...). This portion of thepre-submission manuscript (preprint) may be deposited and made publicly available at a, Recent advances in spoken language technology, artificial intelligence, and conversational interface design, coupled with the emergence of smart devices, have increased the possibilities of using conversational interfaces for a growing range of application domains. These interfaces are currently applied in the healthcare domain in a range of innovative tasks that allow to provide a more natural an user-friendly human-machine communication, promote patient participation in their own care, and help and support medical professionals. This chapter provides a detailed description of the great potential for the use of conversational interfaces for the specific area of mental health by means of the description of their most valuable applications in this domain and the open challenges for future research., This research has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 823907 (MENHIR project https://menhir-project.eu).

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a videobased task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.