Search

The world has become multimodal. In addition to text, we have been sharing a huge amount of multimedia information in the form of images and videos on the Internet. The wide spread use of smart mobile devices has also changed the way we interact with the Internet. It is now natural for us to capture images and videos freely and use as part of a query, in addition to the traditional text and voices. These, along with the rapid advancements in multimedia, natural language processing, information retrieval, and conversation technologies, mean that it is time for us to explore multimodal conversation and its roles in search and recommendation. Multimodal conversation has the potential to help us to uncover and digest the huge amount of multimedia information and knowledge hidden within many systems. It also enables a natural 2-way interactions between humans and machines, with mutual benefits in enriching their respective knowledge. Finally, it opens up the possibilities of disrupting many existing applications and launching new innovative applications. This panel is timely and aims to explore this emerging trend, and discuss its potential benefits and pitfalls to society. The panel will also explore the limitations of current technologies and highlight future research directions towards developing a multimedia conversational system.

Despite the increasing presence of conversational agents (CAs) in our daily lives, the lack of information and technology behind them prevents CAs from answering many questions. One of the most typical problems facing conversational user interfaces today is that they often disappoint people by giving the same answer (e.g., “I don't know”). In this work, we focused on situations in which CAs do not provide a proper answer because of a lack of information. Under these situations, we aimed to find more effective answer strategies for CAs to provide people better user experiences. We tested four different response strategies using different degrees of inferences and information as ground. We found differences in usability and user experience depending on how CAs respond. Our results will help designers understand how people feel about the way CAs respond and create better CA responses in situations where it is difficult to provide accurate answers.

This paper describes how machine learning training data and symbolic knowledge from curators of conversational systems can be used together to improve the accuracy of those systems and to enable better curatorial tools. This is done in the context of a real-world practice of curators of conversational systems who often embed taxonomically-structured meta-knowledge into their documentation. The paper provides evidence that the practice is quite common among curators, that is used as part of their collaborative practices, and that the embedded knowledge can be mined by algorithms. Further, this meta-knowledge can be integrated, using neuro-symbolic algorithms, to the machine learning-based conversational system, to improve its run-time accuracy and to enable tools to support curatorial tasks. Those results point towards new ways of designing development tools which explore an integrated use of code and documentation by machines.

As the complexity of intelligent environments grows, there is a need for more sophisticated and flexible interfaces. Conversational systems constitute a very interesting alternative to ease the users’ workload when interacting with such environments, as they can operate them in natural language. A number of commercial toolkits for their implementation have appeared recently. However, these are usually tailored to specific implementations of the processes involved for processing the user’s utterance and generate the system response. In this paper, we present a modular architecture to develop conversational systems by means of a plug-and-play paradigm that allows the integration of developers’ specific implementations and commercial utilities under different configurations that can be adapted to the specific requirements for each system.

Conversational Systems (CSs) are increasingly being used and receiving investments. However, these systems can present problems that discourage users from using them. This paper aims to verify the opinions of Brazilian users and ex-users of CSs about the use of these systems and their possible difficulties. We conducted an online survey with several questions about the use of CSs and obtained 113 respondents from different knowledge areas, such as law, agronomy, health, computing, among others. Results reveal some issues in the Brazilian context that negatively influence the use of CSs, such as accent, lack of support for the Portuguese language, and internet access problems. Despite this, results show that respondents consider CSs easy to use and satisfactory. We hope our results can contribute to improving these systems for the Brazilian context, providing insights for the exploration of new strategies for designers and developers.

We report the results of a crowdsourcing user study for evaluating the effectiveness of human-chatbot collaborative conversation systems, which aim to extend the ability of a human user to answer another person's requests in a conversation using a chatbot. We examine the quality of responses from two collaborative systems and compare them with human-only and chatbot-only settings. Our two systems both allow users to formulate responses based on a chatbot's top-ranked results as suggestions. But they encourage the synthesis of human and AI outputs to a different extent. Experimental results show that both systems significantly improved the informativeness of messages and reduced user effort compared with a human-only baseline while sacrificing the fluency and humanlikeness of the responses. Compared with a chatbot-only baseline, the collaborative systems provided comparably informative but more fluent and human-like messages.

Museums and Art exhibition spaces are adopting Artificial Intelligence (AI) systems to engage and attract visitors in several contexts. The use of AI can boost visitors’ attention, promote informal learning through conversations in front of the exhibits, and motivate visitors to act socially. In this paper, we describe a voice-based conversational system Iris+, in which visitors are inquired to answer questions to an agent. It is a proactive agent that invites visitors to reflect and take action to improve future world. First, we will describe how this system works. Second, we will show the outcomes of evaluation studies with visitors in situ and, a survey shows how visitors engaged in social action after interacting with IRIS+ and public demographics. Based on those visitors’ studies, we propose a set of challenges to design conversational systems in public spaces and the improvements were incorporated into the redesign of IRIS++.

We present a new solution towards building a crowd knowledge enhanced multimodal conversational system for travel. It aims to assist users in completing various travel-related tasks, such as searching for restaurants or things to do, in a multimodal conversation manner involving both text and images. In order to achieve this goal, we ground this research on the combination of multimodal understanding and recommendation techniques which explores the possibility of a more convenient information seeking paradigm. Specifically, we build the system in a modular manner where each modular construction is enriched with crowd knowledge from social sites. To the best of our knowledge, this is the first work that attempts to build intelligent multimodal conversational systems for travel, and moves an important step towards developing human-like assistants for completion of daily life tasks. Several current challenges are also pointed out as our future directions.

There are several service-oriented models, where services are deployed for the user. The user chooses any one of them. During the operational life cycle of these services, there are several issues that occur. User wants an interface for complaint. This paper uses the sentence boundary detection, NER, document categorization, and sentiment extraction methodologies. The natural language processing generates the training model which is a statistical representation of current system knowledge. When a user enters the input then training model extracts the value of a different parameter, these parameters used by call center model for better understanding of user input. The machine learning model used to generate a logical response. When the time exceeds the size of sample, data should be increased and the model understanding also more and more accurate. The accuracy of the model depends on the size of the sample training data. When training data size increases the statistical model for the call center is updated. When the user interacts with the call center agent then call center agent to extract the value of all parameters based on the current statistical model which is based on the sample training data. The paper uses different parameter such as NER, Document category, and sentiment for making a better user interaction. The probability of correct response is increase n time if n parameters are used for response generation. Call center module to take help from sentence detection, NER, document categorization and sentiment training model for extraction the value of the parameter. These parameter value helpful for extracting the NLP Text meaning. The response correctness also increases whenever anyone parameter is extracted correctly. The maximum entropy approach is used for making statistical modeling. The training data are taken from the heterogeneous source.

Conversational agents have the potential to support users in many tasks. However, support for productivity and well-being in the workplace has received little attention. We present the first design of a conversational system that supports information workers with multiple work-related goals, informed by a survey of the current and potential use of conversational agents in the workplace. The goals of this research include the evaluation of using an agent for scheduling and prioritizing tasks, switching tasks, providing break reminders, dealing with social media distractions and for end of the day reflection on tasks accomplished. We deployed a chat-based intelligent agent, named Amber, in a field study with 24 information workers over the course of 6 days. We present our preliminary findings from the field study and discuss implications for the design of future workplace conversational agents

How does the presence of an audience influence the social interaction with a conversational system in a physical space? To answer this question, we analyzed data from an art exhibit where visitors interacted in natural language with three chatbots representing characters from a book. We performed two studies to explore the influence of audiences. In Study 1, we did fieldwork cross-analyzing the reported perception of the social interaction, the audience conditions (visitor is alone, visitor is observed by acquaintances and/or strangers), and control variables such as the visitor's familiarity with the book and gender. In Study 2, we analyzed over 5,000 conversation logs and video recordings, identifying dialogue patterns and how they correlated with the audience conditions. Some significant effects were found, suggesting that conversational systems in physical spaces should be designed based on whether other people observe the user or not.

In recent years, the task of Question Answering over passages, also pitched as a reading comprehension, has evolved into a very active research area. A reading comprehension system extracts a span of text, comprising of named entities, dates, small phrases, etc., which serve as the answer to a given question. However, these spans of text would result in an unnatural reading experience in a conversational system. Usually, dialogue systems solve this issue by using template-based language generation. These systems, though adequate for a domain specific task, are too restrictive and predefined for a domain independent system. In order to present the user with a more conversational experience, we propose a pointer generator based full-length answer generator which can be used with most QA systems. Our system generates a full length answer given a question and the extracted factoid/span answer without relying on the passage from where the answer was extracted. We also present a dataset of 315000 question, factoid answer and full length answer triples. We have evaluated our system using ROUGE-1,2,L and BLEU and achieved 74.05 BLEU score and 86.25 Rogue-L score.

In this paper, we discuss dialogue failures and how this affects the user experience, specifically for a scenario of a multi-bot conversational system. Additionally, we show how the use of multiple chatbots provide new strategies to overcome misunderstandings and to keep the user in the conversation flow. To inform such conclusions and recommendations, we describe a study with a multi-bot wealth management advice system in which participants conversed with four chatbots simultaneously. We analyzed each conversation log applying thematic network analysis and manually identified the main instances of dialogue failures, usually provoked by chatbot misunderstandings or system breakdowns. We examined the follow-up users’ utterances after each failure, and the users’ strategies to deal with them. We categorize our findings into a list of the most common users’ strategies, and highlight solutions provided by a multi-bot approach in assisting the dialogue failures.

We present a conversational system that aims at calculating the amount of consumed carbohydrates in a meal by diabetic patients. Through a chat input, users can freely describe foods, which are first semantically interpreted and then matched against a nutritionist database for the final calculation of carbohydrates. Specific issues that have been addressed include: large-scale food recognition in Italian, without any restriction; interpretation of fuzzy quantities in relation to food (e.g. a portion of, a dish of, etc.); exploitation of dialogue strategies to revise system mis-interpretations and failures. CH1 integrates innovative neural approaches to language interpretation with rule-based approaches for ontology reasoning. In the paper we provide both experimental evaluations for the main components of the system, as well as qualitative user tests.

This paper presents a model of incremental speech generation in practical conversational systems. The model allows a conversational system to incrementally interpret spoken input, while simultaneously planning, realising and self-monitoring the system response. If these processes are time consuming and result in a response delay, the system can automatically produce hesitations to retain the floor. While speaking, the system utilises hidden and overt self-corrections to accommodate revisions in the system. The model has been implemented in a general dialogue system framework. Using this framework, we have implemented a conversational game application. A Wizard-of-Oz experiment is presented, where the automatic speech recognizer is replaced by a Wizard who transcribes the spoken input. In this setting, the incremental model allows the system to start speaking while the user's utterance is being transcribed. In comparison to a non-incremental version of the same system, the incremental version has a shorter response time and is perceived as more efficient by the users.

In this paper, we present our progress so far in realization of project aimed to create a complex, modular humor-equipped conversational system. By complex, we mean that it should be able to: (1) detect users’ emotions, (2) detect users’ humorous behaviors and react to them properly, (3) generate humor according to users’ emotive states and (4) learn each user’s individual sense of humor. The research is conducted in Japanese. We chose puns as a relatively computable genre of humor. We describe a general outline of our system, as well as its four modules: humor detection module, emotion recognition module, response generator module and individualisation module. We present the algorithm of systems used in each module, along with some evaluation results.

This paper presents a chat-like conversational system, and that generates a reply by selecting an appropriate reply generating module. Such modules consist in selecting a sentence from an article of Web news, retrieving a definition sentence in Wikipedia, question-answering, and so on. A dialogue strategy corresponds to which reply generating module should be chosen according to a user input and the dialogue history, and is learned in the MDP framework. User evaluations showed that our system could learn an appropriate dialogue strategy, and perform natural dialogues.

The evaluation of subjective aspects of HCI, such as human-likeness, likeability or users' emotions towards computers is still quite a neglected issue, especially in the field of non-task oriented conversational systems (chatterbots). In this paper we try to bridge this gap by proposing a new methodology of evaluation. The methods presented were tested in our research on humor-equipped chatterbots. We describe them in details, discuss their drawbacks and usability. In one of the presented methods we used an emotiveness analysis system, which itself can be considered an AI tool, as it was used to detect users' emotions towards conversational systems, and to perform their automatic evaluation. We also propose some methods that we have not used yet, which, however, seem applicable in this field, such as brain scanning techniques. Finally, we give some ideas that should be addressed in the future.

Described in the article are the multiaccess systems on the basis of the algorithmic language BASIC, developed in the N.I. Lobachevski's Nizhny Novgorod state university in the period of 1969-1978. The first of them was the BASIC- batch-processing system that was introduced on an M-20 type ECM (BESM-3, BESM-4, M-220, M-222) in early 1970. It was actively used in academic processes and, unlike other computers of the time, had an incremental compiler. This option, in 1971, has allowed changing the batch-processing system into a multiaceess conversational system entitled "Student". The system was installed free of charge in more than a hundred design organizations and educational institutions of the country. In 1976 the last version of the time sharing operation system appeared that provided an opportunity to conversationally program on a Unified Series ECM under a Unified Series DOS.

In conversation, a speaker sometimes asks questions that relate to another speaker’s detailed personality, such as his/her favorite foods and sports. This behavior also appears in conversations with conversational agents; therefore, agents should be developed that can respond to such questions. In previous agents, this was achieved by creating question-answer pairs defined by hand. However, when a small number of persons create the pairs, we cannot know what types of questions are frequently asked. This makes it difficult to know whether the created questions cover frequently asked questions; therefore, such essential question-answer pairs for conversational agents are possibly overlooked. This study analyzes a large number of question-answer pairs for six personae created by many question-generators, with one answer-generator for each persona. The proposed approach allows many questioners to create questions for various personae, enabling us to investigate the types of questions that are frequently asked. A comparison with questions appearing in conversations between humans shows that 50.2% of the questions were contained in our question-answer pairs and the coverage rate was almost saturated with the 20 recruited question-generators.

In this paper we describe a very preliminary conversational system focused on the automated diagnosis of heart conditions. The system uses a relational Ngram model to extract meaning from user queries and answers, and maintains a belief network consisting of 48 different cardiac conditions, symptoms, and other internal system parameters. The end result is a system that can process an initial user query, ask pertinent questions, and to deduce reasonable medical conclusions. Although several example output conversations are shown, the medical accuracy of the diagnosis is not the focus of this paper. Instead this paper focuses on the system level infrastructure and methodology for building a medical conversation system.

The long term goal of this research is to build artificial conversational intelligence that can set up or participate in the fluent conversational interactions as good as people in order to benefit each other. This paper discusses conversation quanta as a foundation of conversational intelligence. In contrast to conversational systems for which much emphasis has been placed on the symbolic processing and algorithms, our approach is data-intensive, allowing for the conversational system to acquire the depth and proficiency in interaction in an incremental fashion, in addition to the broad coverage and robustness.

In this paper we describe a phonetic classification of Japanese puns (dajare). Basing on real life examples gathered from available sources (books, Internet), we divided Japanese puns into 12 groups with numerous subgroups, according to phonetic changes that occur within them. This classification was prepared for the NLP purpose, i.e. to be used in humor processing. Its usefulness was shown in a research project, aimed at constructing a humor-equipped conversational system for Japanese.

One of the biggest bottlenecks for conversational systems is large-scale provision of suitable content. Our approach readily provides this without the need for custom-crafting. In this demonstration, we present the use of question-answer (QA) pairs mined from online question-and-answer websites to construct system utterances for a conversational agent. Our system uses QA pairs to formulate utterances that drive a conversation in addition to the answering of user questions as has been done in previous work. We use a collection of strategies that specify how and when the different parts of our question-answer pairs can be used and augmented with a small number of generic hand-crafted text snippets to generate natural and coherent system utterances.

In this paper, we describe the effectiveness of utterance generation using causal knowledge for a dialogue system. Recently, there has been a variety of research on non-taskoriented dialogue systems; however, an effective approach has not yet been developed. One of the most important reasons for this is that non-task-oriented dialogue systems lack common sense knowledge, which is not in their databases. As the first step towards solving this problem, we concentrated on causal knowledge containing reasons and effects, which can provide unwritten meanings for utterance understanding and generating modules. In this paper we investigated how an utterance generated with knowledge related to user input can improve an existing conversational system. Experiment results show that utterance generation using causal knowledge can improve a conversational system.

In this chapter we define the notion of multimedia conversational system and provide a classification schema for characterizing the type of content which can be produced, stored and retrieved with these systems. This classification schema will be used in reviewing the capabilities of a number of existing multimedia conversational systems and to assess requirements for advanced indexing and retrieval of conversational content. To meet these requirements, we present new types of indexing techniques and provide an evaluation of their effectiveness on real case studies.

In this paper, we present a new specification, implementation and evaluation of the "guess the card" system. This is a conversational system with spoken Spanish and vision capabilities that plays a game with members of the general public in a permanent stand at a science museum. The system has been built using a methodology and programming environment based on the notion of dialogue model specification and interpretation, that we have been developing over the years. The present system uses the latest version of the formalism and improves considerably over previous versions [8]. In particular, the present version takes advantage of the discourse structure, increases modularity, and incorporates general and flexible recovery strategies. An evaluation of the system is also presented.

This paper contains the results of evaluation experiments conducted to investigate if implementation of a pun generator into a non-task oriented talking system improves the latter's performance. We constructed a simple joking conversational system and conducted one user evaluation experiment and two third person evaluation experiments. The results showed that humor does have a positive influence on the dialogue between humans and computers. The implications of this fact and problems that occurred during the research are discussed. We also propose how they can be solved in the future.

Computer mediated communication (CMC) systems are providing new ways to communicate. Yet many text-based CMC systems do not represent the invisible, interactive practices, such as turn-negotiation, commonly found in face-to-face (FTF) conversations. Designing new text structures may help address these problems. To explore the effects of adding rhythmic, non-verbal cues to computer-mediated communication (CMC), we developed Fugue, a networked environment that creates visualizations of conversations as they occur, dynamically highlighting social presence and turn-negotiating events on a two-dimensional grid. In this paper we examine traditional text-based conversation systems, the role of turn-negotiating, social presence and activity in FTF communication, and how Fugue makes these non-verbal linguistic cues explicit within a graphical display.

Humans use a combination of gesture and speech to convey meaning, and usually do so without holding a device or pointer. We present a system that incorporates body tracking and gesture recognition for an untethered human-computer interface. This research focuses on a module that provides parameterized gesture recognition, using various machine learning techniques. We train the support vector classifier to model the boundary of the space of possible gestures, and train Hidden Markov Models on specific gestures. Given a sequence, we can find the start and end of various gestures using a support vector classifier, and find gesture likelihoods and parameters with a HMM. Finally multimodal recognition is performed using rank-order fusion to merge speech and vision hypotheses.

In multi-agent, multi-user environments, users as well as agents should have a means of establishing who is talking to whom. In this paper, we present an experiment aimed at evaluating whether gaze directional cues of users could be used for this purpose. Using an eye tracker, we measured subject gaze at the faces of conversational partners during four-person conversations. Results indicate that when someone is listening or speaking to individuals, there is indeed a high probability that the person looked at is the person listened (p=88%) or spoken to (p=77%). We conclude that gaze is an excellent predictor of conversational attention in multiparty conversations. As such, it may form a reliable source of input for conversational systems that need to establish whom the user is speaking or listening to. We implemented our findings in FRED, a multi-agent conversational system that uses eye input to gauge which agent the user is listening or speaking to.

We describe Talk'n'Travel, a spoken dialogue language system for making air travel plans over the telephone. Talk'n'Travel is a fully conversational, mixed-initiative system that allows the user to specify the constraints on his travel plan in arbitrary order, ask questions, etc., in general spoken English. The system operates according to a plan-based agenda mechanism, rather than a finite state network, and attempts to negotiate with the user when not all of his constraints can be met.

The notions of dialogue, dialogue systems and intelligence are explored. Eight general aspects of dialogue and dialogue systems are identified to help describe current efforts at making communication with computers more intelligent. Four evaluative dimensions are discussed on which computer dialogue systems can be assessed, and then this descriptive/evaluative scheme is applied to three example dialogue systems: ELIZA , an early conversational system, GUS , a frame-based dialogue system and GUIDON , a medical diagnostic tutor. The paper ends with a few comments on the current state of evaluative work and future directions for our methodology.

Computer systems for clinical consulting on patient management operate on descriptions of medical expertise derived from repositories of systematized knowledge of medicine (textbooks and/or panels of experts) or from empirical situations embedded in medical records. The paper describes MARIS, a conversational system of the latter category, designed to provide relatively powerful consulting services for the management of patients in internal medicine.

The goal of this work is to make human-machine interaction in natural language smooth and efficient. To realize this, the machine has to generate natural language sentences which are understood by users most easily. And it is important for this understandability that each sentence is appropriate to the conversational situation at the time of its utterance. In this paper, we will shed light on this appropriateness and consider how we can formalize the method to generate such appropriate sentences. First, we will define such appropriateness; second we will discuss some surface features of Japanese sentences and consider how people decide to add such features to their sentences depending on situational factors. Then we will show brief methods of deciding how to add suitable surface features to sentences in suitable situations. Finally, we will show an implementation of a conversational system which adopts these methods.

We present a conversational system for the computer analysis of nucleic acid and protein sequences based on the well-known Queen and Korn program (1). The system can be used by persons with only minimal knowledge of computers.

Collection, storage and retrieval of health and safety information is critical in occupational settings in making decisions about safety of employees and the healthful nature of the work environment. Computer surveillance programs have been utilized previously, but have been too complicated, too mathematical, and not directly accessible to health and safety personnel. This paper discusses the Occupational Surveillance Interactive System (OSIS), a conversational system which utilizes English instead of a computer language, and which provides immediate feedback to the health and safety practitioner through fact retrieval and statistical inference. Additionally, an algorithm is introduced which provides a predictive cueing for the development of some disease processes. The hypothetical example illustrated herein introduces a prototypal computer surveillance system, operating under an interactive mode, to provide the health and safety professional with a decision-making tool in occupational settings.