Search

Measurement of interaction quality is a critical task for the improvement of spoken dialog systems. Existing approaches to dialog quality estimation either focus on evaluating the quality of individual turns, or collect dialog-level quality measurements from end users immediately following an interaction. In contrast to these approaches, we introduce a new dialog-level annotation workflow called Dialog Quality Annotation (DQA). DQA expert annotators evaluate the quality of dialogs as a whole, and also label dialogs for attributes such as goal completion and user sentiment. In this contribution, we show that: (i) while dialog quality cannot be completely decomposed into dialog-level attributes, there is a strong relationship between some objective dialog attributes and judgments of dialog quality; (ii) for the task of dialog-level quality estimation, a supervised model trained on dialog-level annotations outperforms methods based purely on aggregating turn-level features; and (iii) the proposed evaluation model shows better domain generalization ability compared to the baselines. On the basis of these results, we argue that having high-quality human-annotated data is an important component of evaluating interaction quality for large industrial-scale voice assistant platforms.

Search for resource-efficient materials in vast compositional spaces is an outstanding challenge in creating environmentally stable perovskite semiconductors. We demonstrate a physics-constrained sequential learning framework to subsequently identify the most stable alloyed organic-inorganic perovskites. We fuse data from high-throughput degradation tests and first-principle calculations of phase thermodynamics into an end-to-end Bayesian optimization algorithm using probabilistic constraints. By sampling just 1.8% of the discretized Cs MA FA PbI (MA, methylammonium; FA, formamidinium) compositional space, perovskites centered at Cs MA FA PbI show minimal optical change under increased temperature, moisture, and illumination with >17-fold stability improvement over MAPbI . The thin films have 3-fold improved stability compared with state-of-the-art multi-halide Cs (MA FA ) Pb(I Br ) , translating into enhanced solar cell stability without compromising conversion efficiency. Synchrotron-based X-ray scattering validates the suppression of chemical decomposition and minority phase formation achieved using fewer elements and a maximum of 8% MA. We anticipate that this data fusion approach can be extended to guide materials discovery for a wide range of multinary systems. Despite recent intensive efforts to improve the environmental stability of halide perovskite materials for energy harvesting and conversion, traditional trial-and-error explorations face bottlenecks in the navigation of vast chemical and compositional spaces. We develop a closed-loop optimization framework that seamlessly marries data from first-principle calculations and high-throughput experimentation into a single machine learning algorithm. This framework enables us to achieve rapid optimization of compositional stability for Cs MA FA PbI perovskites while taking the human out of the decision-making loop. We envision that this data fusion approach is generalizable to directly tackle challenges in de

Traditional goal-oriented dialogue systems rely on various components such as natural language understanding, dialogue state tracking, policy learning and response generation. Training each component requires annotations which are hard to obtain for every new domain, limiting scalability of such systems. Similarly, rule-based dialogue systems require extensive writing and maintenance of rules and do not scale either. End-to-End dialogue systems, on the other hand, do not require module-specific annotations but need a large amount of data for training. To overcome these problems, in this demo, we present Alexa Conversations, a new approach for building goal-oriented dialogue systems that is scalable, extensible as well as data efficient. The components of this system are trained in a data-driven manner, but instead of collecting annotated conversations for training, we generate them using a novel dialogue simulator based on a few seed dialogues and specifications of APIs and entities provided by the developer. Our approach provides out-of-the-box support for natural conversational phenomena like entity sharing across turns or users changing their mind during conversation without requiring developers to provide any such dialogue flows. We exemplify our approach using a simple pizza ordering task and showcase its value in reducing the developer burden for creating a robust experience. Finally, we evaluate our system using a typical movie ticket booking task and show that the dialogue simulator is an essential component of the system that leads to over $50\%$ improvement in turn-level action signature prediction accuracy.

Detecting an Out-of-Domain (OOD) utterance is crucial for a robust dialog system. Most dialog systems are trained on a pool of annotated OOD data to achieve this goal. However, collecting the annotated OOD data for a given domain is an expensive process. To mitigate this issue, previous works have proposed generative adversarial networks (GAN) based models to generate OOD data for a given domain automatically. However, these proposed models do not work directly with the text. They work with the text's latent space instead, enforcing these models to include components responsible for encoding text into latent space and decoding it back, such as auto-encoder. These components increase the model complexity, making it difficult to train. We propose OodGAN, a sequential generative adversarial network (SeqGAN) based model for OOD data generation. Our proposed model works directly on the text and hence eliminates the need to include an auto-encoder. OOD data generated using OodGAN model outperforms state-of-the-art in OOD detection metrics for ROSTD (67% relative improvement in FPR 0.95) and OSQ datasets (28% relative improvement in FPR 0.95) (Zheng et al., 2020)., Comment: NAACL 2021 Industry track

Task oriented dialog agents provide a natural language interface for users to complete their goal. Dialog State Tracking (DST), which is often a core component of these systems, tracks the system's understanding of the user's goal throughout the conversation. To enable accurate multi-domain DST, the model needs to encode dependencies between past utterances and slot semantics and understand the dialog context, including long-range cross-domain references. We introduce a novel architecture for this task to encode the conversation history and slot semantics more robustly by using attention mechanisms at multiple granularities. In particular, we use cross-attention to model relationships between the context and slots at different semantic levels and self-attention to resolve cross-domain coreferences. In addition, our proposed architecture does not rely on knowing the domain ontologies beforehand and can also be used in a zero-shot setting for new domains or unseen slot values. Our model improves the joint goal accuracy by 5% (absolute) in the full-data setting and by up to 2% (absolute) in the zero-shot setting over the present state-of-the-art on the MultiWoZ 2.1 dataset., Comment: Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI 2020)

Data availability is a bottleneck during early stages of development of new capabilities for intelligent artificial agents. We investigate the use of text generation techniques to augment the training data of a popular commercial artificial agent across categories of functionality, with the goal of faster development of new functionality. We explore a variety of encoder-decoder generative models for synthetic training data generation and propose using conditional variational auto-encoders. Our approach requires only direct optimization, works well with limited data and significantly outperforms the previous controlled text generation techniques. Further, the generated data are used as additional training samples in an extrinsic intent classification task, leading to improved performance by up to 5\% absolute f-score in low-resource cases, validating the usefulness of our approach., Comment: EMNLP WNGT workshop

MultiWOZ 2.0 (Budzianowski et al., 2018) is a recently released multi-domain dialogue dataset spanning 7 distinct domains and containing over 10,000 dialogues. Though immensely useful and one of the largest resources of its kind to-date, MultiWOZ 2.0 has a few shortcomings. Firstly, there is substantial noise in the dialogue state annotations and dialogue utterances which negatively impact the performance of state-tracking models. Secondly, follow-up work (Lee et al., 2019) has augmented the original dataset with user dialogue acts. This leads to multiple co-existent versions of the same dataset with minor modifications. In this work we tackle the aforementioned issues by introducing MultiWOZ 2.1. To fix the noisy state annotations, we use crowdsourced workers to re-annotate state and utterances based on the original utterances in the dataset. This correction process results in changes to over 32% of state annotations across 40% of the dialogue turns. In addition, we fix 146 dialogue utterances by canonicalizing slot values in the utterances to the values in the dataset ontology. To address the second problem, we combined the contributions of the follow-up works into MultiWOZ 2.1. Hence, our dataset also includes user dialogue acts as well as multiple slot descriptions per dialogue state slot. We then benchmark a number of state-of-the-art dialogue state tracking models on the MultiWOZ 2.1 dataset and show the joint state tracking performance on the corrected state annotations. We are publicly releasing MultiWOZ 2.1 to the community, hoping that this dataset resource will allow for more effective models across various dialogue subproblems to be built in the future., Comment: Data release writeup

Knowledge graph based simple question answering (KBSQA) is a major area of research within question answering. Although only dealing with simple questions, i.e., questions that can be answered through a single knowledge base (KB) fact, this task is neither simple nor close to being solved. Targeting on the two main steps, subgraph selection and fact selection, the research community has developed sophisticated approaches. However, the importance of subgraph ranking and leveraging the subject--relation dependency of a KB fact have not been sufficiently explored. Motivated by this, we present a unified framework to describe and analyze existing approaches. Using this framework as a starting point, we focus on two aspects: improving subgraph selection through a novel ranking method and leveraging the subject--relation dependency by proposing a joint scoring CNN model with a novel loss function that enforces the well-order of scores. Our methods achieve a new state of the art (85.44% in accuracy) on the SimpleQuestions dataset., Comment: Accepted by The 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019). 11 pages, 1 figure

User interaction with voice-powered agents generates large amounts of unlabeled utterances. In this paper, we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding (SLU) tasks. We use Embeddings from Language Model (ELMo) to take advantage of unlabeled data by learning contextualized word representations. Additionally, we propose ELMo-Light (ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our findings suggest unsupervised pre-training on a large corpora of unlabeled utterances leads to significantly better SLU performance compared to training from scratch and it can even outperform conventional supervised transfer. Additionally, we show that the gains from unsupervised transfer techniques can be further improved by supervised transfer. The improvements are more pronounced in low resource settings and when using only 1000 labeled in-domain samples, our techniques match the performance of training from scratch on 10-15x more labeled in-domain data., Comment: To appear at AAAI 2019

Fast expansion of natural language functionality of intelligent virtual agents is critical for achieving engaging and informative interactions. However, developing accurate models for new natural language domains is a time and data intensive process. We propose efficient deep neural network architectures that maximally re-use available resources through transfer learning. Our methods are applied for expanding the understanding capabilities of a popular commercial agent and are evaluated on hundreds of new domains, designed by internal or external developers. We demonstrate that our proposed methods significantly increase accuracy in low resource settings and enable rapid development of accurate models with less data., Comment: To appear in the Proceedings of NAACL-HLT 2018 (Industry Track)

In this paper we describe the approaches adopted to generate the runs submitted to ImageCLEFPhoto 2009 with an aim to promote document diversity in the rankings. Four of our runs are text based approaches that employ textual statistics extracted from the captions of images, i.e. MMR [1] as a state of the art method for result diversification, two approaches that combine relevance information and clustering techniques, and an instantiation of Quantum Probability Ranking Principle. The fifth run exploits visual features of the provided images to re-rank the initial results by means of Factor Analysis. The results reveal that our methods based on only text captions consistently improve the performance of the respective baselines, while the approach that combines visual features with textual statistics shows lower levels of improvements.

In this paper we describe the approaches adopted to generate the five runs submitted to ImageClefPhoto 2009 by the University of Glasgow. The aim of our methods is to exploit document diversity in the rankings. All our runs used text statistics extracted from the captions associated to each image in the collection, except one run which combines the textual statistics with visual features extracted from the provided images. The results suggest that our methods based on text captions significantly improve the performance of the respective baselines, while the approach that combines visual features with text statistics shows lower levels of improvements.

Power line communication has been emanated as one of the most enduring means of communication for smart grid applications especially while considering the biggest advantage i.e. an already established infrastructure, therefore sending out the control information over the same network will add only a little cost and hence opens the door for a plethora of applications. The communication over Power Line is not so new when we are concerned about generation, transmission or deliverance of power but here our main concern is control and management of power rather than transmission or deliverance of power and this purpose can only be accomplished if we are utilizing the available resources in an efficient manner which in turn is dependent on the fast and effective transmission of data or control information over these channels. To ensure the fulfilment of these requisites there is a requirement to analyse the basic topological connections and the circuit modelling and thus determined the various control and traffic problems associated with the transmission of this information which usually varies according to applications. Therefore OFDM (BPSK, QPSK, and QAM) has been utilized for the purpose of analysis of the channel performance while ensuring the speed and robustness of the channel to be the main criteria for any kind of services or applications .Moreover there usually arises a problem of power failure and reliable communication over remote locations and therefore the solution for it is an interfacing between wired and wireless communication technologies and hence in the thesis work, a comparison of the bit error probability had been shown between the performance of the channel while using OFDM and CDMA and this comparison provides an solution to choose the technology according to the requirement of application .

Power line communication has been emanated as one of the most enduring means of communication for smart grid applications especially while considering the biggest advantage i.e. an already established infrastructure, therefore sending out the control information over the same network will add only a little cost and hence opens the door for a plethora of applications. The communication over Power Line is not so new when we are concerned about generation, transmission or deliverance of power but here our main concern is control and management of power rather than transmission or deliverance of power and this purpose can only be accomplished if we are utilizing the available resources in an efficient manner which in turn is dependent on the fast and effective transmission of data or control information over these channels. To ensure the fulfilment of these requisites there is a requirement to analyse the basic topological connections and the circuit modelling and thus determined the various control and traffic problems associated with the transmission of this information which usually varies according to applications. Therefore OFDM (BPSK, QPSK, and QAM) has been utilized for the purpose of analysis of the channel performance while ensuring the speed and robustness of the channel to be the main criteria for any kind of services or applications .Moreover there usually arises a problem of power failure and reliable communication over remote locations and therefore the solution for it is an interfacing between wired and wireless communication technologies and hence in the thesis work, a comparison of the bit error probability had been shown between the performance of the channel while using OFDM and CDMA and this comparison provides an solution to choose the technology according to the requirement of application .