Search

Recognition errors are common in human communication. Similar errors often lead to unwanted behaviour in dialogue systems or virtual assistants. In human communication, we can recover from them by repeating misrecognized words or phrases; however in human-machine communication this recovery mechanism is not available. In this paper, we attempt to bridge this gap and present a system that allows a user to correct speech recognition errors in a virtual assistant by repeating misunderstood words. When a user repeats part of the phrase the system rewrites the original query to incorporate the correction. This rewrite allows the virtual assistant to understand the original query successfully. We present an end-to-end 2-step attention pointer network that can generate the the rewritten query by merging together the incorrectly understood utterance with the correction follow-up. We evaluate the model on data collected for this task and compare the proposed model to a rule-based baseline and a standard pointer network. We show that rewriting the original query is an effective way to handle repetition-based recovery and that the proposed model outperforms the rule based baseline, reducing Word Error Rate by 19% relative at 2% False Alarm Rate on annotated data., Will be published in Interspeech 2021

When using computer-aided translation systems in a typical, professional translation workflow, there are several stages at which there is room for improvement. The SCATE (Smart Computer-Aided Translation Environment) project investigated several of these aspects, both from a human-computer interaction point of view, as well as from a purely technological side. This paper describes the SCATE research with respect to improved fuzzy matching, parallel treebanks, the integration of translation memories with machine translation, quality estimation, terminology extraction from comparable texts, the use of speech recognition in the translation process, and human computer interaction and interface design for the professional translation environment. For each of these topics, we describe the experiments we performed and the conclusions drawn, providing an overview of the highlights of the entire SCATE project. The research in this project was funded by the Flemish Agency for Innovation and Technology IWT, project number 13007.

Copyright © 2016 ISCA. Recent work has suggested that prominence perception could be driven by the predictability of the acoustic prosodic features of speech. On the other hand, lexical predictability and part of speech information are also known to correlate with prominence. In this paper, we investigate how the bottom-up acoustic and top-down lexical cues contribute to sentence prominence by using both types of features in unsupervised and supervised systems for automatic prominence detection. The study is conducted using a corpus of Dutch continuous speech with manually annotated prominence labels. Our results show that unpredictability of speech patterns is a consistent and important cue for prominence at both the lexical and acoustic levels, and also that lexical predictability and part-of-speech information can be used as efficient features in supervised prominence classifiers. Kakouros S., Pelemans J., Verwimp L., Wambacq P., Räsänen O., ''Analyzing the contribution of top-down lexical and bottom-up acoustic cues in the detection of sentence prominence'', 17th annual conference of the International Speech Communication Association (ISCA) - Interspeech 2016, pp. 1074-1078, September 8-12, 2016, San Francisco, California, USA. ispartof: pages:1074-1078 ispartof: Proceedings Interspeech 2016 vol:08-12-September-2016 pages:1074-1078 ispartof: Annual conference of the International Speech Communication Association (ISCA) - Interspeech 2016 location:San Francisco, California, USA date:8 Sep - 12 Sep 2016 status: published

We present Sparse Non-negative Matrix (SNM) estimation, a novel probability estimation technique for language modeling that can efficiently incorporate arbitrary features. We evaluate SNM language models on two corpora: the One Billion Word Benchmark and a subset of the LDC English Gigaword corpus. Results show that SNM language models trained with n-gram features are a close match for the well-established Kneser-Ney models. The addition of skip-gram features yields a model that is in the same league as the state-of-the-art recurrent neural network language models, as well as complementary: combining the two modeling techniques yields the best known result on the One Billion Word Benchmark. On the Gigaword corpus further improvements are observed using features that cross sentence boundaries. The computational advantages of SNM estimation over both maximum entropy and neural network estimation are probably its main strength, promising an approach that has large flexibility in combining arbitrary features and yet scales gracefully to large amounts of data.

Copyright © 2015 ISCA. We present a novel family of language model (LM) estimation techniques named Sparse Non-negative Matrix (SNM) estimation. A first set of experiments empirically evaluating these techniques on the One BillionWord Benchmark [3] shows that with skip-gram features SNMLMs are able to match the state-of-theart recurrent neural network (RNN) LMs; combining the two modeling techniques yields the best known result on the benchmark. The computational advantages of SNM over both maximum entropy and RNNLM estimation are probably its main strength, promising an approach that has the same flexibility in combining arbitrary features effectively and yet should scale to very large amounts of data as gracefully as n-gram LMs do. Shazeer N., Pelemans J., Chelba C., ''Sparse non-negative matrix language modeling For skip-grams'', Proceedings 16th annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015, pp. 1428-1432, September 6-10, 2015, Dresden, Germany. ispartof: pages:1428-1432 ispartof: Proceedings Interspeech 2015 vol:2015-January pages:1428-1432 ispartof: Annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015 location:Dresden, Germany date:6 Sep - 10 Sep 2015 status: published

Copyright © 2015 ISCA. Direct integration of translation model (TM) probabilities into a language model (LM) with the purpose of improving automatic speech recognition (ASR) of spoken translations typically requires a number of complex operations for each sentence. Many if not all of the LM probabilities need to be updated, the model needs to be renormalized and the ASR system needs to load a new, updated LM for each sentence. In computer-aided translation environments the time loss induced by these complex operations seriously reduces the potential of ASR as an efficient input method. In this paper we present a novel LM adaptation technique that drastically reduces the complexity of each of these operations. The technique consists of LM probability updates using exponential weights based on TM probabilities for each sentence and does not enforce probability renormalization. Instead of storing each resulting language model in its entirety, we only store the update weights which also reduces disk storage and loading time during ASR. Experiments on Dutch read speech translated from English show that both disk storage and recognition time drop dramatically compared to a baseline system that employs a more conventional way of updating the LM. Pelemans J., Vanallemeersch T., Demuynck K., Van hamme H., Wambacq P., ''Efficient language model adaptation for automatic speech recognition of spoken translations'', Proceedings 16th annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015, pp. 2262-2266, September 6-10, 2015, Dresden, Germany. ispartof: pages:2262-2266 ispartof: Proceedings Interspeech 2015 vol:2015-January pages:2262-2266 ispartof: Annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015 location:Dresden, Germany date:6 Sep - 10 Sep 2015 status: published

Copyright © 2015 ISCA. In this paper we present a pruning algorithm and experimental results for our recently proposed Sparse Non-negative Matrix (SNM) family of language models (LMs). We show that when trained with only n-gram features SNMLM pruning based on a mutual information criterion yields the best known pruned model on the One Billion Word Language Model Benchmark, reducing perplexity with 18% and 57% over Katz and Kneser- Ney LMs, respectively. We also present a method for converting an SNMLM to ARPA back-off format which can be readily used in a single-pass decoder for Automatic Speech Recognition. Pelemans J., Shazeer N., Chelba C., ''Pruning sparse non-negative matrix N-gram language models'', Proceedings 16th annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015, pp. 1433-1437, September 6-10, 2015, Dresden, Germany. ispartof: pages:1433-1437 ispartof: Proceedings Interspeech 2015 vol:2015-January pages:1433-1437 ispartof: Annual conference of the International Speech Communication Association (ISCA) - Interspeech 2015 location:Dresden, Germany date:6 Sep - 10 Sep 2015 status: published

© 2015 IEEE. Compounding is one of the most productive word formation processes in many languages and is therefore a main source of data sparsity in language modeling. Many solutions have been suggested to model compound words, most of which break the compound into its constituents and train a new model with them. In earlier work, we argued that this approach is suboptimal and we presented a novel technique that clusters new, domain-specific compound words together with their semantic heads. The clusters were then used to build a class-based n-gram model that enabled a reliable estimation of n-gram probabilities, without the need for additional training data. In this paper, we investigate how this 'semantic head mapping' can best be made an integral part of the language modeling strategy and find that, with some adaptations, our technique is capable of producing more accurate compound probability estimates than a baseline word-based n-gram language model, which lead to a significant word error rate reduction for Dutch read speech. Pelemans J., Demuynck K., Van hamme H., Wambacq P., ''Improving N-gram probability estimates by compound-head clustering'', 40th IEEE international conference on acoustics, speech, and signal processing - ICASSP’2015, pp. 5221-5225, April 19-24, 2015, Brisbane, Australia. ispartof: pages:5221-5225 ispartof: Proceedings ICASSP 2015 vol:2015-August pages:5221-5225 ispartof: IEEE international conference on acoustics, speech, and signal processing - ICASSP’2015 location:Brisbane, Australia date:19 Apr - 24 Apr 2015 status: published

Recurrent neural network language models benefit from the integration of meta-information.Meta-information of various types, word-level, discourse-level, and intrinsic, is valuable.If meta-information can be robustly predicted, it has potential to improve performance.Intrinsic meta-information, word- and sentence-length, is trivial to obtain, but still useful.Approach is validated with the Spoken Dutch Corpus and the Wall Street Journal data set. Due to their advantages over conventional n-gram language models, recurrent neural network language models (rnnlms) recently have attracted a fair amount of research attention in the speech recognition community. In this paper, we explore one advantage of rnnlms, namely, the ease with which they allow the integration of additional knowledge sources. We concentrate on features that provide complementary information w.r.t. the lexical identities of the words. We refer to such information as meta-information. We single out three cases and investigate their merits by means of N-best list re-scoring experiments on a challenging corpus of spoken Dutch (referred to as cgn) as well as on the English Wall Street Journal (wsj) corpus. First, we look at Parts of Speech (POS) tags and lemmas, two sources of word-level linguistic information that are known to make a contribution to the performance of conventional language models. We confirm that rnnlms can benefit from these sources as well. Second, we investigate socio-situational settings (ssss) and topics, two sources of discourse-level information that are also known to benefit language models. ssss are present in the cgn data, and can be seen as a proxy for the language register. For the purposes of our investigation, we assume that information on the sss can be captured at the moment at which speech is recorded. Topics, i.e., treatments of different subjects, are present in the wsj data. In order to predict POS, lemmas, sss and topic, a second rnnlm is coupled to the main rnnlm. We refer to this architecture as a recurrent neural network tandem language model (rnntlm). Our experimental findings show that if high-quality meta-information labels are available, both word-level and discourse-level information improve performance of language models. Third, we investigate sentence length and word length (i.e., token size), two sources of intrinsic information that are readily available for exploitation because they are known at the time of re-scoring. Intrinsic information has been largely overlooked by language modeling research. The results of both experiments on cgn data and wsj data show that integrating sentence length and word length can achieve improvement. rnnlms allow these features to be incorporated with ease, and obtain improved performance.

Copyright © 2014 ISCA. Reading skills of children can be improved with the help of automatic reading tutors (ART), i.e. interactive software with an appealing interface which supports and challenges the child in the reading task, provides instantaneous feedback and automatically assesses its reading skills. For this purpose, ARTs benefit from automatic speech recognition technology for tracking the child's responses and detecting reading miscues (errors). In previous work, a novel speech recognition architecture has been proposed which adopts a two-layered structure: first a phone recognizer uses task-independent acoustic and language models to generate a phone lattice which is then decoded using a lexicon of expected words and task-dependent finite state grammars. This approach has shown significant improvements in reading miscue detection. In this paper, we extend this technique by employing a more flexible decoding scheme that allows substitution, deletion and insertion of phones. Specifically, the phone lattice generated in the first layer is extended based on a phone confusion matrix that models the typical phone confusions in a language. The proposed system has provided improved miscue detection on the CHOREC database compared to a baseline system without a phone confusion model. Yılmaz E., Pelemans J., Van hamme H., ''Automatic assessment of children's reading with the FLaVoR decoding using a phone confusion model'', Proceedings 15th annual conference of the International Speech Communication Association (ISCA) - Interspeech 2014, pp. 969-972, September 14-18, 2014, Singapore. ispartof: pages:969-972 ispartof: Proceedings Interspeech 2014 pages:969-972 ispartof: Annual conference of the International Speech Communication Association (ISCA) - Interspeech 2014 location:Singapore date:14 Sep - 18 Sep 2014 status: published

In this paper we present a novel clustering technique for compound words. By mapping compounds onto their semantic heads, the technique is able to estimate n-gram probabilities for unseen compounds. We argue that compounds are well represented by their heads which allows the clustering of rare words and reduces the risk of over-generalization. The semantic heads are obtained by a two-step process which consists of constituent generation and best head selection based on corpus statistics. Experiments on Dutch read speech show that our technique is capable of correctly identifying compounds and their semantic heads with a precision of 80.25% and a recall of 85.97%. A class-based language model with compound-head clusters achieves a significant reduction in both perplexity and WER. © 2014 IEEE. Pelemans J., Demuynck K., Van hamme H., Wambacq P., ''Coping with language data sparsity: semantic head mapping for compound words'', Proceedings 39th IEEE international conference on acoustics, speech, and signal processing - ICASSP’2014, pp. 141-145, May 4-9, 2014, Florence, Italy. ispartof: pages:141-145 ispartof: Proceedings ICASSP 2014 pages:141-145 ispartof: IEEE international conference on acoustics, speech, and signal processing - ICASSP’2014 location:Florence, Italy date:4 May - 9 May 2014 status: published