Search

International audience; Modern end-to-end (E2E) Automatic Speech Recognition (ASR) systems rely on Deep Neural Networks (DNN) that are mostly trained on handcrafted and pre-computed acoustic features such as Mel-filter-banks or Mel-frequency cepstral coefficients. Nonetheless , and despite worse performances, E2E ASR models processing raw waveforms are an active research field due to the lossless nature of the input signal. In this paper, we propose the E2E-SincNet, a novel fully E2E ASR model that goes from the raw waveform to the text transcripts by merging two recent and powerful paradigms: SincNet and the joint CTC-attention training scheme. The conducted experiments on two different speech recognition tasks show that our approach outperforms previously investigated E2E systems relying either on the raw waveform or pre-computed acoustic features, with a reported top-of-the-line Word Error Rate (WER) of 4.7% on the Wall Street Journal (WSJ) dataset.

Deep neural networks (DNNs) and more precisely recurrent neural networks (RNNs) are at the core of modern automatic speech recognition systems, due to their efficiency to process input sequences. Recently, it has been shown that different input representations, based on multidimensional algebras, such as complex and quaternion numbers, are able to bring to neural networks a more natural, compressive and powerful representation of the input signal by outperforming common real-valued NNs. Indeed, quaternion-valued neural networks (QNNs) better learn both internal dependencies, such as the relation between the Mel-filter-bank value of a specific time frame and its time derivatives, and global dependencies, describing the relations that exist between time frames. Nonetheless, QNNs are limited to quaternion-valued input signals, and it is difficult to benefit from this powerful representation with real-valued input data. This paper proposes to tackle this weakness by introducing a real-to-quaternion encoder that allows QNNs to process any one dimensional input features, such as traditional Mel-filter-banks for automatic speech recognition., Comment: Accepted at INTERSPEECH 2019

Quaternion convolutional neural networks (QCNN) are powerful architectures to learn and model external dependencies that exist between neighbor features of an input vector, and internal latent dependencies within the feature. This paper proposes to evaluate the effectiveness of the QCNN on a realistic theme identification task of spoken telephone conversations between agents and customers from the call center of the Paris transportation system (RATP). We show that QCNNs are more suitable than real-valued CNN to process multidimensional data and to code internal dependencies. Indeed, real-valued CNNs deal with both internal and external relations at the same level since components of an entity are processed independently. Experimental evidence is provided that the proposed QCNN architecture always outperforms real-valued equivalent CNN models in the theme identification task of the DECODA corpus. It is also shown that QCNN accuracy results are the best achieved so far on this task, while reducing by a factor of 4 the number of model parameters.

Recently, the connectionist temporal classification (CTC) model coupled with recurrent (RNN) or convolutional neural networks (CNN), made it easier to train speech recognition systems in an end-to-end fashion. However in real-valued models, time frame components such as mel-filter-bank energies and the cepstral coefficients obtained from them, together with their first and second order derivatives, are processed as individual elements, while a natural alternative is to process such components as composed entities. We propose to group such elements in the form of quaternions and to process these quaternions using the established quaternion algebra. Quaternion numbers and quaternion neural networks have shown their efficiency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with less learning parameters than real-valued models. This paper proposes to integrate multiple feature views in quaternion-valued convolutional neural network (QCNN), to be used for sequence-to-sequence mapping with the CTC model. Promising results are reported using simple QCNNs in phoneme recognition experiments with the TIMIT corpus. More precisely, QCNNs obtain a lower phoneme error rate (PER) with less learning parameters than a competing model based on real-valued CNNs., Accepted at INTERSPEECH 2018

International audience; Identifying and characterizing the dynamics of modern tv series subplots is an open problem. One way is to study the underlying social network of interactions between the characters. Standard dynamic network extraction methods rely on temporal integration, either over the whole considered period, or as a sequence of several time-slices. However, they turn out to be inappropriate in the case of tv series, because the scenes shown onscreen alternatively focus on parallel storylines, and do not necessarily respect a traditional chronology. In this article, we introduce Narrative Smoothing, a novel network extraction method taking advantage of the plot properties to solve some of their limitations. We apply our method to a corpus of 3 popular series, and compare it to both standard approaches. Narrative smoothing leads to more relevant observations when it comes to the characterization of the protagonists and their relationships, confirming its appropriateness to model the intertwined storylines constituting the plots.

International audience; The availability of open-source software is playing a remarkable role in the popularization of speech recognition and deep learning. Kaldi, for instance, is nowadays an established framework used to develop state-of-the-art speech recognizers. PyTorch is used to build neural networks with the Python language and has recently spawn tremendous interest within the machine learning community thanks to its simplicity and flexibility. The PyTorch-Kaldi project aims to bridge the gap between these popular toolkits, trying to inherit the efficiency of Kaldi and the flexibility of PyTorch. PyTorch-Kaldi is not only a simple interface between these software, but it embeds several useful features for developing modern speech recognizers. For instance, the code is specifically designed to naturally plug-in user-defined acoustic models. As an alternative, users can exploit several pre-implemented neural networks that can be customized using intuitive configuration files. PyTorch-Kaldi supports multiple feature and label streams as well as combinations of neural networks, enabling the use of complex neural architectures. The toolkit is publicly-released along with a rich documentation and is designed to properly work locally or on HPC clusters. Experiments, that are conducted on several datasets and tasks, show that PyTorch-Kaldi can effectively be used to develop modern state-of-the-art speech recognizers.

International audience; Online communities have gained considerable importance in recent years due to the increasing number of people connected to the Internet. Moderating user content in online communities is mainly performed manually, and reducing the workload through automatic methods is of great financial interest for community maintainers. Often, the industry uses basic approaches such as bad words filtering and regular expression matching to assist the moderators. In this article, we consider the task of automatically determining if a message is abusive. This task is complex since messages are written in a non-standardized way, including spelling errors, abbreviations, community-specific codes... First, we evaluate the system that we propose using standard features of online messages. Then, we evaluate the impact of the addition of pre-processing strategies, as well as original specific features developed for the community of an online in-browser strategy game. We finally propose to analyze the usefulness of this wide range of features using feature selection. This work can lead to two possible applications: 1) automatically flag potentially abusive messages to draw the moderator's attention on a narrow subset of messages ; and 2) fully automate the moderation process by deciding whether a message is abusive without any human intervention.

International audience; The diachronic nature of broadcast news data leads to the problem of Out-Of-Vocabulary (OOV) words in Large Vocabulary Continuous Speech Recognition (LVCSR) systems. Analysis of OOV words reveals that a majority of them are Proper Names (PNs). However PNs are important for automatic indexing of audio-video content and for obtaining reliable automatic transcriptions. In this paper, we focus on the problem of OOV PNs in diachronic audio documents. To enable recovery of the PNs missed by the LVCSR system, relevant OOV PNs are retrieved by exploiting the semantic context of the LVCSR transcriptions. For retrieval of OOV PNs, we explore topic and semantic context derived from Latent Dirichlet Allocation (LDA) topic models, continuous word vector representations and the Neural Bag-of-Words (NBOW) model which is capable of learning task specific word and context representations. We propose a Neural Bag-of-Weighted Words (NBOW2) model which learns to assign higher weights to words that are important for retrieval of an OOV PN. With experiments on French broadcast news videos we show that the NBOW and NBOW2 models outperform the methods based on raw embeddings from LDA and Skip-gram models. Combining the NBOW and NBOW2 models gives a faster convergence during training. Second pass speech recognition experiments, in which the LVCSR vocabulary and language model are updated with the retrieved OOV PNs, demonstrate the effectiveness of the proposed context models.

Most of the Natural and Spoken Language Processing tasks now employ Neural Networks (NN), allowing them to reach impressive performances. Embedding features allow the NLP systems to represent input vectors in a latent space and to improve the observed performances. In this context, Recurrent Neural Network (RNN) based architectures such as Long Short-Term Memory (LSTM) are well known for their capacity to encode sequential data into a non-sequential hidden vector representation, called sequence embedding. In this paper, we propose an LSTM-based multi-stream sequence embedding in order to encode parallel sequences by a single non-sequential latent representation vector. We then propose to map this embedding representation in a high-dimensional space using a Support Vector Machine (SVM) in order to classify the multi-stream sequences by finding out an optimal hyperplane. Multi-stream sequence embedding allowed the SVM classifier to more efficiently profit from information carried by both parallel streams and longer sequences. The system achieved the best performance, in a multi-stream sequence classification task, with a gain of 9 points in error rate compared to an SVM trained on the original input sequences.

Machine Learning (ML) techniques have allowed a great performance improvement of different challenging Spoken Language Understanding (SLU) tasks. Among these methods, Neural Networks (NN), or Multilayer Perceptron (MLP), recently received a great interest from researchers due to their representation capability of complex internal structures in a low dimensional subspace. However, MLPs employ document representations based on basic word level or topic-based features. Therefore, these basic representations reveal little in way of document statistical structure by only considering words or topics contained in the document as a “bag-of-words”, ignoring relations between them. We propose to remedy this weakness by extending the complex features based on Quaternion algebra presented in [1] to neural networks called QMLP. This original QMLP approach is based on hyper-complex algebra to take into consideration features dependencies in documents. New document features, based on the document structure itself, used as input of the QMLP, are also investigated in this paper, in comparison to those initially proposed in [1]. Experiments made on a SLU task from a real framework of human spoken dialogues showed that our QMLP approach associated with the proposed document features outperforms other approaches, with an accuracy gain of 2% with respect to the MLP based on real numbers and more than 3% with respect to the first Quaternion-based features proposed in [1]. We finally demonstrated that less iterations are needed by our QMLP architecture to be efficient and to reach promising accuracies.

Recently, machine learning methods have provided a broad spectrum of original and efficient algorithms based on Deep Neural Networks (DNN) to automatically predict an outcome with respect to a sequence of inputs. Recurrent hidden cells allow these DNN-based models to manage long-term dependencies such as Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM). Nevertheless, these RNNs process a single input stream in one (LSTM) or two (Bidirectional LSTM) directions. But most of the information available nowadays is from multistreams or multimedia documents, and require RNNs to process these information synchronously during the training. This paper presents an original LSTM-based architecture, named Parallel LSTM (PLSTM), that carries out multiple parallel synchronized input sequences in order to predict a common output. The proposed PLSTM method could be used for parallel sequence classification purposes. The PLSTM approach is evaluated on an automatic telecast genre sequences classification task and compared with different state-of-the-art architectures. Results show that the proposed PLSTM method outperforms the baseline n-gram models as well as the state-of-the-art LSTM approach., Comment: 2016 IEEE Workshop on Spoken Language Technology

International audience; Performance of spoken language understanding applications declines when spoken documents are automatically transcribed in noisy conditions due to high Word Error Rates (WER). To improve the robustness to transcription errors, recent solutions propose to map these automatic transcriptions in a latent space. These studies have proposed to compare classical topic-based representations such as Latent Dirichlet Allocation (LDA), supervised LDA and author-topic (AT) models. An original compact representation, called c-vector, has recently been introduced to walk around the tricky choice of the number of latent topics in these topic-based representations. Moreover, c-vectors allow to increase the robustness of document classification with respect to transcription errors by compacting different LDA representations of a same speech document in a reduced space and then compensate most of the noise of the document representation. The main drawback of this method is the number of sub-tasks needed to build the c-vector space. This paper proposes to both improve this compact representation (c-vector) of spoken documents and to reduce the number of needed sub-tasks, using an original framework in a robust low dimensional space of features from a set of AT models called "Latent Topic-based Sub-space" (LTS). In comparison to LDA, the AT model considers not only the dialogue content (words), but also the class related to the document. Experiments are conducted on the DECODA corpus containing speech conversations from the call-center of the RATP Paris transportation company. Results show that the original LTS representation outperforms the best previous compact representation (c-vector), with a substantial gain of more than 2.5% in terms of correctly labeled conversations.

International audience

The Neural Bag-of-Words (NBOW) model performs classification with an average of the input word vectors and achieves an impressive performance. While the NBOW model learns word vectors targeted for the classification task it does not explicitly model which words are important for given task. In this paper we propose an improved NBOW model with this ability to learn task specific word importance weights. The word importance weights are learned by introducing a new weighted sum composition of the word vectors. With experiments on standard topic and sentiment classification tasks, we show that (a) our proposed model learns meaningful word importance for a given task (b) our model gives best accuracies among the BOW approaches. We also show that the learned word importance weights are comparable to tf-idf based word weights when used as features in a BOWSVM classifier.

International audience; Proper name recognition is a challenging task in information retrieval from large audio/video databases. Proper names are semantically rich and are usually key to understanding the information contained in a document. Our work focuses on increasing the vocabulary coverage of a speech transcription system by automatically retrieving proper names from contemporary diachronic text documents. We proposed methods that dynamically augment the automatic speech recognition system vocabulary using lexical and temporal features in diachronic documents. We also studied different metrics for proper name selection in order to limit the vocabulary augmentation and therefore the impact on the ASR performances. Recognition results show a significant reduction of the proper name error rate using an augmented vocabulary.

This paper proposes a new prediction process to explain and predicts popularity evolution of YouTube videos. We exploit prior study on the classification of YouTube videos in order to predict the evolution of videos' view-count. This classification allows to identify important factors of the observed popularity dynamics. In particular, we use this classification as filtering method allowing to identify the factors responsible for this popularity evolution. Results given by extensive experiments show that the proposed prediction process is able to reduce the average prediction errors compared to a state-of-the-art baseline model. We also evaluate the impact of adding popularity criteria in the classification.

International audience; This paper presents a method for speech analytics that integrates topic-space based representation into a feed-forward artificial neural network (FFANN), working as a document classifier. The proposed method consists in configuring the FFANN's topology and in initializing the weights according to a previously estimated topic-space. Setup based on thematic priors is expected to improve the efficiency of the FFANN's weight optimization process, while speeding-up the training process and improving the classification accuracy. This method is evaluated on a spoken dialogue categorization task which is composed of customer-agent dialogues from the call-centre of Paris Public Transportation Company. Results show the interest of the proposed setup method, with a gain of more than 4 points in terms of classification accuracy, compared to the baseline. Moreover, experiments highlight that performance is weakly dependent to FFANN's topology with the LDA-based configuration, in comparison to classical empirical setup.

Presented as poster at Spatial Statistics Conference 2015, Avignon, France, June 2015; International audience; Social Networks became a major actor in information propagation. Using the Twitter popular platform, mobile users post or relaymessages from different locations. The tweet content, meaning and location show how an event-such as the bursty one“JeSuisCharlie'” happened in France in January 2015 is comprehended in different countries. This research aims at clustering thetweets according to the co-occurrence of their terms, including the country, and forecasting the probable country of a non locatedtweet, knowing its content. First, we present the process of collecting a large quantity of data from the Twitter website. Wefinally have a set of 2.189 located tweets about “Charlie'', from the 7th to the 14th of January. We describe an original methodadapted from the Author-Topic (AT) model based on the Latent Dirichlet Allocation method (LDA). We define a homogeneousspace containing both lexical content (words) and spatial information (country). During a training process on a part of the sample,we provide a set of clusters (topics) based on statistical relations between lexical and spatial terms. During a clustering task, weevaluate the method effectiveness on the rest of the sample that reaches up to 95% of good assignment.

International audience; This paper describes a multi-modal person recognition system for video broadcastdeveloped for participating to the REPERE challenge, that was organized jointly by the DGA and the ANR (French Research National Agency). The main track of this challenge targets the identification of all persons occurring in a video either. The main scientific issue addressed by this challenge is the combination of audio and video information extraction processes for improving the extraction performance in both modalities. In this paper, we present a strategy for speaker identification based on enriching the speaker diarization by features related to the ”understanding” of the video scenes: text overlay transcription and analysis, automatic situation identification (TV set, report), the amount of people visible, TV set disposition and even the camera when available. Experiments on the REPERE corpus show interest of the proposed approach.; Cet article présente un système d’identification de personnes dans des flux multimédia.Ce système a été engagé dans le défi REPERE, co-organisé par l’ANR et la DGA et qui s’est terminé en 2014. La tâche principale du défi consistait à identifier des individus apparaissant dans au moins une des modalités portées par la vidéo, qu’il s’agisse de locuteurs audibles ou de visages visibles à l’écran. Un des verrous scientifiques majeurs de cette tâche est lié à la combinaison des modalités audio et vidéo. Cet article présente une stratégie pour la reconnaissance des personnes basée sur une identification du locuteur reposant sur des descripteurs dehaut niveau, modélisant différents aspects de la scène filmée : la transcription et l’analyse des textes incrustés, l’identification du type de la scène filmée (reportage, plateau, ...), le nombre de personnes présentes, la disposition des caméras... Nos expériences sur le corpus REPERE montrent l’intérêt de l’approche proposée.

International audience; Performance of Automatic Speech Recognition (ASR) systems drops dramatically when transcribing conversations recorded in noisy conditions. Speech analytics suffer from this poor automatic transcription quality. To tackle this difficulty , a solution consists in mapping transcriptions into a space of hidden topics. This abstract representation allows to substantiate the drawbacks of the ASR process. The well-known and commonly used one is the topic-based representation from a Latent Dirichlet Allocation (LDA). Several studies demonstrate the effectiveness and reliability of this high-level representation. During the LDA learning process, distribution of words into each topic is estimated automatically. Nonetheless, in the context of a classification task, no consideration is made for the targeted classes. Thus, if the targeted application is to find out the main theme related to a dialogue, this information should be taken into consideration. In this paper, we propose to compare a classical topic-based representation of a dialogue, with a new one based not only on the dialogue content itself (words), but also on the theme related to the dialogue. This original representation is based on the author-topic (AT) model. The effectiveness of the proposed representation is evaluated on a classification task from automatic dialogue transcriptions between an agent and a customer of the Paris Transportation Company. Experiments confirmed that this author-topic model approach outperforms by far the classical topic representation, with a substantial gain of more than 7% in terms of correctly labeled conversations.

International audience; Social networks become a major actor in massive information propagation. In the context of the Twitter platform, its popularity is due in part to the capability of relaying messages (i.e. tweets) posted by users. This particular mechanism, called retweet, allows users to massively share tweets they consider as potentially interesting for others. In this paper, we propose to study the behavior of tweets that have been massively retweeted in a short period of time. We first analyze specific tweet features through a Principal Component Analysis (PCA) to better understand the behavior of highly forwarded tweets as opposed to those retweeted only a few times. Finally, we propose to automatically detect the massively retweeted messages. The qualitative study is used to select the features allowing the best classification performance. We show that the selection of only the most correlated features, leads to the best classification accuracy (F-measure of 65.7%), with a gain of about 2.4 points in comparison to the use of the complete set of features.

International audience; Various studies highlighted that topic-based approaches give a powerful spoken content representation of documents. Nonetheless, these documents may contain more than one main theme, and their automatic transcription inevitably contains errors. In this study, we propose an original and promising framework based on a compact representation of a textual document , to solve issues related to topic space granularity. Firstly, various topic spaces are estimated with different numbers of classes from a Latent Dirichlet Allocation. Then, this multiple topic space representation is compacted into an elementary segment , called c-vector, originally developed in the context of speaker recognition. Experiments are conducted on the DECODA corpus of conversations. Results show the effectiveness of the proposed multi-view compact representation paradigm. Our identification system reaches an accuracy of 85%, with a significant gain of 9 points compared to the baseline (best single topic space configuration).

International audience; The main objective of this paper is to identify themes from dialogues of telephone conversations in a real-life customer care service. In order to capture significant semantic content in spite of high expression variability, features are extracted in a large number of hidden spaces constructed with a Latent Dirichlet Allocation (LDA) approach. Multiple views of a spoke document can then be represented with several hidden topic models. Nonetheless, the model diversity due to the multi-model approach introduces a new type of variability. An approach is proposed based on features extracted in a common homogenous subspace with the purpose of reducing the multi-span representation variability. A Gaussian Mixture Model subspace model, inspired by previous work on speaker identification, is proposed for theme identification. This representation, novel for theme classification, is compared with the direct application of multiple topic-model representations. Experiments are reported using a corpus collected in the call center of the Paris Transportation Service. Results show the effectiveness of the proposed representation paradigm with a theme identification accuracy of 78.8%, showing a significant improvement with respect to previous results on the same corpus.

The main objective of this paper is to identify themes from dialogues of telephone conversations in a real-life customer care service. In this task, the word semantic variability contained in these conversations may impact the classification performance by retaining the noise in their vectorial representation. In this article, we propose an original method to compensate this semantic variability using the Factor Analysis (FA) paradigm, initially designed for speech processing tasks to compensate the acoustic variability, mainly in Speaker Verification (SV) and Automatic Speech Recognition (ASR). In our proposal, we used the FA paradigm to estimate the semantic variability as an additive component located in a subspace of low dimension (with respect to the super-vector space). This additive semantic variability is estimated in Factor Analysis model space. From this estimation, a specific vector transformation is obtained and is applied to vectors of dialogue representation. Experiments are reported using a corpus collected in the call center of the Paris Transportation Service. Results show the effectiveness of the proposed representation paradigm with a theme identification accuracy of 80.0%, showing a significant improvement with respect to previous results on the same corpus. Index Terms: Human/Human conversation representation, Semantic variability, Factor analysis, Variability compensation, Automatic classification, Latent Dirichlet Allocation.

International audience; In this paper, we study the impact of dialogue representations and classification methods in the task of theme identification of telephone conversation services having highly imperfect automatic transcriptions. Two dialogue representations are firstly compared: the classical Term Frequency-Inverse Document Frequency with Gini purity criteria (TF-IDF-Gini) method and the Latent Dirichlet Allocation (LDA) approach. We then propose to study an original classification method that takes advantage of the LDA topic space representation , highlighted as the best dialogue representation. To do so, two assumptions about topic representation led us to choose a Gaussian process (GP) based method. This approach is compared with a Support Vector Machine (SVM) classification method. Results show that the GP approach is a better solution to deal with the multiple theme complexity of a dialogue, no matter the conditions studied (manual or automatic transcriptions). We finally discuss the impact of the topic space reduction on the classification accuracy.

This paper describes a multi-modal person recognition system for video broadcast developed for participating in the DefiRepere challenge. The main track of this challenge targets the identification of all persons occurring in a video either in the audio modality (speakers) or the image modality (faces). This system is developed by the PERCOL team involving 4 research labs in France and was ranked first at the 2014 Defi-Repere challenge. The main scientific issue addressed by this challenge is the combination of audio and video information extraction processes for improving the extraction performance in both modalities. In this paper, we present the strategy followed by the PERCOL team for speaker identification based on enriching the speaker diarization with features related to the ”understanding” of the video scenes: text overlay transcription and analysis, automatic situation identification (TV set, report), the amount of people visible, TV set disposition and even the camera when available. Experiments on the REPERE corpus show interesting results on the speaker identification system enriched by the scene understanding features and the usefulness of the speaker to identify faces.

International audience; The performance of Automatic Speech Recognition (ASR) systems drops dramatically when used in noisy environments. Speech analytics suffer from this poor quality of automatic transcriptions. In this paper, we seek to identify themes from dialogues of telephone conversation services using multiple topic-spaces estimated with a Latent Dirichlet Allocation (LDA) approach. This technique consists in estimating several topic models that offer different views of the document. Unfortunately, such a multi-model approach also introduces additional vari-abilities due to the model diversity. We propose to extract the useful information from the full model-set by using an i-vector based approach, previously developed in the context of speaker recognition. Experiments are conducted on the DECODA corpus , that contains records from the call center of the Paris Transportation Company. Results show the effectiveness of the proposed representation paradigm, our identification system reaching an accuracy of 84.7%, with a gain of 3.3 points compared to the baseline.

In this article, we are interested in spoken term detection task, with a particular focus on Person Name (PN) spotting in automatic speech recognition (ASR) system outputs. We propose a two-step method that combines an acoustic matching based on a Phoneme Confusion Network (PCN) with a semantic rescoring based on the Latent Dirichlet Allocation (LDA) models. The first module allows to find, in the PCN, potential PN candidates in speech segments, while the second is in charge of ranking the competing PN, according to a LDA topic model. The proposed LDA-based approach outperforms significantly the baseline system based on a search in the ASR phoneme lattice, obtaining a F-measure score of 77.04% on PN detection.

International audience; The paper introduces new features for describing possible focus variation in a human/human conversation. The application considered is a real-life telephone customer care service. The purpose is to hypothesize the dominant theme of conversations between a casual customer calling. Conversations are processed by an automatic speech recognition system that provides hypotheses used for extracting word frequency. Features are extracted in different, broadly defined and partially overlapped, time segments. Combinations of each feature in different segments are represented in a quaternion algebra framework. The advantage of the proposed approach is made evident by the statistically significant improvements in theme classification accuracy .

International audience; This article presents a method for automatic tagging of Youtube videos. The proposed method combines an automatic speech recognition (ASR) system, that extracts the spoken contents, and a keyword extraction component that aims at finding a small set of tags representing a video. In order to improve the robustness of the tagging system to the recognition errors, a video transcription is represented in a topic space obtained by a Latent Dirichlet Allocation (LDA), in which each dimension is automatically characterized by a list of weighted terms. Tags are extracted by combining the weighted word list of the best LDA classes. We evaluate this method by employing the user-provided tags of Youtube videos as reference and we investigate the impact of the topic model granularity. The obtained results demonstrate the interest of such model to improve the robustness of the tagging system.

International audience; —We present a method to detect social events in a set of pictures from an image hosting service (Flickr). This method relies on the analysis of user-generated tags, by using statistical models trained on both a small set of manually annotated data and a large data set collected from the Internet. Social event modeling relies on multi-span topic model based on LDA (Latent Dirichlet Allocation). Experiments are conducted in the experimental setup of MediaEval'2011 evaluation campaign. The proposed system outperforms significantly the best system of this benchmark, reaching a F-measure score of about 71%.

International audience; The detection and characterization, in audiovisual documents, of speech utterances where person names are pronounced, is an important cue for spoken content analysis. This paper tackles the problematic of retrieving spoken person names in the 1-Best ASR outputs of broadcast TV shows. Our assumption is that a person name is a latent variable produced by the lexical context it appears in. Thereby, a spoken name could be derived from ASR outputs even if it has not been proposed by the speech recognition system. A new context modelling is proposed in order to capture lexical and structural information surrounding a spoken name. The fundamental hypothesis of this study has been validated on broadcast TV documents available in the context of the REPERE challenge.

International audience; Combining automatic speech recognition (ASR) systems generally relies on the posterior merging of the outputs or on acoustic cross-adaptation. In this paper, we propose an integrated approach where outputs of secondary systems are integrated in the search algorithm of a primary one. In this driven decoding algorithm (DDA), the secondary systems are viewed as observation sources that should be evaluated and combined to others by a primary search algorithm. DDA is evaluated on a subset of the ESTER I corpus consisting of 4 hours of French radio broadcast news. Results demonstrate DDA significantly outperforms vote-based approaches: we obtain an improvement of 14.5% relative word error rate over the best single-systems, as opposed to the the 6.7% with a ROVER combination. An in-depth analysis of the DDA shows its ability to improve robustness (gains are greater in adverse conditions) and a relatively low dependency on the search algorithm. The application of DDA to both A* and beam-search-based decoder yields similar performances.

This paper presents a confidence measure for speech indexing that aims to predict the indexing quality of a speech document for a Spoken Document Retrieval (SDR) task. We first introduce how the indexing quality of a speech document is evaluated. Then, we present our method to predict the indexing quality of a speech document. It is based on confidence measure provided by an automatic speech recognition system and the detection of semantic outliers implemented with the Latent Dirichlet Allocation (LDA) model. Experiments are conducted on the French Broadcast news campaign ESTER2 in a classical SDR scenario where users submit text-queries to a search engine. Results demonstrate an overall improvement when the detection is done with the LDA model. The detection rate is always above 70%. Index Terms: speech indexing, confidence measure, spoken document retrieval, latent dirichlet allocation

International audience; A research on the analysis of human/human conversations in a call centre is described. The purpose of the research is to provide short reports of each conversation with information useful for monitoring the call centre efficiency. Data from real users discussing over the telephone with agents are processed by an automatic speech recognition (ASR) system. Reports are grouped into classes by the agents based on predefined taxonomy. A train set of manually transcribed data is used for training the extraction of features relevant to the application and the classification of the conversations. The use of all the words of the application vocabulary, of automatically selected key_words, and of automatically learned sentence chunks containing semantic classes of words are compared and evaluated with a totally different test set. The results show a significant increase in performance when chunks are used even in comparison with the use of bags of words obtained with a boosting algorithm.

—This paper focuses on automatic speech recognition systems combination based on driven decoding paradigms. The driven decoding algorithm (DDA) involves the use of a 1-best hypothesis provided by an auxiliary system as another knowledge source in the search algorithm of a primary system. In previous studies, it was shown that DDA outperforms ROVER when the primary system is guided by a more accurate system. In this paper we propose a new method to manage auxiliary transcriptions which are presented as a bag-of-n-grams (BONG) without temporal matching. These modifications allow to make easier the combination of several hypotheses given by different auxiliary systems. Using BONG combination with hypotheses provided by two auxiliary systems, each of which obtained more than 23% of WER on the same data, our experiments show that a CMU Sphinx based ASR system can reduce its WER from 19.85% to 18.66% which is better than the results reached with DDA or classical ROVER combination.

International audience; This paper presents a semantic confidence measure that aims to predict the relevance of automatic transcripts for a task of Spoken Document Retrieval (SDR). The proposed predicting method relies on the combination of Automatic Speech Recognition (ASR) confidence measure and a Semantic Com-pacity Index (SCI), that estimates the relevance of the words considering the semantic context in which they occurred. Experiments are conducted on the French Broadcast news corpus ESTER, by simulating a classical SDR usage scenario : users submit text-queries to a search engine that is expected to return the most relevant documents regarding the query. Results demonstrate the interest of using semantic level information to predict the transcription indexability.

In a previous paper we proposed Web-based language models relying on the possibility theory. These models explicitly represent the possibility of word sequences. In this paper we propose to find the best way of combining this kind of model with classical probabilistic models, in the context of automatic speech recognition. We propose several combination approaches, depending on the nature of the combined models. With respect to the baseline, the best combination provides an absolute word error rate reduction of about 1% on broadcast news transcription , and of 3.5% on domain-specific multimedia document transcription. Index Terms: language models, world wide web, possibility measure, automatic speech recognition

Statistical classifiers operate on features that generally include both useful and useless information. These two types of information are difficult to separate in the feature domain. Recently, a new paradigm based on a Latent Factor Analysis (LFA) proposed a model decomposition into usefull and useless components. This method was successfully applied to speaker and language recognition tasks. In this paper, we study the use of LFA for video genre classification by using only the audio channel. We propose a classification method based on short-term cep-stral features and Gaussian Mixture Models (GMM) or Support Vector Machine (SVM) classifiers, that are combined with Factor Analysis (FA). Experiments are conducted on a corpus composed of 5 types of video (musics, commercials, cartoons, movies and news). The relative classification error reduction obtained by using the best factor analysis configuration with respect to the baseline system, Gaussian Mixture Model Universal Background Model (GMM-UBM), is about 56%, corresponding to a correct identification rate of about 90%.

Usually, language models are built either from a closed corpus, or by using World Wide Web retrieved documents, which are considered as a closed corpus themselves. In this paper we propose several other ways, more adapted to the nature of the Web, of using this resource for language modeling. We first start by improving an approach consisting in estimating n-gram probabilities from Web search engine statistics. Then, we propose a new way of considering the information extracted from the Web in a probabilistic framework. Then, we also propose to rely on Possibility Theory for effectively using this kind of information. We compare these two approaches on two automatic speech recognition tasks: (i) transcribing broadcast news data, and (ii) transcribing domain-specific data, concerning surgical operation film comments. We show that the two approaches are effective in different situations. Index Terms: language modeling, World Wide Web, possibility measure, automatic speech recognition

International audience; no abstract

International audience; With the purpose of improving Spoken Language Un- derstanding (SLU) performance, a combination of different acoustic speech recognition (ASR) systems is proposed. State a posteriori probabilities obtained with systems using different acoustic feature sets are combined with log-linear inter- polation. In order to perform a coherent combination of these probabilities, acoustic models must have the same topology (i.e. same set of states). For this purpose, a fast and efficient twin model training protocol is proposed. By a wise choice of acoustic feature sets and log-linear interpolation of their like- lihood ratios, a substantial Concept Error Rate (CER) reduction has been observed on the test part of the French MEDIA corpus.

International audience; Most of the Web-based methods for lexicon augmenting consist in capturing global semantic features of the targeted domain in order to collect relevant documents from the Web. We suggest that the local context of the out-of-vocabulary (OOV) words contains relevant information on the OOV words. With this information, we propose to use the Web to build locally-augmented lexicons which are used in a final local decoding pass. Our experiments confirm the relevance of the Web for the OOV word retrieval. Different methods are proposed to retrieve the hypothesis words. Finally we present the integration of new words in the transcription process based on part-of-speech models. This technique allows to recover 7.6% of the significant OOV words and the accuracy of the system is improved.

International audience; Driven Decoding Algorithm (DDA) is initially an integrated approach for the combination of 2 speech recognition (ASR) systems. It consists in guiding the search algorithm of a primary ASR system by the one-best hypothesis of an auxiliary system. In this paper , we generalize DDA to confusion-network driven decoding and we propose new combination schemes for multiple system combination. Since previous experiments involved 2 ASR systems on broadcast news data, the proposed extended DDA is evaluated using 3 ASR systems from different labs. Results show that generalized-DDA outperforms significantly ROVER method: we obtain a 15.7% relative word error rate improvement with respect to the best single system, as opposed to 8.5% with the ROVER combination.

International audience; Automatic transcript aligned and corrected This competition is arbitrated by a matching score Wi. Experimental context :-First experiments assessed on 3 hours of radio ESTER (with exact transcript and a 10% WER transcripts)-Second experiments assessed on 11 hours of RTBF on wich time stamps where manually added.-All words available in database are added to the language model-Language model : about 67000 words trained on « lemonde »-Speech recognition system : SPEERAL, an asynchronous decoder based on the A* algorithm. Results : Conclusions :-On ESTER tests approximative transcripts bring a WER gain of about 14% relative, while exact ones allows a WER gain close to 24% relative.-Spotting performance is good; more than 95.3% of segments have been found, with a precision of about 96.7%.-On RTBF tests, spotting performance is good; more than 95.3% of segments have been found, with a precision of about 96.7%. Fast-match to transcript island-The principle of the proposed method is close to approaches used in the field of information retrieval.-In our case, the hypothesis is a query which may be answered by one of the transcript island.-The lexicon is represented by a lexical space Ls where each dimension is associated to a word. The coefficients of these vectors represent the frequencies of words in the document.-As the current hypothesis is developed, a set of word clusters Ci is built and updated.-These clusters result from the intersection of hc and the transcript island Ii.-For each new word added to the hypothesis hc, transcript islands are considered as candidates for guiding the search.

In the recent years, deep learning has become the leading approach to modern artificial intelligence (AI). The important improvement in terms of processing time required for learning AI based models alongside with the growing amount of available data made of deep neural networks (DNN) the strongest solution to solve complex real-world problems. However, a major challenge of artificial neural architectures lies on better considering the high-dimensionality of the data.To alleviate this issue, neural networks (NN) based on complex and hypercomplex algebras have been developped. The natural multidimensionality of the data is elegantly embedded within complex and hypercomplex neurons composing the model. In particular, quaternion neural networks (QNN) have been proposed to deal with up to four dimensional features, based on the quaternion representation of rotations and orientations. Unfortunately, and conversely to complex-valued neural networks that are nowadays known as a strong alternative to real-valued neural networks, QNNs suffer from numerous limitations that are carrefuly addressed in the different parts detailled in this thesis.The thesis consists in three parts that gradually introduce the missing concepts of QNNs, to make them a strong alternative to real-valued NNs. The first part introduces and list previous findings on quaternion numbers and quaternion neural networks to define the context and strong basics for building elaborated QNNs.The second part introduces state-of-the-art quaternion neural networks for a fair comparison with real-valued neural architectures. More precisely, QNNs were limited by their simple architectures that were mostly composed of a single and shallow hidden layer. In this part, we propose to bridge the gap between quaternion and real-valued models by presenting different quaternion architectures. First, basic paradigms such as autoencoders and deep fully-connected neural networks are introduced. Then, more elaborated convolutional and recurrent neural networks are extended to the quaternion domain. Experiments to compare QNNs over equivalents NNs have been conducted on real-world tasks across various domains, including computer vision, spoken language understanding and speech recognition. QNNs increase performances while reducing the needed number of neural parameters compared to real-valued neural networks.Then, QNNs are extended to unconventional settings. In a conventional QNN scenario, input features are manually segmented into three or four components, enabling further quaternion processing. Unfortunately, there is no evidence that such manual segmentation is the representation that suits the most to solve the considered task. Morevover, a manual segmentation drastically reduces the field of application of QNNs to four dimensional use-cases. Therefore the third part introduces a supervised and an unsupervised model to extract meaningful and disantengled quaternion input features, from any real-valued input signal, enabling the use of QNNs regardless of the dimensionality of the considered task. Conducted experiments on speech recognition and document classification show that the proposed approaches outperform traditional quaternion features., Au cours des dernières années, l’apprentissage profond est devenu l’approche privilégiée pour le développement d’une intelligence artificielle moderne (IA). L’augmentation importante de la puissance de calcul, ainsi que la quantité sans cesse croissante de données disponibles ont fait des réseaux de neurones profonds la solution la plus performante pour la resolution de problèmes complexes. Cependant, la capacité à parfaitement représenter la multidimensionalité des données réelles reste un défi majeur pour les architectures neuronales artificielles.Pour résoudre ce problème, les réseaux de neurones basés sur les algèbres des nombres complexes et hypercomplexes ont été développés. En particulier, les réseaux de neurones de quaternions (QNN) ont été proposés pour traiter les données tridi- mensionnelles et quadridimensionnelles, sur la base des quaternions représentant des rotations dans notre espace tridimensionnel. Malheureusement, et contrairement aux réseaux de neurones à valeurs complexes qui sont de nos jours acceptés comme une alternative aux réseaux de neurones réels, les QNNs souffrent de nombreuses lacunes qui sont en partie comblées par les différents travaux détaillés par ce manuscrit.Ainsi, la thèse se compose de trois parties qui introduisent progressivement les concepts manquants, afin de faire des QNNs une alternative aux réseaux neuronaux à valeurs réelles. La premiere partie présente et répertorie les précédentes découvertes relatives aux quaternions et aux réseaux de neurones de quaternions, afin de définir une base pour la construction des QNNs modernes.La deuxième partie introduit des réseaux neuronaux de quaternions état de l’art, afin de permettre une comparaison dans des contextes identiques avec les architectures modernes traditionnelles. Plus précisément, les QNNs étaient majoritairement limités par leurs architectures trop simples, souvent composées d’une seule couche cachée comportant peu de neurones. Premièrement, les paradigmes fondamentaux, tels que les autoencodeurs et les réseaux de neurones profonds sont présentés. Ensuite, les très répandus et étudiés réseaux de neurones convolutionnels et récurrents sont étendus à l’espace des quaternions. De nombreuses experiences sur différentes applications réelles, telles que la vision par ordinateur, la compréhension du langage parlé ainsi que la reconnaissance automatique de la parole sont menées pour comparer les modèles de quaternions introduits aux réseaux neuronaux conventionnels. Dans ces contextes bien spécifiques, les QNNs ont obtenus de meilleures performances ainsi qu’une réduction importante du nombre de paramètres neuronaux nécessaires à la phase d’apprentissage.Les QNNs sont ensuite étendus à des conditions d’entrainement permettant de traiter toutes les représentations en entrée des modèles de quaternions. Dans un scénario traditionnel impliquant des QNNs, les caractéristiques d’entrée sont manuellement segmentées en quatre composants, afin de correspondre à la representation induite par les quaternions. Malheureusement, il est difficile d’assurer qu’une telle segmentation est optimale pour résoudre le problème considéré. De plus, une segmentation manuelle réduit fondamentalement l’application des QNNs à des tâches naturellement définies dans un espace à au plus quatre dimensions. De ce fait, la troisième partie de cette thèse introduit un modèle supervisé et un modèle non supervisé permettant l’extraction de caractéristiques d’entrée désentrelacées et significatives dans l’espace des quaternions, à partir de n’importe quel type de signal réel uni-dimentionnel, permettant l’utilisation des QNNs indépendamment de la dimensionnalité des vecteurs d’entrée et de la tâche considérée. Les expériences menées sur la reconnaissance de la parole et la classification de documents parlés montrent que les approches proposées sont plus performantes que les représentations traditionnelles de quaternions.

In the same way as TV channels, data streams are represented as a sequence of successive events that can exhibit chronological relations (e.g. a series of programs, scenes, etc.). For a targeted channel, broadcast programming follows the rules defined by the channel itself, but can also be affected by the programming of competing ones. In such conditions, event sequences of parallel streams could provide additional knowledge about the events of a particular stream. In the sphere of machine learning, various methods that are suited for processing sequential data have been proposed. Long Short-Term Memory (LSTM) Recurrent Neural Networks have proven its worth in many applications dealing with this type of data. Nevertheless, these approaches are designed to handle only a single input sequence at a time. The main contribution of this thesis is about developing approaches that jointly process sequential data derived from multiple parallel streams. The application task of our work, carried out in collaboration with the computer science laboratory of Avignon (LIA) and the EDD company, seeks to predict the genre of a telecast. This prediction can be based on the histories of previous telecast genres in the same channel but also on those belonging to other parallel channels. We propose a telecast genre taxonomy adapted to such automatic processes as well as a dataset containing the parallel history sequences of 4 French TV channels. Two original methods are proposed in this work in order to take into account parallel stream sequences. The first one, namely the Parallel LSTM (PLSTM) architecture, is an extension of the LSTM model. PLSTM simultaneously processes each sequence in a separate recurrent layer and sums the outputs of each of these layers to produce the final output. The second approach, called MSE-SVM, takes advantage of both LSTM and Support Vector Machines (SVM) methods. Firstly, latent feature vectors are independently generated for each input stream, using the output event of the main one. These new representations are then merged and fed to an SVM algorithm. The PLSTM and MSE-SVM approaches proved their ability to integrate parallel sequences by outperforming, respectively, the LSTM and SVM models that only take into account the sequences of the main stream. The two proposed approaches take profit of the information contained in long sequences. However, they have difficulties to deal with short ones. Though MSE-SVM generally outperforms the PLSTM approach, the problem experienced with short sequences is more pronounced for MSE-SVM. Finally, we propose to extend this approach by feeding additional information related to each event in the input sequences (e.g. the weekday of a telecast). This extension, named AMSE-SVM, has a remarkably better behavior with short sequences without affecting the performance when processing long ones., Les flux de contenus audiovisuels peuvent être représentés sous forme de séquences d’événements (par exemple, des suites d’émissions, de scènes, etc.). Ces données séquentielles se caractérisent par des relations chronologiques pouvant exister entre les événements successifs. Dans le contexte d’une chaîne TV, la programmation des émissions suit une cohérence définie par cette même chaîne, mais peut également être influencée par les programmations des chaînes concurrentes. Dans de telles conditions,les séquences d’événements des flux parallèles pourraient ainsi fournir des connaissances supplémentaires sur les événements d’un flux considéré.La modélisation de séquences est un sujet classique qui a été largement étudié, notamment dans le domaine de l’apprentissage automatique. Les réseaux de neurones récurrents de type Long Short-Term Memory (LSTM) ont notamment fait leur preuve dans de nombreuses applications incluant le traitement de ce type de données. Néanmoins,ces approches sont conçues pour traiter uniquement une seule séquence d’entrée à la fois. Notre contribution dans le cadre de cette thèse consiste à élaborer des approches capables d’intégrer conjointement des données séquentielles provenant de plusieurs flux parallèles.Le contexte applicatif de ce travail de thèse, réalisé en collaboration avec le Laboratoire Informatique d’Avignon et l’entreprise EDD, consiste en une tâche de prédiction du genre d’une émission télévisée. Cette prédiction peut s’appuyer sur les historiques de genres des émissions précédentes de la même chaîne mais également sur les historiques appartenant à des chaînes parallèles. Nous proposons une taxonomie de genres adaptée à de tels traitements automatiques ainsi qu’un corpus de données contenant les historiques parallèles pour 4 chaînes françaises.Deux méthodes originales sont proposées dans ce manuscrit, permettant d’intégrer les séquences des flux parallèles. La première, à savoir, l’architecture des LSTM parallèles(PLSTM) consiste en une extension du modèle LSTM. Les PLSTM traitent simultanément chaque séquence dans une couche récurrente indépendante et somment les sorties de chacune de ces couches pour produire la sortie finale. Pour ce qui est de la seconde proposition, dénommée MSE-SVM, elle permet de tirer profit des avantages des méthodes LSTM et SVM. D’abord, des vecteurs de caractéristiques latentes sont générés indépendamment, pour chaque flux en entrée, en prenant en sortie l’événement à prédire dans le flux principal. Ces nouvelles représentations sont ensuite fusionnées et données en entrée à un algorithme SVM. Les approches PLSTM et MSE-SVM ont prouvé leur efficacité dans l’intégration des séquences parallèles en surpassant respectivement les modèles LSTM et SVM prenant uniquement en compte les séquences du flux principal. Les deux approches proposées parviennent bien à tirer profit des informations contenues dans les longues séquences. En revanche, elles ont des difficultés à traiter des séquences courtes.L’approche MSE-SVM atteint globalement de meilleures performances que celles obtenues par l’approche PLSTM. Cependant, le problème rencontré avec les séquences courtes est plus prononcé pour le cas de l’approche MSE-SVM. Nous proposons enfin d’étendre cette approche en permettant d’intégrer des informations supplémentaires sur les événements des séquences en entrée (par exemple, le jour de la semaine des émissions de l’historique). Cette extension, dénommée AMSE-SVM améliore remarquablement la performance pour les séquences courtes sans les baisser lorsque des séquences longues sont présentées.

The diachronic nature of broadcast news causes frequent variations in the linguisticcontent and vocabulary, leading to the problem of Out-Of-Vocabulary (OOV)words in automatic speech recognition. Most of the OOV words are found tobe proper names whereas proper names are important for automatic indexingof audio-video content as well as for obtaining reliable automatic transcriptions.New proper names missed by the speech recognition system can be recovered by adynamic vocabulary multi-pass recognition approach in which new proper namesare added to the speech recognition vocabulary based on the context of the spokencontent. Existing methods for vocabulary selection rely on web search engines andadaptation corpora and choose the new vocabulary words using term-documentfrequency and co-occurrence based features. Open vocabulary systems based onsub-word units are an interesting solution but they face the problem of producinga reliable text transcription. The goal of this thesis is to model the semantic andtopical context of new proper names in order to retrieve those which are relevantto the spoken content in the audio document. Training semantic/topic models is achallenging problem in this task because (a) several new proper names come witha low amount of data and (b) the context model should be robust to word errorsin the automatic transcription. Probabilistic topic models and word embeddingsfrom neural network models are explored for the task of retrieval of relevantproper names. A thorough evaluation of contextual representations from thesemodels is performed. It is argued that these representations, which are learnedin an unsupervised manner, are not the best for the given retrieval task. Neuralnetwork context models trained with an objective to maximise the retrieval performanceare proposed. A Neural Bag-of-Words (NBOW) model trained to learncontext vector representations at a document level is shown to outperform thegeneric representations. The proposed Neural Bag-of-Weighted-Words (NBOW2)model learns to assign a degree of importance to input words and has the abilityto capture task specic key-words. Experiments on automatic speech recognitionon French broadcast news videos demonstrate the eectiveness of the proposedmodels. Further evaluation of the NBOW2 model on standard text classicationtasks, including movie review sentiment classication and newsgroup topic classication, shows that it learns interesting information about the task and givesthe best classication accuracies among the bag-of-words models.; La nature diachronique des bulletins d'information provoque de fortes variationsdu contenu linguistique et du vocabulaire, conduisant au problème demots hors vocabulaire (Out-Of-Vocabulary OOV) dans le cadre de la reconnaissanceautomatique de la parole. La plupart des mots OOV se trouvent êtredes noms propres; ces noms propres sont importants pour l'indexation automatiquede ces contenus audio-vidéo, ainsi que pour obtenir des transcriptions fiables. Le but de cette these est de retrouver les noms propres manquants enutilisant le contexte an de les intégrer dans le lexique du système de reconnaissancepour effectuer une deuxièeme passe de reconnaissance. Les méthodesexistantes pour la sélection de nouveaux mots a ajouter au vocabulaire utilisentdes documents textuels récuperés sur le internet et choisissent les nouveaux motsde vocabulaire en utilisant des matrices terme-document ou des co-occurrencesde mots. Dans cette these nous allons modéliser le contexte sémantique etutiliser des informations thématiques contenus dans le document audio a transcrirean de récupérer les noms propres pertinents. Des modèles probabilistesde thème (topic model) et des projections dans un espace continu obtenues al'aide de réseaux de neurones (word embeddings) sont explorés pour la tâchede récuperation des noms propres pertinents. Une évaluation approfondie deces représentations contextuelles a été réalisée. Le contexte est estimé a l'aidede réseaux de neurones appris avec l'objectif de maximiser la récuperation desnoms propres pertinents. Un modèle de neurones (Neural Bag-of-Words, NBOW)entra^ne pour apprendre des représentations contextuelles au niveau du documentobtient de meilleures performances que les modèles classiques. Le nouveaumodèle que nous proposons (Bag-of-Weighted-Words, NBOW2) estime un degréd'importance pour chacun des mots du document et a la capacité de capturerdes mots importants spéciques. Des expériences de reconnaissance automatiquede bulletins d'information télévisés démontrent l'efficacité des modèles proposes.Une évaluation plus poussée du modèle NBOW2 sur les tâches de classicationde texte, d'analyse de critique de lm et de classication thématique de texteissus de groupe de discussion, démontre la capacité d'apprentissage du modèle etsurpasse les modèles utilisant des sac-de-mots (Bag-Of-Words, BOW).

The diachronic nature of broadcast news causes frequent variations in the linguisticcontent and vocabulary, leading to the problem of Out-Of-Vocabulary (OOV)words in automatic speech recognition. Most of the OOV words are found tobe proper names whereas proper names are important for automatic indexingof audio-video content as well as for obtaining reliable automatic transcriptions.New proper names missed by the speech recognition system can be recovered by adynamic vocabulary multi-pass recognition approach in which new proper namesare added to the speech recognition vocabulary based on the context of the spokencontent. Existing methods for vocabulary selection rely on web search engines andadaptation corpora and choose the new vocabulary words using term-documentfrequency and co-occurrence based features. Open vocabulary systems based onsub-word units are an interesting solution but they face the problem of producinga reliable text transcription. The goal of this thesis is to model the semantic andtopical context of new proper names in order to retrieve those which are relevantto the spoken content in the audio document. Training semantic/topic models is achallenging problem in this task because (a) several new proper names come witha low amount of data and (b) the context model should be robust to word errorsin the automatic transcription. Probabilistic topic models and word embeddingsfrom neural network models are explored for the task of retrieval of relevantproper names. A thorough evaluation of contextual representations from thesemodels is performed. It is argued that these representations, which are learnedin an unsupervised manner, are not the best for the given retrieval task. Neuralnetwork context models trained with an objective to maximise the retrieval performanceare proposed. A Neural Bag-of-Words (NBOW) model trained to learncontext vector representations at a document level is shown to outperform thegeneric representations. The proposed Neural Bag-of-Weighted-Words (NBOW2)model learns to assign a degree of importance to input words and has the abilityto capture task specic key-words. Experiments on automatic speech recognitionon French broadcast news videos demonstrate the eectiveness of the proposedmodels. Further evaluation of the NBOW2 model on standard text classicationtasks, including movie review sentiment classication and newsgroup topic classication, shows that it learns interesting information about the task and givesthe best classication accuracies among the bag-of-words models.; La nature diachronique des bulletins d'information provoque de fortes variationsdu contenu linguistique et du vocabulaire, conduisant au problème demots hors vocabulaire (Out-Of-Vocabulary OOV) dans le cadre de la reconnaissanceautomatique de la parole. La plupart des mots OOV se trouvent êtredes noms propres; ces noms propres sont importants pour l'indexation automatiquede ces contenus audio-vidéo, ainsi que pour obtenir des transcriptions fiables. Le but de cette these est de retrouver les noms propres manquants enutilisant le contexte an de les intégrer dans le lexique du système de reconnaissancepour effectuer une deuxièeme passe de reconnaissance. Les méthodesexistantes pour la sélection de nouveaux mots a ajouter au vocabulaire utilisentdes documents textuels récuperés sur le internet et choisissent les nouveaux motsde vocabulaire en utilisant des matrices terme-document ou des co-occurrencesde mots. Dans cette these nous allons modéliser le contexte sémantique etutiliser des informations thématiques contenus dans le document audio a transcrirean de récupérer les noms propres pertinents. Des modèles probabilistesde thème (topic model) et des projections dans un espace continu obtenues al'aide de réseaux de neurones (word embeddings) sont explorés pour la tâchede récuperation des noms propres pertinents. Une évaluation approfondie deces représentations contextuelles a été réalisée. Le contexte est estimé a l'aidede réseaux de neurones appris avec l'objectif de maximiser la récuperation desnoms propres pertinents. Un modèle de neurones (Neural Bag-of-Words, NBOW)entra^ne pour apprendre des représentations contextuelles au niveau du documentobtient de meilleures performances que les modèles classiques. Le nouveaumodèle que nous proposons (Bag-of-Weighted-Words, NBOW2) estime un degréd'importance pour chacun des mots du document et a la capacité de capturerdes mots importants spéciques. Des expériences de reconnaissance automatiquede bulletins d'information télévisés démontrent l'efficacité des modèles proposes.Une évaluation plus poussée du modèle NBOW2 sur les tâches de classicationde texte, d'analyse de critique de lm et de classication thématique de texteissus de groupe de discussion, démontre la capacité d'apprentissage du modèle etsurpasse les modèles utilisant des sac-de-mots (Bag-Of-Words, BOW).

These past ten years, TV series became increasingly popular. In contrast to classical TV series consisting of narratively self-sufficient episodes, modern TV series develop continuous plots over dozens of successive episodes. However, the narrative continuity of modern TV series directly conflicts with the usual viewing conditions: due to modern viewing technologies, the new seasons of TV series are being watched over short periods of time. As a result, viewers are largely disengaged from the plot, both cognitively and emotionally, when about to watch new seasons. Such a situation provides video summarization with remarkably realistic use-case scenarios, that we detail in Chapter 1. Furthermore, automatic movie summarization, long restricted to trailer generation based on low-level features, finds with TV series an unprecedented opportunity to address in well-defined conditions the so-called "semantic gap": summarization of narrative media requires content-oriented approaches capable to bridge the gap between low-level features and human understanding. We review in Chapter 2 the two main approaches adopted so far to address automatic movie summarization. Chapter 3 is dedicated to the various subtasks needed to build the intermediary representations on which our summarization framework relies: Section 3.1 focuses on video segmentation, whereas the rest of Chapter 3 is dedicated to the extraction of different mid-level features, either saliency-oriented (shot size, background music), or content-related (speakers). In Chapter 4, we make use of Social Network Analysis as a possible way to model the plot of modern TV series: the narrative dynamics can be properly captured by the evolution over time of the social network of interacting characters. Nonetheless, we have to address here the sequential nature of the narrative when taking instantaneous views of the state of the relationships between the characters. We show that standard time-windowing approaches can not properly handle this case, and we detail our own method for extracting dynamic social networks from narrative media. Chapter 5 is dedicated to the final generation and evaluation of character-oriented summaries, both able to reflect the plot dynamics and to emotionally re-engage viewers into the narrative. We evaluate our framework by performing a large-scale user study in realistic conditions.; Ces dix dernières années, les séries télévisées sont devenues de plus en plus populaires. Par opposition aux séries TV classiques composées d'épisodes auto-suffisants d'un point de vue narratif, les séries TV modernes développent des intrigues continues sur des dizaines d'épisodes successifs. Cependant, la continuité narrative des séries TV modernes entre directement en conflit avec les conditions usuelles de visionnage : en raison des technologies modernes de visionnage, les nouvelles saisons des séries TV sont regardées sur de courtes périodes de temps. Par conséquent, les spectateurs sur le point de visionner de nouvelles saisons sont largement désengagés de l'intrigue, à la fois d'un point de vue cognitif et affectif. Une telle situation fournit au résumé de vidéos des scénarios d'utilisation remarquablement réalistes, que nous détaillons dans le Chapitre 1. De plus, le résumé automatique de films, longtemps limité à la génération de bande-annonces à partir de descripteurs de bas niveau, trouve dans les séries TV une occasion inédite d'aborder dans des conditions bien définies ce qu'on appelle le "fossé sémantique" : le résumé de médias narratifs exige des approches orientées contenu, capables de jeter un pont entre des descripteurs de bas niveau et le niveau humain de compréhension. Nous passons en revue dans le Chapitre 2 les deux principales approches adoptées jusqu'ici pour aborder le problème du résumé automatique de films de fiction. Le Chapitre 3 est consacré aux différentes sous-tâches requises pour construire les représentations intermédiaires sur lesquelles repose notre système de génération de résumés : la Section 3.1 se concentre sur la segmentation de vidéos, tandis que le reste du chapitre est consacré à l'extraction de descripteurs de niveau intermédiaire, soit orientés saillance (échelle des plans, musique de fond), soit en relation avec le contenu (locuteurs). Dans le Chapitre 4, nous utilisons l'analyse des réseaux sociaux comme une manière possible de modéliser l'intrigue des séries TV modernes : la dynamique narrative peut être adéquatement capturée par l'évolution dans le temps du réseau des personnages en interaction. Cependant, nous devons faire face ici au caractère séquentiel de la narration lorsque nous prenons des vues instantanées de l'état des relations entre personnages. Nous montrons que les approches classiques par fenêtrage temporel ne peuvent pas traiter convenablement ce cas, et nous détaillons notre propre méthode pour extraire des réseaux sociaux dynamiques dans les médias narratifs. Le Chapitre 5 est consacré à la génération finale de résumés orientés personnages, capables à la fois de refléter la dynamique de l'intrigue et de ré-engager émotionnellement les spectateurs dans la narration. Nous évaluons notre système en menant à une large échelle et dans des conditions réalistes une enquête auprès d'utilisateurs.