Search

Whether on TV or on the internet, video content production is seeing an unprecedented rise. Not only is video the dominant medium for entertainment purposes, but it is also reckoned to be the future of education, information and leisure. Nevertheless, the traditional paradigm for multimedia management proves to be incapable of keeping pace with the scale brought about by the sheer volume of content created every day across the disparate distribution channels. Thus, routine tasks like archiving, editing, content organization and retrieval by multimedia creators become prohibitively costly. On the user side, too, the amount of multimedia content pumped daily can be simply overwhelming; the need for shorter and more personalized content has never been more pronounced. To advance the state of the art on both fronts, a certain level of multimedia understanding has to be achieved by our computers. In this research thesis, we aim to go about the multiple challenges facing automatic media content processing and analysis, mainly gearing our exploration to three axes: 1. Representing multimedia: With all its richness and variety, modeling and representing multimedia content can be a challenge in itself. 2. Describing multimedia: The textual component of multimedia can be capitalized on to generate high-level descriptors, or annotations, for the content at hand. 3. Summarizing multimedia: we investigate the possibility of extracting highlights from media content, both for narrative-focused summarization and for maximising memorability., Que ce soit à la télévision ou sur internet, la production de contenu vidéo connaît un essor sans précédent. La vidéo est devenu non seulement le support dominant pour le divertissement, mais elle est également considérée comme l'avenir de l'éducation, l'information et le loisir. Néanmoins, le paradigme traditionnel de la gestion du multimédia s'avère incapable de suivre le rythme imposé par l'ampleur du volume de contenu créé chaque jour sur les différents canaux de distribution. Ainsi, les tâches de routine telles que l'archivage, l'édition, l'organisation et la recherche de contenu par les créateurs multimédias deviennent d'un coût prohibitif. Du côté de l'utilisateur, la quantité de contenu multimédia distribuée quotidiennement peut être écrasante ; le besoin d'un contenu plus court et plus personnalisé n'a jamais été aussi prononcé. Pour faire progresser l'état de l'art sur ces deux fronts, un certain niveau de compréhension du multimédia doit être atteint par nos ordinateurs. Dans cette thèse, nous proposons d'aborder les multiples défis auxquels sont confrontés le traitement et l'analyse automatique de contenu multimédia, en orientant notre exploration autour de trois axes : 1. la représentation: avec toute sa richesse et sa variété, la modélisation et la représentation du contenu multimédia peut être un défi en soi. 2. la description: La composante textuelle du multimédia peut être exploitée pour générer des descripteurs de haut niveau (annotation) pour le contenu en question. 3. le résumé: où nous étudions la possibilité d'extraire les moments d'intérêt de ce contenu, à la fois pour un résumé centré sur la narration et pour maximiser la mémorabilité.

Les approches traditionnelles de modélisation de sujets (Topic Modeling) s'appuient généralement sur des statistiques de cooccurrence entre termes et documents pour trouver des sujets latents dans une collection de documents. Cependant, le fait de s'appuyer uniquement sur ces statistiques peut donner des résultats incohérents ou difficiles à interpréter pour les utilisateurs finaux dans de nombreuses applications où l'intérêt réside dans l'interprétation des sujets résultants (e.g. l'étiquetage de documents, la comparaison de corpus, orienter l'exploration du contenu..). Nous proposons de tirer parti des connaissances externes de sens commun, c'est-à-dire des informations du monde réel audelà de la cooccurrence des mots, pour trouver des topics plus cohérents et plus facilement interprétables par les humains. Nous présentons le "Common Sense Topic Model" (CSTM), une approche nouvelle et efficace qui augmente le clustering avec des connaissances extraites du graphe de connaissances ConceptNet. Nous évaluons cette approche sur plusieurs jeux de données en comparaison avec des modèles couramment utilisés, en utilisant une évaluation automatique et humaine, et nous montrons comment elle montre une corrélation supérieure au jugement humain. Cet article a été déjà publié à K-CAP 2021[4].

openaire: EC/H2020/780069/EU//MeMAD

Pre-trained word embeddings constitute an essential building block for many NLP systems and applications, notably when labeled data is scarce. However, since they compress word meanings into a fixed-dimensional representation, their use usually lack interpretability beyond a measure of similarity and linear analogies that do not always reflect real-world word relatedness, which can be important for many NLP applications. In this paper, we propose a model which extracts topics from text documents based on the common-sense knowledge available in ConceptNet [Speer et al., 2017] - a semantic concept graph that explicitly encodes real-world relations between words - and without any human supervision. When combining both ConceptNet’s knowledge graph and graph embeddings, our approach outperforms other baselines in the zero-shot setting, while generating a human-understandable explanation for its predictions through the knowledge graph. We study the importance of some modeling choices and criteria for designing the model, and we demonstrate that it can be used to label data for a supervised classifier to achieve an even better performance without relying on any humanly-annotated training data. We publish the code of our approach at https://github.com/D2KLab/ZeSTE and we provide a user friendly demo at https://zeste.tools.eurecom.fr/., OASIcs, Vol. 93, 3rd Conference on Language, Data and Knowledge (LDK 2021), pages 17:1-17:15

Text segmentation is a traditional task in NLP where a document is broken down into smaller, coherent segments. While several methods and benchmarks exist for well-formed, and clean textual documents that can be found in long articles or synthetic datasets, segmenting media content comes with different challenges such as the errors produced by the procedure of Automatic Speech Recognitionand the lack of sentence end markers that are found in written text (e.g. punctuation marks or HTML tags). Many radio or TV programs are also conversational in nature (e.g. interview and debate), and thus, rely less on repeated words unlike encyclopedia text (e.g. Wikipedia articles) that are frequently used for textual segmentation evaluation. This is even further compounded when working with non-English content. In this work, we present an approach to content segmentation that leverages topical coherence, language modeling and word embeddings to detect change of topics. We evaluate our approach on a real production dataset of French programs that have been manually annotated for segments. We also show how, when a ground truth summary of the content is provided such as segment titles, we can align them to their corresponding segments using the same representations.&nbsp

openaire: EC/H2020/780069/EU//MeMAD

This deliverable primarily describes the MeMAD ontology which builds on top of the EBU Core data model. A number of additional classes and properties are defined in order to cope with the original set of metadata delivered by the MeMAD data providers (INA and Yle). Furthermore, this deliverable describes two tools that enable to convert the legacy metadata coming from both INA and Yle into RDF, the W3C standard for representing knowledge graph on the web, following the MeMAD ontology. During this process, metadata is harmonized and enriched semantically enabling to perform queries across sources. This deliverable describes also a number of tools that perform named entity recognition and disambiguation on both automatic transcription and true subtitles of TV programs. These tools existed prior to the beginning of MeMAD but they have been further improved and developed during the first year of the project (e.g. development of API, specific training to MeMAD audiovisual material). The converter tools are developed in open source and are publicly available from the MeMAD GitHub account. A scientific publication is already appended to this report which describes a novel ensemble methods to extract and disambiguate entities from different kinds of text including timed text (subtitles of TV programs) and which has been investigated during the first 6 months of the project. Several improvements and formal evaluations of those tools are already foreseen to happen during the second year of the project.