Search

The popularity of platforms for storing and transmitting video content has created a substantial volume of video data. Given a set of actors present in a video, generating metadata with the temporal determination of the interval in which each actor is present and their spatial 2D localization in each frame in these intervals can facilitate video retrieval and recommendation. In this work, we investigate Video Face Clustering for this spatio-temporal localization of actors in videos. We first describe our method for Video Face Clustering in which we take advantage of face detection, embeddings, and clustering methods to group similar faces of actors in different frames and provide the spatio-temporal localization of them. Then, we explore, propose, and investigate innovative applications of this spatio-temporal localization in three different tasks: (i) Video Face Recognition, (ii) Educational Video Recommendation and (iii) Subtitles Positioning in 360-video.

Recently the Brazilian DTV system standards have been upgraded, called TV 2.5, in order to provide a better integration between broadcast and broadband services. The next Brazilian DTV system evolution, called TV 3.0, will address more deeply this convergence of TV systems not only at low-level network layers but also at the application layer. One of the new features to be addressed by this future application layer is the use of Artificial Intelligence technologies. Recently, there have been practical applications using Artificial Intelligence (AI) deployed to improve TV production efficiency and correlated cost reduction. The success in operationalize and evaluate these applications is a strong indication of the interest and relevance of AI in TV. This paper presents TeleMídia Lab’s future vision on interactive and intelligent TV Systems, with particular focus on edge AI. Edge AI means use in-device capabilities to run AI applications instead of running them in cloud.

A learning object (LO) is an entity, digital or not, that can be used and reused or referenced during a technological support process for teaching and learning. Despite mainly being multimedia, with audio, video, text and images synchronized with each other, LOs can help disseminate knowledge even only in educational texts. However, creating these texts can be costly in time and effort, creating the need to seek new ways to generate this content. This article presents a solution for the generation of text-based LOs generated through summaries supported by Deep Learning models. The present work was evaluated in a supervised experiment in which volunteers rate computer educational texts generated by three types of summarizers. The results presented are positive and allow us to compare the performance of summaries as LO generators in text format. The findings also suggest that using post-processing in the output of models can improve the readability of generated content.

Ad hoc teamwork is a research topic in multi-agent systems whereby an agent (the "ad hoc agent") must successfully collaborate with a set of unknown agents (the "teammates") without any prior coordination or communication protocol. However, research in ad hoc teamwork is predominantly focused on agent-only teams, but not in agent-human teams, which we believe is an exciting research avenue and has enormous application potential in human-robot teams. This paper will tap into this potential by proposing HOTSPOT, the first framework for ad hoc teamwork in human-robot teams. Our framework comprises two main modules, addressing the two key challenges in the interaction between a robot acting as the ad hoc agent and human teammates. First, a decision-theoretic module that is responsible for all task-related decision making (task identification, teammate identification, and planning). Second, a communication module that uses natural language processing in order to parse all communication between the robot and the human. To evaluate our framework, we use a task where a mobile robot and a human cooperatively collect objects in an open space, illustrating the main features of our framework in a real-world task.

Audio quality degradation can have many causes. For musical applications, this fragmentation may lead to highly unpleasant experiences. Restoration algorithms may be employed to reconstruct missing parts of the audio in a similar way as for image reconstruction --- in an approach called audio inpainting. Current state-of-the art methods for audio inpainting cover limited scenarios, with well-defined gap windows and little variety of musical genres. In this work, we propose a Deep-Learning-based (DL-based) method for audio inpainting accompanied by a dataset with random fragmentation conditions that approximate real impairment situations. The dataset was collected using tracks from different music genres to provide a good signal variability. Our best model improved the quality of all musical genres, obtaining an average of 12.9 dB of PSNR, although it worked better for musical genres in which acoustic instruments are predominant.

A crucial task to overall video understanding is the recognition and localisation in time of different actions or events that are present along the scenes. To address this problem, action segmentation must be achieved. Action segmentation consists of temporally segmenting a video by labeling each frame with a specific action. In this work, we propose a novel action segmentation method that requires no prior video analysis and no annotated data. Our method involves extracting spatio-temporal features from videos in samples of 0.5s using a pre-trained deep network. Data is then transformed using a positional encoder and finally a clustering algorithm is applied with the use of a silhouette score to find the optimal number of clusters where each cluster presumably corresponds to a different single and distinguishable action. In experiments, we show that our method produces competitive results on Breakfast and Inria Instructional Videos dataset benchmarks.

NCL is the declarative programming language used to develop TV applications in IPTV systems and Terrestrial TV standardized by ITU and Brazilian TV Forum, respectively. Its main characteristics are: defining temporal synchronization among media assets and viewer interactions; layout reuse facilities; support multi-device presentation; support embed HTML code and scripts in the lightweight scripting language Lua; and an API for life-cycle controls (start, pause, resume, stop) and modifying applications on-the-fly called NCL editing command. This talk briefly introduces NCL, highlights its recent advances, and discuss the future of the language.

Discovering and accessing specific content within educational video bases is a challenging task, mainly because of the abundance of video content and its diversity. Recommender systems are often used to enhance the ability to find and select content. But, recommendation mechanisms, especially those based on textual information, exhibit some limitations, such as being error-prone to manually created keywords or due to imprecise speech recognition. This paper presents a method for generating educational video recommendations using deep face-features of lecturers without identifying them. More precisely, we use an unsupervised face clustering mechanism to create relations among the videos based on the lecturer's presence. Then, for a selected educational video taken as a reference, we recommend the ones where the presence of the same lecturers is detected. Moreover, we rank these recommended videos based on the amount of time the referenced lecturers were present. For this task, we achieved an mAP value of 99.165%.

Instant messaging apps have billions of monthly active users, which made this environment a good medium for purposes other than communication, such as education. In this context, one of the most appropriate technologies to encourage learning in these applications is chatbots. They consist of conversational applications that make use of artificial intelligence and natural language processing to interact with the user in a similar way to a human. In this scenario, this article presents the AGATE a chatbot developed for the Environmental Management Office (AGA / UEMA) in order to disseminate knowledge on environmental education, focusing on the problems of water and energy waste. The chatbot was developed for Telegram through the pyTelegramBotAPI implementation. In addition, conversation flows were created using DialogFlow. A multidisciplinary team composed of 14 people participated in the evaluation of the project, with a greater participation of professionals in the field of Biology. The initial results focused on the user experience were satisfactory.

People with visual impairments suffer from the incapacity to understand contextual information in videos, such as the place where characters are, or any other non-spoken actions in general. Some content creators address this issue by providing a secondary audio to describe such information, called Audio Descriptions (ADs). How- ever, some works in the literature have highlighted that people with visual impairment are usually not able to completely understand scene changes based only on characters’ voices or traditional ADs. Moreover, traditional ADs do not completely describe some of the important visual information, such as the background scenery (e.g. colors, furniture) and characters’ details (e.g. blond woman using a red dress). In this work, we propose incrementing the traditional AD techniques with the usage of interactive video features present in TV systems. More precisely, the proposed interactivity enables users to access specialized AD for different visual information (e.g., scene, scenario, character). To support the development of such interactive content, we present an application template, which helps to create the final interactive-enhanced video application. Asa proof of concept for our approach, we created an interactive AD for an independent video mainly composed of visual information, with only a few talks.