Search

Audio quality degradation can have many causes. For musical applications, this fragmentation may lead to highly unpleasant experiences. Restoration algorithms may be employed to reconstruct missing parts of the audio in a similar way as for image reconstruction --- in an approach called audio inpainting. Current state-of-the art methods for audio inpainting cover limited scenarios, with well-defined gap windows and little variety of musical genres. In this work, we propose a Deep-Learning-based (DL-based) method for audio inpainting accompanied by a dataset with random fragmentation conditions that approximate real impairment situations. The dataset was collected using tracks from different music genres to provide a good signal variability. Our best model improved the quality of all musical genres, obtaining an average of 12.9 dB of PSNR, although it worked better for musical genres in which acoustic instruments are predominant.

A crucial task to overall video understanding is the recognition and localisation in time of different actions or events that are present along the scenes. To address this problem, action segmentation must be achieved. Action segmentation consists of temporally segmenting a video by labeling each frame with a specific action. In this work, we propose a novel action segmentation method that requires no prior video analysis and no annotated data. Our method involves extracting spatio-temporal features from videos in samples of 0.5s using a pre-trained deep network. Data is then transformed using a positional encoder and finally a clustering algorithm is applied with the use of a silhouette score to find the optimal number of clusters where each cluster presumably corresponds to a different single and distinguishable action. In experiments, we show that our method produces competitive results on Breakfast and Inria Instructional Videos dataset benchmarks.

Face recognition systems are present in many modern solutions and thousands of applications in our daily lives. However, current solutions are not easily scalable, especially when it comes to the addition of new targeted people. We propose a cluster-matching-based approach for face recognition in video. In our approach, we use unsupervised learning to cluster the faces present in both the dataset and targeted videos selected for face recognition. Moreover, we design a cluster matching heuristic to associate clusters in both sets that is also capable of identifying when a face belongs to a non-registered person. Our method has achieved a recall of 99.435% and a precision of 99.131% in the task of video face recognition. Besides performing face recognition, it can also be used to determine the video segments where each person is present., Comment: 13 pages

Conversational user interfaces have been an increasingly popular way to obtain quick information in natural language, especially with advances in the area of natural language processing (NLP) fueled by the surge of deep neural networks. Real-world cases include IBM Watson and Azure Microsoft chatbots, among others. Such examples, however, are most present in the commercial spectrum, with few being actively used by the scientific community. In this work, we develop an Interactive Question Answering System (IQA) for the Sucupira platform, Brazil's biggest open platform of postgraduate content, deployed as conversational interfaces for two popular messaging platforms: Telegram and Discord. We also propose the use of multilingual embeddings for ordering answers by a language-independent measuer of semantic similarity. Our IQA was evaluated with an study with 16 participants. The results were mostly positive, indicating that the use of conversational user interfaces, along with NLP techniques, may be a valid alternative for retrieving scientific information from the Sucupira plataform.

Many recent works have successfully applied some types of Convolutional Neural Networks (CNNs) to reduce the noticeable distortion resulting from the lossy JPEG compression technique. Most of them are built upon the processing made on the spatial domain. In this work, we propose a JPEG image decoder that is purely based on the frequency-to-frequency domain: it reads the quantized DCT coefficients received from a low-quality image bitstream and, using a deep learning-based model, predicts the missing coefficients in order to recompose the same image with enhanced quality. In experiments with two datasets, our best model was able to improve from images with quantized DCT coefficients corresponding to a Qualityz Factor (QF) of 10 to enhanced quality images with QF slightly higher than 20.

Most of information technologies courses are focused on exposing the traditional static data (e.g., SGBD) processing model and lack on presenting the data stream model. Such model can be applied to fields such as Internet of Things, busyness and finances, logistics and smart cities) of the industry. Complex Event Processing consists of a programming approach to handle Data Stream Processing. It provide primitives to process and detect the occurrence of patterns in data streams. This short course has the objective of presenting the complex event processing (CEP) programming model as a means of dealing with the specificities of data flows.

The broad use of video capture and services for its storage and transmission has enabled the production of a massive volume of video data. This usage presents a challenge in controlling the type of content that is loaded for these video storage services. The Internet slang NSFW (Not Safe For Work) is often used as a warning for media that contain inappropriate content, such as nudity, intense sexuality, violence, gore or other potentially disturbing subject matter. Convolutional Neural Network (CNNs) architectures, or ConvNets, have become the primary method used for audio-visual pattern recognition. In this work, we intend to: (1) create a CNN based model for video features extraction; And (2), validate these video features with baselines models for NSFW video classification using a multi-modal approach. In initial experimentation, our best model achieves a recall of 96.6% for NSFW class.

Methods based on Deep Learning became state-of-the-art in several Multimedia challenges. However, there is a gap of professionals to perform Deep Learning in the industry. Therefore, this tutorial aims to present the grounds and ways to develop applications that uses methods based on Deep Learning for video analysis tasks. Likewise, this tutorial is an opportunity for students and information technology professionals to qualify themselves. The main focus of this short course is to present fundamentals and technologies to develop such methods of DL.

An ergonomic evaluation is an observation of a person in order to identify musculoskeletal disorders (WMSDs) caused by prolonged or repeated harmful poses that a person adopts during work tasks. Nowadays, an ergonomist or other health professional perform such evaluations based on a set of posture rules and checklists, which can be subjective and thus lead to erroneous risk classifications. Moreover, this professional usually perform such evaluation in the patient work environment. In order to make those evaluations more objective and concise we propose a evaluation method using a mobile depth sensor. Different from other methods based in fixed depth sensors (e.g. Kinect), our method enable professionals easily perform it in the patient work environment. More precisely, we present an experiment that uses a smartphone from Google's Tango project and the Ovako Working Posture Analysing System (OWAS) method. To evaluate our approach, we also perform the ergonomic assessment using the Kinect sensor, a device that has a good reliability in the automated ergonomic evaluation. Both evaluations involved a set of 34 poses performed by 3 volunteers and annotated by an ergonomist. The Kinect has achieved accuracy of 57,08% on torso classification, 58,33% for arms and 25,00% for legs positions. While the approach using the mobile depth sensor has achieved 35,41% on torso classification, 93,05% for arms and 66,23% for legs positions on the same set of poses. Although the small sample, the achieved results may indicate that our mobile depth sensor approach can be as viable as methods based fixed depth sensor.

Recent advances in hardware and software technologies have given rise to a new class of human-computer interfaces that both explores multiple modalities and allows for multiple collaborating users. When compared to the development of traditional single-user WIMP (windows, icons, menus, pointer)-based applications, however, applications supporting the seamless integration of multimodal-multiuser interactions bring new specification and runtime requirements. With the aim of assisting the specification of multimedia applications that integrate multimodal-multiuser interactions, this paper: (1) proposes the MMAM (Multimodal-Multiuser Authoring Model); (2) presents three different instantiations of it (in NCL, HTML, and a block-based syntax); and (3) evaluates the proposed model through a task-based user study. MMAM enables programmers to design and ponder different solutions for applications with multimodal-multiuser requirements. The proposed instantiations served as proofs of concept about the feasibility of our model implementation and provided the basis for practical experimentation, while the performed user study focused on capturing evidence of both the user understanding and the user acceptance of the proposed model. We asked developers to perform tasks using MMAM and then answer a TAM (Technology Acceptance Model)-based questionnaire focused on both the model and its instances. As results, the study indicates that the participants easily understood the model (most of them performed the required tasks with minor or no errors) and found it both useful and easy to use. 94.47% of the participants gave positive answers to the block-based representation TAM questions, whereas 75.17% gave positive answers to the instances-related questions.

Methods based on Deep Learning became state-of-the-art in several Multimedia challenges. However, there is a gap of professionals to perform Deep Learning in the industry. Therefore, this short course aims to present the grounds and ways to develop multimedia applications using methods based on Deep Learning. Likewise, this short course is an opportunity for students and IT professionals can qualify yourselves.

We describe how we modified an NCL player to accept as input, in addition to NCL documents, Lua scripts. These Lua scripts evaluate to a table in a particular format, called NCL-ltab, which is a Lua table encoding of the NCL player's internal model. One advantage of our modifications is that they allow the NCL parsing to occur in the Lua script, i.e., outside the formatter but integrated in its execution flow. The same applies for parsers of dialects of NCL or similar languages. Another advantage is that new parsers can be plugged dynamically into the formatter (if they are written in Lua or can be called from Lua). In this paper, we detail the internal model of the NCL player we are using and the NCL-ltab input format. To evaluate our proposal, we present two parser-integration use cases, one for NCL itself, using the DietNCL parsing toolkit, and another for sNCL, an alternative, user-friendlier syntax for NCL.

In this paper, we present the conversion of NCL to Smix and discuss its main implications. NCL is a declarative language for the specification of interactive multimedia presentations which was adopted by the ITU-T H.761 recommendation for interoperable IPTV services. Smix is a synchronous domain-specific language with a similar purpose, but with a simpler and more precise semantics. By implementing NCL over Smix, we bring to the former the notions of reaction and execution instants, and with them some benefits. From a practical perspective, we fix the semantics of the converted documents, get a leaner NCL player (the Smix interpreter), and simplify further conversions. From a systems-design perspective, the structured conversion of NCL to Smix helps us tame the complexity of mapping the user-oriented constructs of NCL into the machine-oriented primitives that realize them as a multimedia presentation. In the paper, we present NCL and Smix, discuss related work on document conversion, and detail the conversion process and a prototype implementation.

Nowadays, tablets and smartphones are commonly used by children for both entertainment and education purposes. In special, interactive multimedia eBooks running on those devices allow a richer experience when compared to traditional text-only books, being potentially more engaging and entertaining to readers. However, to explore the most exciting features in these environments, authors are currently left alone in the sense that there is no high level (less technical) support, and these features are usually accessible only through programming or some other technical skill. In this work, we aim at extracting the main features on enhanced children's eBooks and propose a model, named Fabulas - the Portuguese word for fables -that allows authors to create interactive multimedia children's eBooks declaratively. The model was conceived by taking, as a starting point, a systematic analysis of the common concepts, with the focus on identifying and categorizing recurring characteristics and pointing out functional and non-functional requirements that establish a strong orientation towards the set of desirable abstractions of an underlying model. Moreover, the paper presents a case study for the implementation of Fabulas on the Web, and discusses the authoring of a complete interactive story over it.

Learning Objects (LOs) are entities that can be used, reused, or referred during the teaching process. They are commonly embedded into documents that establish spatial and temporal relationships on their contents. Hypervideos LOs allow students to individualize their learning experience with non-linear browsing mechanisms and content adaptation. This paper presents a survey of features for a set of documents representing such LOs as well as desirable aspects that should be expressed during the authoring phase. Also, this paper presents a conceptual model that fits such requirements. The model is implemented by SceneSync, a domain specific language focused on the synchronization and temporal behavior of LOs. As a result of the work, we present a set of LOs specified in SceneSync and a discussion about the identified features, which confirm the expressiveness and applicability of the model.

Recent advances in technologies for speech, touch and gesture recognition have given rise to a new class of user interfaces that does not only explore multiple modalities but also allows for multiple interacting users. Even so, current declarative multimedia languages e.g. HTML, SMIL, and NCL?support only limited forms of user input (mainly keyboard and mouse) for a single user. In this paper, we aim at studying how the NCL multimedia language could take advantage of those new recognition technologies. To do so, we revisit the model behind NCL, named NCM (Nested Context Model), and extend it with first-class concepts supporting multiuser and multimodal features. To evaluate our approach, we instantiate the proposal and discuss some usage scenarios, developed as NCL applications with our extended features.