Search

We propose a new method for the visual quality assessment of 360-degree (omnidirectional) videos. The proposed method is based on computing multiple spatio-temporal objective quality features on viewports extracted from 360-degree videos. A new model is learnt to properly combine these features into a metric that closely matches subjective quality scores. The main motivations for the proposed approach are that: 1) quality metrics computed on viewports better captures the user experience than metrics computed on the projection domain; 2) the use of viewports easily supports different projection methods being used in current 360-degree video systems; and 3) no individual objective image quality metric always performs the best for all types of visual distortions, while a learned combination of them is able to adapt to different conditions. Experimental results, based on both the largest available 360-degree videos quality dataset and a cross-dataset validation, demonstrate that the proposed metric outperforms state-of-the-art 360-degree and 2D video quality metrics.

We propose a new viewport-based multi-metric fusion (MMF) approach for visual quality assessment of 360-degree (omnidirectional) videos. Our method is based on computing multiple spatio-temporal objective quality metrics (features) on viewports extracted from 360-degree videos, and learning a model that combines these features into a metric, which closely matches subjective quality scores. The main motivations for the proposed method are that: 1) quality metrics computed on viewports better captures the user experience than metrics computed on the projection domain; 2) no individual objective image quality metric always performs best for all types of visual distortions, while a learned combination of them is able to adapt to different conditions and produce better results overall. Experimental results, based on the largest available 360-degree videos quality dataset, demonstrate that the proposed metric outperforms state-of-the-art 360-degree and 2D video quality metrics.

The recent availability of consumer-level head-mounted displays and omnidirectional cameras has been driving an explosion of 360 video content. Transforming the original recorded 360 video content in meaningful interactive multimedia presentations that support viewers in tasks such as learning, entertainment, and telepresence, however, is not trivial and requires new tools. Such tools must provide an easy-to-use authoring model for the integration of different media objects and active user interface elements.In this paper, based on real-world scenarios for interactive 360 videos, we gather a set of requirements and propose a declarative authoring model to support authors in the process of designing and creating 360-degree interactive video presentations. As a case study, we implement different applications showing the expressiveness and completeness of the model in the scope of the target scenarios.

In this paper, we give an overview of the semantic gap problem in multimedia and discuss how machine learning and symbolic AI can be combined to narrow this gap. We describe the gap in terms of a classical architecture for multimedia processing and discuss a structured approach to bridge it. This approach combines machine learning (for mapping signals to objects) and symbolic AI (for linking objects to meanings). Our main goal is to raise awareness and discuss the challenges involved in this structured approach to multimedia understanding, especially in the view of the latest developments in machine learning and symbolic AI., 1st International Workshop on Bridging the Gap between Semantics and Multimedia Processing (SeMP 2019): http://semp.mybluemix.net/2019/. arXiv admin note: text overlap with arXiv:1911.09606

Unlike traditional planar 2D visual content, immersive 360-degree images and videos undergo particular processing steps and are intended to be consumed via head-mounted displays (HMDs). To get a deeper understanding on the perception of 360-degree visual distortions when consumed through HMDs, we perform an exploratory task-based subjective study in which we have asked subjects to define the first noticeable difference and break-in-presence points when incrementally adding specific compression artifacts. The results of our study: give insights on the range of allowed visual distortions for 360-degree content; show that the added visual distortions are more tolerable in mono than in stereoscopic 3D; and identify issues with current 360-degree objective quality metrics.

Historically, the Multimedia community research has focused on output modalities, through studies on timing and multimedia processing. The Multimodal Interaction community, on the other hand, has focused on user-generated modalities, through studies on Multimodal User Interfaces (MUI). In this paper, aiming to assist the development of multimedia applications with MUIs, we propose the integration of concepts from those two communities in a unique high-level programming framework. The framework integrates user modalities --both user-generated (e.g., speech, gestures) and user-consumed (e.g., audiovisual, haptic)-- in declarative programming languages for the specification of interactive multimedia applications. To illustrate our approach, we instantiate the framework in the NCL (Nested Context Language) multimedia language. NCL is the declarative language for developing interactive applications for Brazilian Digital TV and an ITU-T Recommendation for IPTV services. To help evaluate our approach, we discuss a usage scenario and implement it as an NCL application extended with the proposed multimodal features. Also, we compare the expressiveness of the multimodal NCL against existing multimedia and multimodal languages, for both input and output modalities.

Visual distortions in processed 360-degree visual content and consumed through head-mounted displays (HMDs) are perceived very differently when compared to traditional 2D content. To better understand how compression-related artifacts affect the overall perceived quality of 360-degree videos, this paper presents a subjective quality assessment study and analyzes the performance of objective metrics to correlate with the gathered subjective scores. In contrast to previous related work, the pro- posed study focuses on the equiangular cubemap projection and includes specific visual distortions (blur, blockiness, H.264 compression, and cubemap seams) on both monoscopic and stereoscopic sequences. The objective metrics performance analysis is based on metrics computed in both the projection domain and the viewports, which is closer to what the user sees. The results show that overall objective metrics computed on viewports are more correlated with the subjective scores in our dataset than the same metrics computed in the projection domain. Moreover, the proposed dataset and objective metrics analysis serve as a bench- mark for the development of new perception-optimized quality assessment algorithms for 360-degree videos, which is still a largely open research problem.

O livro Minicursos do XXIV Simposio Brasileiro de Sistemas Multimidia e Web oferece a sua comunidade textos relacionados a temas que norteiam os ultimos avancos na area de multimidia e web, trazendo informacoes sobre novas tecnologias e topicos atuais de pesquisa em areas correlatas ao tema. Assim, este volume oferece uma excelente oportunidade para familiarizacao dos interessados com novos temas de pesquisa que podem vir a ser uteis em suas vidas profissionais.

Nowadays, tablets and smartphones are commonly used by children for both entertainment and education purposes. In special, interactive multimedia eBooks running on those devices allow a richer experience when compared to traditional text-only books, being potentially more engaging and entertaining to readers. However, to explore the most exciting features in these environments, authors are currently left alone in the sense that there is no high level (less technical) support, and these features are usually accessible only through programming or some other technical skill. In this work, we aim at extracting the main features on enhanced children's eBooks and propose a model, named Fabulas - the Portuguese word for fables -that allows authors to create interactive multimedia children's eBooks declaratively. The model was conceived by taking, as a starting point, a systematic analysis of the common concepts, with the focus on identifying and categorizing recurring characteristics and pointing out functional and non-functional requirements that establish a strong orientation towards the set of desirable abstractions of an underlying model. Moreover, the paper presents a case study for the implementation of Fabulas on the Web, and discusses the authoring of a complete interactive story over it.

Learning Objects (LOs) are entities that can be used, reused, or referred during the teaching process. They are commonly embedded into documents that establish spatial and temporal relationships on their contents. Hypervideos LOs allow students to individualize their learning experience with non-linear browsing mechanisms and content adaptation. This paper presents a survey of features for a set of documents representing such LOs as well as desirable aspects that should be expressed during the authoring phase. Also, this paper presents a conceptual model that fits such requirements. The model is implemented by SceneSync, a domain specific language focused on the synchronization and temporal behavior of LOs. As a result of the work, we present a set of LOs specified in SceneSync and a discussion about the identified features, which confirm the expressiveness and applicability of the model.

Recent advances in technologies for speech, touch and gesture recognition have given rise to a new class of user interfaces that does not only explore multiple modalities but also allows for multiple interacting users. Even so, current declarative multimedia languages e.g. HTML, SMIL, and NCL?support only limited forms of user input (mainly keyboard and mouse) for a single user. In this paper, we aim at studying how the NCL multimedia language could take advantage of those new recognition technologies. To do so, we revisit the model behind NCL, named NCM (Nested Context Model), and extend it with first-class concepts supporting multiuser and multimodal features. To evaluate our approach, we instantiate the proposal and discuss some usage scenarios, developed as NCL applications with our extended features.

This paper discusses the importance of non-functional requirements in the design of hypermedia authoring tools, which typically provides multiple graphical abstractions (views). It focuses on creating products and services that operate robustly across a broad range of environments, and that take into account the changeable needs of their users over time, as they become more familiar with the tool. In order to meet these non-functional aspects, this paper proposes a microkernel-based architecture for authoring tools, where the microkernel is responsible for instantiating the requested extensions (plugins), maintaining the core data model that represents the hypermedia document under development, and notifying changes in this model to plugins interested in them. Based on the proposed architecture, a new version of Composer (an NCL authoring tool) is presented, rewritten from scratch. Results from experiments show that the discussed non-functional requirements are adequately met.

PONTIFÍCIA UNIVERSIDADE CATÓLICA DO RIO DE JANEIRO COORDENAÇÃO DE APERFEIÇOAMENTO DO PESSOAL DE ENSINO SUPERIOR FUNDAÇÃO DE APOIO À PESQUISA DO ESTADO DO RIO DE JANEIRO Tecnologias de vídeos bidimensionais (2D) têm evoluído rapidamente nos últimos anos. Apesar disso, elas não permitem uma visão realista e imersiva do mundo, pois não oferecem importantes dicas de profundidade para o sistema visual humano. Tecnologias de vídeo tridimensionais (3D) têm como objetivo preencher essa lacuna, provendo representações que permitem a reprodução de informações de profundidade em displays 3D. Embora a representação baseada em vídeos estereoscópicos ainda seja a mais utilizada até o momento, novas representações de vídeo 3D têm emergido, tais como MVV (Multi-view video), 2D plus Z (2D plus depth), MVD (Multi-view plus depth) e LDV (Layered-depth video). A integração de aplicações multimídia com mídias 3D tem o potencial de permitir novos conteúdos interativos, novas experiências com o usuário e novos modelos de negócio. Nesta tese, duas abordagens para a integração de aplicações multimídia em cadeias de transmissão de vídeo 3D fim-a-fim são propostas. Primeiro, uma abordagem que é compatível com cadeias de transmissão de vídeo 3D baseado em vídeos estereoscópicos é discutida. A proposta consiste em extensões para linguagens multimídia 2D e um processo de conversão de aplicações multimídia 2D para sua versão estereoscópica. Essa proposta não requer nenhuma alteração no exibidor de linguagens multimídia 2D para a apresentação de mídias estereoscópicas. Em uma segunda abordagem, extensões adicionais a linguagens multimídia também são propostas visando a integração de aplicações multimídia em cadeias de vídeo 3D baseado em profundidade (2D plus Z ou LDV). Além disso, uma arquitetura para a composição gráfica dessas aplicações, baseada no conceito de LDV e que permite a integração de objetos de mídia baseado em profundidade em exibidores de aplicações multimídias é apresentada. Como um exemplo de aplicação prática das proposta desta tese, ambas são implementadas e integradas em um sistema de vídeo 3D fim-a-fim baseado no Sistema Brasileiro de TV Digital. Two-dimensional video technologies have evolved quickly in the last few years. Even so, they do not achieve a realistic and immersive view of the world since they do not offer important depth cues to the human vision system. Three-dimensional video (3DV) technologies try to fulfill this gap through video representations that enable 3D displays to provide those additional depth cues. Although CSV (Conventional Stereoscopic Video) has been the most widely-used 3DV representation, other 3DV representations have emerged during the last years. Examples of those representations include MVV (Multi-view video), 2D plus Z (2D plus depth), MVD (Multi-view plus depth), and LDV (Layered-depth Video). Although end-to-end 3DV delivery chains based on those 3DV formats have been studied, the integration of interactive multimedia applications into those 3DV delivery chains has not yet been explored enough. The integration of multimedia applications with 3D media using those new representations has the potential of allowing new rich content, user experiences and business models. In this thesis, two approaches for the integration of multimedia applications into 3DV end-to-end delivery chains are proposed. First, a backward-compatible approach for integrating CSV-based media into 2D-only multimedia languages is discussed. In this proposal, it is possible to add depth information to 2D-only media objects. The proposal consists of extensions to multimedia languages and a process for converting the original multimedia application into its stereoscopic version. It does not require any change on the language player and is ready-to-run in current CSV-based 3DV delivery chains and digital receiver s hardware. Second, extensions to multimedia languages based on layered-depth media are proposed and a software architecture for the graphics composition of multimedia applications using those extensions is presented. As an example, both proposals are implemented and integrated into an end-to-end 3DV delivery chain based on the Brazilian Digital TV System.