Search

In this position paper, we outline our research challenges in Affective Interactive Systems, and present recent work on visualizing avatar biosignals for social VR entertainment. We highlight considerations for how biosignals animations in social VR spaces can (falsely) indicate users' availability status., Comment: Position paper for CHI '22 Workshop on "Social Presence in Virtual Event Spaces"

The rise of capturing systems for objects and scenes in 3D with increased fidelity and immersion has led to the popularity of volumetric video contents that can be seen from any position and angle in 6 degrees of freedom navigation. Such contents need large volumes of data to accurately represent the real world. Thus, novel optimization solutions and delivery systems are needed to enable volumetric video streaming over bandwidth-limited networks. In this chapter, we discuss theoretical approaches to volumetric video streaming optimization, through compression solutions, as well as network and user adaptation, for high-end and low-powered devices. Moreover, we present an overview of existing end-to-end systems, and we point to the future of volumetric video streaming.

Visualizing biosignals can be important for social Virtual Reality (VR), where avatar non-verbal cues are missing. While several biosignal representations exist, designing effective visualizations and understanding user perceptions within social VR entertainment remains unclear. We adopt a mixed-methods approach to design biosignals for social VR entertainment. Using survey (N=54), context-mapping (N=6), and co-design (N=6) methods, we derive four visualizations. We then ran a within-subjects study (N=32) in a virtual jazz-bar to investigate how heart rate (HR) and breathing rate (BR) visualizations, and signal rate, influence perceived avatar arousal, user distraction, and preferences. Findings show that skeuomorphic visualizations for both biosignals allow differentiable arousal inference; skeuomorphic and particles were least distracting for HR, whereas all were similarly distracting for BR; biosignal perceptions often depend on avatar relations, entertainment type, and emotion inference of avatars versus spaces. We contribute HR and BR visualizations, and considerations for designing social VR entertainment biosignal visualizations.

Behaviour in virtual environments might be informed by our experiences in physical environments, but virtual environments are not constrained by the same physical, perceptual, or social cues. Instead of replicating the properties of physical spaces, one can create virtual experiences that diverge from reality by dynamically manipulating environmental, aural, and social properties. This paper explores digital proxemics, which describe how we use space in virtual environments and how the presence of others influences our behaviours, interactions, and movements. First, we frame the open challenges of digital proxemics in terms of activity, social signals, audio design, and environment. We explore a subset of these challenges through an evaluation that compares two audio designs and two displays with different social signal affordances: head-mounted display (HMD) versus desktop PC. We use quantitative methods using instrumented tracking to analyse behaviour, demonstrating how personal space, proximity, and attention compare between desktop PC and HMDs.

Immersive reality technologies, such as Virtual and Augmented Reality, have ushered a new era of user-centric systems, in which every aspect of the coding--delivery--rendering chain is tailored to the interaction of the users. Understanding the actual interactivity and behaviour of the users is still an open challenge and a key step to enabling such a user-centric system. Our main goal is to extend the applicability of existing behavioural methodologies for studying user navigation in the case of 6 Degree-of-Freedom (DoF). Specifically, we first compare the navigation in 6-DoF with its 3-DoF counterpart highlighting the main differences and novelties. Then, we define new metrics aimed at better modelling behavioural similarities between users in a 6-DoF system. We validate and test our solutions on real navigation paths of users interacting with dynamic volumetric media in 6-DoF Virtual Reality conditions. Our results show that metrics that consider both user position and viewing direction better perform in detecting user similarity while navigating in a 6-DoF system. Having easy-to-use but robust metrics that underpin multiple tools and answer the question ``how do we detect if two users look at the same content?" open the gate to new solutions for a user-centric system.

We are facing increasingly pressure on reducing travel and working remotely. Tools that support effective remote communication and collaboration are much needed. Social Virtual Reality (VR) is an emerging medium, which invites multiple users to join a collaborative virtual environment (VE) and has the potential to support remote communication in a natural and immersive way. We successfully organized a CHI 2020 Social VR workshop virtually on Mozilla Hubs, which invited researchers and practitioners to have a fruitful discussion over user representations and ethics, evaluation methods, and interaction techniques for social VR as an emerging immersive remote communication tool. In this CHI 2021 virtual workshop, we would like to organize it again on Mozilla Hubs, continuing the discussion about proxemics, social cues and VE designs, which were identified as important aspects for social VR communication in our CHI 2020 workshop.

In order to create an improved experience in variable network delivery conditions, immersive multimedia content can be delivered over existing network environments, from multiple sources. These sources are normally servers located in the cloud, in various locations. Storytelling and certain related content, such as the immersive opera multimedia data in the context of the European Horizon2020 project TRACTION, require multimedia players to be able to receive content simultaneously from several locations, and at times, merge the content, creating new content in real-time. For instance, 360° recordings and polygonal 3D content can be delivered from different locations, and the end-user receives the unified content on his or her device. This paper introduces a study of how devices can be analysed, in terms of metrics, when receiving multimedia content from multiple sources, as the network and the devices have constraints regarding performance and video quality

Point clouds denote a prominent solution for the representation of 3-D photo-realistic content in immersive applications. Similarly to other imaging modalities, quality predictions for point cloud contents are vital for a wide range of applications, enabling trade-off optimizations between data quality and data size in every processing step from acquisition to consumption. In this work, we focus on use cases that consider human end-users consuming point cloud contents and, hence, we concentrate on visual quality metrics. In particular, we propose a set of perceptually-relevant descriptors based on Principal Component Analysis (PCA) decomposition that is applied to both geometry and texture data for full-reference point cloud quality assessment. Statistical features are derived from these descriptors to characterize local shape and appearance properties for both a reference and a distorted point cloud. They are subsequently compared to provide corresponding predictions of visual quality for the latter. As part of our method, a learning-based approach is proposed to fuse these individual quality predictors to a unified perceptual score. Various regression models are additionally evaluated for this task and shown to be effective in harnessing the predictors' strength. We validate the accuracy of the individual quality predictors, as well as the unified quality scores obtained after any regression model against subjectively-annotated datasets, and we show that non-linear regression models exhibit notable gains with respect to current literature. A software implementation of the proposed metric is made available at the following link: https://github.com/cwi-dis/pointpca., Comment: 10 pages, 4 figures, 3 tables

The document represents the reference for the software integration and the experiments performed in the VR-Together project. It aims to describe the requirements, the architecture and how the experimental work envisaged to implement the main paradigm outlined in VR-Together: the creation of a platform, and the corresponding media content, that allows two, or more users to feel as if they were together in a virtual environment. The togetherness feeling is provided delivering photorealistic content, both for media and end-user representations. 3.0 - Final version updating the one officially approved in M25 (V1.1) during the 2nd year review. V3.0 has been issued to update the changes in the requirements and architecture in the last year for exploring a diversity of case studies and improve accessibility to the platform as suggested by the review panel

Virtual Reality (VR) and Augmented Reality (AR) applications have seen a drastic increase in commercial popularity. Different representations have been used to create 3D reconstructions for AR and VR. Point clouds are one such representation characterized by their simplicity and versatility, making them suitable for real time applications, such as reconstructing humans for social virtual reality. In this study, we evaluate how the visual quality of digital humans, represented using point clouds, is affected by compression distortions. We compare the performance of the upcoming point cloud compression standard against an octree-based anchor codec. Two different VR viewing conditions enabling 3- and 6 degrees of freedom are tested, to understand how interacting in the virtual space affects the perception of quality. To the best of our knowledge, this is the first work performing user quality evaluation of dynamic point clouds in VR; in addition, contributions of the paper include quantitative data and empirical findings. Results highlight how perceived visual quality is affected by the tested content, and how current data sets might not be sufficient to comprehensively evaluate compression solutions. Moreover, shortcomings in how point cloud encoding solutions handle visually-lossless compression are discussed.

Thedocument reports the work of the project on user experience. It overviews the planning, execution and evaluation of the pilot actions of the project, provides initial results regarding the established connected user labs, and details the advisory board and the focus groups to gather requirements from professionals. The deliverable as well discusses the metrics and methodologies used in the user experience research.

While traditional live-broadcasting is typically comprised of a handful of well-defined workflows, these become insufficient when targeting multiple screens and interactive companion devices on the viewer side. In this case study, we describe the development of an end-to-end system enabling immersive and interactive experiences using an object-based broadcasting approach. We detail the deployment of this system during the live broadcast of the FA Cup Final at Wembley Stadium in London in May 2018. We also describe the trials and interviews we ran in the run-up to this event, the infrastructure we used, the final software developed for controlling and rendering on-screen graphics and the system for generating and configuring the live broadcast-objects. In this process, we learned about the workflows inside an OB truck during live productions through an ethnographic study and the challenges involved in running an object-based broadcast over the Internet, which we discuss alongside other gained insights.

Frequent variations in throughput make mobile networks a challenging environment for video streaming. Current video players deal with those variations by matching video quality to network throughput. However, this adaptation strategy results in frequent changes of video resolution and bitrate, which negatively impacts the users' streaming experience. Alternatively, keeping the video quality constant would improve the experience, but puts additional demand on the network. Downloading high quality content when channel quality is low requires additional resources, because data transfer efficiency is linked to channel quality. In this paper, we present a predictive Channel Quality based Buffering Strategy (CQBS) that lets the video buffer grow when channel quality is good, and relies on this buffer when channel quality decreases. Our strategy is the outcome of a Markov Decision Process. The underlying Markov chain is conditioned on 377 real-world LTE channel quality traces that we have collected using an Android mobile application. With our strategy, mobile network providers can deliver constant quality video streams, using less network resources.

We present a software solution for creating and playing back interactive multi-screen experiences. The system consists of a pre-production application for editing layout and timing of interactive media objects and a live-triggering software for in- serting on-demand content during live streams of these edited experiences. The system is governed by a hierarchical file for- mat that defines the temporal relationship and synchronisation of media objects. We also briefly introduce the concept of DMApp Components, an open specification which is used to describe and create custom interactive media objects.

[EN] Low-latency and media sync are essential requirements to enable interactive multi-party services, such as Social TV. In this work, we present an open-source and customizable framework that allows measuring end-to-end (e2e) video delays and provides support for different types of media sync, including Inter-Destination Media Sync (IDMS). This framework can be used by researchers to investigate the suitability of different techniques for optimizing the system performance in terms of e2e delays and media sync., This work has been funded, partially, by UPV under its R&D Support Program in PAID-01-10 Project and by CWI under EU/FP7 REVERIE Project (ICT-2011-7-287723).

D-105: A customizable open-source framework for measuring and equalizing e2e delays in shared video watching

The TV is dead motto of just a few years ago has been replaced by the prospect of Internet Protocol (IP) television experiences over converged networks to become one of the great technology opportunities in the next few years. As an introduction to the Special Issue on Smart, Social and Converged Television, this extended editorial intends to review the current IP television landscape in its many realizations: operator-based, over-the-top, and user generated. We will address new services like social TV and recommendation engines, dissemination including new paradigms built on peer to peer and content centric networks, as well as the all important quality of experience that challenges services and networks alike. But we intend to go further than just review the existing work by proposing areas for the future of television research. These include strategies to provide services that are more efficient in network and energy usage while being socially engaging, novel services that will provide consumers with a broader choice of content and devices, and metrics that will enable operators and users alike to define the level of service they require or that they are ready to provide. These topics are addressed in this survey paper that attempts to create a unifying framework to link them all together. Not only is television not dead, it is well alive, thriving and fostering innovation and this paper will hopefully prove it., 18 pages, 1 figure

In recent years social networking and social interactions have challenged old conceptions in the television landscape. Web applications that offer video content, networked television sets and set-top boxes, and online TV widgets are – or, will be – radically transforming how people watch and interact around television content. Since the wealth of existing solutions and approaches might be daunting to newcomers, this paper surveys previous and current efforts in the area of social television. The contribution of this paper is a framework that categorizes the most salient features of existing applications. The resulting framework is a valuable contribution for better understanding the present, and a useful tool for evaluating and analyzing future developments in the field. The final goal is to provide a structured categorization that helps research, industry and entrepreneurs in analyzing the current shift on how people socialize around television content. ispartof: pages:94-99 ispartof: Proceedings of the Networked and Electronic Media Summit (NEM Summit 2011), Torino, Italy, September 27-29, 2011 pages:94-99 ispartof: NEM Summit location:Torino, Italy status: published

FONDECYT FONDECYT

This document describes a PhD research track on the disclosure of audiovisual digital archives. The domain of audiovisual material is introduced as well as a problem description is formulated. The main research objective is to investigate the gap between the different users and the digital archives. Next, design research is proposed as a methodology for this research. Lastly, the social and scientific relevance is briefly discussed.

nrpages: 190 status: published

International audience; In spite of the high profile of media types such as video, audio and images, many multimedia presentations rely extensively on text content. Text can be used for incidental labels, or as subtitles or captions that accompany other media objects. In a multimedia document, text content is not only constrained by the need to support presentation styles and layout, it is also constrained by the temporal context of the presentation. This involves intra-text and extra-text timing synchronization with other media objects. This paper describes a new timed-text representation language that is intended to be embedded in a non-text host language. Our format, which we call aText (for the Ambulant Text Format), balances the need for text styling with the requirement for an efficient representation that can be easily parsed and scheduled at runtime. aText, which can also be streamed, is defined as an embeddable text format for use within declarative XML languages. The paper presents a discussion of the requirements for the format, a description of the format and a comparison with other existing and emerging text formats. We also provide examples for aText when embedded within the SMIL and MLIF languages and discuss our implementation experiences of aText with the Ambulant Player.

International audience; This paper presents a model and architecture for non-intrusive user interfaces in the interactive digital TV domain. The model is based on two concepts: non-monolithic rendering for content consumption and actions descriptions for user interaction. In the first case, subsets of the multimedia content can be delivered to different rendering components (e.g., video to the TV screen and extra information to a handheld device). In the second case, we differentiate between actions, handlers, and activators. An action is the description of the user intentions, a handler implements that action, and an activator is the user interface of the action. Because we define actions instead of user interfaces, the implementation of the activators can take multiple forms: conventional user interfaces (using gestures or speech) and intelligent interfaces, in which the actions are derived from a set of parameters (e.g., number of people in the room or distance to the TV).

FONDECYT 17 FONDECYT

Fine-grained emotion recognition is the process of automatically identifying the emotions of users at a fine granularity level, typically in the time intervals of 0.5s to 4s according to the expected duration of emotions. Previous work mainly focused on developing algorithms to recognize only one emotion for a video based on the user feedback after watching the video. These methods are known as post-stimuli emotion recognition. Compared to post-stimuli emotion recognition, fine-grained emotion recognition can provide segment-by-segment prediction results, making it possible to capture the temporal dynamics of users’ emotions when watching videos. The recognition result it provides can be aligned with the video content and tell us which specific content in the video evokes which emotions. Most of the previous works on fine-grained emotion recognition require fine-grained emotion labels to train the recognition algorithm. However, the experiments to collect these fine-grained emotion labels are usually costly and time-consuming. Thus, this thesis focuses on investigating whether we can accurately predict the emotions of users at a fine granularity level with only a limited amount of emotion ground truth labels for training. We start our technical contribution in Chapter 3 by building up the baseline methods which are trained using fine-grained emotion labels. This can help us understand how accurate the recognition can be if we take advantage of the fine-grained emotion labels. We propose a correlation-based emotion recognition algorithm (CorrNet) to recognize the valence and arousal (V-A) of each instance (fine-grained segment of signals) using physiological signals. CorrNet extracts features both inside each fine-grained signal segment (instance) and between different instances for the same video stimuli (correlation-based features). We found out that, compared to sequential learning, correlation-based instance learning offers advantages of higher recognition accuracy, less overfitting and less computational complexity.Compared to collecting fine-grained emotion labels, it is easier to collect only one emotion label after the user watched that stimulus (i.e., the post-stimuli emotion labels). Therefore, in the second technical chapter (Chapter 4) of the thesis, we investigate whether the emotions can be recognized at a fine granularity level by training with only post-stimuli emotion labels (i.e., labels users annotated after watching videos), and propose an Emotion recognition algorithm based on Deep Multiple Instance Learning (EDMIL). EDMIL recognizes fine- grained valence and arousal (V-A) labels by identifying which instances represent the post-stimuli V-A annotated by users after watching the videos. Instead of fully-supervised training, the instances are weakly-supervised by the post-stimuli labels in the training stage. Our experiments show that weakly supervised learning can reduce overfitting caused by the temporal mismatch between fine-grained annotations and input signals.Although the weakly-supervised learning algorithm developed in Chapter 4 can obtain accurate recognition results with only few annotations, it can only identify the annotated (post-stimuli) emotion from the baseline emotion (e.g., neutral) because only post-stimuli labels are used for training. The non-annotated emotions are all categorized as part of the baseline. To overcome this, in Chapter 5, we propose an Emotion recognition algorithm based on Deep Siamese Networks (EmoDSN). EmoDSN recognizes fine-grained valence and arousal (V-A) labels by maximizing the distance metric between signal segments with different V-A labels. According to the experiments we run in this chapter, EmoDSN achieves promising results by using only 5 shots (5 samples in each emotion category) of training data.Reflecting on the achievements reported in this thesis, we conclude that the fully-supervised algorithm (Chapter 3) can result in more accurate fine-grained emotion recognition results if the annotation quantity is sufficient. The weakly-supervised learning method (Chapter 4) can result in better recognition results at the instance level compared to fully-supervised methods. We also found that the weakly-supervised learning methods can perform the best if users annotate their most salient, but short emotions or their overall and longer-duration (i.e., persisting) emotions. The few-shot learning method (Chapter 5) can obtain more emotion categories (more than the weakly-supervised learning) by using less amount of samples for training (better than the fully-supervised learning). However, the limitation of it is that accurate recognition results can only be achieved by a subject-dependent model.

For this special issue on Multimedia Systems, Languages, and Infrastructures for Interactive Television the four best papers on multimedia systems and infrastructures were invited to extend their conference contribution in the form of a journal paper. These papers cover a wide range of current challenges for multimedia systems: content recommendation, participatory multimedia genres, evaluation of mobile media usage, and digital media narratives.

In this paper, we look at how television genres can play a role in the use of social interactive television systems (social iTV). Based on a user study of a system for sending and receiving enriched video fragments to and from a range of devices, we discuss which genres are preferred for talking while watching, talking about after watching and for sending to users with different devices. The results show that news, soap, quiz and sport are genres during which our participants talk most while watching and are thus suitable for synchronous social iTV systems. For asynchronous social iTV systems film, news, documentaries and music programs are potentially popular genres. The plot structure of certain genres influences if people are inclined to talk while watching or not, and to which device they would send a video fragment. We also discuss how this impacts the design and evaluation of social iTV systems. ispartof: pages:71-80 ispartof: ACM International Conference Proceeding Series, Proceeding of the 1st international conference on Designing interactive user experiences for TV and video vol:291 pages:71-80 ispartof: uxTV 2008 location:Mountain View, California date:22 Oct - 24 Oct 2008 status: published

Promoting a sense of presence is often identified as a prerequisite for mediated interaction. To do so, however, we need a thorough understanding of what presence encompasses and how it can be influenced. The goal of this paper is to elaborate on the different aspects of the sense of presence as identified in the literature, while illustrating whether and how these aspects are promoted in three virtual world cases. We hope to evoke reflection on the link between promoting presence and supporting mediated interaction.