Search

Extended Reality (XR) has emerged as a transformative and immersive technology with versatile applications in content creation and consumption. As XR gains popularity, companies eager to adopt it often possess a surface-level understanding, investing significant resources without effectively addressing the genuine needs of end-users. This study explores the current workflows of XR production companies, and the potential of social XR in mitigating challenges throughout the XR production workflow. We present the outcomes of three respective focus group workshops conducted with three XR production companies and their experts (N=17). The results indicate that at every stage of the production, namely pre-production, production, post-production, and post-release, there are communication challenges between producers and clients, as well as different production and post-production specialists. We discuss various aspects of XR concerning the problem and propose novel opportunities offered by social XR to ameliorate those challenges, improving communication and making development more agile.

Point clouds denote a prominent solution for the representation of 3D photo-realistic content in immersive applications. Similarly to other imaging modalities, quality predictions for point cloud contents are vital for a wide range of applications, enabling trade-off optimizations between data quality and data size in every processing step from acquisition to rendering. In this work, we focus on use cases that consider human end-users consuming point cloud contents and, hence, we concentrate on visual quality metrics. In particular, we propose a set of perceptually relevant descriptors based on principal component analysis (PCA) decomposition, which is applied to both geometry and texture data for full-reference point cloud quality assessment. Statistical features are derived from these descriptors to characterize local shape and appearance properties for both a reference and a distorted point cloud. The extracted statistical features are subsequently compared to provide corresponding predictions of visual quality for the distorted point cloud. As part of our method, a learning-based approach is proposed to fuse these individual predictors to a unified perceptual score. We validate the accuracy of the individual predictors, as well as the unified quality scores obtained after regression against subjectively annotated datasets, showing that our metric outperforms state-of-the-art solutions. Insights regarding design decisions are provided through exploratory studies, evaluating the performance of our metric under different parameter configurations, attribute domains, color spaces, and regression models. A software implementation of the proposed metric is made available at the following link: https://github.com/cwi-dis/pointpca.

In the new area of immersive multimedia environments, understanding and manipulating visual attention are crucial for enhancing user experience. This study introduces an innovative framework that extends traditional 2D saliency maps to the analysis of 3D point clouds, a step forward in adapting saliency prediction to more complex and immersive environments. Our framework centers on the orthographic projection of 3D point clouds onto 2D planes, enabling the application of established 2D saliency models to this novel context. We further delve into the evaluation of these models on a 3D point cloud eye-tracking dataset, exploring various projection settings and thresholding techniques to maintain the integrity of saliency information in the transition from 2D to 3D. This research not only bridges a gap in applying visual attention models to 3D data but also offers insights into the optimization of quality of experience in immersive multimedia systems.

The latest social VR technologies have enabled users to attend traditional media and arts performances together while being geographically removed, making such experiences accessible despite budget, distance, and other restrictions. In this work, we aim at improving the way remote performances are shared by designing and evaluating a VR theatre lobby which serves as a space for users to gather, interact, and relive the common experience of watching a virtual opera. We conducted an initial test with experts (N=10, i.e., designers and opera enthusiasts) in pairs using our VR lobby prototype, developed based on the theoretical lobby design concept. A unique aspect of our experience is its highly realistic representation of users in the virtual space. The test results guided refinements to the VR lobby structure and implementation, aiming to improve the user experience and align it more closely with the social VR lobby's intended purpose. With the enhanced prototype, we ran a between-subject controlled study (N=40) to compare the user experience in the social VR lobby between individuals and paired participants. To do so, we designed and validated a questionnaire to measure the user experience in the VR lobby. Results of our mixed-methods analysis, including interviews, questionnaire results, and user behavior, reveal the strength of our social VR lobby in connecting with other users, consuming the opera in a deeper manner, and exploring new possibilities beyond what is common in real life All supplemental materials are available at https://github.com/cwi-dis/IEEEVR2024-VRLobby.

Immersive technologies like eXtended Reality (XR) are the next step in videoconferencing. In this context, understanding the effect of delay on communication is crucial. This article presents the first study on the impact of delay on collaborative tasks using a realistic Social XR system. Specifically, we design an experiment and evaluate the impact of end-to-end delays of 300, 600, 900, 1,200, and 1,500 ms on the execution of a standardized task involving the collaboration of two remote users that meet in a virtual space and construct block-based shapes. To measure the impact of the delay in this communication scenario, objective and subjective data were collected. As objective data, we measured the time required to execute the tasks and computed conversational characteristics by analyzing the recorded audio signals. As subjective data, a questionnaire was prepared and completed by every user to evaluate different factors such as overall quality, perception of delay, annoyance using the system, level of presence, cybersickness, and other subjective factors associated with social interaction. The results show a clear influence of the delay on the perceived quality and a significant negative effect as the delay increases. Specifically, the results indicate that the acceptable threshold for end-to-end delay should not exceed 900 ms. This article additionally provides guidelines for developing standardized XR tasks for assessing interaction in Social XR environments.

Point clouds (PCs) have attracted researchers and developers due to their ability to provide immersive experiences with six degrees of freedom (6DoF). However, there are still several open issues in understanding the Quality of Experience (QoE) and visual attention of end users while experiencing 6DoF volumetric videos. First, encoding and decoding point clouds require a significant amount of both time and computational resources. Second, QoE prediction models for dynamic point clouds in 6DoF have not yet been developed due to the lack of visual quality databases. Third, visual attention in 6DoF is hardly explored, which impedes research into more sophisticated approaches for adaptive streaming of dynamic point clouds. In this work, we provide an open-source Compressed Point cloud dataset with Eye-tracking and Quality assessment in Mixed Reality (ComPEQ - MR). The dataset comprises four compressed dynamic point clouds processed by Moving Picture Experts Group (MPEG) reference tools (i.e., VPCC and GPCC), each with 12 distortion levels. We also conducted subjective tests to assess the quality of the compressed point clouds with different levels of distortion. The rating scores are attached to ComPEQ - MR so that they can be used to develop QoE prediction models in the context of MR environments. Additionally, eye-tracking data for visual saliency is included in this dataset, which is necessary to predict where people look when watching 3D videos in MR experiences. We collected opinion scores and eye-tracking data from 41 participants, resulting in 2132 responses and 164 visual attention maps in total. The dataset is available at https://ftp.itec.aau.at/datasets/ComPEQ-MR/.

In recent years, a large variety of online communication tools have emerged, including social Virtual Reality (VR) platforms for interacting in a virtual world with participants being represented as virtual avatars. Given their popularity, an active area of research focuses on improving the user experience in these virtual experiences. To enable experimentation at large scale on online platforms, it is however essential to collect behavioural data (e.g. movements and audio information). In this work, we present a toolchain that enables the running of experiments using a modified version of the social VR platform Mozilla Hubs. Specifically, our toolkit enables collection and tracking or user positions and movements at a central location, enabling fine-grained analysis of user behaviour during a social VR experience.

Introduction: In the study of neurodegenerative diseases, the possibility to follow the fate of specific cells or molecules within the whole body would be a milestone to better understand the complex evolution of disease mechanisms and to monitor the effects of therapies. The techniques available today do not allow the visualization of disease-relevant cells within the whole tridimensional biological context at high spatial resolution. Methods: Here we show the results from the first validation steps of a novel approach: by combining the conjugate nanobodies anti-glial fibrillary acidic protein (GFAP) and metal-nanoparticles (i.e. 2 nm gold NP) with X-ray phase contrast tomography (XPCT) we would be able to obtain a tridimensional visualization and identification of cells of interest together with the surrounding tissue and the vascular and neuronal networks. Results: By exploiting the X-ray attenuation properties of metal nanoparticles and the specific targeting capabilities of nanobodies, we could give XPCT the specificity it presently lacks, making it no longer a pure morphological but a molecular and targeted imaging technique. In our case, we synthesized and characterized Gold-NP/GFAP nanobody to target the astrocytes of mouse brain. Discussion: The results of the first tests presented in this paper have provided us with information on the feasibility of the approach, encouraging us to carry out further experiments in order to achieve the ultimate goal of setting up this new imaging technique.

A computationally-simplified and descriptor-richer Point Cloud Quality Assessment (PCQA) metric, namely PointPCA+, is proposed in this paper, which is an extension of PointPCA. PointPCA proposed a set of perceptually-relevant descriptors based on PCA decomposition that were applied to both the geometry and texture data of point clouds for full reference PCQA. PointPCA+ employs PCA only on the geometry data while enriching existing geometry and texture descriptors, that are computed more efficiently. Similarly to PointPCA, a total quality score is obtained through a learning-based fusion of individual predictions from geometry and texture descriptors that capture local shape and appearance properties, respectively. Before feature fusion, a feature selection module is introduced to choose the most effective features from a proposed super-set. Experimental results show that PointPCA+ achieves high predictive performance against subjective ground truth scores obtained from publicly available datasets.

Subjective quality assessment serves as a method to evaluate the perceptual quality of 3D point clouds. These evaluations can be conducted using lab-based or remote or crowdsourcing tests. The lab-based tests are time-consuming and less cost-effective. As an alternative, remote or crowd tests can be used, offering a time and cost-friendly approach. Remote testing enables larger and more diverse participant pools. However, this raises the question of its applicability due to variability in participants' display devices and environments for the evaluation of the point cloud. In this paper, the focus is on investigating the applicability of remote testing by using the Absolute Category Rating (ACR) test method for assessing the subjective quality of point clouds in different tests. We compare the results of lab and remote tests by replicating lab-based tests. In the first test, we assess the subjective quality of a static point cloud geometry for two different types of geometrical degradations, namely Gaussian noise, and octree-pruning. In the second test, we compare the performance of two different compression methods (G-PCC and V-PCC) to assess the subjective quality of coloured point cloud videos. Based on the results obtained using correlation and Standard deviation of Opinion Scores (SOS) analysis, the remote testing paradigm can be used for evaluating point clouds.

The office repo for QAVA-DPC: Eye-Tracking Based Quality Assessment and Visual Attention Dataset for Dynamic Point Cloud in 6 DoF ISMAR 2023

This repository contains the files for building and running the container for the Objective Quality Metrics Grand Challenge.

Despite remarkable advances, current Extended Reality (XR) ap- plications are in their majority local and individual experiences. A plethora of interactive applications, such as teleconferencing, telesurgery, interconnection in new buildings project chain, Cul- tural Heritage, and Museum contents communication, are well on their way to integrating immersive technologies. However, in- terconnected, and interactive XR, where participants can virtually interact across vast distances, remains a distant dream. In fact, three great barriers stand between current technology and remote im- mersive interactive life-like experiences, namely (i) content realism, (ii) motion-to-photon latency, and accurate (iii) human-centric qual- ity assessment and control. Overcoming these barriers will require novel solutions at all elements of the end-to-end transmission chain. This workshop focuses on the challenges, applications, and major advancements in multimedia, networks, and end-user infrastruc- tures to enable the next generation of interactive XR applications and services.

Perceptual quality assessment of Dynamic Point Cloud (DPC) contents plays an important role in various Virtual Reality (VR) applications that involve human beings as the end user, understanding and modeling perceptual quality assessment is greatly enriched by insights from visual attention. However, incorporating aspects of visual attention in DPC quality models is largely unexplored, as ground-truth visual attention data is scarcely available. This paper presents a dataset containing subjective opinion scores and visual attention maps of DPCs, collected in a VR environment using eye-tracking technology. The data was collected during a subjective quality assessment experiment, in which subjects were instructed to watch and rate DPCs at various degradation levels under 6 degrees-of-freedom inspection, using a head-mounted display. The dataset comprises 5 reference DPC contents, with each reference encoded at 3 distortion levels using 3 different codecs, amounting to a total of 9 degraded DPC contents. Moreover, it includes 1,000 gaze trials from 40 participants, resulting in 15,000 visual attention maps in total. The curated dataset can serve as authentic benchmark data for assessing the performance of objective DPC quality metrics. Additionally, it establishes a link between quality assessment and visual attention within the context of DPC. This work deepens our understanding of DPC quality and visual attention, driving progress in the realm of VR experiences and perception.

Extended Reality (XR) has been hailed as the new frontier of media, ushering new possibilities for societal areas such as communications, training, entertainment, gaming, and cultural heritage. However, despite the remarkable technical advances, current XR applications are in their majority local and individual experiences. In fact, three great barriers stand between current technology and remote immersive interactive life-like experiences, namely content realism, by means of Artificial Intelligence (AI) techniques, motion-to-photon latency, and accurate human-centric driven experiences able to map real and virtual worlds seamlessly. Overcoming these barriers will require novel solutions at all elements of the end-to-end transmission chain. In this panel, together with the leading experts of the SIGMM community, we will explore the challenges and opportunities to unlock the next generation of interactive XR applications and services.

It is our great pleasure to welcome you to the second workshop on Interactive eXtended Reality (IXR 2023). After the success of the inaugural edition of this workshop in 2022, our purpose is to make this the premier forum for the presentation of research results and experience reports on the topic of interactive extended reality (XR), including contributions in terms of use cases, applications, novel protocols, and algorithms. Moreover, IXR 2023 gives researchers and practitioners a unique opportunity to share their perspectives with others interested in the various aspects of XR and how to make it ready for the future.

Immersive reality technologies, such as Virtual and Augmented Reality, have ushered a new era of user-centric systems, in which every aspect of the coding-delivery-rendering chain is tailored to the interaction of the users. Understanding the actual interactivity and behaviour of the users is still an open challenge and a key step to enabling such a user-centric system. Our main goal is to extend the applicability of existing behavioural methodologies for studying user navigation in the case of 6 Degree-of-Freedom (DoF). Specifically, we first compare the navigation in 6-DoF with its 3-DoF counterpart highlighting the main differences and novelties. Then, we define new metrics aimed at better modelling behavioural similarities between users in a 6-DoF system. We validate and test our solutions on real navigation paths of users interacting with dynamic volumetric media in 6-DoF Virtual Reality conditions. Our results show that metrics that consider both user position and viewing direction better perform in detecting user similarity while navigating in a 6-DoF system. Having easy-To-use but robust metrics that underpin multiple tools and answer the question "how do we detect if two users look at the same content?"open the gate to new solutions for a user-centric system.

The development and widespread adoption of immersive XR applications has led to a renewed interest in representations that are capable of reproducing real-world objects and scenes with high fidelity. Among such representations, point clouds have attracted the interest of industry and academia alike, and new compression solutions have been developed to facilitate their adoption in mainstream applications. To ensure the best quality of experience for the end-user in limited bandwidth scenarios, new full-reference objective quality metrics have been proposed, promoting features designed specifically for point cloud contents. However, the performance of such features to predict the quality of point cloud contents when the reference is not available is largely unexplored. In this paper, we evaluate the performance of features commonly used to model point cloud distortions in a no-reference framework. The obtained features are integrated into a quality value through a support vector regression model. Results demonstrate the potential of full-reference features for no-reference assessment.

Virtual Reality telecommunication systems promise to overcome the limitations of current real-time teleconferencing solutions, by enabling a better sense of immersion and fostering more natural interpersonal interactions. Many solutions that currently enable immersive teleconferencing employ synthetic avatars to represent their users. However, photorealistic reconstructions have been shown to increase the sense of presence with respect to synthetic avatars in tele-immersive scenarios. In this paper, we present VR2Gather, a costumizable end-to-end system to transmit volumetric contents in multi-party real-time communication. We present the architecture and evaluate the costs and benefits of using different modules and transport mechanisms in terms of CPU usage, latency, and bandwidth. Moreover, we report the user experience based on applications the system has been used for, and how it was customised to meet the requirements using different acquisition and rendering modules.

The rise of capturing systems for objects and scenes in 3D with increased fidelity and immersion has led to the popularity of volumetric video contents that can be seen from any position and angle in 6 degrees of freedom navigation. Such contents need large volumes of data to accurately represent the real world. Thus novel optimization solutions and delivery systems are needed to enable volumetric video streaming over bandwidth-limited networks. In this chapter, we discuss theoretical approaches to volumetric video streaming optimization, through compression solutions, as well as network and user adaptation, for high-end and low-powered devices. Moreover, we present an overview of existing end-to-end systems, and we point to the future of volumetric video streaming.