Search

Critique sessions are an essential educational activity at the center of many design disciplines, especially those involving the creation of physical mockups. Conventional approaches often require the students and the instructor to be in the same space to jointly view and discuss physical artifacts. However, in remote learning contexts, available tools (such as videoconferencing) are insufficient due to ineffective, inefficient spatial referencing. This paper presents ARCritique, a mobile Augmented Reality application that allows users to 1) scan physical artifacts, generate corresponding 3D models, and share them with distant instructors; 2) view the model simultaneously in a synchronized virtual environment with remote collaborators; and 3) point to and draw on the model synchronously to aid communication. We evaluated ARCritique with seven Industrial Design students and three faculty to use the app in a remote critique setting. The results suggest that direct support for spatial communication improves collaborative experiences. Published version

In terms of the education system, developing countries of the world are still characterized with unequal school systems due to perpetuating inequality, and poverty. It is said that children who are admitted into the top schools are bound to achieve good grades in mathematics and sciences, unlike children in the next 6600 schools combined. This is a result of the majority of learners not having access to proper educational facilities, such as scientific labs and other technological means that would be in favor of the learner, unlike the rest of their counterparts in private or privileged schools. This paper presents a virtual reality science application, specifically focusing on the concept of inertia, which was achieved using various techniques such as 3D visualization and computer graphics to provide a platform for training and education of the scientific concept. Studies show that with the use of this technology, users were able to improve their subject scores, and it assisted with memory retention as well. With just the aid of a cell phone and virtual reality glasses, the ordinary learner will now be able to navigate his way around certain scientific concepts and experiments. Furthermore, the learner will be able to comprehend the scientific concept that is presented in front of their eyes, and this will ensure that every learner has equitable access to the learning content.

The rise of online learning poses unique challenges in music education, where live demonstration and musical synchronization are critical for student success. We present HoloKeys, a music education interface which allows instructors to play remotely located pianos using an augmented reality headset and wifi-enabled microcontrollers. This approach allows students to receive distance education which is more direct, immersive, and comprehensive than conventional video conferencing allows for. HoloKeys enables remote students to observe live instructional demonstration on a physical keyboard in their immediate environment just as they would in traditional settings. HoloKeys consists of two separate components: an augmented reality user interface and a piano playing apparatus. Our system aims to extend online music education beyond desktop platforms into the physical world, thereby addressing crucial obstacles encountered by educators and students transitioning into online education.

In this work, we present a system that adds augmented reality interaction and 3D-space utilization to educational videoconferencing for a more engaging distance learning experience. We developed infrastructure and user interfaces that enable the use of an instructor’s physical 3D space as a teaching stage, promote student interaction, and take advantage of the flexibility of adding virtual content to the physical world. The system is implemented using hand-held mobile augmented reality to maximize device availability, scalability, and ready deployment, elevating traditional video lectures to immersive mixed reality experiences. We use multiple devices on the teacher’s end to provide different simultaneous views of a teaching space towards a better understanding of the 3D space.

In-stadium sports spectating delivers a unique social experience in a variety of sports. However, in contrast to broadcast delivery, it lacks the provision of real-time information augmentation, like game statistics overlaid on screen. In an earlier iteration, we developed ARSpectator, a prototypical, mobile system which can be brought to the stadium to experience both, the live sport action and situated infographics spatially augmented into the scene. In some situations it is difficult or often impossible to go to the stadium though, for instance because of limited stadium access during pandemics or when wanting to conduct controlled user studies. We address this by turning our ARSpectator system into an indirect augmented reality experience deployed to an immersive, virtual reality head-mounted display: The live stadium experience is delivered by way of a surrounding 360 video recording while maintaining and extending the provision of interactive, situated infographics. With our XRSpectator demo prototype presented here, users can have an ARSpectator experience of a rugby game in our local stadium.

Due to the evolution of motion capture devices, natural user interfaces have been applied in several areas, such as neuromotor rehabilitation supported by virtual environments. This paper presents a smartphone application that allows the user to interact with the virtual environment and enables the captured data to be stored, processed, and used in machine learning models. The application submits the recordings to the remote database with information about the movement and in order to apply supervised machine learning. As a proof of concept, we generated a dataset capturing movement data using our application with 232 instances divided into 8 classes of movements. Moreover, we used this dataset for training models that classifies these movements. The remarkable accuracy of the models shows the feasibility of using body articulation data for a classification task after some data transformations.

Urban information discovery has a long history in geography and other related disciplines. Routing and Point-of-Interest (POI) searching services have been implemented in many Geographic Information Systems (GISs)-based applications, where spatial information (e.g., POI data) is projected to a digital map as a source of auxiliary data to support intelligent spatial decision-making. In recent years, Augmented Reality (AR) has become a growing trend to visualize spatial objects through digital visual elements. However, a research gap exists between GIS and AR systems in terms of computational efficiency and interoperability. This paper aims to develop a geospatial cyberinfrastructure that integrates urban information and AR to enhance spatial knowledge visualization and discovery. Our experiments demonstrate how spatial analysis and urban sentiment computing can efficiently be integrated into the AR-based framework to support real-time decision-making.

Human speech often contains ambiguity stemming from the use of demonstrative pronouns (DPs), such as “this” and “these.” While we can typically decipher which objects of interest DPs are referring to based on context, modern day voice assistants (VAs – such as Google Assistant and Siri) are yet unable to process queries containing such ambiguity. For instance, to humans, a question such as “how much is this?” can be clarified through visual reference (e.g., a buyer gestures to the seller the object they would like to purchase). To bridge this gap between human and machine cognition, we built and examined a touch + voice multimodal VA prototype that enables users to select key spatial information to embed as context and query the VA. The prototype converts results of mobile, real-time object recognition and optical character recognition models into augmented reality buttons that represent features. Users can interact with and modify the selected features through a word grid. We conducted a study to investigate: 1) how touch performs as an additional modality to resolve ambiguity in queries, 2) how users use DPs when interacting with VAs, and 3) how users perceive a VA that can understand DPs. From this procedure we found that as the query becomes more complex, users prefer the multimodal VA over the standard VA without experiencing elevated cognitive load. Additionally, even though it took some time getting used to, many participants eventually became comfortable with using DPs to interact with the multimodal VA and appreciated the improved human-likeness of human-VA conversations.

Reconstructing point clouds from images would extremely benefit many practical CV applications, such as robotics, automated vehicles, and Augmented Reality. Fueled by the advances of deep neural network, many deep learning frameworks are proposed to address this problem recently. However, these frameworks generally rely on a large amount of labeled training data (e.g., image and point cloud pairs). Although we usually have numerous 2D images, corresponding 3D shapes are insufficient in practice. In addition, most available 3D data covers only a limited amount of classes, which further restricts the models' generalization ability to novel classes. To mitigate these issues, we propose a novel few-shot single-view point cloud generation framework by considering both class-specific and class-agnostic 3D shape priors. Specifically, we abstract each class by a prototype vector that embeds class-specific shape priors. Class-agnostic shape priors are modeled by a set of learnable shape primitives that encode universal 3D shape information shared across classes. Later, we combine the input image with class-specific prototypes and class-agnostic shape primitives to guide the point cloud generation process. Experiments on the popular ModelNet and ShapeNet datasets demonstrate that our method outperforms state-of-the-art methods in the few-shot setting.

The advancements in terms of networking, image resolution, computer vision (CV) and mobile cloud computing performances are transforming Mobile Augmented Reality (MAR) into a technology which may be put to good use in a variety of everyday contexts. To make this point, we here consider an artisanal craft rooted back into the past, key locksmithing, and show how today MAR capabilities may simplify and ameliorate the performances of such ancient trade. To this aim, we introduce the requirements posed by such craft and present a MAR-based workflow which may be implemented to support and speed its execution. This could also impact the everyday lives of ordinary citizens, since it pose the bases of remote locksmithing activities.

Augmented Reality (AR) enables smartphone users to interact with virtual content spatially overlaid on a continuously captured physical world. Under the current permission enforcement model in popular operating systems, AR apps are given Internet permission at installation time, and request camera permission and external storage write permission at runtime through a user's approval. With these permissions granted, any Internet-enabled AR app could silently collect camera frames and derived visual information for malicious intent without a user's awareness. This raises serious concerns about the disclosure of private user data in their living environments. To give users more control over application usage of their camera frames and the information derived from them, we introduce LensCap, a split-process app design framework, in which the app is split into a camera-handling visual process and a connectivity-handling network process. At runtime, LensCap manages secured communications between split processes, enacting fine-grained data usage monitoring. LensCap also allows both processes to present interactive user interfaces. With LensCap, users can decide what forms of visual data can be transmitted to the network, while still allowing visual data to be used for AR purposes on device. We prototype LensCap as an Android library and demonstrate its usability as a plugin in Unreal Engine. Performance evaluation results on five AR apps confirm that visual privacy can be preserved with an insignificant latency penalty (

The paper discusses the creation of a new laser projection system intended for continuous operation. Appropriate application scenarios are considered and some of the problems related to the design and implementation of the system are discussed. The paper proposes and tests an innovative technical approach to the control of the thermoelectric cooling module used in the system. A new electronic unit is designed, developed and tested. It is a major functional block of the system and its main tasks are the temperature regulation, the control of the laser diode brightness and the provision of communication capabilities for remote monitoring and management. Our initial experiments show that the temperature control works very well during long-term operation. The protocol for remote monitoring and management and the desktop software program developed by us provide nice insights into the performance of our temperature management algorithms in the firmware of the system.

The current practice for planning complex cardiovascular surgeries includes printing and cutting physical heart models. Unfortunately, such cuts are permanent, thus it is not possible to interactively experiment with different cuts, slowing the planning process. In collaboration with Children’s Healthcare of Atlanta Heart Center (CHOA), we are exploring new ways to improve cardiovascular, or heart, surgical planning through augmented reality (AR). We are developing CardiacAR , an iOS AR application that enables interactive surgical planning on mobile devices. CardiacAR offers powerful and flexible tools critical for surgical planning, such as omni-directional slicing of patients’ 3D heart models and virtual annotation to assist planning. We believe the ubiquity of iOS devices will help broaden access to the CardiacAR technology and streamline its deployment.

Recent years have seen an explosion in Extended Reality (XR) experiences. Until now, researchers and developers were limited to relatively expensive, closed-source, and consumer-level hardware when creating high-fidelity XR experiences, as these were the only devices that can create a high-fidelity XR experience out of the box with minimal setup times. In this paper, we present Project Esky, a complete open-source modular software framework that enables the rapid creation of high-fidelity XR experiences on any combination of display and tracker hardware. Our framework includes several components that handle 6DOF head tracking with head pose prediction to minimize latency, hand interactions with virtual content, and network components to facilitate between-user co-located experiences. Our framework also includes an asynchronous rendering pipeline that simplifies the representation of optical distortions, as well as a simplified planar-based temporal re-projection technique to minimize rendering latency. It is our hope that the techniques described in this paper, and the open source software implementations that support it, will be a stepping stone towards bringing high-fidelity XR content to researchers and hobbyist users alike.

This demonstration presents ShowMeAround, a video conferencing system designed to allow people to give virtual tours over live 360-video. Using ShowMeAround a host presenter walks through a real space and can live stream a 360-video view to a small group of remote viewers. The ShowMeAround interface has features such as remote pointing and viewpoint awareness to support natural collaboration between the viewers and host presenter. The system also enables sharing of pre-recorded high resolution 360 video and still images to further enhance the virtual tour experience.

People with visual impairments face challenges in scene and object recognition, especially in unknown environments. We combined the mobile scene detection framework Apple ARKit with MobileNet-v2 and 3D spatial audio to provide an auditory scene description to people with visual impairments. The combination of ARKit and MobileNet allows keeping recognized objects in the scene even if the user turns away from the object. An object can thus serve as an auditory landmark. With a search function, the system can even guide the user to a particular item. The system also provides spatial audio warnings for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can find items without visual feedback using the proposed application. The study also reveals that the range of local object detection through MobileNet-v2 was insufficient, which we aim to overcome using more accurate object detection frameworks in future work (YOLOv5x).

This demonstration showcases the improvements made to Project Esky[2], an open source Extended Reality (XR) software framework capable of high fidelity natural hand-interactions with virtual content. In combination with showing our temporal re-projection rendering technique in realtime, we also showcase several live desktop demonstrations, such as a virtually controlled powerpoint presentation, an e-learning example with an animated car engine, a fire simulation and a model painting application. It is our hope that the demonstrations presented will inspire others to work with Project Esky to build their own experiences, and will be a stepping stone towards bringing high-fidelity XR content to researchers and hobbyists.

Virtual and augmented reality offer comparative performance in terms of remote usability testing to lab-based co-located settings. However, direct contact with a researcher is still required to provide setup, troubleshoot, and training. In this paper, we present Surrogate-Aloud as a remote ideation and usability method that establishes a surrogate relationship between participants and a facilitating researcher through video conferencing. The researcher wears a VR headset and shares their viewpoint through video conferencing to a remote participant, who applies think-aloud protocol to express movement and interaction commands to be executed by the researcher, alongside their thought process as they interact with virtual prototypes or scenarios. We conducted a preliminary study to evaluate the Surrogate-Aloud method for remote usability evaluation and ideation of a new instructional technique with volumetric recordings. Results show that Surrogate-Aloud leverages the surrogate’s technical expertise and enables sufficient capability to conduct truly remote usability evaluation and ideation.

Over the past several years artificial intelligence (AI) techniques have gained a considerable presence in immersive media. From assistance with real-time digital production to emerging novel forms of creative expression, applications of AI are becoming ubiquitous in the fields of human-computer interaction (HCI), computer graphics, and media art. As we are interested in how novel computational techniques are shaping the state of creativity in immersive and interactive technologies, we organize this special interest group (SIG) to stimulate a discussion among AI, HCI, immersive media, and art communities. The goal of this SIG includes outlining existing and emerging areas of cross-disciplinary collaborations, proposing a roadmap of future goals and challenges for creative immersive AI research, and establishing a diverse group of researchers and practitioners involved in creative applications of AI in immersive and interactive media. Refereed/Peer-reviewed

HCI and social science experimentation that explores or uses extended reality (XR) has been particularly impacted by the recent Covid-19 pandemic. This is due to typical deployment of XR experiments inside laboratories, and a paucity of research into how to effectively conduct remote XR experimentation. This first CHI Remote XR workshop aims to explore the current state of the art around three main themes of remote XR experimentation: (i) participant recruitment and screening; (ii) data collection, including limitations and affordances of existing research and XR tools; and (iii) software frameworks and requirements for the effective design of encapsulated remote XR user studies. This workshop brings together researchers and practitioners in XR to explore these recently emerged themes and to imagine how effective future remote XR research might be conducted.

Distance learning is facing a critical moment finding a balance between high quality education for remote students and engaging them in hands-on learning. This is particularly relevant for project-based classrooms and makerspaces, which typically require extensive trouble-shooting and example demonstrations from instructors. We present RobotAR, a teleconsulting robotics toolkit for creating Augmented Reality (AR) makerspaces. We present the hardware and software for an AR-compatible robot, which behaves as a student’s voice assistant and can be embodied by the instructor for teleconsultation. As a desktop-based teleconsulting agent, the instructor has control of the robot’s joints and position to better focus on areas of interest inside the workspace. Similarly, the instructor has access to the student’s virtual environment and the capability to create AR content to aid the student with problem-solving. We also performed a user study which compares current techniques for distance hands-on learning and an implementation of our toolkit.

Edge computing brings services and data closer to their users. It targets applications where low latency is key, such as cloud-based gaming, augmented reality, and connected cars. In these scenarios mobile users are often the norm, and services need to follow their users to continuously ensure the benefits of edge computing. Mobile edge computing addresses this demand, however, we identified three open challenges: First, edge computing facilities are sparse so far and achieving good connectivity requires a seamless transfer of workloads between different edge providers. Second, migration of edge services needs to be lightweight to make relocation transparent to users. Third, using geo-distributed edge resources of various providers demands for security measures to ensure service integrity and confidentiality.; B@We present E'dgedancer, a platform that offers infrastructure support for portable, provider-independent, and secure migration of edge services. E'dgedancer offers a lightweight and generic execution environment by utilising WebAssembly, which features an efficient, easy to transfer bytecode format. To make edge services self-migratable, and independent from the provider, E'dgedancer supports the notion of mobile agents offering weak and strong migration support. It utilises trusted execution to ensure the security of edge services during execution and migration. Finally, we shows that E'dgedancer features a lower migration time compared to previously proposed virtual machine migration while offering better security and platform independence.

People with disabilities rarely visit museums due to mobility issues in reaching and navigating museum buildings and difficulties in accessing artworks. This work addresses the latter problem, considering people with low vision. To this end, we present MusA, an inclusive mobile app aimed at providing interactive artwork descriptions to museum visitors in AR. This contribution describes the mobile app design process, consisting in two iterations with end-users. The evaluation, conducted with participants with low vision, shows that AR is an effective support for accessing visual artworks. MusA is more engaging than traditional audio guides and it is also highly usable. It provides useful functionalities and it raises interest of low vision people in visiting museums.

Video sharing (e.g., YouTube, Vimeo, Facebook, TikTok) accounts for the majority of internet traffic, and video processing is also foundational to several other key workloads (video conferencing, virtual/augmented reality, cloud gaming, video in Internet-of-Things devices, etc.). The importance of these workloads motivates larger video processing infrastructures and – with the slowing of Moore’s law – specialized hardware accelerators to deliver more computing at higher efficiencies. This paper describes the design and deployment, at scale, of a new accelerator targeted at warehouse-scale video transcoding. We present our hardware design including a new accelerator building block – the video coding unit (VCU) – and discuss key design trade-offs for balanced systems at data center scale and co-designing accelerators with large-scale distributed software systems. We evaluate these accelerators “in the wild" serving live data center jobs, demonstrating 20-33x improved efficiency over our prior well-tuned non-accelerated baseline. Our design also enables effective adaptation to changing bottlenecks and improved failure management, and new workload capabilities not otherwise possible with prior systems. To the best of our knowledge, this is the first work to discuss video acceleration at scale in large warehouse-scale environments.

Augmented Reality (AR), with its potential to bridge the virtual and real environments, creates new possibilities to develop more engaging and productive learning experiences. Evidence is beginning to emerge that this sophisticated technology offers new ways to improve the learning process for better interaction and engagement with students. Recently, AR has garnered much attention as an interactive technology that facilitates direct interaction with virtual objects in the real world. These virtual objects can be surrogates for real world teaching resources, allowing for virtual labs. Thus AR could allow learning experiences that would not be possible in impoverished educational systems around the world. Interestingly though, concepts such as virtual hand interaction and techniques such as machine learning are still not widely investigated in the domain of AR learning. The need for touchless interaction technologies has exceptionally increased in this post-COVID world. There are also existing pedagogical approaches that have not been explored in great detail in this new medium, such as Kinesthetic learning or ”Learning by Doing”. Using the touchless interaction hand interaction technology and machine learning agents, this research aims to address this gap by exploring these underutilised technologies to demonstrate the efficiency of AR learning. It will explore the different hand tracking APIs to integrate the virtual hand interaction, testing the devices’ compatibility with these APIs and integrating machine learning agents using reinforcement learning to develop an AR learning framework that can provide more productive and interactive learning experiences.

Extended Reality (XR) devices have great potential to become the next wave in mobile interaction. They provide powerful, easy-to-use Augmented Reality (AR) and/or Mixed Reality (MR) in conjunction with multimodal interaction facilities using gaze, gesture, and speech. However, current implementations typically lack a coherent semantic representation for the virtual elements, backend-communication, and dialog capabilities. Existing devices are often restricted to mere command and control interactions. To improve these shortcomings and realize enhanced system capabilities and comprehensive interactivity, we have developed a flexible modular approach that integrates powerful back-end platforms using standard API interfaces. As a concrete example, we present our distributed implementation of a multimodal dialog system on the Microsoft Hololens®. It uses the SiAM-dp multimodal dialog platform as a back-end service and an Open Semantic Framework (OSF) back-end server to extract the semantic models for creating the dialog domain model.

Augmented Reality (AR) learning has successfully been proven to support new learning pedagogies. The pedagogical approach of Kinesthetic Learning or "Learning by Doing" using AR combined with virtual touchless hand interaction and machine learning agents has not been explored in great detail. Fundamentally this exploration of touchless technologies and learning intelligence has taken on new importance in the new post-COVID world which created a need for more interactive and personalized learning solutions. This research presented the implementation of touchless hand interaction and machine learning agents in an AR learning application for teaching the PC assembly in the resource constraint environment.

Intelligent Computing for Interactive System Design provides a comprehensive resource on what has become the dominant paradigm in designing novel interaction methods, involving gestures, speech, text, touch and brain-controlled interaction, embedded in innovative and emerging human–computer interfaces. These interfaces support ubiquitous interaction with applications and services running on smartphones, wearables, in-vehicle systems, virtual and augmented reality, robotic systems, the Internet of Things (IoT), and many other domains that are now highly competitive, both in commercial and in research contexts. This book presents the crucial theoretical foundations needed by any student, researcher, or practitioner working on novel interface design, with chapters on statistical methods, digital signal processing (DSP), and machine learning (ML). These foundations are followed by chapters that discuss case studies on smart cities, brain–computer interfaces, probabilistic mobile text entry, secure gestures, personal context from mobile phones, adaptive touch interfaces, and automotive user interfaces. The case studies chapters also highlight an in-depth look at the practical application of DSP and ML methods used for processing of touch, gesture, biometric, or embedded sensor inputs. A common theme throughout the case studies is ubiquitous support for humans in their daily professional or personal activities. In addition, the book provides walk-through examples of different DSP and ML techniques and their use in interactive systems. Common terms are defined, and information on practical resources is provided (e.g., software tools, data resources) for hands-on project work to develop and evaluate multimodal and multi-sensor systems. In a series of short additions to each chapter, an expert on the legal and ethical issues explores the emergent deep concerns of the professional community, on how DSP and ML should be adopted and used in socially appropriate ways, to most effectively advance human performance during ubiquitous interaction with omnipresent computers. This carefully edited collection is written by international experts and pioneers in the fields of DSP and ML. It provides a textbook for students and a reference and technology roadmap for developers and professionals working on interaction design on emerging platforms.

Travelling is one of the most popular activity among people. With the advancement of technology nowadays, there are many applications that has been implemented to support the traveler such as smart tour guide system. Smart tour guide is a system which used to guide and give information about the tourism spot to tourist or traveler. As for current Augmented Reality (AR) application has been implemented in tourism to provide some historical or cultural information through text, picture or even virtual object form. Although the existing application is good enough for tourist usage, however there are some improvement that can be done to increase the efficiency and user interactivity, decrease the task and time spend for the trip. Therefore, this research gathered the requirement of the smart tour guide application with AR. This study gathered the requirement through preliminary study and survey result conducted in UCSI University, Kuala Lumpur, Malaysia comprises of 28 female and 22 male respondents with 39 respondents in the age of 18-23 years old, 9 people in 24-29 years old and 2 people are in 30-35 years old. The result shows although the existing application such as Google Map, Waze, Trip Advisor, and Klook satisfy most of the traveler respondents, the respondents still requires different application to support their trip especially for providing navigation and tour search services. As the result of the survey and preliminary study, most of participants prefer to have smart tour guide application with AR.

Three-Dimensional (3D) scene reconstruction using depth cameras is ubiquitous in Augmented Reality, Robotics, and Medical Imaging. Although many gradient-based highly computational reconstruction methods have been proposed in the literature, there is hardly any attempt at using meta-heuristic optimization techniques like the Genetic Algorithm (GA) for performing global camera pose estimation in a scene reconstruction framework. In this paper, we develop a 3D scene reconstruction framework that uses a combination of local image features and outlier removal technique (RANSAC) for performing local camera pose estimation. Further, we formulate a GA based global camera pose estimation approach. The 3D model is represented using an efficient and salable voxel-based representation. The paper presents the influence of various parameters on the local and global camera pose estimation techniques and the prescription for best-suited parameter values to achieve near-optimal performances.

When establishing a visual connection between a virtual reality user and an augmented reality user, it is important to consider whether the augmented reality user faces a surplus of information. Augmented reality, compared to virtual reality, involves two – not one – planes of information: the physical and the virtual. We propose SelectVisAR, a selective visualisation system of virtual environments in augmented reality. Our system enables an augmented reality spectator to perceive a co-located virtual reality user in the context of four distinct visualisation conditions: Interactive, Proximity, Everything, and Dollhouse. We explore an additional two conditions, Context and Spotlight, in a follow-up study. Our design uses a human-centric approach to information filtering, selectively visualising only parts of the virtual environment related to the interactive possibilities of a virtual reality user. The research investigates how selective visualisations can be helpful or trivial for the augmented reality user when observing a virtual reality user.

This research explores the utilization of a new media known as Virtual Reality (VR) and Augmented Reality (AR) to enhance interior design (ID) students’ presentation at the College of Fine Arts and Design (CFAD), University of Sharjah. Through a survey, we identify the most popular media used by the students for their presentations and investigates the potential barrier of using new media such as AR and VR technology from the students’ perspective. The study reveals that the use of VR and AR technology for project presentations is minimal, although the students did show interest in the technology. Among the identified reason for not using this technology is lack of exposure, obsolete syllabi, and lack of expertise. Finally, we propose a feasible approach to employ AR/VR for ID project presentation purposes. We believe that AR/VR is a potential tool that can open a new dimension of experiencing and engaging with visuals of design by using interactive and immersion features.

Virtual reality and teleportation, to telepresence, augmented reality, and remotely‑controlled robotics are future applications that are bound to achieve unprecedented development for society, economics and culture and to revolutionize the way we live, learn, work and play. Unfortunately, today’s Internet is simply not able to provide the stringent performance requirements needed by such applications to run smoothly and to offer a perfect quality of experience. This is due to several fundamental limitations in the design of the current network architecture and communication protocols. As a result, it is now the time to put into question the tenets of today’s Internet and think novel architecture and protocols that take into account recent technological advances in cloud computing, virtualization and network softwarization to design the future Tactile Internet. In this talk, we start by analyzing the characteristics and requirements of future networking applications and highlight the limitations of the today’s Internet architecture and protocols. We then provide an overview of FlexNGIA, a Flexible Next-Generation Internet Architecture that leverages cloud computing infrastructures, services and technologies as the keystone of the future Internet providing high performance and fully flexibility for future Internet applications and services. We also discuss through some use-cases how FlexNGIA could ensure the performance guarantees required by some of the future network applications.

Serious games have been progressing for decades in both digital and non-digital forms. Using a game approach to train and educate people is steadily rising, mainly based on the market of virtual and augmented reality. Fire training is an important subject because of participants’ safety, the required costs, the limitation of the scenarios’ complexity, and the ecological concerns. Nowadays, the reduced prices of hardware and software expedited the rise of the VR/AR immersive training experiences. The development of a virtual environment for fire training purposes is a challenging project. In this work, we present a fire spread simulation environment for training employees, in indoor fire cases, which runs in a CAVE (Cave Automatic Virtual Environment). This immersive environment has been developed in the Unity 3D engine and fire spread modeling was performed with FDS (Fire Dynamics Simulator). The developed platform allows the user to observe the fire propagation, roam through the scene, and intervene with the proper extinguisher to blow out the fire.

Project Pump fit focuses on the ways how to improve and make gym exercises, workouts more effective and less strainful. By using the method of Augmented reality(AR) in which a virtual instructor will be showing how to do the exercise and will correct if the person is doing it wrong. Giving a more close monitoring to each individual making it more efficient and useful.In this project, the research was conducted in the preliminary phase followed by literature reviews, market analysis and user surveys. The data was analysed using card sorting technique. In the ideation phase, UI had two directions to continue the project of which I choose the most affordable and versatile one. With the direction confirmed, explorations were done for the UI after Information Architecture and Wireframing. The prototype was made in Adobe XD followed by Unity for the AR and VR content. Heuristic analysis was conducted to evaluate and improve the experience as well as the visual part. Throughout the timeline of the project, a lot of challenges were faced, mainly COVID-19 being one of them. Due to which interviews were conducted through online platforms. User testing part was replaced by Heuristic Analysis There is always a scope of improvement which is noted in the conclusion.

Project Educo focuses on the upliftment of the subject History & Civics of class 10th. The subject is considered as one of the most boring subjects due to its theoretical approach and lack of interactivity. Through immersive technologies like AR and VR, the approach to the subject, as well as the number of students preferring to take the subject for further education, is supposed to increase at an exponential rate. In this project, the research was conducted in the preliminary phase followed by literature reviews, market analysis and user surveys (N=26). The data was analysed using card sorting technique. In the ideation phase, User Interface had two directions to continue the project of which we choose the most affordable and versatile one. With the direction confirmed, explorations were done for the UI after Information Architecture and Wireframing. The prototype was made in Adobe XD followed by Unity for the AR and VR content. Heuristic analysis was conducted to evaluate and improve the experience as well as the visual part.

In this work, we describe the use of a digital docent: a 3D avatar, presented using virtual and augmented reality, as a means for providing interactive storytelling experiences at a living history museum. To allow flexibility depending on the user’s location and access to technology, the app is designed to provide a common experience supporting a variety of different delivery modalities including AR devices, mobile AR, and VR on the Web.

We present a case study on the use of mixed reality (MR) spatial computing in a fully remote classroom. We conducted a 10-week undergraduate class fully online, using a combination of traditional teleconferencing software and MR spatial computing (Magic Leap One headsets) using an avatar-mediated social interaction application (Spatial). The class culminated in a virtual poster session, using Spatial in MR to present project results, and we conducted a preliminary investigation of students experiences via interviews and questionnaires. Students reported that they had a good experience using MR for the poster session and that they thought it provided advantages over 2D video conferencing. Particular advantages cited were a stronger sense that they were in the presence of other students and instructors, an improved ability to tell where others were directing their attention, and a better ability to share 3D project content and collaborate.

Many Deaf and Hard-of-Hearing (DHH) individuals rely on sign language interpreting to communicate with hearing peers. If on-site interpreting is not available, DHH individuals may use remote interpreting over a smartphone video-call. However, this solution requires the DHH individual to give up either 1) the use of one signing hand by holding the smartphone or 2) their ability to multitask and move around by propping the smartphone up in a fixed location. We explore this problem within the context of the workplace, and present a prototype hands-free device using augmented reality glasses with a hat-mounted fisheye camera and mic/speaker. To explore the validity of our design, we conducted 1) a video interpretability experiment, and 2) a user study with 18 participants (9 DHH, 9 hearing) in a workplace environment. Our results suggest that a hands-free device can support accurate interpretation while enhancing personal interactions.

Current wearable AR devices create an isolated experience with a limited field of view, vergence-accommodation conflicts, and difficulty communicating the virtual environment to observers. To address these issues and enable new ways to visualize, manipulate, and share virtual content, we introduce Augmented Augmented Reality (AAR) by combining a wearable AR display with a wearable spatial augmented reality projector. To explore this idea, a system is constructed to combine a head-mounted actuated pico projector with a Hololens AR headset. Projector calibration uses a modified structure from motion pipeline to reconstruct the geometric structure of the pan-tilt actuator axes and offsets. A toolkit encapsulates a set of high-level functionality to manage content placement relative to each augmented display and the physical environment. Demonstrations showcase ways to utilize the projected and head-mounted displays together, such as expanding field of view, distributing content across depth surfaces, and enabling bystander collaboration.

In the past 20 years the "Quality of Experience" (QoE) has increasingly become a necessary aspect to be taken into account by the multimedia industry. In fact, the QoE has allowed to extend the concept of image quality by considering other aspects to evaluate "the delight or annoyance of a customer's experiences with a service" [1]. The concept of QoE is extremely important since it has a big impact on several fields, ranging from the acquisition to the rendering of multimedia contents. Many works can be found in literature that address the problem of evaluating the QoE for 2D and 3D contents but nowadays we are facing a new challenging task: to study the QoE for immersive media. When talking about immersive media we are referring to different types of multimedia that allow the users to explore the content in their personal way thus introducing a new and important variable to be considered while evaluating the QoE. In particular, recent years have witnessed the spread of Virtual and Augmented Reality that allow the users to be involved in a more realistic and deep way in the artificial multimedia content or to interact with virtual objects [2]. In this new scenario we are moving to the concept of "personal" QoE that encompasses many open questions that are still unanswered such as: which is the impact of the rendering system on the QoE, how important are the viewing conditions, how can we account for the immersive media content way of exploration, can the study of multimedia saliency help in understanding the QoE [3]? In this talk we will address these open questions to explore what are the current research findings and trends and I will give you an insight of what will come next.

With the emergence of new 360-degree cameras, ambisonic microphones, and VR/AR display devices, more diverse multi-modal content has become available, and with it the demand for the capability of streaming 360-degree videos to enhance users? 360-multimedia experience on mobile devices such as mobile phones and head-mounted displays. The big issue for the mobile 360-multimedia delivery systems is the huge resource demand on the underlying networks and devices to deliver 360-multimedia content with high quality of experience. In this talk, we will discuss the research challenges of 360-degree video delivery systems such as the large bandwidth, low latency, users? disorientation, and cyber-sickness, and opportunities to solve these challenges including rate adaptation algorithms of tiles videos, view prediction algorithms, content navigation, enhancement of DASH streaming for 360-videos, and control of Quality of Experience (QoE) [1]. We will briefly dive into more details of the concept of navigation graphs for 360-degree videos and present the opportunity of navigation graphs to organize 360-video content that can help in viewing navigation, caching and improvements of QoE [2]. We will show how navigation graphs are serving as models for viewing behaviors in the temporal and spatial domains, and can assist with view predictions, bandwidth, and latency control. Our experimental results are encouraging [3] and support the intuition that if we can encapsulate viewing patterns of 360-degree videos into navigation graphs at multiple levels of contextual details, we will be able to stream "need-to-see" 360-content to wireless HMD devices in timely manner within bandwidth-constrained environments, and enhance viewing quality experience of 360-degree videos in augmented reality applications.

Virtual Reality (VR) and Augmented Reality (AR) technologies have become popular in recent years. Encoding and transmitting the omni-directional or $360^\circ $ video is critical and challenging for those applications. The $360^\circ $ video requires much higher bandwidth than the traditional planar video. A premium quality $360^\circ $ video with 120 frames per second (fps) and 24K resolution can easily consume bandwidth in the range of Gigabits-per-second~\cite1. On the other hand, at any given time, a user only watches a small portion of the $360^\circ$ scope within her Field-of-View (FoV). An effective way to reduce the bandwidth requirement of $360^\circ $ video is through FoV-adaptive streaming, which codes and delivers the predicted FoV region at higher quality, and discards or codes at lower quality the remaining regions. Such strategy has been quite extensively studied for video-on-demand \citefov_adapt_2,fov_adapt_3,1,tile_based_3,qian2016optimizing and live video streaming applications\citelive_1,live_2,live_3, sun2020flocking. Interactive applications, such as conferencing, gaming, and remote collaboration, can also benefit from $360^\circ $ video by creating an immersive environment for participants to interact with each other citeinteractive_gamming \citevr_conferencing \citelee2015outatime. However, realtime coding and streaming of $360^\circ $ video with extremely low latency, required for interactive applications, has not been sufficiently addressed. This work focuses on developing low-latency and FoV-adaptive coding and streaming strategies for interactive $360^\circ$ video streaming. We assume the sender and the receiver are connected by a network path with dynamically varying throughput without short-latency guarantee. The sender is either the video source, or a proxy server relaying the source video. The receiver is either the end user device that directly renders the video, or a local edge server that renders the video and transmit to the end user \citeHou2017.

Nowadays, people spend dramatically more time on watching videos through different devices. The advanced hardware technology and network allow for the increasing demands of users viewing experience. Thus, enhancing the Quality of Experience of end-users in advanced multimedia is the ultimate goal of service providers, as good services would attract more consumers. Quality assessment is thus important. The first workshop on "Quality of Experience (QoE) in visual multimedia applications" (QoEVMA'20) focuses on the QoE assessment of any visual multimedia applications both subjectively and objectively. The topics include 1)QoE assessment on different visual multimedia applications, including VoD for movies, dramas, variety shows, UGC on social networks, live streaming videos for gaming/shopping/social, etc. 2)QoE assessment for different video formats in multimedia services, including 2D, stereoscopic 3D, High Dynamic Range (HDR), Augmented Reality (AR), Virtual Reality (VR), 360, Free-Viewpoint Video(FVV), etc. 3)Key performance indicators (KPI) analysis for QoE. This summary gives a brief overview of the workshop, which took place at October 16, 2020 in Seattle (U.S.), as a half-day workshop.

XR (eXtended Reality) technologies play an increasing role in education. Physics education suggests great opportunities for the practical realization of XR teaching. Observation is one of the most important research method used by physics. The primary role of an XR technology user is to be an observer. This makes it easy to adopt them for physics learning. By this reason XR technologies turn out in the center of the physics teachers' attention. The paper deals with the methodological and practical aspects of XR observations in physics classes. It describes definitions of observation, the procedure of observations, types of XR observations, tools of XR observations, labworks based on XR observations, the opportunity of making simplest observational XR managers by students, and the students’ opinion about this version of XR technology. The wide availability of XR technologies allows us to apply them in physics classes right now. Both students and teachers can use it easily.

Battle for Survival: Reptopia is an immersive augmented reality experience using the Magic Leap One device developed by Pixeldust Studios at Pixeldust Labs. Pixeldust Studios put together this pilot project for Mandai park development and the Singapore Zoo. This pilot ‘Battle for Survival: Reptopia’ experience was specifically designed to appeal to two key personas at the Zoo: The Indulgent Parent and The Educator. This Magic Leap experience deploys critical components of the magic leap experience and API like spatial 3d environments, spatial 3d audio, controller touch and trigger and head tracking. For purposes of a public experience, we deployed basic dependable and efficient features only, that would be user friendly and easy to adopt quickly for even a novice user of the headset. The experience is comprised of 5 stations/image-based triggers that will bring to life 5 unique immersive and interactive photoreal 3d habitat worlds in the middle of your room or table, that will showcase its species in action. The experience is narrated by a 3d host character “Icon” who is a panther chameleon. He will guide the user inside the AR exhibit, sometimes in person and at times as background voice. At each station, he will introduce one or two of the reptiles/amphibians by including a brief initial non-interactive storyline that the users can watch and listen and learn various crucial info on the critters including their special skills and hunting mechanisms. Next the users will be allowed to interact with some of these qualities for a couple minutes using the Magic Leap controller and finally they will get to play a fun immersive game for some stations. Below is a brief breakdown of each of the stations AR activities. Each station roughly takes about 3-7 minutes to complete.

This work presents a process to create an Augmented Reality (AR) smartphone application which aims to enhance the way that contents of posters are delivered to viewers. Background and literature reviews provide context for the development of this project. The project consists of two parts: the poster and the mobile application. The app is created in Unity using the Vuforia engine. User will be able to download and install the app. This will allow any viewer to interact with the poster via AR technology.

Due to the rising popularity of streaming services, television networks are experiencing pressure to keep the attention of the younger audience. Especially in the field of Edutainment, platforms like YouTube or TED are serious competitors and require broadcasters to come up with novel ideas to engage viewers in their program. In this work, we present the augmented reality (AR) SpaceStation application, designed to supplement the viewing of educational videos about the ISS. We evaluated users’ experience during the interaction with the app in a within-subject user study (N = 31) and assessed their workload. During the interaction with the SpaceStation App, participants experienced a higher workload compared to a video-only condition; nonetheless, they considered AR a valuable and enjoyable addition. This paper concludes with a discussion from the perspectives of viewers, content creators, and hosts, and states initial ideas on how to design television programs with AR content, without creating information overload.

The Product, Multimedia and Graphic design represent vital areas of the design, which human need in all daily life, as in many other areas of design, such as accessories, jewelry, clothes, fabrics, furniture, ceramics, sculpture, architecture, urban planning... etc. The Product, multimedia, and graphic design careers were greatly influenced by the emergence of new information's technologies, including virtual and augmented reality (50% of mobile users now make electronically shopping using augmented reality techniques), which contributed to the emergence of new areas of design practices and 3D modeling in the labor market, these new changes have led to the urgent need for new generation and one teamwork of product, multimedia, and graphic designers who have the new innovative capabilities and familiarity with many of modern sciences like cinematic tricks, 3D motion graphics, application programming, 3D modeling, virtual models... etc., which all are associated with such new technologies. The virtual and augmented reality makes such design careers closer to each other, so this paper discusses the expected and increased relation of product, multimedia, and graphic design professions in the light of using these new technologies in the product design, advertising, marketing... etc., and its effect on the labor market and the design process practicing..

The healthcare industry is regularly seeking for innovative ways to leverage technology in order to create or improve the experiences of their professionals and their patients. With advancements in Head-worn display (HWD) technology such as the Microsoft Hololens, we must look into the feasibility of leveraging the technology to aid us in the ongoing challenge of creating these meaningful and educational experiences. In this research, by targeting the bag and mask step in the Neonatal Resuscitation training, we evaluate the efficiency and performance of using data from sensor devices and immersive technology to help provide users with useful feedback. Two studies were conducted to evaluate the potential of these technologies as well as the most effective method of delivering the feedback. We have found that between four different stimuli, a visual stimulus resulted in the best performance in the timing of the bag compressions in both our studies with non-healthcare professionals and healthcare professionals. Based on the results of the study, we also express the importance of having a proper seal between the mask and the baby in order to achieve the appropriate pressure during manual ventilation.