Search

The internet-of-things (IoT) refers to the growing field of interconnected pervasive computing devices and the networking that supports smart, embedded applications. The IoT has multiple human-computer interaction challenges due to its many formats and interlinked components, and central to these is the need to provide sensory information and situational context pertaining to users in a more human-friendly, easily understandable format. This work addresses this by applying mixed reality toward expressing the underlying behaviors and states internal to IoT devices and IoT-enabled objects. It extends the authors’ previous research on IoT Avatars (mixed reality character representations of physical IoT devices), presenting a new head-mounted display framework and interconnection architecture. This contributes i) an exploration of mixed reality for smart spaces, ii) an approach toward expressive avatar behaviors using fuzzy inference, and iii) an early functional prototype of a hybrid physical and mixed reality IoT-enabled object. This approach is a step toward new information presentation, interaction, and engagement capabilities for smart devices and environments.

In the modern era, preoperative planning is substantially facilitated by artificial reality technologies, which permit a better understanding of patient anatomy, thus increasing the safety and accuracy of surgical interventions. In the field of orthopedic surgery, the increase in safety and accuracy improves treatment quality and orthopedic patient outcomes. Artificial reality technologies, which include virtual reality (VR), augmented reality (AR), and mixed reality (MR), use digital images obtained from computed tomography or magnetic resonance imaging. VR replaces the user's physical environment with one that is computer generated. AR and MR have been defined as technologies that permit the fusing of the physical with the virtual environment, enabling the user to interact with both physical and virtual objects. MR has been defined as a technology that, in contrast to AR, enables users to visualize the depth and perspective of the virtual models. We aimed to shed light on the role that MR can play in the visualization of orthopedic surgical anatomy. The literature suggests that MR could be a valuable tool in orthopedic surgeon's hands for visualization of the anatomy. However, we remark that confusion exists in the literature concerning the characteristics of MR. Thus, a more clear description of MR is needed in orthopedic research, so that the potential of this technology can be more deeply understood.

Critique sessions are an essential educational activity at the center of many design disciplines, especially those involving the creation of physical mockups. Conventional approaches often require the students and the instructor to be in the same space to jointly view and discuss physical artifacts. However, in remote learning contexts, available tools (such as videoconferencing) are insufficient due to ineffective, inefficient spatial referencing. This paper presents ARCritique, a mobile Augmented Reality application that allows users to 1) scan physical artifacts, generate corresponding 3D models, and share them with distant instructors; 2) view the model simultaneously in a synchronized virtual environment with remote collaborators; and 3) point to and draw on the model synchronously to aid communication. We evaluated ARCritique with seven Industrial Design students and three faculty to use the app in a remote critique setting. The results suggest that direct support for spatial communication improves collaborative experiences. Published version

Virtual and augmented reality allows the utilization of natural user interfaces, such as realistic finger interaction, even for purposes that were previously dominated by the WIMP paradigm. This new form of interaction is particularly suitable for applications involving manipulation tasks in 3D space, such as CAD assembly modeling. The objective of this paper is to evaluate the suitability of natural interaction for CAD assembly modeling in virtual reality. An advantage of the natural interaction compared to the conventional operation by computer mouse would indicate development potential for user interfaces of current CAD applications. Our approach bases on two main elements. Firstly, a novel natural user interface for realistic finger interaction enables the user to interact with virtual objects similar to physical ones. Secondly, an algorithm automatically detects constraints between CAD components based solely on their geometry and spatial location. In order to prove the usability of the natural CAD assembly modeling approach in comparison with the assembly procedure in current WIMP operated CAD software, we present a comparative user study. Results show that the VR method including natural finger interaction significantly outperforms the desktop-based CAD application in terms of efficiency and ease of use.

The brain–computer interface (BCI) technique represents one of the furthermost active interdisciplinary study domains and includes a wide knowledge spectrum from a different disciplines such as medicine, neuroscience, machine learning and rehabilitation. The motor imagery (MI) technique based on BCI has been broadly applied in rehabilitation especially for upper limb motor movement where people with disabilities need to restore or improve their walking capability. Nowadays, virtual reality is a beneficial scheme for BCI users because it proposes a relatively cost-effective, safe way for BCI users to train and explain themselves in using BCI in a computer-generated environment earlier than in a real-life scenario. Depicting the whole picture for signal processing techniques and methods utilised in MI-based BCI training environments is difficult. In addition, numerous challenges and open issues regarding signal processing and pattern recognition exist in the literature of the current topic; however, to the best of our knowledge, this is the first attempt to highlight these challenges and open issues in signal processing methods, techniques and pattern recognition in smart BCI training environments. This work illustrates the effect of the theoretical perspectives associated with BCI works for research development in smart training environments. Consequently, this research copes with these issues via a systematic review protocol to help the large community of BCI users, especially people with disabilities. Fundamentally, four substantial databases, namely, IEEE, ScienceDirect, Scopus and PubMed contain a considerable amount of technical and scientific articles relevant to smart BCI training systems. A set of 375 articles is collected from 2010 to 2020 to reveal a clear picture and a better understanding of all the academic literature through a final set of 25 articles. In addition, this research provides the state of the art for signal processing, feature extraction, classification techniques and smart training environment characteristics for MI-based BCI applications. This study also reports the challenges and issues identified by the researchers as well as recommended solutions to solve the persistent problems. This study introduces the state-of-the art virtual and augmented reality environments as a smart platform and the neurofeedback schemes used for MI-based smart BCI training systems. Moreover, this study highlights for the first time 10 concepts of smart training in a virtual environment applied in MI and BCI, and investigates the evaluation of these concepts against the literature to gain only 45.55%. Collectively, the implication of this study will offer the opportunity of deploying an efficient smart BCI training system in terms of data acquisition and recording, pattern recognition and smart environment for BCI users and rehabilitation programmes.

The purpose of this article is to discuss how to improve the ways the Taiwanese Ground Forces use the common operational picture (COP) for training to improve the efficacy of tactical training and ...

Visualization can be a motivating way to teach students about molecules. Nowadays, the available experimental data and accurate computational results allow students to build realistic and accurate molecular models. These models include the representation of complex systems such as proteins, membranes, or nanotubes. However, the visualization of these three-dimensional (3D) structures can be challenging, complex, or even abstract for most people. To make the visualization easier and more motivating, BioSIM Augmented Reality (BioSIMAR) was developed as a free online software program to provide a new way to visualize and interact with 3D molecular models in a virtual environment. Using augmented reality (AR) and quick response code technologies, BioSIMAR provides tangible models of molecules to help illustrate chemical concepts. BioSIMAR runs on any electronic device, including mobile phones, tablets, laptops, or desktop computers with a camera and an internet connection, without requiring the installation of additional software. This software program helps people visualize and understand concepts about atoms and molecules, and their characteristics. BioSIMAR is freely available at https://ar.biosim.pt/.

BACKGROUND AND PURPOSE: Physician training and onsite proctoring are critical for safely introducing new biomedical devices, a process that has been disrupted by the pandemic. A teleproctoring concept using optical see-through head-mounted displays with a proctor's ability to see and, more important, virtually interact in the operator's visual field is presented. MATERIALS AND METHODS: Test conditions were created for simulated proctoring using a bifurcation aneurysm flow model for WEB device deployment. The operator in the angiography suite wore a Magic Leap-1 optical see-through head-mounted display to livestream his or her FOV to a proctor's computer in an adjacent building. A Web-based application (Spatial) was used for the proctor to virtually interact in the operator's visual space. Tested elements included the quality of the livestream, communication, and the proctor's ability to interact in the operator's environment using mixed reality. A hotspot and a Wi-Fi-based network were tested. RESULTS: The operator successfully livestreamed the angiography room environment and his FOV of the monitor to the remotely located proctor. The proctor communicated and guided the operator through the procedure over the optical see-through head-mounted displays, a process that was repeated several times. The proctor used mixed reality and virtual space sharing to successfully project images, annotations, and data in the operator's FOV for highlighting any device or procedural aspects. The livestream latency was 0.71 (SD, 0.03) seconds for Wi-Fi and 0.86 (SD, 0.3) seconds for the hotspot (P = .02). The livestream quality was subjectively better over the Wi-Fi. CONCLUSIONS: New technologies using head-mounted displays and virtual space sharing could offer solutions applicable to remote proctoring in the neurointerventional space.

Biometrics In conjunction with the new development of the Internet of Things (IoT), augmented reality (AR) systems are evolving to visualize 3D virtual models of the real world into an intelligent and interactive virtual reality environment that facilitates physical identification of objects and defines their specifications efficiently. The integration between AR and IoT in a complementary way helps identify network-related items' specifications and interact with the Internet of Things more efficiently. An identity is a dedicated, publicly known attribute or set of names for an individual device. Typically, identifiers operate within a specific area or network, making it difficult to identify things globally. This paper explores the use of Augmented Reality (AR) Technology for identifying devices and displaying relevant information about the device to the user. Based on the developed model network, the developed system of identification of IoT devices was tested. Also, the traffic generated by the AR device when generating requests to the organization server was investigated. According to the test results, the system is undemanding to the main network indicators. The system-generated traffic is self-similar. The test results show that the server software can solve the problems of identifying IoT devices through interaction with augmented reality devices.

Augmented reality to qualify a new form of interaction between the user and the machine based on the association of real objects, from the user's environment, and virtual. In Morocco the use of this technology in the learning context remains in the embryonic phase that tries to overcome several issues (Profile of the designers, Users, and the learning scenario). Scripting is primarily a work of content design, resource organization, activity planning and mediations to induce and accompany learning. The integration of augmented reality (AR) requires the contribution of the different specialists who work on the design and realization of the scenario. While the CRMEF AR project. CRMEF is at the end of a first phase of development, this contribution answers the following research question: What are the steps of the scenario of learning with augmented reality? How to harmonize the mastery of data integration in scientifically validated 3D in the context of learning and the use of learners? For this purpose, the project aims to model and simplify the integration of AR in Regional Centers for Education and Training through anAR BANK OF OBJECTS validated by the National Scientific Committeein 3D format.

The development of information and communication technology has an impact on globalization which will affect all aspects of human life, especially in the world of education. This requires educators to make innovation in learning. One of the innovations that following technological developments, namely Augmented Reality (AR). Augmented Reality is a technique of combining real and virtual worlds and is based on 3-Dimensions (3D). The objects used can be 2D or 3D. This research is entitled Augmented Reality for Building Materials using Unity 3D, Vuforia SDK, and Blender Applications. In this research, the application was developed by involving 3 main software, namely vuforia, unity 3D, and Blender. Vuforia is a plugin for a software called unity 3D. Augmented Reality and Unity 3D are two platforms that make it easier for users to produce 3D graphics that will be displayed if the graphic that has been used as a marker is highlighted with a camera from an SP. The results of this study have succeeded in developing applications with building markers using vuforia, unity 3D, and Blender.

Augmented reality (AR) is a technology that allows overlaying of virtual elements on top of the physical environment. This enhances the perception and conveys additional information to the user. With the emergence of industry 4.0 concepts in manufacturing landscape, AR found its way to improve existing Human-Machine Interfaces (HMI) on the shop-floor. The industrial setting has a wide variety of application opportunities from AR, ranging from training and digital work instructions to quality inspection and remote maintenance. Even though its implementation in the industry is rising in popularity, it is still mainly restricted to large companies due the limited availability of resources in Small and Medium Size Enterprises (SME). However, SMEs can benefit from AR solutions in its production processes. Therefore, this research aims to develop and present the results of comparison of two simple and cost-effective AR software frameworks for Hand Held Device (HHD) and a Head Mounted Device (HMD), which can be applied for developing AR applications for manufacturing. Two AR applications are developed using these software frameworks which are presented in the case study section. Android device is chosen as a HHD and HoloLens is the HMD used in the case study. The development structure can be reproduced by a wider range of enterprises with diverse needs and resource availability.

Software architecture and applications for the connected car can process and share a variety of digital content, among which high-definition video and augmented reality (AR) content, toward enhanced driving assistance, navigation and infotainment systems and services. However, several technical challenges need to be overcome to make such systems and services viable and efficient, including dealing effectively with a variety of types of systems, devices and platforms, either installed inside the vehicle or represented by the personal mobile and wearable devices of the drivers and passengers. In this paper, we outline these technical challenges and propose a software solution in the form of an event-based middleware layer by modeling the smart, connected car as a specific type of a smart environment. We employ an adapted version of Euphoria, a recently introduced software architecture for general-purpose smart environments, to implement asynchronous communications among heterogeneous input/output devices inside the vehicle. We also adapt Euphoria to fit into the four-layer infrastructure model of the connected car. We conduct a technical evaluation of the request-response time performance achieved with the Euphoria middleware for streaming digital content from 1 Mbps (360p@30fps) to 32 Mbps (4K@30fps) on various devices, either integrated in the vehicle, not integrated but used inside the vehicle and devices outside the vehicle (the control condition). Our results show effective live streaming achieved for 2K content at 30fps with the 600 Mbps network (i.e., the connected car) and for 4K content at 30fps with the 1.7 Gbps network envisioned for hyper-connected vehicles. These results open up opportunities for high-definition video and AR applications in the automotive industry.

Recovering human pose from RGB images and videos has drawn increasing attention in recent years owing to minimum sensor requirements and applicability in diverse fields such as human-computer interaction, robotics, video analytics, and augmented reality. Although a large amount of work has been devoted to this field, 3D human pose estimation based on monocular images or videos remains a very challenging task due to a variety of difficulties such as depth ambiguities, occlusion, background clutters, and lack of training data. In this survey, we summarize recent advances in monocular 3D human pose estimation. We provide a general taxonomy to cover existing approaches and analyze their capabilities and limitations. We also present a summary of extensively used datasets and metrics, and provide a quantitative comparison of some representative methods. Finally, we conclude with a discussion on realistic challenges and open problems for future research directions.

This work addresses recent research in the area of pedestrian navigation aids that aims at finding alternatives to the widely used map-based turn-by-turn navigation systems in the context of Smart City environments. Four different approaches of pedestrian navigation systems were compared to each other in a user experiment that was conducted in a virtual environment: (1) map-based, (2) landmark-based, (3) augmented reality, and (4) public display navigation. The results of the experiment with 45 participants conducted in a virtual environment suggest that the augmented reality navigation performs best concerning efficiency and effectiveness and the landmark-based navigation performs worst in the context of Smart Cities.

In the paper, the author notes that the development of modern technologies, including artificial intelligence, unmanned transport, robotics, portable and embedded digital devices, already has a great impact on the daily life of a person and can fundamentally change the existing social order in the near future.Virtual reality as a technology was born in the cross-section of research in the field of three-dimensional computer graphics and human-machine interaction. The spectrum of mixed reality includes the real world itself, the one that is before our eyes, the world of augmented reality — an improved reality that results from the introduction of sensory data into the field of perception in order to supplement information about the surrounding world and improve the perception of information; the world of virtual reality, which is created using technologies that provide full immersion in the environment. In some studies, augmented virtuality is also included in the spectrum, which implies the addition of virtual reality with elements of the real world (combining the virtual and real world).The paper substantiates the conclusion that in the near future both the legislator and judicial practice will have to find a balance between the interests of the creators of virtual worlds and virtual artists exclusive control over their virtual works, on the one hand, and society in using these virtual works and their development, on the other hand. It is necessary to allow users to participate, interact and create new forms of creative expression in the virtual environment.The author concludes that a broader interpretation of the fair use doctrine should be applied in this area, especially for those virtual worlds and virtual objects that imitate the real world and reality. However, it is necessary to distinguish between cases where the protection of such objects justifies licensing and those where it is advisable to encourage unrestricted use of the results for the further development of new technologies.

This research discusses the use of Augmented Reality, Virtual Reality and Mixed Reality technology applications in the learning process of relevant content to the Computer Science area. This systematic review aims to identify applications that use technologies to represent virtual environments and support the teaching and learning of Computer Science subjects. A protocol was elaborated and executed, resulting in the final selection of 14 papers from four databases, published from 2010 to 2018. The examined papers presented information that categorized technology applications in terms of tools used. Contents addressed to the identification of applied instructional strategies and techniques, and the recognition of effects on the learning process. As a result, we found virtual environments that show potential to teaching basic content in courses related to Computer Science. In addition, the application of virtual environments in this educational scenario has provided positive effects on the learning process, such as increased interactivity, easier content absorption, increased motivation and interest in the subjects, providing greater understanding and improving efficiency in content transmission.

With the advent of the 5G and Artificial Intelligence of Things (AIoT) era, related technologies such as the Internet of Things, big data analysis, cloud applications, and artificial intelligence have brought broad prospects to many application fields, such as smart homes, autonomous vehicles, smart cities, healthcare, and smart campus. At present, most university campus app is presented in the form of static web pages or app menus. This study mainly developed a Deep Neural Network (DNN) based emotionally aware campus virtual assistant. The main contributions of this research are: (1) This study introduces the Chinese Word Embedding to the robot dialogue system, effectively improving dialogue tolerance and semantic interpretation. (2) The traditional method of emotion identification must first tokenize the Chinese sentence, analyze the clauses and part of speech, and capture the emotional keywords before being interpreted by the expert system. Different from the traditional method, this study classifies the input directly through the convolutional neural network after the input sentence is converted into a spectrogram by Fourier Transform. (3) This study is presented in App mode, which is easier to use and economical. (4) This system provides a simple voice response interface, without the need for users to find information in complex web pages or app menus.

Based on literature review it was established that the use of augmented reality as an innovative technology of student training occurs in following directions: 3D image rendering; recognition and marking of real objects; interaction of a virtual object with a person in real time. The main advantages of using AR and VR in the educational process are highlighted: clarity, ability to simulate processes and phenomena, integration of educational disciplines, building an open education system, increasing motivation for learning, etc. It has been found that in the field of physical process modelling the Proteus Physics Laboratory is a popular example of augmented reality. Using the Proteus environment allows to visualize the functioning of the functional nodes of the computing system at the micro level. This is especially important for programming systems with limited resources, such as microcontrollers in the process of training future IT professionals. Experiment took place at Borys Grinchenko Kyiv University and Sumy State Pedagogical University named after A. S. Makarenko with students majoring in Computer Science (field of knowledge is Secondary Education (Informatics)). It was found that computer modelling has a positive effect on mastering the basics of microelectronics. The ways of further scientific researches for grounding, development and experimental verification of forms, methods and augmented reality, and can be used in the professional training of future IT specialists are outlined in the article.

Product assembly simulation is considered as one of the key technologies in the process of complex product design and manufacturing. Virtual assembly realizes the assembly process design, verification, and optimization of complex products in the virtual environment, which plays an active and effective role in improving the assembly quality and efficiency of complex products. In recent years, augmented reality (AR) and digital twin (DT) technology have brought new opportunities and challenges to the digital assembly of complex products owing to their characteristics of virtual reality fusion and interactive control. This paper expounds the concept and connotation of AR, enumerates a typical AR assembly system structure, analyzes the key technologies and applications of AR in digital assembly, and notes that DT technology is the future development trend of intelligent assembly research. Keywords: Augmented reality, Digital twin, Digital assembly, Virtual simulation, Assembly analysis

Virtual Humans are becoming a commodity in computing technology and lately have beenutilized in the context of interactive presentations in Virtual Cultural Heritage environments and exhibitions. To this end, this research work underlines the importance of aligning and fine-tuning Virtual Humans’ appearance to their roles and highlights the importance of affective components. Building realistic Virtual Humans was traditionally a great challenge requiring a professional motion capturing studio and heavy resources in 3D animation and design. In this paper, a workflow for their implementation is presented, based on current technological trends in wearable mocap systems and advancements in software technology for their implementation, animation, and visualization. The workflow starts from motion recording and segmentation to avatar implementation, retargeting, animation, lip synchronization, face morphing, and integration to a virtual or physical environment. The testing of the workflow occurs in a use case for the Mastic Museum of Chios and the implementation is validated both in a 3D virtual environment accessed through Virtual Reality and on-site at the museum through an Augmented Reality application. The findings, support the initial hypothesis through a formative evaluation, and lessons learned are transformed into a set of guidelines to support the replication of this work.

Human speech often contains ambiguity stemming from the use of demonstrative pronouns (DPs), such as “this” and “these.” While we can typically decipher which objects of interest DPs are referring to based on context, modern day voice assistants (VAs – such as Google Assistant and Siri) are yet unable to process queries containing such ambiguity. For instance, to humans, a question such as “how much is this?” can be clarified through visual reference (e.g., a buyer gestures to the seller the object they would like to purchase). To bridge this gap between human and machine cognition, we built and examined a touch + voice multimodal VA prototype that enables users to select key spatial information to embed as context and query the VA. The prototype converts results of mobile, real-time object recognition and optical character recognition models into augmented reality buttons that represent features. Users can interact with and modify the selected features through a word grid. We conducted a study to investigate: 1) how touch performs as an additional modality to resolve ambiguity in queries, 2) how users use DPs when interacting with VAs, and 3) how users perceive a VA that can understand DPs. From this procedure we found that as the query becomes more complex, users prefer the multimodal VA over the standard VA without experiencing elevated cognitive load. Additionally, even though it took some time getting used to, many participants eventually became comfortable with using DPs to interact with the multimodal VA and appreciated the improved human-likeness of human-VA conversations.

We present a tool for Auto-generating Storyboard And Previz for screenwriters and filmmakers, called ASAP. Our system allows users to easily simulate their stories in the form of 3D animated/visual scenes with virtual humans in a virtual environment. We only ask users to write their script using Final Draft, an exclusive screenwriting tool, and upload them to our system. The uploaded script is parsed into paragraphs of the action, character, and dialogue. From those paragraphs (i.e., text data), our system uses a combination of deep learning, data-driven, and rule-based approaches to instantly generate virtual human’s physical motions and co-speech gestures, presenting natural behavior/dialogue scenes. Thus, users can observe automatically generated pre-visualized animations (i.e., previz) from the script and can create the storyboard by capturing scenes being played. Our ASAP can minimize time-and-money consuming and labor-intensive work in the early stages of filmmaking, and do it as soon as possible. We believe that our tool and approach have a good potential for wide dissemination in the film industry.

This research investigates the application of positive computing principles to Virtual Reality (VR) training scenarios where the Virtual Environment (VE) has not a direct influence on operator learning. We propose to place the 3D models of the only objects needed for the task in a VE consisting of 360° panoramas of natural environments. We made a preliminary evaluation of the user experience which showed that the hedonic quality is significantly higher with this VE than a 3D modeled empty room. However, we also observed a reduction of the pragmatic quality, due to potential distractions. Thus, further research is needed to demonstrate the efficacy of our positive computing approach in training against a traditional one based on the faithful 3D reproduction of the real environment.

Various redirected walking (RDW) techniques have been proposed, which unwittingly manipulate the mapping from the user’s physical locomotion to motions of the virtual camera. Thereby, RDW techniques guide users on physical paths with the goal to keep them inside a limited tracking area, whereas users perceive the illusion of being able to walk infinitely in the virtual environment. However, the inconsistency between the user’s virtual and physical location hinders passive haptic feedback when the user interacts with virtual objects, which are represented by physical props in the real environment.In this paper, we present a novel reinforcement learning approach towards RDW with passive haptics. With a novel dense reward function, our method learns to jointly consider physical boundary avoidance and consistency of user-object positioning between virtual and physical spaces. The weights of reward and penalty terms in the reward function are dynamically adjusted to adaptively balance term impacts during the walking process. Experimental results demonstrate the advantages of our technique in comparison to previous approaches. Finally, the code of our technique is provided as an open-source solution.

Bodily signals can complement subjective and behavioral measures to analyze human factors, such as user engagement or stress, when interacting with virtual reality (VR) environments. Enabling widespread use of (also the real-time analysis) of bodily signals in VR applications could be a powerful method to design more user-centric, personalized VR experiences. However, technical and scientific challenges (e.g., cost of research-grade sensing devices, required coding skills, expert knowledge needed to interpret the data) complicate the integration of bodily data in existing interactive applications. This paper presents the design, development, and evaluation of an open-source software framework named Excite-O-Meter. It allows existing VR applications to integrate, record, analyze, and visualize bodily signals from wearable sensors, with the example of cardiac activity (heart rate and its variability) from the chest strap Polar H10. Survey responses from 58 potential users determined the design requirements for the framework. Two tests evaluated the framework and setup in terms of data acquisition/analysis and data quality. Finally, we present an example experiment that shows how our tool can be an easy-to-use and scientifically validated tool for researchers, hobbyists, or game designers to integrate bodily signals in VR applications.

In remote collaboration using Augmented Reality (AR), speech and gesture are major communication methods for the general public. However, the Deaf and Hard-of-Hearing (DHH) population cannot join in the communication due to the absence of a sign language interface which is their primary language. Recent works have tried to augment spoken language with sign language animations or captions, but the research to convey sign language with spoken language is still very limited. In this paper, we propose a novel multi-modal communication system that integrates sign language translation, speech recognition, and shared object manipulation in the mobile AR environment. Though the system is currently under development, we demonstrated a rapid prototype of the telemedicine app leveraging the video prototyping method to integrate the system modules. We performed preliminary interviews about our approach with DHH users, a sign language interpreter, and a physician. We discuss the insights into the future design of the DHH communication support in the AR collaboration system. This study has a socio-cultural, economic impact on the DHH population as a barrier-free design of a remote collaboration system in a practical scenario. Another contribution of this work is that we suggested a novel user-centered system for DHH users in AR by integrating the existing technologies.

With the rapid development of virtual reality hardware, virtual tours in galleries and museums are more and more popular in recent years. However, users sometimes get lost in the virtual space and miss displaying items during free exploration of the virtual scene. Therefore, proper navigation assistance is very important to help the user finish the tour and concentrate on the exhibits. Embodied agent can help users explore the scene effectively with higher interactivity and enhance user presence, but it cannot ensure completeness of visiting.In this paper, we investigate the effect of embodied agent on providing navigation assistance in virtual environments. We focus on the motion control of the agent and experiment on different speed, stay time and interactivity settings of the agent. The result shows that interactive agent can enhance visiting completeness and user presence. With these findings, we provide important considerations for the design of embodied agent for navigation in virtual environments.

People get disoriented and detached from the real world when immersed in a virtual environment; this makes them lose track of rotation in the real world. This paper studies people’s ability to maintain perception of spatial orientation in the real environment while engaged in a virtual experience and explores how visual cues affect the results. Twelve participants performed targeting tasks with rotations, followed by pointing in a known direction to observe the error in perception of real world orientation. Error was measured in three VR environments: visual cues consistent with real world rotation; visual cues slowly changing to become inconsistent with real world; no rotational visual cues. We found that visual cues are essential for people to perceive real-world orientation and removing cues results in drastic disorientation. Moreover, altering visual cues deliberately can be used to control people’s orientation perception to disorientate people in the direction we desire; in our experiment participants did not notice this manipulation. Manipulation of the presentation of visual cues may allow designers to control, correct and manipulate people’s cognitive representation of their orientation and position not only in the virtual world, but also in the real world, be it for in-place redirection or "redirected standing", or corrective redirection for safety.

New opportunities for immersive storytelling experiences have arrived through the technology in mobile phones, including the ability to overlay or register digital content on a user’s real world surroundings, to greater immerse the user in the world of the story. This raises questions around the methods and freedom to interact with the digital elements, that will lead to a more immersive and engaging experience. To investigate these areas the Augmented Virtuality (AV) mobile phone application Home Story was developed for iOS devices. It allows a user to move and interact with objects in a virtual environment displayed on their phone, by physically moving in the real world, completing particular actions to progress a story. A mixed methods study with Home Story either guided participants to the next interaction, or offered them increased agency to choose what object to interact with next. Virtual objects could also be interacted with in one of three ways; imagining the interaction, an embodied interaction using the user’s free hand, or a virtual interaction performed on the phone’s touchscreen. Similar levels of immersion were recorded across both study conditions suggesting both can be effective, though highlighting different issues in each case. The embodied free hand interactions proved particularly memorable, though further work is required to improve their implementation, arising from their novelty and lack of familiarity.

Augmented reality (AR) is the real world to which a virtual world including images sounds, etc. Augmented reality can be defined as a medium connecting digital information with the real world spatially and temporally and is instantly interactive. Virtual reality (VR) is a new technology that allows users to be present and interact with a computer simulation environment. Most virtual reality environments are devices for human-to-computer interaction with computers. Advances in technology in today's world have made a lot of information about our environment turn into a virtual environment. A type of information that has a special place in education today is augmented reality. This form of the virtual world is a living, direct or indirect physical view that adds elements around people's real world. This paper presents the study conducted by data collection and library studies and looks at how augmented reality and virtual reality are used as digital technology in school classrooms around the world. It seems to be one of the most appropriate ways to benefit from education in the age of Covid-19 and the closure of schools in many parts of the world and forcing students to enter the virtual world, which may continue and use these methods.

We introduce Haplets, a wearable, low-encumbrance, finger-worn, wireless haptic device that provides vibrotactile feedback for hand tracking applications in virtual and augmented reality. Haplets are small enough to fit on the back of the fingers and fingernails while leaving the fingertips free for interacting with real-world objects. Through robust physically-simulated hands and low-latency wireless communication, Haplets can render haptic feedback in the form of impacts and textures, and supplements the experience with pseudo-haptic illusions. When used in conjunction with handheld tools, such as a pen, Haplets provide haptic feedback for otherwise passive tools in virtual reality, such as for emulating friction and pressure-sensitivity. We present the design and engineering for the hardware for Haplets, as well as the software framework for haptic rendering. As an example use case, we present a user study in which Haplets are used to improve the line width accuracy of a pressure-sensitive pen in a virtual reality drawing task. We also demonstrate Haplets used during manipulation of objects and during a painting and sculpting scenario in virtual reality. Haplets, at the very least, can be used as a prototyping platform for haptic feedback in virtual reality.

Civil engineering is a primary domain for new augmented reality technologies. In this work, the area of subsurface utility engineering is revisited, and new methods tackling well-known, yet unsolved problems are presented. We describe our solution to the outdoor localization problem, which is deemed one of the most critical issues in outdoor augmented reality, proposing a novel, lightweight hardware platform to generate highly accurate position and orientation estimates in a global context. Furthermore, we present new approaches to drastically improve realism of outdoor data visualizations. First, a novel method to replace physical spray markings by indistinguishable virtual counterparts is described. Second, the visualization of 3D reconstructions of real excavations is presented, fusing seamlessly with the view onto the real environment. We demonstrate the power of these new methods on a set of different outdoor scenarios Civil engineering is a primary domain for new augmented reality technologies. In this work, the area of subsurface utility engineering is revisited, and new methods tackling well-known, yet unsolved problems are presented. We describe our solution to the outdoor localization problem, which is deemed one of the most critical issues in outdoor augmented reality, proposing a novel, lightweight hardware platform to generate highly accurate position and orientation estimates in a global context. Furthermore, we present new approaches to drastically improve realism of outdoor data visualizations. First, a novel method to replace physical spray markings by indistinguishable virtual counterparts is described. Second, the visualization of 3D reconstructions of real excavations is presented, fusing seamlessly with the view onto the real environment. We demonstrate the power of these new methods on a set of different outdoor scenarios.

Over the past numerous years, the training structure is applying outdated strategies that keep schools from moving to competency-based models. Since most of the parts are instructor-oriented, the students are the detached party in the classroom which prompts less communication between students and the instructor. Augmented Reality (AR) is an interactive technology that reshapes the classroom learning environments by providing a means to interact with the course and the instructor dynamically. The learning process can be made more effective, efficient, and expressive when AR is implemented in education. Recently, many research efforts have been made for merging AR with teaching because of its ability to provide an immersive and interactive environment. Furthermore, this work discusses the benefits of AR compared to customary technical platforms (such as e-learning, MOOC courses, and virtual reality) and traditional schooling approaches (chalk and talk and old-style books). Additionally, this work presents the studies that have been carried out on AR, outlines and provides a chronological review of the different approaches of AR being executed in training and education. The paper also highlights the present state of the art research, how the technology trend has changed over the years, and which subjects and their corresponding domain have been explored.

Currently, the route planning functions in 2D/3D campus navigation systems in the market are unable to process indoor and outdoor localization information simultaneously, and the UI experiences are not optimal because they are limited by the service platforms. An ARCore-based augmented reality campus navigation system is designed in this paper in order to solve the relevant problems. Firstly, the proposed campus navigation system uses ARCore to enhance reality by presenting 3D information in real scenes. Secondly, a visual inertial ranging algorithm is proposed for real-time locating and map generating in mobile devices. Finally, rich Unity3D scripts are designed in order to enhance users’ autonomy and enjoyment during navigation experience. In this paper, indoor navigation and outdoor navigation experiments are carried out at the Lingang campus of Shanghai University of Electric Power. Compared with the AR outdoor navigation system of Gaode, the proposed AR system can achieve increased precise outdoor localization by deploying the visual inertia odometer on the mobile phone and realizes the augmented reality function of 3D information and real scene, thus enriching the user’s interactive experience. Furthermore, four groups of students have been selected for system testing and evaluation. Compared with traditional systems, such as Gaode map or Internet media, experimental results show that our system could facilitate the effectiveness and usability of learning on campus.

Many scholars and exhibition halls in recent years have become engaged in the research and development of augmented reality and virtual reality navigation systems. However, most of these navigation systems are still based on handheld mobile devices. Thus, users are likely to experience intermittent visits owing to repeated viewing of the exhibits, the navigation information on the mobile devices, and limited display and interactive functions. Therefore, this study used the Microsoft HoloLens device to implement a head-mounted museum navigation system by integrating such technologies as mixed reality, gesture recognition, and location awareness. This system presents diversified and highly interactive 3D and multimedia navigation information and supports indoor and outdoor location awareness functions to help visitors understand exhibits and exhibition venues, thus further enhancing the display and interaction functions of the mobile navigation system and increasing the pleasure of visiting.

Robots are no longer working isolated in safety fences and Human-Robot Collaboration (HRC) is becoming one of the most promising topic of research to improve the efficiency in many application scenarios. Sharing the same workspace, both human and robot should clearly understand the intentions and motions of each other, in order to enable an efficient and effective interaction. In this work we propose an AR-based system to show the robot planned motion and target to the worker. We focused on representing this information in an intuitive way for inexperienced users. We introduced a multi-modal communication feedback in order to enable the user to agree with or change the robot plan using gestures and speech. The effectiveness of the system has been evaluated with test cases performed by a group of testers with no robotic experience. The results showed that the system helped the user to better understand the robot intentions and planned motion, improving the ergonomics and trust in the interaction. Furthermore, the evaluation included the rating of the different input modalities provided, in order to compare the different ways of communication proposed.

Augmented Reality (AR) enables smartphone users to interact with virtual content spatially overlaid on a continuously captured physical world. Under the current permission enforcement model in popular operating systems, AR apps are given Internet permission at installation time, and request camera permission and external storage write permission at runtime through a user's approval. With these permissions granted, any Internet-enabled AR app could silently collect camera frames and derived visual information for malicious intent without a user's awareness. This raises serious concerns about the disclosure of private user data in their living environments. To give users more control over application usage of their camera frames and the information derived from them, we introduce LensCap, a split-process app design framework, in which the app is split into a camera-handling visual process and a connectivity-handling network process. At runtime, LensCap manages secured communications between split processes, enacting fine-grained data usage monitoring. LensCap also allows both processes to present interactive user interfaces. With LensCap, users can decide what forms of visual data can be transmitted to the network, while still allowing visual data to be used for AR purposes on device. We prototype LensCap as an Android library and demonstrate its usability as a plugin in Unreal Engine. Performance evaluation results on five AR apps confirm that visual privacy can be preserved with an insignificant latency penalty (

This paper inspects importance of augmented reality (AR) technology and 3D interior models for designing purpose. We are launching augmented reality technology for interior and exterior design. Architectural projects also require virtual information. This paper represents the use of AR technology for interior and exterior designing. AR helps in various fields like digital architectural design and construction of buildings. Finally, the purpose of this technology is to apply new way of AR to the work in interior designing, where user can observe virtual design and transform a 3D virtual environment design using information, with a active and supple user interface Can interact easily. To solve this issue, we have evolved a technique that automatically computes all the appropriate point to increase the understanding of the location design and allow the user to easily understand to this approach. This paper helps to improve and evaluate the design project; a complex decoration modeling structure is created to ensure that the user can connect with every points of the project design to achieve a perfect coordination between structure and Function. Extended materials and receptive info of adorning materials-designing-tools for simple designing be modeled using special stereoscopic tools as prototypes for the AR3D Interior design in the ARID (augmented reality interior design) technique to increase customer conception and involvement in a personalized interior and exterior design project. Hence this Technique helps to understand the use of new appearance of virtual reality (VR) application called Augmented Reality.

New geospatial technologies and ubiquitous sensing allow new insights into people’s spatial practices and experiences of public spaces. These tools offer new data streams for analysis and interpretation of social phenomena. Mobile augmented reality tools such as smartphones and wearables merge the experience of entangled online and offline spaces in citizen’s daily life. This paper demonstrates a concept that combines eye-tracking tools with innovative mapping in order to enhance the interpretability of real outdoor environmental experiences. Through videogrammetry, a participants’ head posture can be reconstructed. Subsequently the fixations measured through eye-tracking are projected onto a 3D point cloud of the surrounding environment. The presented methodological approach is implemented in the interdisciplinary project DigitAS – The Digital, Affects and Space – which investigates the perception of public places as spaces of recreation, security or fear. The project’s Mixed Methods approach combined qualitative, mobile, in-situ and reconstructive methods with eye-tracking in an outdoor setting. Potentials of the geospatial mapping concept for social science research is discussed.

Avatar use on video-conference platforms has found dual purpose in recent times as a potential method for ensuring privacy and improving subjective engagement with remote meeting, provided one can also ensure a minimal loss in the quality of social interaction and sense of personal presence. This work focuses on interactions of this sort through real-time motion captured 3D personalized virtual avatars in a 2D video-conferencing context. Our experiments were designed with the intention of exploring previously defined perceptual illusions that occur with avatar-use in Virtual and Augmented Reality settings, outside of the immersive technological domains where they are normally measured. The research described here was aimed at empirically evaluating three separate dimensions of human-avatar interaction. The first was humans-as-avatars, with experimental conditions that were designed to measure changes to subjective perceptions of self-face ownership and self-concept. The second focus was other-perception, with the unique design of the studies outlined below among the first to measure social presence in a video-call between two human-driven avatars. The third emphasis was on the experiential content involved in avatar use, as there were measurements for emotion induction, fatigue and behavior change included in the data collection. The results describe some evidence for face and body ownership, while participants also reported high levels of social presence with the other avatar, indicating that avatar cameras could be a favorable alternative to non-camera feeds in video conferencing. There were also some useful insights gained regarding emotion elicitation in non-video vs. avatar conditions, as well as avatar-induced behavior change.

This chapter leads the reader through the development of the AR-HRC system from concept and background through the design of the necessary set of interfaces required to enhance human-robot interaction. It thus began by introducing the need for human-robot collaborative teams in terms of current and emerging application spaces requiring collaboration to achieve or significantly improve outcomes. In particular, the area of space exploration will require human-robot interaction at levels well beyond current state of the art or understanding. Similar terrestrial applications are outlined that will be significantly enhanced, as well. However, it was also shown that little attention has been paid to research in this field. All of these issues provided the impetus for the creation of the Augmented Reality Human-Robot Collaboration (AR-HRC) system described here. A discussion of the related work in HRI has shown that an effective system should transfer the interaction mechanisms natural for humans to the precision required for machine information. Previous work in HRI has also shown that the autonomy level of an HRI system should be variable so that it can match the needs of a given situation. In this manner the system is able to capitalize on the problem solving skills of a human while also effectively balancing that with the speed and dexterity of a robot. Prior work in HRI also highlighted the importance of situational awareness. The lack of situational awareness has been shown to decrease performance and, in certain cases, can lead to catastrophic failures. Use of natural speech has also been shown to be effective in HRI. However, speech alone is not enough to complete the grounding process in the exchange between human and robot, leading to a reduced ability to communicate as a result. Therefore, a multi-modal interface is shown to provide a more effective approach. By combining speech with gesture, a more natural interface and the requisite grounding is achieved. The multi-modal medium used for the AR-HRC system presented here is Augmented Reality, which affords both speech and gestural communication channels. The literature review therefore includes an introduction to AR and the state of the art of AR in the context of using it in a multi-modal human-robot interaction system. AR has been shown to provide a shared work space that is conducive to collaboration and at the same time increases situational awareness, enhancing its potential in this situation. AR also supports a tangible user interface, essentially allowing a person to use a real world object to

Augmented Reality has started becoming a booming industry nowadays. Augmented reality is used for educational purposes, businesses and now in navigation of huge multi-complexes too. Augmented Reality makes work easier and more interesting with its various features, and a smaller number of hardware components at the same time. Augmented Reality when used in navigation, can provide a user with augmented directions from the user's current position to his/her desired location, in an undetermined surrounding. The aim of our project is to look for all feasible technology in terms of Indoor Navigation and present a modest prototype as a solution. For Indoor Navigation, Augmented Reality uses its inbuilt characteristics to determine a person's position and provide the necessary navigation directive. The area of the project's prototype is a huge Campus, that would provide students and guests with usability advantages and location finding features in the indoors of the college itself, particularly where the target is located. This project makes use of technologies like Unity, ARCore and Navmesh etc.

During the last decade, there has been a surge in research studies exploring the adoption of Augmented Reality (AR) in educational settings. Within these multiple research studies, AR's capability to extend the teaching and learning environment with augmented 3D learning objects with enhanced interactive capabilities have been demonstrated. This new technology has not been widely adopted in the mainstream but with the recent unprecedented circumstances of COVID-19, there has been an increasing societal willingness to adopt these technologies. AR has been a desirable technology due to its inherent touchless nature which facilitates social distancing at this time but AR applications crucially offer so much more. They can provide interactive functionality through augmentation of the teaching and learning environment within an immersive user experience including 3D interactions with learning objects, gestures, hand interaction, tangible and multi-modal interaction. This paper presents the results of a review of touchless interaction studies in educational applications and proposes the implementation of real-time touchless hand interaction within kinesthetic learning and utilization of machine learning agents. The architecture of two AR applications with real-time hand interaction and machine learning agents are demonstrated within this paper enabling engaged kinesthetic learning as an alternative learning interface.

Recent years have seen an explosion in Extended Reality (XR) experiences. Until now, researchers and developers were limited to relatively expensive, closed-source, and consumer-level hardware when creating high-fidelity XR experiences, as these were the only devices that can create a high-fidelity XR experience out of the box with minimal setup times. In this paper, we present Project Esky, a complete open-source modular software framework that enables the rapid creation of high-fidelity XR experiences on any combination of display and tracker hardware. Our framework includes several components that handle 6DOF head tracking with head pose prediction to minimize latency, hand interactions with virtual content, and network components to facilitate between-user co-located experiences. Our framework also includes an asynchronous rendering pipeline that simplifies the representation of optical distortions, as well as a simplified planar-based temporal re-projection technique to minimize rendering latency. It is our hope that the techniques described in this paper, and the open source software implementations that support it, will be a stepping stone towards bringing high-fidelity XR content to researchers and hobbyist users alike.

People with visual impairments face challenges in scene and object recognition, especially in unknown environments. We combined the mobile scene detection framework Apple ARKit with MobileNet-v2 and 3D spatial audio to provide an auditory scene description to people with visual impairments. The combination of ARKit and MobileNet allows keeping recognized objects in the scene even if the user turns away from the object. An object can thus serve as an auditory landmark. With a search function, the system can even guide the user to a particular item. The system also provides spatial audio warnings for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can find items without visual feedback using the proposed application. The study also reveals that the range of local object detection through MobileNet-v2 was insufficient, which we aim to overcome using more accurate object detection frameworks in future work (YOLOv5x).

Virtual and augmented reality offer comparative performance in terms of remote usability testing to lab-based co-located settings. However, direct contact with a researcher is still required to provide setup, troubleshoot, and training. In this paper, we present Surrogate-Aloud as a remote ideation and usability method that establishes a surrogate relationship between participants and a facilitating researcher through video conferencing. The researcher wears a VR headset and shares their viewpoint through video conferencing to a remote participant, who applies think-aloud protocol to express movement and interaction commands to be executed by the researcher, alongside their thought process as they interact with virtual prototypes or scenarios. We conducted a preliminary study to evaluate the Surrogate-Aloud method for remote usability evaluation and ideation of a new instructional technique with volumetric recordings. Results show that Surrogate-Aloud leverages the surrogate’s technical expertise and enables sufficient capability to conduct truly remote usability evaluation and ideation.

Distance learning is facing a critical moment finding a balance between high quality education for remote students and engaging them in hands-on learning. This is particularly relevant for project-based classrooms and makerspaces, which typically require extensive trouble-shooting and example demonstrations from instructors. We present RobotAR, a teleconsulting robotics toolkit for creating Augmented Reality (AR) makerspaces. We present the hardware and software for an AR-compatible robot, which behaves as a student’s voice assistant and can be embodied by the instructor for teleconsultation. As a desktop-based teleconsulting agent, the instructor has control of the robot’s joints and position to better focus on areas of interest inside the workspace. Similarly, the instructor has access to the student’s virtual environment and the capability to create AR content to aid the student with problem-solving. We also performed a user study which compares current techniques for distance hands-on learning and an implementation of our toolkit.