Search

In-lab driving simulators have been favored by human-vehicle interaction researchers and interface designers due to their capabilities in unfettered construction of surrounding environments. On-road driving simulators, meanwhile, were developed for experiments in a real-world context. We introduce new methods and tools for simulating driving via virtual vehicle in real-world environments. The proposed system fully exploits the benefits of vivid sensory information in real road environments without sacrificing the unlimited potential of environment construction for it. In this paper, we explain the system design of the platform, and we verify its capabilities as a driving simulator through a pilot study (N=10). The proposed platform is perceived as highly realistic and can easily be augmented to test new urban landscapes and futuristic interfaces before occupancies.

A serious trouble of virtual bicycle ride has to be faced currently, that is the visual information of scene motion perceived by riders doesn’t synchronize with the proprioceptive information of actual ride. This phenomenon makes riders deeply depressing in immersion and interaction. This paper proposes three different interpolation algorithms of Catmull- Rom curve to evaluate the consistency of visual and proprioceptive perceptions for virtual bicycle ride. Experimental results show that the realtime interpolation algorithm of Catmull-Rom curve shall help to match visual and proprioceptive perceptions, and improve the immersion and interaction of virtual bicycle ride.

Order picking is one of the most important tasks in modern warehouses, while augmented reality (AR) is a friendly alternative by conveying manual picking information into visual guidance to improve the picking performance. Nevertheless, there is still a lack of scalable and systematic consensus to deploy the pick-by-AR method associated with actual warehouse workplaces. This article describes a comprehensive methodology for the use of wearable AR in the order picking process. To enable a robust AR assisted manual order picking system in a warehouse floor, the marker-based global map about the actual shelves is established automatically, empowering a scalable, low-cost and long-term tracking capacity for pick-by-AR within the warehouse floor and allowing the operator to move freely in the actual workplace. The system is evaluated in the automobile assembly line, and experimental results illustrate that the proposed work can provide a systematic solution for pick-by-AR applications in actual warehouse fields.

This article shows an overview of an immersive VR application to explore, to analyze and to mark large-scale photogrammetric 3D models. The objective is to interrogate the effectiveness of the proposed navigation process and annotation tools for spatial understanding. Due to the amount of interactions, different kinds of menus were necessary: desktop coordination menu, radial menu attached to the controller and 3D spatial menu for creating markers. Besides the menus, the navigation tasks through different perception space scales required a great number of interactions metaphors, patterns and techniques displaying the complexity of the user experience in Virtual Reality for understanding and analyzing urban digital twins. Those interactions allowed by the user interface were then analysed and classifyed according to a theoretical background and were experimented in preliminary tests with end users. Although designed for particular needs of the army, the tools and interactions can be adapted for city models explorations and urban planning. For future steps of the research, a usability study is going to be performed to test the performance of the interface and to have more end users feedback.

Head mounted display (HMD) is an important part of the virtual reality (VR) system and the main interface of virtual world. During the process of using HMD, there is a certain time gap between the user's movement and the triggered change of display, which is referred to herein as a " Motion-to- Photon " latency. The latency may cause cybersickness which has an adverse effect on user experience and even physical health, so it is necessary to evaluate the latency of an HMD. We propose an automated and universal HMD latency measurement system. By simulating a user’s motion, we are able to measure "Motion-to-Photon" latency using the generated periodic signals in HMD, with accuracy and reliability confirmed by our experiments. The system is fully automatic and easy to use, and suitable for the HMD with inside-out or outside-in positioning methods. It helps to improve both the detection level of existing HMDs and the user experience. It is of guiding significance for the design and manufacture of new HMD.

In this paper, we investigate the effectiveness of augmented reality-based peephole interaction on a smartphone which fixes a virtual screen on a flat surface in real space and that enables a user to change the viewing area rapidly by moving or tilting the device. Since the virtual screen is superimposed in real space, the user can easily grasp the sense of distance to the virtual screen and move the device to a proper position rapidly. In addition, it is expected that the user can remember the positions of target objects easily by associating the objects with real space information. We conducted an experiment to compare the proposed augmented reality-based peephole interface with a conventional non-augmented reality-based peephole interface, and the conventional touchscreen interface. Although there was no significant difference in task completion time, the results of the questionnaire showed that the proposed interface and the conventional peephole interface were preferred to the conventional touchscreen interface and that it could be possible to show the effectiveness of the proposed interface in further investigations.

Water on road is a potential hazard for road vehicles. In complex traffic, it is difficult for drivers and autonomous navigation systems to safely share the road with other vehicles and pedestrians while detecting and avoiding water puddles and pot holes. Detection of water puddles needs to be both accurate and fast enough for realtime assistance or planning. We present a new fast deep-learning based water detection network that exploits temporal information to achieve an accuracy comparable to the state-of-the-art accuracy while requiring less than half memory and CUDA cores. We also show how the detection results are visualised in 3D for AR/VR applications.

Accurate indoor positioning has attracted a lot of attention for a variety of indoor location-based applications, with the rapid development of mobile devices and their onboard sensors. A hybrid indoor localization method is proposed based on single off-the-shelf smartphone, which takes advantage of its various onboard sensors, including camera, gyroscope and accelerometer. The proposed approach integrates three components: visual-inertial odometry (VIO), point-based area mapping, and plane-based area mapping. A simplified RANSAC strategy is employed in plane matching for the sake of processing time. Since Apple's augmented reality platform ARKit has many powerful high-level APIs on world tracking, plane detection and 3D modeling, a practical smartphone app for indoor localization is developed on an iPhone that can run ARKit. Experimental results demonstrate that our plane-based method can achieve an accuracy of about 0.3 meter, which is based on a much more lightweight model, but achieves more accurate results than the point-based model by directly using ARKit's area mapping. The size of the plane-based model is less than 2KB for a closed-loop corridor area of about 45m*15m, comparing to about 10MB of the point-based model.

In this paper, a novel Deep HDR Merger network, which is called MergeNet, is proposed to reconstruct a HDR image from a single filtered LDR image. Filtered images are adopted as input since they contain more dynamic range than traditional ones. By learning the correlation between filtered LDR images and HDR images, the MergeNet successfully achieves HDR reconstruction of filtered images. We used five evaluation methods to make qualitative and quantitative comparisons to show that our method produced excellent results. Experimental results show that the proposed method performs favorably against state-of-the-art HDR image reconstruction methods.

In recent years, the availability of affordable mobile Virtual Reality (VR) viewers has resulted in strong interests to incorporate Immersive Virtual Reality (IVR) within classrooms. However, studies on the effect of IVR on primary schoolers' learning are few, and they have often used equipment and settings far removed from everyday classroom instruction. We explored the role of interactivity and immersion in learning in a primary school classroom with 36 children aged 11-13 years, using commercially available devices that are ready-to-scale. We co-created content with different levels of immersion and interactivity together with teachers and investigated student engagement and learning. We present and discuss the use of multiple data sources (performance, physiological responses, observations and self-report) in a real-world classroom evaluation.

Interactive educational environments are one of the prime applications for the use of Augmented Reality (AR). A large variety of such systems has been proposed in the past for various areas of education. However, in most cases the number of users these AR systems can support is limited. Only few systems have been developed that support a large number of co-located users to jointly collaborate in a dynamic and interactive learning environment. Multi-user AR collaboration presents a unique setting with distinct challenges and requirements for user interaction and information sharing. In this paper, we present VesARlius, a novel AR system for collaborative and interactive anatomy learning in a large group of co-located users. Our system employs a set of multi-user collaboration paradigms allowing users to engage in an interactive AR learning environment. We evaluated the collaborative features of our system in a user study with 16 medical students. Results demonstrate the potential of the VesARlius system to be used effectively for large-group AR anatomy learning. From our lessons learned, we provide a set of design guidelines for developing similar AR systems to enable large-group collaboration in other application domains.

Augmented reality is a technology which extends measureless contents in the virtual space and makes actual objects not limited in the physical properties. As the most popular cultural media among the campus, the poster have its shortcoming of offering restricted information. In this paper, we introduce a responsive poster-reading system which generates diverse contents step by step with body-based and hand-based interactions, and a design guideline which applies for corresponding posters. The proposed method is able to provide users with a user-friendly, immersive and intuitive experience.

The military training systems based on augmented reality(AR) present challenges for cooperation training and confrontation training. In this paper, we propose a multi-vehicle cooperative military training simulation system based on video see-through AR. Our system combines robust tracking algorithm, realistic AR display and network module to overlay virtual forces accurately on each soldier's real-world view. We present our tracking algorithm using a monocular camera in combination with an inertial measurement unit(IMU) and GPS in a loosely coupled Extended Kalman Filter(EKF)-based framework to obtain robust and real-time pose estimation. Our AR display provides real-time dynamic painting of virtual forces, battlefield situational awareness, and visual feedback for a variety of training activities. The inserted virtual forces can move in a pre-defined trajectory and shoot real vehicles in the specified range. The network module is used to support multi-vehicle real-time cooperative training. We evaluate the accuracy of the tracking algorithm by comparing with DGPS. The consistency of rendering on different soldiers' AR displays is analyzed. In addition, we test our distinctive multi-vehicle cooperative confrontation training system online in a small team.

Previous studies have suggested that different types of human-computer interactions (HCI) have different impact on use's emotional responses in virtual environment. In this study, we investigat the effects of two different types of HCI, including controller-based interaction and speech-based interaction, on user's emotion in a virtual counseling environment. To achieve this end, we first develop a virtual counseling environment which enable each participant acted as a counselor to help a depressed virtual woman. We then assess the emotional intensity of each participant after they experienced 20-30 minutes' consultation in either controller-based or speech-based interaction condition. Our results show that, although there is no significant difference in emotional intensity between two different interaction conditions, participants in speech-based condition show relatively stronger emotion than these in controller-based condition.

In many complex tasks, a remote expert may need to assist a local user or to guide his or her actions in the local user's environment. Existing solutions also allow multiple users to collaborate remotely using high-end Augmented Reality (AR) and Virtual Reality (VR) head-mounted displays (HMD). In this paper, we propose a portable remote collaboration approach, with the integration of AR and VR devices, both running on mobile platforms, to tackle the challenges of existing approaches. The AR mobile platform processes the live video and measures the 3D geometry of the local environment of a local user. The 3D scene is then transited and rendered in the remote side on a mobile VR device, along with a simple and effective user interface, which allows a remote expert to easily manipulate the 3D scene on the VR platform and to guide the local user to complete tasks in the local environment.

Low-cost mobile mixed reality solutions for head tracking, as well as tracking of interaction targets, can be enhanced by multiple inertial measurement units (IMUs). Since microelectro-mechanical systems (MEMS) were introduced, IMUs have become smaller, they need less power and cost less. Yet, as any sensor, IMUs produce sensor error and mitigate the accuracy of the tracking system. This is particularly the case for low-cost IMUs. In this poster, we investigate whether the joint use of three IMUs can reduce the overall error. We present a solution fusing three independently calibrated low-cost IMUs on a planar and non-planar grid. We show that we can achieve the tracking quality of a single high-cost IMU this way. For the comparison, we offer several one Degree of Freedom measurement setups for mechanical rotation and translation movements. Afterwards, we discuss some concepts towards designing non-planar arrangements of Multi-IMUs in a Grid that may be suitable for HMD tracking.

This paper discusses the design of a wearable display in the form of compact eyeglasses, supporting a fair field of view, correct focus cue, and optical see-through capacity. Based on integral imaging, our proposal comprises a discrete transparent microdisplay array as the image source and a discrete lenslet array as the spatial light modulator, without the need for a pre-imaging system or special prism. We designed an annular aperture array to eliminate stray light, controlled within an imperceptible limit. Through a stray light simulation and an imaging simulation, the system was proved to provide a good image quality for both virtual and real information.

With the rapid development of relevant hardware and software, mixed reality has been widely applied in gaming, industrial production, tourism et al.. However, the environment understanding cannot fulfill the need of a mobile mixed reality application in some occasions. In this paper, we propose Perceptual MR Space, a semi-automatic environment understanding technique for complete, consistent and efficient environment understanding in mobile mixed reality applications. We use several environment reconstruction tasks to evaluate the effectiveness of the technique. A mixed reality game "Run!Tom" is developed to show the application and advantage of the technique.

In high-paced applications like air traffic control, disaster management, and medical triage, information is often routed through a central location such as a dispatch or control center. In this work, we introduce a set of color-coded and gaze-aware visualizations designed to simultaneously manage both information priority and user attention. These notifications are unique in that they not only respond to whether or not they have been observed by the user, but they use color to represent the priority of the object to assist processing of other incoming information in the environment. We built both heads-up display (HUD)-based and in-situ information management paradigms and combined these with gaze and color visualizations. In an interactive virtual reality (VR) control center simulation, HUD visualizations slightly reduced unnecessary head movement without sacrificing performance, but were distracting to participants.

Inter-brain connectivity between pairs of people was explored during a finger tracking task in the real-world and in Virtual Reality (VR). This was facilitated by the use of a dual EEG set-up that allowed us to use hyperscanning to simultaneously record the neural activity of both participants. We found that similar levels of inter-brain synchrony can be elicited in the real-world and VR for the same task. This is the first time that hyperscanning has been used to compare brain activity for the same task performed in real and virtual environments.

Over the last decade, there have been countless virtual reality rehabilitation systems make their way from the laboratories into the real world. For one reason or another, this same trend has not occurred for mixed-reality rehabilitation systems. Even less so, are either of these rehabilitation systems making their way to patients’ homes for home rehabilitation. As mixed-reality hardware becomes more easily accessible, affordable and accepted; it is becoming more feasible for mixed reality rehabilitation systems to be placed in patients’ homes. This brings researchers exciting new possibilities regarding patients’ treatments but also new challenges regarding their design/implementation. This paper will discuss adapting one such mixed-reality system, The Augmented Reflection Technology System, in order to allow patients to carry out their clinician recommended rehabilitation at their own home. We present a demonstration system that can be used for immersive home-rehabilitation and discuss future possibilities in the field.

Text entry is a challenge for Virtual Reality (VR) applications. In the context of immersive VR Head-Mounted Displays, text entry has been investigated for standard physical keyboards as well as for various hand representations. Specifically, prior work has indicated that minimalistic fingertip visualizations is an efficient hand representation. However, these representations typically require external tracking systems. Touch-sensitive physical keyboards allow for on-surface interaction, with sensing integrated into the keyboard itself. However, they have not been thoroughly investigated within VR. We close this gap by comparing text entry on a standard physical keyboard and a touch-sensitive physical keyboard in a controlled user study (n=26). Our results indicate that text entry using touch-sensitive physical keyboards can be as efficient as the fingertip visualization, but that results vary between experienced and inexperienced typists.

E-learning systems are still not very accessible for persons with disabilities, particularly those with cognitive impairments. It's well known that the training deficit is one of the cause of lower employment rates. In the past, we have addressed this issue by working on the accessibility of MOOCs. We have developed Aiana, a MOOC player with accessibility features based on the fragmentation of information streams and enabling user interface self-configuration. We are starting a new research program focused on the accessibility of immersive Serious Games for persons with cognitive impairments by first transposing some of Aiana's design principles. We believe that immersive Serious Games can provide effective assistance to learning for PWDs and we want to demonstrate this rigorously through large field studies. More generally, we wonder about the questions raised by the accessibility of Mixed Reality tools in immersive e-learning systems.

This paper presents a novel method of reconstructing indoor scenes, both structures and objects, by a single panorama photo. The method combines room structure estimation, furniture detection, models selection, as well as 3D positions reasoning. Compare with others, our preliminary results show this method could get almost the same performance with a simpler procedure.

We present a novel strategy for image-level interaction that is applicable to a single image without any prior structural knowledge, such as object status or reconstructed 3D models. By training sets of input image and interaction pairs using a target image, our model can generate result images by applying the desired interaction to new unseen images. The proposed method is differentiated from previous approaches for changing poses, which requires absolute statuses for training images or estimated 3D model with reconstruction errors. Based on the conceptual analysis of encoder-decoder networks, we propose a novel generator network architecture containing a feature converter network, which is suitable for applying interactions to images. We also implement a discriminator network for training, which is a well-known technique for generative adversarial networks. Experimental results demonstrate that the proposed method successfully generates result images with applied interactions without any prior knowledge. We expect that our method will provide insights into novel interaction schemes for augmented reality by reflecting interactions onto real scenes and providing more realistic user experiences.

Image inpainting aims to restore texture of missing regions in scene from an RGB image. In this paper, we aim to restore not only the texture but also the geometry of the missing regions in scene from a pair of RGB and depth images. Inspired by the recent development of generative adversarial network, we employ an encoder–decoderbased generative adversarial network with the input of RGB and depth image. The experimental results show that our method restores the missing region of both RGB and depth image.

For the development of industrial Augmented Reality (AR) applications beyond the scope of a marketing showcase for trade shows, the practical usability of researched technologies needs to be considered. In this paper we show the design process for a Microsoft HoloLens* application, with the goal of demonstrating the viability of AR-guidance in industrial business. The design is consistently focussing on the benefit of the application as well as usability in an industrial environment. This includes specifically aspects of image and object recognition, which are impaired in a dirty environment or with objects frequently subjected to wear and tear.

A large-scale optical tracking system is proposed in this paper. It uses synchronous laser coding to realize optical locating. Inertial sensors are integrated to assist the calculation of 6-DOF pose. The encoding of the base station in the system can be customized and we designed a specific layout of base stations to prevent signal interference. Based on the above layout scheme, the tracking range of the system can be expanded to hundreds of square meters. The tracking system proposed in this paper benefits promoting the development of large scale optical tracking equipment with low cost and high precision.

Facial expressions are an important part of human communication. However, children with autism spectrum disorders (ASD) are often suffering from difficulties of understanding non-verbal cues and form appropriate responses. Traditional approaches including labeling formatted photographs of human facial expressions from a third person's perspective could help them learn and improve such skills. Yet such training systems are often in lack of real time feedback. As Optical See-Through (OST) Augmented Reality (AR) headsets possess the advantages of near-eyes display and better depth alignment between virtual renderings and the environment, we decided on an OST AR approach for the system. In this paper, we present a system designed for OST AR headsets that occludes the subject's facial expressions with an emotion-presenting 3D emoji model. We hope this system could help us understand how children with ASD perceive emotions through standard emotion presenting systems and help them enhance their skills of understanding facial expressions.

Deaf infants born to hearing parents are at risk of language deprivation, which may lead to life-long impact on linguistic, cognitive and socio-emotional development. It remains demanding for hearing parents to provide meaningful and linguistic-rich interaction with their deaf and hard of hearing (DHH) children, due to lack of sign language fluency and insufficient communication strategies. In this study, we present a proof-of-concept visual augmentation prototype utilizing the Augmented Reality (AR) lamp metaphor that aims to support context-aware and non-intrusive parent-child interaction using American Sign Language (ASL), with adaptation to joint-attention strategies that match with the child’s communication modality. The proposed prototype enables future studies to collect in-depth design critiques and preliminary usability evaluation from domain experts, novice ASL learners, and hearing parents with DHH children.

Systems for remote collaboration on physical tasks generally use AR/VR technologies to create a shared visual space for collaborators to perform tasks together. The shared space often comes from a single camera view. Prior research has not reported on the benefits of using multiple cameras for remote collaboration. On the contrary, there seems to be some usability issues, which must be addressed, when designing remote collaboration systems that use multiple cameras to capture different areas and perspectives of a task space. To be usable, a multi-camera remote collaboration system must indicate to the local user which camera the remote user is looking at and vice versa, the system must make it fast and easy for the remote user to obtain the right camera view for a given collaborative task. We present SceneCam, an AR prototype with which we explore different techniques for improving the usability of multi-camera remote collaboration by making camera selection easier and faster. Specifically, SceneCam implements two camera selection techniques. The first technique nudges the remote user to manually select an optimal camera view of the local user's actions. The second technique automatically selects an optimal camera view of the local user and shows it to the remote user. Additionally, SceneCam implements two focus-in-context views (exocentric and egocentric views) that provide the remote user with a spatial overview of the local user's whereabouts in relation to the multiple task space areas and direct visual access to the camera views of said areas. Camera selection techniques (manual point-and-click, nudging, automatic), and focus-in-context views (no focus-in-context view, exocentric, egocentric) make up the two dimensions in a design space for multi-camera remote collaboration. We describe how SceneCam spans this design space. Lastly, as part of future work we discuss some hypotheses regarding the effects of the proposed camera selection techniques, focus-in-context views and combinations hereof on the usability of multi-camera remote collaboration.

The severity and development of psoriasis is difficult to describe and discuss between patients and doctors. Such phenomenon mainly caused by traditional medical communication methods (language and pictures) cannot visually show the disease status and the comprehension largely depend on doctors’ (variable) skills to describe the anomaly. In this paper, we propose a solution based on augmented reality (AR). Patients can see psoriasis develop on their face by making accurate and reliable psoriasis maps. Heuristic analysis and pilot testing of 24 participants demonstrated that the solution can significantly increase comprehension and promote the patient's willingness to receive medical support. We believe that AR can be an effective tool to assist the treatment of psoriasis.

Since the introduction of augmented reality (AR) technology, in-situ instructions for manual tasks have been a central use case for a large body of previous work. However, most implementations provide identical sets of instructions to each user disregarding the user's current mental load. This is a major issue since previous work has shown the importance and potential of an adapted instruction fidelity for manual tasks such as playing an instrument. To implement a low-cost mental load adaptation for AR instructions, we evaluated a mobile off-the-shelf electroencephalographic (EEG) device for its suitability and feasibility to measure mental load while wearing a video see-through AR head-mounted display (HMD). In a first user experiment (n=12), data of EEG power band values and proprietary performance metrics of the manufacturer were collected and analysed regarding their validity to estimate the user's mental load. Our results indicate that our setup successfully induced different levels of mental effort. The proprietary performance metrics, however, only partially reflected the participants' current mental effort and require further analysis.

The six degree-of-freedom (6DoF) pose estimation is an important task in Augmented Reality, especially for initializing or recovering from the failure of 3D tracking for the textureless object, since it still encounters the insufficient accuracy problems because of cluttered backgrounds, occasionally quick movements, and other factors. We propose a simple but effective method to cutout the interested textureless object in a single RGB image with clear contour, which can be employed to directly estimate 6Dof poses with relatively high precision, and then help to remove most of the disturbing edges of clutter background for further refinement of pose estimation. To achieve this task, we propose a novel convolutional neural network, similar to an autoencoder, to reconstruct arbitrary scenes containing the object of interest, and extract the object area. We evaluated our method on objects from the LINEMOD dataset, and the results show that our approach is superior to the baseline as well as some advanced methods.

We present a wearable Mixed Reality (MR) remote collaboration system called Wearable RemoteFusion. The system supports spatial annotation and view frustum sharing, and enables natural non-verbal communication cues (eye gaze and hand gesture) for visual assistance in a stitched live dense scene. We describe the design and implementation details of the prototype system, and report on a pilot user study investigating how sharing natural gaze and gesture cues affects collaborative performance and the user experience. We found that by sharing augmented natural cues like the local eye gaze and remote hand gesture, participants had a stronger feeling of Co-presence, and the remote user could guide the local user to complete tasks with less physical workload. We discuss implications for collaborative interface design and directions for future research.

Technology in education is rapidly changing the way that students learn. This allows for the creation of learning tools that provide better interaction, creative engagement and adaptability to a learner. Augmented Reality (AR) is one of these emerging technologies which can facilitate the development of new learning tools. AR has successfully been proven to allow new types of learning pedagogies by providing human-centered learning environments. In particular, the pedagogical approach of Kinesthetic learning or ”Learning by Doing” has not been explored in great detail in combination with Augmented Reality. This approach is to physically act out an activity to aid in the learning process and has been previously proven as one of the most successful approaches. For a successful application of this pedagogy, the student must get precise feedback and be guided through a process, thus some form of intelligent guide needs to be actively monitoring the learning environment. This paper presents the exploration of this concept through the presentation of an initial prototype system that was developed and implemented based on an adaptive learning methodology within an AR application, with the prospect that in the future will use intelligent agents.

Cystic fibrosis (CF) is a genetic disorder and chronic lung disease that affect between 70,000 and 100,000 people worldwide. Patients have to take medication and perform daily airway clearance exercises in order to mobilize sticky mucus from the lungs caused by CF. These exercises are rather repetitive and not very engaging causing patients to skip despite the health repercussions. In collaboration with a physiotherapist, we have developed a virtual reality diving game that diegetically incorporates the breathing exercise by translating it into movement through the underwater world. Initial user tests have shown that our combination of physiological input and virtual reality creates a compelling and immersive experience. However, it was also shown to be a task with widely differing perceived difficulty so that further steps need to be undertaken to adjust the game to various user groups.

In this paper, we introduce a hand interaction guidance system (HIGS) which can automatically author video guidance from 'one-shot' expert demonstration recording, and decompose the step instructions and deliver feedback according to the users' operation. The system observes hand-object interactions with a RGB-D camera from an egocentric view. In the guidance recording stage, HIGS is able to detect the global positions and moments of hand-object interactions and recover the 3D-hand poses during the operation. In the guidance stage, the recorded guidance video is automatically progressed as the user performs the task and the steps are used to assess task progress and completion. Our system operates in real time for both authoring and for monitoring and providing guidance. We see this work as a step towards fully automated learning and guidance systems, in mixed reality settings.

Vocabulary is an essential part in second language learning. The traditional way of learning vocabulary is dull and boring in class with thin context. Nevertheless, a rich context for memorizing words is necessary according to the situated learning theory. Thanks to virtual reality (VR) technology, such a rich context can be achieved by creating a virtual simulation scenario. "Words In Kitchen" is a VR system designed as an educational tool for second language learners to learn vocabulary. With the help of the proposed system, a user can interact with virtual objects in the virtual scene to learn the spelling and pronunciation of specific vocabularies. We aim to make full use of immersion and interaction of VR and propose a prototype of vocabulary learning in VR, so that the system can promote learners' common vocabulary learning as a supplement to class and textbooks. A pilot study was conducted to identify its usability and enjoyment and the result indicated that the system could help remember words and stimulate interests.

Virtual simulation is very important for engineering application. Traditional simulation based on DELMIA platform has some shortcomings, such as limited capability of detailed observation, low precision and time-consuming. VR system simulation can effectively solve these problems. However, in engineering applications, the large model simulation in VR system cannot meet the real-time requirements, and collision detection is often slow or even collapsed. To solve this problem, this paper proposes a dual-model approach for engineering collision detection in the CAVE environment: a display model and a collision detection model are established respectively. The display model contains a large model, and the collision detection model contains a manikin and the hidden large model. For visual rendering, the display model uses caching strategy, and the collision detection model uses real-time strategy. The two sets of models are combined and presented in CAVE. Because the dual-model approach separates the visual rendering of the large model from the collision detection calculation, the large model in the collision detection model can be lightweight, or the area of interest can be selected for the collision detection calculation. Experiments show that the dual-model approach can effectively guarantee the real-time performance of visual display and collision detection, and provides a complete solution for engineering collision detection using the CAVE system, which can be applied in the fields of industrial virtual maintenance, assembly, and pre-design evaluation.

The use of augmented reality technology is becoming a necessity of many cultural heritage sites to stay competitive and attractive to tourists. In this paper, we present an outdoor augmented reality guidance system for ancient rock painting aiming to enhance tourist experience and introduce the culture and historical background. We propose a two-stage approach that combines GPS and feature-based recognition methods. In the first stage, we build a navigation map in advance, which consists of the distribution information of major scenic spots, then visit these spots guided by GPS data on the standard smartphone. In the second stage, we adopt a feature-based recognition and tracking method to identify rock painting accurately and augment them with interactive contextual annotations like 3D textual labels, images, and videos. The reliability of this system is verified through on-site demonstrations - RockArt of the Helan Mountain, which is one of National Key Historical and Cultural Sites in China.

This research investigates applying Augmented Reality (AR) visualisation of spatial cues in first-person view task instruction videos. Instructional videos are becoming popular, and are not only used in formal education and training, but even in everyday life as more people seek for how-to videos when they need help with instructions. However, video clips are 2D visualisation of the task space, sometimes making it hard for the viewer to follow and match the objects in the video to those in the real-world task space. We propose augmenting task instruction videos with 3D visualisation of spatial cues to overcome this problem, focusing on creating and viewing first-person view instruction videos. As a proof of concept, we designed and implemented a prototype system, called AR Tips, which allows users to capture and watch first-person view instructional videos on a wearable AR device, augmented with 3D visual cues shown in-situ at the task environment. Initial feedback from potential end users indicate that the prototype system is very easy to use and could be applied to various scenarios. Refereed/Peer-reviewed

Projector-camera systems are commonly used for projector calibration. Conventional methods usually use stationary cameras and temporal coded structured light (SL). In this paper, we propose a projector calibration method using a mobile camera and spatial coded SL. Our method allows the users to use a handheld camera to carry out projector calibration and therefore reduce the effort and time required for camera setup. Although the decoding of temporal coded SL can be error-prone in a real-world situation, our method can achieve robust calibration results by taking advantage of the multi-view observations of the projection thanks to the mobility of the camera. Experiments show that the result of our method is comparable with that of a checkerboard-based approach.

Recording medical surgery operations is important for sharing the various operating techniques. In most operating rooms, fixed surgery cameras are already installed, but it is almost impossible to capture the surgical field because of occlusion by the surgeon’s head and body. In order to capture the surgical field in real surgery operations, we proposed the installation of multiple cameras in a surgical lighting system, so that at least one camera can capture the target surgical field even when the surgeon’s head and body occlude other cameras. In this paper, we present a method for automatic viewpoint selection from multi-view surgical videos, so that the surgical field can always be recorded in the output video. We employ a method for learning-based object detection from videos for automatic evaluation of the surgical field area from multiple input videos. By selecting the viewpoint with the largest area of the surgical field, we can virtually reduce the area of the surgeon’s head and hands. In general, frequent camera switching degrades the video quality of view (QoV). Therefore, we apply the Dijkstra method widely used in the shortest path problem as a combinatorial optimization method for this problem. Our camera scheduling method is that the camera switching is not performed for a certain period of time, and the surgical field observed in the entire video is maximized.

We propose a concept in this poster paper, the Kuroko Paradigm, which is able to enhance user engagement during interaction with an augmented reality (AR) avatar by adding a physical object to the interaction with the avatar. With the development of AR and VR, interactions between users and AR avatars have been realized with different approaches. However, most of such interactions and experiences are passive, from which users do not expect a high level of engagement. We hypothesize that by introducing a reality actuator, such as a robot or a drone, to handle a physical object triggered by the user without being noticed, and rendering AR avatars as interacting with the physical object at the same time, user engagement during the experience will be enhanced. To prove this concept, we conducted an experiment emulating a classic game of catch. In the experiment, a user will try to throw a ball to an AR avatar, and the ball will be caught by a reality actuator. From the user's perspective, the ball is caught by the AR avatar. In the future, we plan to extend the experiment by adding control groups with differing conditions.

We present an idea that uses a tracked graphics tablet attached a virtual or physical object to grasp in VR for direct 3D sketching on virtual objects, giving users the feeling of holding a physical object and drawing on it. We developed a prototype system and evaluated the effect of different tablet manipulation modes on user performance and experience when drawing strokes on virtual objects in VR. Results suggested that having a physical handle or object to grasp enhanced the participants' sense of realism, compared to conditions where there was no tablet or a simple tablet interface. Refereed/Peer-reviewed

Many real world applications for Microsoft HoloLens*-based applications suffer the problem of reliably recognizing and identifying movable objects within an environment. While the HoloLens is perfectly able to discern already known rooms, it still has troubles with reflecting surfaces or identically shaped objects. Using dedicated recognition libraries for each task poses the issue of shared resource access in the rather controlled HoloLens environment-In this poster we present a solution for scenario with hard to track objects and similarly shaped objects for an electrical cabinet assembly task, where the reflective cabinet is tagged with a marker and the prefabricated cables are differentiated by text-based labels.

In this paper we present a Virtual Reality (VR) system (simulated kitchen; Sim:Kitchen) designed and developed for use in the rehabilitation of adults with neurogenic communication disorders. Communication disorders are commonly associated with neurological conditions such as stroke, brain injury, and Parkinson’s disease and require long-term rehabilitation with a speech pathologist. Traditional speech pathology management involves therapy designed to address the communication impairment and to teach strategies to overcome barriers to successful communication. These treatments are delivered within a clinic setting with limited opportunity for practice in realworld situations. VR has the potential to provide a safe, supported lifelike environment in which people undergoing communication rehabilitation could practice therapy tasks and gain confidence before entering actual real-world communication contexts. The aim of this paper is to investigate the perspectives of speech pathologists about the use of VR and the simulated kitchen tool for people with communication disorders as a means to optimise rehabilitation engagement and outcomes.

Despite recent technological advanced in Augmented Reality (AR), the majority of state-of-the-art head-mounted displays (HMDs) still suffers from a limited field of view. Hence, there is a high probability that virtual objects are located outside of the user’s view. Several approaches to mitigate the problem of out-of-view objects have been proposed in the past by using visualization techniques to guide the user’s attention towards such objects. However, none of these techniques provides any context about the real environment. Current state-of-the-art HMDs are equipped with various sensors to create a 3D map of the environment. In this work, we leverage such 3D reconstructions and developed a set of different 3D minimap visualizations for conveying information about out-of-view objects in AR. We propose a novel kind of minimap using stereographic fisheye projection and compare it to more traditional, bird’s-eye view minimaps. Preliminary results show that our stereographic fisheye minimap offers a set of distinct advantages over bird’s-eye view minimaps that facilitate the localization of virtual objects in AR environments.

Nowadays unbalanced crop yield and population growth have led to people worry about a potential future “Agricultural Crisis”. To increase the arable land use efficiency, this research has focused on the development of Augmented Reality based Agricultural information visualization and fieldwork navigation tool. This tool has been developed to provide professional aids for farmers’ daily fieldwork. To provide continuous fieldwork assistance, this system integrates non-distraction field navigation, the overall visualization of fields and subfields in the form of overlaid polygons, and the detail sensor data visualization for pertinent subfields in the form of Point of Interests. Due to the limitation of current commercially available AR headsets, we first validate this test bench on Virtual Reality video see-through headsets and AR tablets. At this stage, this test bench has only been validated with simulated data. In future, Situated Analytics concepts are planned to be integrated into the system to in-situ visualize real-time datachunks acquired from local sensors, so as to provide timely-accurate information and decision support even when the whole dataset is in a high volume.