Search

For many applications, remote guidance and telerobotics provide great advantages. For example, tele-ultrasound can bring much-needed expert healthcare to isolated communities. However, existing tele-guidance methods have serious limitations including either low precision for video conference-based systems, or high complexity and cost for telerobotics. A new concept called human teleoperation leverages mixed reality, haptics, and high-speed communication to provide tele-guidance that gives an expert nearly-direct remote control without requiring a robot. This paper provides an overview of the human teleoperation concept and its application to tele-ultrasound. The concept and its impact are discussed. A new approach to remote streaming and control of point-of-care ultrasound systems independent of their manufacturer is described, as is a high-speed communication system for the HoloLens 2 that is compatible with ResearchMode API sensor stream access. Details of these systems are shown in supplementary video demonstrations. Novel interaction methods enabled by HoloLens 2-based pose tracking are also introduced and tests of the communication and user interaction are presented. The results show continued improvement of the system compared to previous work in instrumentation, HCI, and communication. The system thus has good potential for tele-ultrasound, as well as possible other applications of human teleoperation including remote maintenance, inspection, and training. The remote ultrasound streaming and control application is made available open source. [Display omitted] • System improvements to human teleoperation demonstrate its feasibility. • Other devices such as the Nreal Light can be used for implementation. • New device-agnostic remote ultrasound streaming and control demonstrated. • HoloLens pose tracking enables human teleoperation with limited compute resources. • HoloLens-based communication system provides effective sensor data streaming. [ABSTRACT FROM AUTHOR]

Augmented Reality (AR) is a key technology in the transition to Industry 4.0 and smart manufacturing, gaining reputation in a wide range of industrial fields One promising application is in another field of Industry 4.0, the data monitoring, where AR can be used to visualize and interact with complex and big data in real time, potentially improving the efficiency and accuracy of decision-making In this work, we propose a Pervasive AR tool for data monitoring, developed in collaboration with industry partners. An web application of the same data monitoring function is also created for comparison purposes A Human-Centered Design (HCD) methodology was used to identify the needs and requirements of industrial analysts, which led to the development of these systems Preliminary user studies, with 17 participants having distinct levels of expertise in industry, data analysis and computer science, were conducted to collect opinions and suggestions for improvements, as well as, quantitative data regarding the technologies considered A succeeding user study was then prepared, in which 12 participants used two conditions (C1 — Hands-free Pervasive AR tool for HMDs and C2 — Web tool for tablet devices) to fulfill a set of data monitoring tasks in an industry assembly line The result of these studies confirm the potential of Pervasive AR for data monitoring, as it engages users, promote environmental awareness, contextualizes data and allows fast self-localization. [Display omitted] • Two user studies with more than 10 participants each, industrial settings tasks. • Understand which method (Web or AR) stands out according to the task characteristic. • The findings reveal that certain methods were more effective for types of task. • Display methods significantly influenced participants across analysis. • These studies confirm the potential of Pervasive AR for data monitoring. [ABSTRACT FROM AUTHOR]

The perception of size in virtual objects in Augmented Reality (AR) and Virtual Reality (VR) is a not trivial issue, as the effectiveness of manipulating and interacting with virtual content depends on the accuracy of size perception. However, there are missing straightforward comparisons between VR and AR in terms of size perception for the deep understanding of size perceptual differences. Understanding these perceptual differences can inform designers on how to adapt content when transitioning between these two spatial computing platforms. In this paper, we conducted two psychophysical experiments to measure the perceptual thresholds of size discrimination for virtual objects. Our results indicated that users are more sensitive to size changes in VR than in video see-through AR, suggesting that size differences are easier to be perceived in VR than in AR. Additionally, for increase or decrease of sizes, the accuracy of judgments showed an asymmetric trend in video see-through AR. [Display omitted] • A comparative experiment to understand the difference in size perception in AR vs VR. • The thresholds for size discrimination in AR and VR are not the same. • The accuracy of judgments is asymmetric for increases and decreases of sizes in AR. [ABSTRACT FROM AUTHOR]

The retrieval of 3D objects has gained significant importance in recent years due to its broad range of applications in computer vision, computer graphics, virtual reality, and augmented reality. However, the retrieval of 3D objects presents significant challenges due to the intricate nature of 3D models, which can vary in shape, size, and texture, and have numerous polygons and vertices. To this end, we introduce a novel SHREC challenge track that focuses on retrieving relevant 3D animal models from a dataset using sketch queries and expedites accessing 3D models through available sketches. Furthermore, a new dataset named ANIMAR was constructed in this study, comprising a collection of 711 unique 3D animal models and 140 corresponding sketch queries. Our contest requires participants to retrieve 3D models based on complex and detailed sketches. We receive satisfactory results from eight teams and 204 runs. Although further improvement is necessary, the proposed task has the potential to incentivize additional research in the domain of 3D object retrieval, potentially yielding benefits for a wide range of applications. We also provide insights into potential areas of future research, such as improving techniques for feature extraction and matching and creating more diverse datasets to evaluate retrieval performance. • ANIMAR dataset was constructed with 711 3D animal models and 140 sketch queries. • SHREC challenge track focusing on Text-based 3D ANIMAl model fine-grained Retrieval. • We receive satisfactory results from eight teams and 204 runs. [ABSTRACT FROM AUTHOR]

In augmented reality (AR) applications, detecting and tracking real-world objects remains a challenge. Another challenge is making the loaded virtual objects truly reflect the lighting and shadow conditions of the actual environment in which they are located. In this paper, we proposed an improved method for real-time light estimation of environmental lighting in AR applications, tracking the movement of physical objects, and making the corresponding virtual objects reflect the lighting and shadows of the actual environment. First, we proposed an effective method for estimating light in an AR scene by training a deep neural network once on different scenes from the input of RGB-D images to determine the direction of the brightest light. Second, we experimented with advanced methods of detecting and tracking different types of real objects (flat surfaces, simple geometries, and complex geometries) for visualizing virtual materials on the top of these surfaces, where virtual materials will acquire the conditions of the real place in terms of lighting, reflection, and shadows depending on our proposed method for light estimation. Our method of light estimation was tested via experiments on the three types of real objects mentioned above. Results indicate that our method achieves higher resolution and lower error rate compared with other work. Our methods have achieved very good results in terms of detection accuracy, tracking accuracy, realistic visualization of materials, matching real shade, and lighting conditions. [Display omitted] [ABSTRACT FROM AUTHOR]

This research paper explores the impact of augmented reality (AR) tracking characteristics, specifically an AR head-worn display's tracking registration accuracy and precision, on users' spatial abilities and subjective perceptions of trust in and reliance on the technology. Our study aims to clarify the relationships between user performance and the different behaviors users may employ based on varying degrees of trust in and reliance on AR. Our controlled experimental setup used a 360° field-of-regard search-and-selection task and combines the immersive aspects of a CAVE-like environment with AR overlays viewed with a head-worn display. We investigated three levels of simulated AR tracking errors in terms of both accuracy and precision (+0°, +1°, +2°). We controlled for four user task behaviors that correspond to different levels of trust in and reliance on an AR system: AR-Only (only relying on AR), AR-First (prioritizing AR over real world), Real-Only (only relying on real world), and Real-First (prioritizing real world over AR). By controlling for these behaviors, our results showed that even small amounts of AR tracking errors had noticeable effects on users' task performance, especially if they relied completely on the AR cues (AR-Only). Our results link AR tracking characteristics with user behavior, highlighting the importance of understanding these elements to improve AR technology and user satisfaction. [Display omitted] • Augmented Reality (AR) tracking factors include accuracy and precision. • Search-and-selection task tested effects of tracking factors on performance, trust. • Results indicate even small AR tracking errors impact user performance, trust. • Negative performance effects can be mitigated if users do not rely solely on AR. [ABSTRACT FROM AUTHOR]

Immersive environments with head mounted displays (HMD) and hand-held controllers, either in Virtual or Augmented Reality (VR/AR), offer new possibilities for the creation of artistic 3D content. Some of them are exploited by mid-air drawing applications: the user's hand trajectory generates a set of stylized curves or ribbons in space, giving the impression of painting or drawing in 3D. We propose a method to extend this approach to the sketching of surfaces with a VR controller. The idea is to favor shape exploration, offering a tool, where the user creates a surface just by painting ribbons. These ribbons are not constrained to form patch boundaries for example or to completely cover the shape. They can be very sparse, disordered, overlap or not, intersect or not. The shape is computed simultaneously, starting with the first piece of ribbon drawn by the user and continuing to evolve in real-time as long as the user continues sketching. Our method involves minimizing an energy function based on the projections of the ribbon strokes on a proxy surface by taking the controller's orientations into account. The current implementation considers elevation surfaces. In addition to many examples, we evaluate the time performance of the dynamic shape modeling with respect to an increasing number of input ribbon strokes. Finally, we present images of an artistic creation that combines stylized curve drawings in VR with our surface sketching tool created by a professional artist. [Display omitted] • A novel VR sketch-based surface modeling tool is presented. • Ribbon strokes are drawn as if imitating the artist's gesture while painting. • On-the-fly creation and updating of surfaces from disordered ribbon strokes. • Can process sparse, disordered, possibly overlapping ribbon strokes. • Interactive frame rates give immediate visual feedback on the sketching gestures. [ABSTRACT FROM AUTHOR]

In this paper, we present a systematic review of the applications of (1) Extended Reality (XR), (2) Augmented Reality (AR), and (3) Virtual Reality (VR) technologies to enhance emotion recognition and emotion expression in people with Autism Spectrum Disorder (ASD). ASD can affect various abilities, and poses challenges to the recognition of emotions in others, which is often referred to as "social blindness". Treating this condition typically requires intensive one-on-one or small-group therapy sessions, which can be costly and limited in terms of availability. With the growing number of diagnoses of ASD, concerns have risen regarding a potential "lost generation" that may face difficulties in fulfilling its potential. Through this comprehensive review, we aim to provide an overview of innovative approaches that use XR technologies to improve the learning experience of individuals with ASD. [Display omitted] • Review revealed high interest in XR technologies and wellbeing. • XR technologies significantly improved emotion skills learning. • XR applications greatly enhance user motivation. • A user-centered approach is crucial for maximizing learning efficiency. • Further long-term studies and gamification are needed to sustain motivation. [ABSTRACT FROM AUTHOR]

Laparoscopic surgery techniques are complex and impose postural and communication constraints on the surgeon that may affect surgery outcomes. This paper explores the possibilities of designing intraoperative AR interfaces for laparoscopy surgery. We suggest that the laparoscopic video be displayed on an AR headset and that surgeons consult preoperative image data on that display. Interaction with these elements is necessary. Thus, we propose a head-gaze and hand clicker approach that is effective and minimalist, as well as the implementation of a prototype. We conduct a user study to evaluate the prototype and to comprehend the impact and improvements that headgaze average filtering and the scale method can bring to perform annotations in the laparoscopy video feed on a virtual monitor positioned straight in front of the user. The user experiment was performed in a between-subject protocol with 32 volunteers from the Institute of Informatics, and the proposed task involves communication in the interface through drawing annotations with proposed interaction approaches: A hand device for input confirmations and the head-gaze stabilization methods to pointing and selection. The study found that the users were confident about their performance and demonstrated low physical and temporal demand. The proposed head-gaze methods showed that independent of the stabilization applied, the difference between the error sensibility of the axis of the head-gaze in annotation positioning is significant. The vertical axis presented a higher error rate than the horizontal axis. When we compared other variables, we found some differences in specific circumstances, but overall, the interaction with HL1 is very distributed. [Display omitted] • Laparoscopy imposes posture strain and poor preoperative data access to surgeons. • AR technology may solve these issues if an appropriate interaction is available. • We propose an AR UI with panels to see lap-video and preoperative images. • Interaction is centered on head gaze pointing and a ring under the surgical glove. • An experiment with 32 users characterizes usability and user experience with our UI. [ABSTRACT FROM AUTHOR]

The ubiquity of smartphone and tablet devices, combined with the increasing availability of serious games, has enabled students to learn various abstract concepts in an appealing and convenient manner. While several researchers have explored the use of Augmented Reality (AR) in serious games, many of these games have not been critically explained or evaluated. To that end, we employed game-based learning methodologies and Game Learning Analytics (GLA) to systematize the design and evaluation of an AR-based serious game to teach programming. We evaluated our game for usability and effectiveness by conducting a user study on twenty-seven undergraduate students. The evaluation primarily consisted of a learning test conducted twice – before and after playing the game – along with a usability questionnaire that players completed after playing the game. Our results showed that players made significant progress after playing the game. The game helped players improve their basic programming skills, especially for the group having lower prior programming skills. The results highlighted various ways in which GLA can be used to benefit different stakeholders in the game. Based on players' qualitative responses, we also identified several areas of improvement, most prominently the trade-off between ease of use and game complexity. We provide suggestions and discuss implications for future work. • Design and evaluation procedures for an Augmented Reality-based serious game to teach programming are presented. • An empirical study with twenty-seven undergraduate students is conducted. • Students with lesser prior programming skills show significant improvement after playing the game. • A Game Learning Analytics framework is incorporated in the game for various stakeholder benefits. • Suggestions and implications for future work are presented. [ABSTRACT FROM AUTHOR]

With recent technological advancements in sports broadcasting, remote spectators are presented with an enriched experience. These enriched experiences include additional content such as statistics and graphics that support game understanding. However, these technological advancements are often not accessible to on-site sports spectators. In this paper, we explore the opportunities of using situated visualization to enrich on-site sports spectating. Situated visualization techniques allow us to display information in a spatial context to its physical reference and have the potential to close the gap between on-site spectating and content access. With regards to this, we developed a framework for situated visualization focusing on sports spectating. We identified components needed for such a use case and developed two novel situated visualization approaches based on the proposed framework: (1) situated broadcast-styled visualization and (2) situated infographics. To evaluate the visualization approaches and explore user preferences, we conducted a lab study and a subsequent on-site study in a stadium. [Display omitted] • Conceptual framework describing components of on-site situated sports spectating. • Main components of sports situated visualization are Canvas, Content and Context. • Framework led to situated broadcast-style visualization and situated infographics. • Positive responses towards situated infographics and an improved game understanding. • No increase in mental demand recorded for situated infographics despite the benefits. [ABSTRACT FROM AUTHOR]

Remote collaboration using Augmented Reality (AR) has potential to support physically distributed team-members that need to achieve a common goal by increasing knowledge retention, improving understanding and awareness of the problem and its context. In this vein, the path to achieve usable, realistic and impactful solutions must entail an explicit understanding regarding how remote collaboration occurs through AR and how it may help contribute to a more effective work effort. Thus, characterization and evaluation of the collaborative process is paramount, but a particularly challenging endeavor, due to the multitude of aspects that define the collaboration effort. In this context, the work presented here contributes with a critical analysis, discussing current evaluation efforts, identifying limitations and opportunities. Then, we outline a conceptual framework to support researchers in conducting evaluations in a more structured manner. To instrument this vision, an evaluation toolkit is proposed to support contextual data collection and analysis in such scenarios and obtain an additional perspective on selected dimensions of collaboration. We illustrate the usefulness and versatility of the toolkit through a case study on remote maintenance, comparing two distinct methods: sharing of video and AR-based annotations. Last, we discuss the results obtained, showing the proposed vision allows to have an additional level of insights to better understand what happened, eliciting a more complete characterization of the work effort. [Display omitted] • How evaluation of the collaborative process is conducted in AR-remote scenarios. • Conceptual framework for conducting evaluations in a more structured manner. • Creation of an evaluation toolkit for improved data collection and analysis. • User study demonstrates the toolkit capabilities in a remote maintenance scenario. • Toolkit elicits a more complete characterization of the collaborative process. [ABSTRACT FROM AUTHOR]

The use of technology in cultural heritage sites for end-users, like virtual audio-visual reconstructions, has become a common solution to improve the communication and perception between these spaces and their visitors. Since humans perceive the world with different senses and in real-time, to evoke more than one or two senses at a time can bring benefits for the user perception. Accordingly, some very different implementations unleashing multisensory experiences have been made. Aiming to understand how different stimuli are being evoked in multisensory experiences to enhance cultural heritage experiences, and how these implementations are being evaluated, this paper presents a systematic review of technological multisensory applications in cultural heritage. Thus, the collected and analyzed data, focused on technologies used, purpose of the experience, stimuli explored, evaluation process, main findings obtained, and limitations found, will provide valuable information for further implementations. The overall results unleash the wide diversity available for multisensory implementations, from technological solutions to available content for users. Covering the pros and cons of such diversity, this study sustains the use of multisensory applications in cultural heritage as a powerful tool to enrich users' visits. [Display omitted] • Systematic review on Multisensory VR and AR implementations in Cultural Heritage. • Analysis of evoked stimuli and of what technologies were used to evoke each stimulus. • Detailed analysis of the purpose of each implementation and of conducted evaluations. • Mention to studies' main findings and main limitations found. • Discussion related to analyzed multisensory studies. [ABSTRACT FROM AUTHOR]

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user's ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user's performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training. [Display omitted] • We investigate the effects of three overlay FOV (OFOV) sizes on spatial memorization. • We highlight the impact of OFOV and object proximity on memorization. • OFOV size did not affect spatial memorization, but wide OFOV reduced head rotation. • Our results imply that a larger OFOV may not necessarily benefit training outcomes. [ABSTRACT FROM AUTHOR]

This paper presents an Augmented Reality (AR)-based network cabling tutoring system that trains users how to interconnect cables within a Virtual Local Area Network (VLAN) on a physical switching rack. AR arrows combined with text-based instruction and a checklist provided assistance during practical learning. When learners made a mistake, an Intelligent Tutoring System (ITS) identified the source of the mistake and provided real-time feedback using text, AR and text-to-speech mechanisms. The design was motivated by the human-cognitive architecture and its five evolutionary principles (proposed by Sweller and Sweller (2006)). Users performed four consecutive network cabling training tasks with assistance from our ITS. We found that users made fewer errors when the AR cues, text-based instruction and checklist solutions were replaced with summarised information and then removed completely in the final task compared to those who used an identical system with fixed instruction. Cognitive Load Theory (CLT) explains our results by suggesting that the instructional mechanisms become redundant as knowledge increases. Implications of the study are discussed as well as how AR can help facilitate knowledge transfer in the network cabling domain. [Display omitted] • Several cognitive load considerations for AR-based network training are proposed. • A novel AR training system combining fading with an ITS is proposed. • Our AR training system reduces mistakes during network security training. • In network security training, AR should be faded out as expertise increases. [ABSTRACT FROM AUTHOR]

The representation of consistent mixed reality (XR) environments requires adequate real and virtual illumination composition in real-time. Estimating the lighting of a real scenario is still a challenge. Due to the ill-posed nature of the problem, classical inverse-rendering techniques tackle the problem for simple lighting setups. However, those assumptions do not satisfy the current state-of-art in computer graphics and XR applications. While many recent works solve the problem using machine learning techniques to estimate the environment light and scene's materials, most of them are limited to geometry or previous knowledge. This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images. We show in the experiments that the CNN architecture can predict the environment lighting with an average mean squared error (MSE) of 7.85 × 10−4 when comparing SH lighting coefficients. We validate our model in a variety of mixed reality scenarios. Furthermore, we present qualitative results comparing relights of real-world scenes. [Display omitted] • Automatic end-to-end method to estimate the environment lighting in XR applications. • A CNN architecture that learns a latent-space of the environment lighting. • A methodology to generate egocentric mixed-reality-views from HDR panoramas. • Real-time lighting estimation that does not make assumptions about the XR scene. [ABSTRACT FROM AUTHOR]

Remote Collaboration mediated by Mixed and Augmented Reality (MR/AR) shows great potential in scenarios where physically distributed collaborators need to establish a common ground to achieve a shared goal. So far, most research efforts have been devoted to creating the enabling technology, overcoming engineering hurdles and proposing methods to support its design and development. To contribute to more in-depth knowledge on how remote collaboration occurs through these technologies, it is paramount to understand where the field stands and how characterization and evaluation have been conducted. In this vein, this work reports the results of a literature review which shows that evaluation is frequently performed in ad-hoc manners, i.e., disregarding adapting the evaluation methods to collaborative AR. Most studies rely on single-user methods, which are not suitable for collaborative solutions, falling short of retrieving the necessary amount of contextualized data for more comprehensive evaluations. This suggests minimal support of existing frameworks and a lack of theories and guidelines to guide the characterization of the collaborative process using AR. Then, a critical analysis is presented in which we discuss the maturity of the field and a roadmap of important research actions is proposed, that may help address how to improve the characterization and evaluation of the collaboration process moving forward and, in consequence, improve MR/AR based remote collaboration. [Display omitted] • How evaluation of the collaborative process is conducted in remote scenarios using AR. • Integration of characterization and evaluation methods is of paramount importance. • Critical analysis to discuss the maturity of remote collaboration using AR. • AR remote collaboration is in the Replication-Empiricism phases of the BRETAM model. • Roadmap of research actions to elicit characterization of the collaboration process. [ABSTRACT FROM AUTHOR]

• The proposed solution can reconstruct indoor scenes using only a single frame and an RGB camera. • An indoor scene composed by five objects can be reconstructed in just 21 seconds. • The proposed solution obtains a maximum error of 18% for the scaling factor, less than 9cm for the position and less than 18 ∘ for the rotation. • The innovative methodology can be employed for VR and AR applications, thus bridging the gap between the real and virtual words. [Display omitted] Indoor environment reconstruction is a challenging task in Computer Vision and Computer Graphics, especially when Extended Reality (XR) technologies are considered. Current solutions that employ dedicated depth sensors require scanning of the environment and tend to suffer from low resolution and noise, whereas solutions that rely on a single photo of a scene cannot predict the actual position and scale of objects due to scale ambiguity. The proposed system addresses these limitations by allowing the user to capture single views of objects using an Android smartphone equipped with a single RGB camera and supported by Google ARCore. The system includes 1) an Android app tracking the smartphone's position relative to the world, capturing a single RGB image for each object and estimating depth information of the scene, 2) a program running on a server that classifies the framed objects, retrieves the corresponding 3D models from a database and estimates their position, vertical rotation, and scale factor without deforming the shape. The system has been assessed measuring the translational, rotational and scaling errors of the considered objects with respect to the physical ones acting as a ground truth. The main outcomes show that the proposed solution obtains a maximum error of 18% for the scaling factor, less than nine centimeters for the position and less than 18 ∘ for the rotation. These results suggest that the proposed system can be employed for XR applications, thus bridging the gap between the real and virtual worlds. [ABSTRACT FROM AUTHOR]

Invisible fiducial markers are introduced for localization of Areas Of Interest (AOIs) in mobile eye tracking applications. Fiducial markers are made invisible through the use of film passing Infra-Red (IR) light while blocking the visible spectrum. An IR light source is used to illuminate the markers which are then detected by an IR-sensitive camera, but which are imperceptible by the human eye. We provide the first empirical study that demonstrates such invisible markers are not distracting to a given task, as demonstrated in a flight simulator where distraction of visible and invisible markers are compared between experienced and novice pilots. Fixation frequency and subjective distraction scores showed that visible markers disrupted natural gaze behavior, particularly in novice pilots. Our findings show that invisible markers should be used when there is a need for them to remain inconspicuous. [Display omitted] • Visible markers were shown to disrupt gaze behaviors, especially in novices. • Marker placement may be a contributing factor to gaze behavior disruption. • Novice pilots demonstrate more frequent gaze tunneling behaviors. [ABSTRACT FROM AUTHOR]

Head-mounted Augmented Reality (AR) displays overlay digital information on physical objects. Through eye tracking, they provide insights into user attention, intentions, and activities, and allow novel interaction methods based on this information. However, in physical environments, the implications of using gaze-enabled AR for human activity recognition have not been explored in detail. In an experimental study with the Microsoft HoloLens 2, we collected gaze data from 20 users while they performed three activities: Reading a text, Inspecting a device, and Searching for an object. We trained machine learning models (SVM, Random Forest, Extremely Randomized Trees) with extracted features and achieved up to 89.6% activity-recognition accuracy. Based on the recognized activity, our system—GEAR—then provides users with relevant AR feedback. Due to the sensitivity of the personal (gaze) data GEAR collects, the system further incorporates a novel solution based on the Solid specification for giving users fine-grained control over the sharing of their data. The provided code and anonymized datasets may be used to reproduce and extend our findings, and as teaching material. [ABSTRACT FROM AUTHOR]

Obtaining feedback from time-constrained end-users is a major challenge in evaluating novel systems for specialised applications. The performance and feedback of engineers and surgeons was evaluated through an experiment where participants were asked to identify tumour locations within an anatomically realistic silicon liver model across three different conditions of an Augmented Reality (AR) prototype system (Baseline, Split AR and Full AR). Our findings show that engineers and surgeons share some similarities in their performance, feedback and behaviour, particularly when reliance on the AR system is high for both groups. However, engineers typically focus more on accuracy of the image alignment and are more accurate in their responses when supported by AR. Senior surgeons typically perform faster and use AR as supplementary information, while the performance of junior surgeons is more closely aligned to the performance of engineers. We conclude that engineers could be involved in preliminary evaluations of a surgical system or in evaluations of systems which are aimed at training junior surgeons, but that it is essential to involve surgeons in later evaluations, where ecological validity is a more important consideration. • Level of reliance on a system and expectations around system accuracy can differ between engineers and surgeons. Where surgeons (particularly senior surgeons) were faster and use augmented reality (AR) as supplementary information and engineers were more accurate when using AR system. • We found that testing with engineers could be an ideal alternative when testing digital systems during preliminary stages. • This work recommends that engineers should not replace testing technologies with experienced surgeons, rather this method should only be used as an early indicator in the preliminary stages of a research project. • Engineers could also be involved in evaluations of systems which are aimed at training junior surgeons, but it is essential to involve surgeons in later evaluations, where ecological validity is a more important. [ABSTRACT FROM AUTHOR]

• We propose an augmented crowd simulation system using automatic determination and reconstruction of navigable areas in static, surveillancealike videos. • We utilize pedestrian trajectory data and use deep learning-based semantic segmentation methods to identify navigable areas. • We simulate artificial agents over the reconstructed navigable area together with real agents in the video via collision avoidance. • We demonstrate the accuracy and applicability of the proposed navigable area reconstruction approach on various crowded outdoor scenarios. [Display omitted] Crowd simulations imitate the group dynamics of individuals in different environments. Applications in entertainment, security, and education require augmenting simulated crowds into videos of real people. In such cases, virtual agents should realistically interact with the environment and the people in the video. One component of this augmentation task is determining the navigable regions in the video. In this work, we utilize semantic segmentation and pedestrian detection to automatically locate and reconstruct the navigable regions of surveillance-like videos. We place the resulting flat mesh into our 3D crowd simulation environment to integrate virtual agents that navigate inside the video avoiding collision with real pedestrians and other virtual agents. We report the performance of our open-source system using real-life surveillance videos, based on the accuracy of the automatically determined navigable regions and camera configuration. We show that our system generates accurate navigable regions for realistic augmented crowd simulations. [ABSTRACT FROM AUTHOR]

• We present a design space that articulates the role of gaze in AR across three factors. • Factor 1: Reality–virtuality continuum transitions between reality and AR content. • Factor 2: Information level transitions from concise to detailed AR content. • Factor 3: Task transitions in response to standard gaze input. • We successfully design, implement, and evaluate three AR applications based on these. [Display omitted] Augmented Reality (AR) headsets extended with eye-tracking, a promising input technology for its natural and implicit nature, open a wide range of new interaction capabilities for everyday use. In this paper we present ARtention, a design space for gaze interaction specifically tailored for in-situ AR information interfaces. It highlights three important dimensions to consider in the UI design of such gaze-enabled applications: transitions from reality to the virtual interface, from single- to multi-layer content, and from information consumption to selection tasks. Such transitional aspects bring previously isolated gaze interaction concepts together to form a unified AR space, enabling more advanced application control seamlessly mediated by gaze. We describe these factors in detail. To illustrate how the design space can be used, we present three prototype applications and report informal user feedback obtained from different scenarios: a conversational UI, viewing a 3D visualization, and browsing items for shopping. We conclude with design considerations derived from our development and evaluation of the prototypes. We expect these to be valuable for researchers and designers investigating the use of gaze input in AR systems and applications. [ABSTRACT FROM AUTHOR]

• Development and user assessment of an Augmented Situated Visualization of EMC data. • Significant effect of the interaction methods on the time taken for data extraction. • Significant effect of the interaction methods used on walking steps. • Significant effect of the interaction and visualization methods on task correctness. • Results suggest that specific techniques may be used depending on the task requisites. In electrical engineering, hardware experts often need to analyze electromagnetic radiation data to detect any external interference or anomaly. The field that studies this sort of assessment is called electromagnetic compatibility (EMC). As a way to support EMC analysis, we propose the use of Augmented Situated Visualization (ASV) to supply professionals with visual and interactive information that helps them to comprehend that data, however situating it where it is most relevant in its spatial context. Users are able to interact with the visualization by changing the attributes being displayed, comparing the overlaps of multiple fields, and extracting data, as a way to refine their search. The solutions being proposed in this work were tested against each other in comparable 2D and 3D interactive visualizations of the same data in a series of data-extraction assessments with users, as a means to validate the approaches. Results exposed a correctness-time trade-off between the interaction methods. The hand-based techniques (Hand Slider and Touch Lens) were the least error-prone, being near to half as error-inducing as the gaze-based method. Touch Lens also performed as the least time-consuming method, taking in average less than half of the average time required by the others. For the visualization methods tested, the 2D ray casts presented a higher usability score and lesser workload index than the 3D topology view , however exposing over two times the error ratio. Ultimately, this work exposes how AR can help users to have better performances in a decision-making context, particularly in EMC related tasks, while also furthering the research in the ASV field. [ABSTRACT FROM AUTHOR]

• Dark shadows can be represented even on optical see-through head-mounted displays. • Users actually perceive virtual shadows darker than without augmentations. • Virtual shadows can improve the photo-reality of even transparent objects. In this paper, we present two effects of illusion-based virtual shadows on a real surface for optical see-through head-mounted displays without devices that selectively reduce or intercept light rays from the real scene. The virtual shadows are based on the simultaneous brightness-contrast illusion in which a central region surrounded by a lighter one is perceived as darker than one without such a lighter region. To display a virtual shadow, a virtual object with a real image texture is placed around the shadow region as the lighter region to provide a sense of a darker area. We conducted two kinds of experiments to validate the feasibility of our method. The first experiment showed that users tend to perceive the virtual shadow part as darker than the surrounding area with the same physical luminance. The second experiment provided evidence that there are cases in which the virtual shadows can improve the reality of virtual objects. [ABSTRACT FROM AUTHOR]

• A gaze-based method that enables pure hands-free object positioning in a virtual world for head-mounted displays. • Our method can function with both pure head gaze and pure eye gaze. • Experimental results show that eye and head gaze-based control with our method can achieve 78% and 96% success rates, respectively, when compared to a hand-held controller. In virtual and augmented reality, gaze-based methods have been explored for decades as effective user interfaces for hands-free interaction. Though several well-known gaze-based methods exist for simple interactions such as selection, no solutions exist for 3D manipulation tasks requiring a higher degree of freedom (DoF). In this paper, we introduce OrthoGaze, a novel user interface that allows users to intuitively manipulate the three-dimensional position of a virtual object using only their eye or head gaze. Our approach makes use of three selectable, orthogonal planes, where each plane not only helps guide the user's gaze in an arbitrary virtual space, but also allows for 2-DoF manipulations of object position. To evaluate our method, we conducted two user studies involving aiming and docking tasks in virtual reality to evaluate the fundamental characteristics of sustained gaze aiming and to determine which type of gaze-based control performs best when combined with OrthoGaze. Results showed that eye gaze was more accurate than head gaze for sustained aiming. Additionally, eye and head gaze-based control for 3D manipulations achieved 78% and 96% performance, respectively, in comparison with a hand-held controller. Subjective results also suggest that gaze-based manipulation can comprehensively cause more fatigue than controller-based. From the experimental results, we expect OrthoGaze to become an effective method for pure hands-free object manipulation in head-mounted displays. [ABSTRACT FROM AUTHOR]

• Proposal of a computational glasses for visual field defect. • Design of a whac-a-mole task for empirical performance evaluation. • Optimal combinations of size, position, and opacity for overlaid window. Visual field defect (VFD) is a type of ophthalmic disease that causes the loss of part of the patient's field of view (FoV). In this paper, we propose a method to enlarge the restricted FoV with an optical see-through head-mounted display (OST-HMD) equipped with a camera that captures an overview and overlays it on the persisting FoV. Because the overview window occludes the real background scene, it is important to create a balance between the augmented contextual information and the unscreened local information. We recruited twelve participants and conducted an experiment to seek the best size, position, and opacity for the overview window through a Whac-A-Mole task (a touchscreen game). We found that the performance was better when the overview window was of medium size (FoV of 9.148∘ × 5.153∘, nearly one third of FoV of the used OST-HMD) and placed lower in the visual field. Either too large or too small a size decreases the performance. The performance increases with increased opacity. The obtained results can legitimate the default setting for the overview window. [ABSTRACT FROM AUTHOR]

• Text, audio and image feedback are effective tools to guide correct movement performance. • Audio feedback showed a higher efficacy on correct movement guidance based on its success rate and user approval. • The increase in success rate over time demonstrates that the user has learned and understood the instructions and mechanics of the game to make the correct moves. • Isolated feedback evaluation enables to detect its advantages and disadvantages. • Feedback presentation sequence did not influence their effects. This paper aims to evaluate text, image and audio feedback regarding usability, exercise correction and user preferences when using Augmented Reality (AR) to guide movements execution in rehabilitation applications. For that, this study used an AR rehabilitation tool, named ARkanoidAR, which attempts to correct the wrong movement performed by a user through text, image and audio feedback, while he/she is playing a game controlled based on the therapy protocol defined by a physiotherapist for that particular user. The ARkanoidAR uses a Kinect sensor to track the human body and its movements, and it is completely configurable according to patient therapy needs. To evaluate feedback effects individually, this study collected quantitative data to measure success rate, which represents the amount of time that the user executed the movements correctly when using the application with each feedback individually activated. Experiments were composed of five phases: control, text, image, audio and control. The feedback sequence was randomized before the tests in order to avoid bias. Quantitative results show that users understood the instructions for performing the movements correctly since the success rate along time increased. Tests also included observation and qualitative analysis through questionnaires and a semi-structured interview to evaluate the usability perception of users. The qualitative data helped to detect different types of problems for each feedback based on users' answers. This study was carried out with healthy people, and did not predict the particularities and limitations of physiotherapy patients. [ABSTRACT FROM AUTHOR]

• Our work is the first to explore the domain of LSHF CAR. • We establish a design space specific to LSHF CAR. • We improve the accuracy of synchronized poses between multiple SLAM systems. • We create the first instance of a LSHF CAR experience. • Our results show that virtual repellers are effective in LSHF CAR contexts. In recent years, there has been an increasing amount of Collaborative Augmented Reality (CAR) experiences, classifiable by the deployed scale and the fidelity of the experience. In this paper, we create HoloRoyale, the first large scale high fidelity (LSHF) CAR experience. We do this by first exploring the LSHF CAR design space, drawing on technical implementations and design aspects from AR and video games. We then create and implement a software architecture that improves the accuracy of synchronized poses between multiple users. Finally, we apply our target experience and technical implementation to the explored design space. A core design component of HoloRoyale is the use of visual repellers as crowd control elements to guide players away from undesired areas. To evaluate the effectiveness of the employed visual repellers in a LSHF CAR context we conducted a user study, deploying HoloRoyale in a 12.500 m2 area. The results from the user study suggest that visual repellers are effective crowd control elements that do not significantly impact the user's overall immersion. Overall our main contribution is the exploration of a design space, discussing several means to address the challenges of LSHF CAR, the creation of a system capable of LSHF CAR interactions along with an experience that has been fitted to the design space, and an indepth study that verifies a key design aspect for LSHF CAR. As such, our work is the first to explore the domain of LSHF CAR and provides insight into designing experiences in other AR domains. [ABSTRACT FROM AUTHOR]

Highlights • Edible fiducial markers are expected to be important when one try to apply VR/AR technology on foods. • This paper reports an edible retroreflector made from transparent foodstuffs, and an edible fiducial marker made from the edible retroreflector. • Dynamic projection mapping on foods utilizing these prototypes were demonstrated to show the practical application example. Graphical abstract Abstract In Augmented Reality (AR) and Mixed Reality (MR), it is important to measure the position and orientation of objects in the real world. However, there were many difficulties in recognizing objects from images along, especially considering real-time processing. To make the recognition task easier in such cases, markers are commonly utilized. In particular, retroreflectors are often used as markers since they become very bright in captured images and are easy to be detected by image processing. However, most conventional retroreflectors are made from glass or plastic. Thus, it has been difficult to apply them to some fields such as foods or inspection of the inner wall of digestive organs. In this paper, we propose an edible retroreflector made from transparent foodstuff. We found that kanten , or Japanese agar, which is a traditional Japanese cooking ingredient used to form a transparent jelly, was suitable for forming such optical devices. A prototype successfully retroreflected incident light. An edible fiducial marker made of this edible retroreflector material is also proposed. Dynamic projection mapping on food was demonstrated based on two prototypes to show examples of their potential applications. [ABSTRACT FROM AUTHOR]

Highlights • The effectiveness of Intelligent Tutoring Systems in psychomotor domains can potentially be enhanced by combining them with Augmented Reality paradigms. • Due to ITS complexity and design, work on the use of ITSs with AR is significantly limited and how these paradigms work together is not well understood. • Existing work shows that combining ITSs with AR enhances significantly enhances performance and learning. • Combining AR with ITS requires a gateway interface and networking capabilities in both the ITS and the AR application, due to non-standardised mechanisms used in these systems. • Additional research on the design of the interface; authoring systems for ITSs in AR and their learning effectiveness is required. Graphical abstract Abstract Augmented Reality overlays virtual objects on the real world in real-time and has the potential to enhance education, however, few AR training systems provide personalised learning support. Combining AR with intelligent tutoring systems has the potential to improve training outcomes by providing personalised learner support, such as feedback on the AR environment. This paper reviews the current state of AR training systems combined with ITSs and proposes a series of requirements for combining the two paradigms. In addition, this paper identifies a growing need to provide more research in the context of design and implementation of adaptive augmented reality tutors (ARATs). These include possibilities of evaluating the user interfaces of ARAT and potential domains where an ARAT might be considered effective. [ABSTRACT FROM AUTHOR]

Highlights • Formulation of 3D object tracking as an optimization problem. • 3D object tracker from RGB-D images using particle swarm optimization. • GPU-based fitness function and strategies for particle swarm creation/evaluation. • Better accuracy than state-of-the-art optimization-based tracking techniques. • Runtime analyses that demonstrate the impact of GPU usage on processing time. Graphical abstract Abstract 3D object tracking allows augmented reality applications to add virtual content to the real world in a coherent way. With the popularization of RGB-D sensors, new 6 degree-of-freedom tracking techniques that use features extracted from 3D point clouds have been developed, providing more accurate results. This work proposes the use of particle swarm optimization to handle multiple pose hypotheses during top-down tracking from RGB-D images. A fitness function based on 3D point coordinates, color, and normal information was designed, which is able to handle partial object occlusions by applying a threshold to the Euclidean distance between 3D points. The best particle found for a given frame is kept to compose the set of particles of the next swarm and its components are used as an initial pose to define the boundaries of the search space for tracking in the next frame. In addition, we have taken advantage of GPU processing to reduce the running time. Experiments with a publicly available dataset showed that the use of GPU allowed fast object tracking and that the proposed method presents more accurate tracking results in comparison to state-of-the-art optimization-based techniques, especially in situations where objects are partially occluded. [ABSTRACT FROM AUTHOR]

Highlights • Novel Deep learning based method that estimates the lighting condition of the real scene in interactive time from a raw image. • The method does not require any special equipment or prior knowledge of the scene. • The lighting estimation is an area light that can represent multiple light sources of distinct intensities and directions. • The Method is suitable for indoor and outdoor environments. • A mixed reality framework that incorporates the lighting estimation process to mitigate the lighting mismatch problem in real time. Graphical abstract Abstract The recent developments in virtual and mixed reality by the video game and entertainment industries are responsible for increasing user's visual immersion and provide a better user experience in games and other interactive simulations. However, the interaction between the user and simulated environment still relies on game controllers or other unnatural handheld devices. In the mixed reality context, the usage of more natural and immersive alternative to the game controllers, such as the user's hands, may drastically increase the game interface experience, allowing a personalized visual feedback of the user's interactions in the real-time simulation. There are basically two approaches for including the user's hand: a 3D reconstruction based method, typically based on depth cameras, or an image-based approach, composing the virtual scene with the real images of the user's hands. In the composition of the user's hands and virtual elements, perceptual discrepancies in the illumination of objects may occur, generating an inconsistency in the illumination of the mixed reality environment. A consistent illumination of the environment greatly improves the user's immersion in the mixed reality application. One way to ensure consistent illumination is by estimating the real-world illumination and use this information to adapt the virtual world lighting setting. We present the Spherical Harmonics Light Probe Estimator, a deep learning based technique that estimates the lighting setting of the real-world environment. The method uses a single RGB image and does not requires prior knowledge of the scene. The estimator outputs a light probe of the real-world lighting, represented by 9 spherical harmonics coefficients. The estimated light probe is used to create a composite image containing both real and virtual elements in an environment with a consistent illumination. We validate the technique through synthetic tests achieving an RMS error of 0.0573. We show the usage of the method in an augmented virtuality application. [ABSTRACT FROM AUTHOR]

Highlights • Modality for tactile communication integrating proprioception and cutaneous feedback. • Design of a wearable apparatus for tactile communication in virtual environments. • Assessment of performance during gesture interaction with a back-of-HMD interface. • Design of a tactile vocabulary for mutual guidance in a collaborative setup. Graphical abstract Abstract In real life, the sense of touch provides information about the environment that surrounds us. It is an essential sense and a social one, as it typically implies an interaction with another person. However, even though tactile feedback is used to enhance the user experience in Virtual and Augmented Reality applications (VR/AR), tactile cues are usually rendered reactively by the system. In this paper, we design and assess a touch user interface that is built on a Head-Mounted Display (HMD) to allow peers to exchange tactile signals in VR applications. First, we explored the interface capabilities by assessing different gestures to be adopted in the reproduction of tactile signals. Then, we assessed its use in conjunction with a vibrotactile HMD in a collaborative virtual setup where users could communicate only through vibrotactile cues. Results have shown that accuracy in target localization increases as target density decreases, that swipe gestures are performed with more accuracy when supported by anchor-points, and that accuracy of rhythm reproduction decreases with pace. When applied to a collaborative task, the interface presented high usability and allowed subjects to fully use tactile feedback as a means for intercommunication with their partners. [ABSTRACT FROM AUTHOR]

Visual Simultaneous Localization and Mapping (SLAM) has been used for markerless tracking in augmented reality applications. Distributed SLAM helps multiple agents to collaboratively explore and build a global map of the environment while estimating their locations in it. One of the main challenges in distributed SLAM is to identify local map overlaps of these agents, especially when their initial relative positions are not known. We developed a collaborative AR framework with freely moving agents having no knowledge of their initial relative positions. Each agent in our framework uses a camera as the only input device for its SLAM process. Furthermore, the framework identifies map overlaps of agents using an appearance-based method. We also proposed a quality measure to determine the best keypoint detector/descriptor combination for our framework. [ABSTRACT FROM AUTHOR]

This paper presents a mirror-like augmented reality (AR) system to display the internal anatomy of the current user. Using a single Microsoft V2.0 Kinect (later on referenced as the Kinect ), we animate in real-time a user-specific model of internal anatomy according to the user’s motion and we superimpose it onto the user’s color map. Users can visualize their anatomy moving as if they where looking inside their own bodies in real-time. A new calibration procedure to set up and attach a user-specific anatomy to the Kinect body tracking skeleton is introduced. At calibration time, the bone lengths are estimated using a set of poses. By using Kinect data as input, the practical limitation of skin correspondence in prior work is overcome. The generic 3D anatomical model is attached to the internal anatomy registration skeleton, and warped on the depth image using a novel elastic deformer subject to a closest-point registration force and anatomical constraints. The noise in Kinect outputs precludes direct display of realistic human anatomy. Therefore, to enforce anatomical plausibility, a novel filter to reconstruct plausible motions based on fixed bones lengths as well as realistic angular degrees of freedom (DOFs) and limits are introduced. Anatomical constraints, applied to the Kinect body tracking skeleton joints, are used to maximize the physical plausibility of the anatomy motion while minimizing the distance to the raw data. At run-time, a simulation loop is used to attract the bones toward the raw data. Skinning shaders efficiently drag the resulting anatomy to the user’s tracked motion. Our user-specific internal anatomy model is validated by comparing the skeleton with segmented MRI images. A user study is established to evaluate the believability of the animated anatomy. As an extension of [1] , we also propose an image-based algorithm that corrects accumulated inaccuracy of the system steps: motion capture, anatomy transfer, image generation and animation. These inaccuracies show up as occlusion and self-occlusion misalignments of the anatomy regions when superimposed between them and on top of the color map. We also show that the proposed work can efficiently reduce these inaccuracies. [ABSTRACT FROM AUTHOR]

Mobile audio-guides are currently being used in museums to enhance the visitors’ experience. Over the past few years, some Augmented Reality (AR) solutions have been explored in this context, making it possible to provide augmented visual and sonic stimuli through the use of smartphones. However, the special conditions of museums (e.g. artificial markers cannot be used, small, crowded rooms, etc.) and the limited computational capacity of smartphones mean that there are important restrictions in the use of this technology. In this paper we introduce a novel mobile augmented guide for the Casa Batlló museum (Barcelona, Spain) which is based on a combination of traditional and indirect AR. Through the proposed solution, we further explore the use of indirect AR for the interior of buildings as, at present, this technology is usually used for outdoor environments. The developed application is currently being used by Casa Batlló visitors. Retrieval of user experience shows some of the benefits of the proposed solution and opens the door for other similar solutions in museums. [ABSTRACT FROM AUTHOR]

With the ongoing introduction of wide-FOV VR head-worn displays into the consumer market, the application of VR 3D UIs to professional work environments is attracting increasing attention. One of the most conspicuous concepts is immersive 3D modeling and content creation. In spite of the long research history and multitude of proposed systems, there have been very few analyses of the effect of 3D UIs on productivity in 3D design. In this work, we explore the effect of positional head-tracking and its accompanying parallax depth cues on task performance in 3D object selection and transformation in a setting modeled after 3D design work. Previous studies have come to different conclusions on the importance of positional head-tracking and did not investigate professional 3D modeling tools. In contrast, we performed a user study with design students using professional software on a task that closely emulates their work. Surprisingly, we did not find a significant effect of positional head-tracking on task-completion time, neither when using a traditional 2D mouse nor when using a pinch glove as a 3D input device. Furthermore, we found that the users worked significantly faster with the mouse. We discuss possible explanations and implications for the design of 3D UIs. [ABSTRACT FROM AUTHOR]

We present CubeBuilder, a system for interactive, tangible 3D shape modeling. CubeBuilder allows the user to create a digital 3D model by placing physical, non-interlocking cubic blocks. These blocks may be placed in a completely arbitrary fashion and combined with other objects. In effect, this turns the task of 3D modeling into a playful activity that hardly requires any learning on the part of the user. The blocks are registered using a depth camera and entered into the cube graph where each block is a node and adjacent blocks are connected by edges. From the cube graph, we transform the initial cubes into coherent structures by generating smooth connection geometry for some edges of the graph. Based on an analysis of the cube graph, we identify subgraphs that match given graph templates. These subgraph templates map to predefined geometric refinements of the basic shape. This, in turn, allows the user to tangibly build structures of greater details than the blocks provide in and of themselves. We show a number of shapes that have been modeled by users and are indicative of the expressive power of the system. Furthermore, we demonstrate the scalability of the tangible interface which appears to be limited only by the number of blocks available. [ABSTRACT FROM AUTHOR]

Tracking is an important task that is used for several applications, such as navigation assistance and augmented reality. The improvement and popularization of mobile devices in recent years allowed these applications to be executed on such devices. Thus, several tracking techniques proposed lately take into consideration the benefits and limitations of handheld devices. Therefore, the goal of this work is to perform a systematic mapping in order to provide trends and classification regarding the recent publications in the area of tracking for mobile devices. This study collected 2276 papers from three scientific databases using an open-source crawler, from which 360 were selected to be classified according to four properties: tracking type, degree of freedom, tracking platform and research type. The analysis of these data resulted in a map of the research field, which was presented under three perspectives: the distribution and trends over time of each classification property and the relationship between them. Besides the visual map, the full list of classified papers is available through an open-source web-based catalog. The results showed that the number of publications is increasing every year, which shows a growing interest in this field. Moreover, most works use the device׳s sensors for tracking in location-based applications and almost all of them calculate a 2D or 2D + θ pose. There are also several papers about vision-based techniques to compute the device׳s pose and in the majority of them a full 6D pose is computed. Beyond that, there is a clear preference for systems that calculate the pose locally on the device and only a few use a remote server to assist in this task. Moreover, more than 92% of all papers propose a new technique or use existing ones to create a solution. [ABSTRACT FROM AUTHOR]

Handheld Augmented Reality (HAR) has the potential to introduce Augmented Reality (AR) to large audiences due to the widespread use of suitable handheld devices. However, many of the current HAR systems are not considered very practical and they do not fully answer to the needs of the users. One of the challenging areas in HAR is the in-situ AR content creation where the correct and accurate positioning of virtual objects to the real world is fundamental. Due to the hardware limitations of handheld devices and possible restrictions in the environment, the correct 3D positioning of objects can be difficult to achieve we are unable to use AR markers or correctly map the 3D structure of the environment. We present SlidAR, a 3D positioning for Simultaneous Localization And Mapping (SLAM) based HAR systems. SlidAR utilizes 3D ray-casting and epipolar geometry for virtual object positioning. It does not require a perfect 3D reconstruction of the environment nor any virtual depth cues. We have conducted a user experiment to evaluate the efficiency of SlidAR method against an existing device-centric positioning method that we call HoldAR. Results showed that SlidAR was significantly faster, required significantly less device movement, and also got significantly better subjective evaluation from the test participants. SlidAR also had higher positioning accuracy, although not significantly. [ABSTRACT FROM AUTHOR]

Focus plus context visualization can be used in augmented reality to improve the visual perception of the augmented scene. In the scope of in situ or on-patient medical data visualization, the focus plus context paradigm is used to improve depth perception for physicians showing the patient׳s anatomy as a focus region in the context of the patient׳s body. Volume clipping is one technique to realize focus plus context visualization. However, some of the existing methods for focus plus context visualization based on volume clipping do not run in full real time or are prone to artifacts. In this paper, we present an extension for two of these techniques to improve performance and image quality of the original approaches. We validate all the techniques in a markerless augmented reality environment. A 3D reference model is tracked by the application, and volumetric medical data are shown to the user at the position of the patient׳s anatomy. Our technique is able to handle multiple anatomic regions, although the main region of interest used in this paper is the face. Moreover, tracking accuracy is improved by the use of a hierarchical approach. From an evaluation of the proposed techniques, the results obtained highlight that all of them are free of artifacts, optimized for real-time performance, and improve the visual quality of the augmented scene. [ABSTRACT FROM AUTHOR]

We present a novel approach for guided light-field photography using off-the-shelf smartphones. In contrast to previous work that requires the user to decide where next to position a mobile camera, we actively compute and visualize during runtime a recommendation for the next sampling position and orientation taking into account the current camera pose and required camera alignments. This supports efficient capture of various types of large-field-of-view light fields in just a matter of minutes and without specialized camera equipment. To further reduce the overall capture time, we describe an extension of our guidance algorithm to collaborative light-field photography by small groups of users. [ABSTRACT FROM AUTHOR]

Although the concept of Augmented Reality (AR) has already been proposed more than 20 years ago, most AR-applications are still limited to simple visualization of virtual objects onto spatially limited scenes. Ideas such as the “augmented reality manual”, showing step-by-step instructions to a user wearing a Head-Mounted Display (HMD), have been implemented, but the developed systems did not pass the barrier of demonstration prototypes. One major reason, beside remaining ergonomic and hardware limitations, consists of the large effort required for creating the content of such virtual instructions and for building models allowing accurate tracking. In this paper, we introduce the concept of Cognitive Augmented Reality, which radically revises existing approaches on AR-based assistance systems and proposes a fundamentally new paradigm exploiting prior visual observation and learning of a complete manipulative workflow. More precisely, we present in this paper a complete approach for creating augmented reality content for procedural tasks from video examples, and give then details about the presentation of such content at runtime. We show that this new approach, in spite of its very challenging aspects, is scalable, and valuable from a practicable point of view. [ABSTRACT FROM AUTHOR]

In order to improve registration performance of mobile augmented reality initialization, a new CAD-based recognition method of 3D objects in monocular images is proposed in this paper. Instead of estimating camera pose directly, our method attempts to estimate 3D object pose in the camera coordinate system. The geometric correspondence between the object mass center and its projection on 2D camera image is used to estimate object position. To hypothesize possible azimuths of the object, the 3D CAD model is off-screen rendered with OpenGL ES at different azimuths under an assumed inclination constraint provided by the inertial sensors of mobile devices, and compared with input image by contours matching. Unlike most recognition methods based on machine learning and natural features detection, which require large databases of real images and heavy computational consumption, our method only needs the CAD model of the 3D object in offline stage, and is computationally suitable for mobile devices. This paper describes the details of the method and shows its effectiveness with experiments. [ABSTRACT FROM AUTHOR]

In this paper, we propose a method for estimating the camera pose for an environment in which the intrinsic camera parameters change dynamically. In video see-through augmented reality (AR) technology, image-based methods for estimating the camera pose are used to superimpose virtual objects onto the real environment. In general, video see-through-based AR cannot change the image magnification that results from a change in the camera׳s field-of-view because of the difficulty of dealing with changes in the intrinsic camera parameters. To remove this limitation, we propose a novel method for simultaneously estimating the intrinsic and extrinsic camera parameters based on an energy minimization framework. Our method is composed of both online and offline stages. An intrinsic camera parameter change depending on the zoom values is calibrated in the offline stage. Intrinsic and extrinsic camera parameters are then estimated based on the energy minimization framework in the online stage. In our method, two energy terms are added to the conventional marker-based method to estimate the camera parameters: reprojection errors based on the epipolar constraint and the constraint of the continuity of zoom values. By using a novel energy function, our method can accurately estimate intrinsic and extrinsic camera parameters. We confirmed experimentally that the proposed method can achieve accurate camera parameter estimation during camera zooming. [ABSTRACT FROM AUTHOR]

Abstract: Augmented Reality has a wide-ranging presentation space. In addition to presenting virtual information in a 3D space, such information can also be placed in relation to physical objects, locations or events. Decomposing this presentation space – or more exactly, the principles of how information is represented in Augmented Reality – into unique and independent dimensions provides a fundamental spectrum of options. First, this decomposition facilitates a fine-grained analysis of effects on human understanding. Second, multiple factors, given by multiple differences between different presentation systems with respect to more than one such principle, can be determined and properly addressed. Third, this decomposition facilitates a determination of new fields of research by identifying not-yet-used concepts. Since the beginning of Augmented Reality research, a growing number of applications have emerged that exploit various ways to represent information. This paper resumes this development and presents a set of independent dimensions covering representation principles of virtual information related to a physical environment: the temporality of virtual information, dimensionality, the frame of reference, mounting/registration and the type of reference. The suitability of the devised dimensions is tested by categorizing a wide variety of AR applications. The categorized data is analyzed for the most-often and less-frequently used combinations of classes. In particular, the classes that have not yet been used exhibit the potential to allow future work that investigates new options for information presentation. [Copyright &y& Elsevier]

Abstract: We propose a real-time solution for modeling and tracking multiple 3D objects in unknown environments for Augmented Reality. The proposed solution consists of both scalable tracking and interactive modeling. Our contribution is twofold: First, we show how to scale with the number of objects using keyframes. This is done by combining recent techniques for image retrieval and online Structure from Motion, which can be run in parallel. As a result, tracking 50 objects in 3D can be done within 6–35ms per frame, even under difficult conditions for tracking. Second, we propose a method to let the user add new objects very quickly. The user simply has to select in an image a 2D region lying on the object. A 3D primitive is then fitted to the features within this region, and adjusted to create the object 3D model. We demonstrate the modeling of polygonal and circular-based objects. In practice, this procedure takes less than a minute. [Copyright &y& Elsevier]

Abstract: Recently, mobile-phone based outdoor augmented reality (AR) systems have become readily available. One of the most popular applications are AR browsers that show virtual points of interest (POIs) overlaid on top of the phone''s camera view. These virtual cues can be used to guide people to the POIs. However, the usefulness of AR systems for guiding users to POI has not yet been evaluated, especially when compared to map interfaces. In this paper we present results of a user study comparing navigation with information typically provided by currently available handheld AR browsers, to navigation with a digital map, and a combined map and AR condition. We found no overall difference in task completion time, but found evidence that AR browsers are less useful for navigation in some environment conditions. We also found that navigation performance differed significantly with gender for the Map and AR+Map interfaces, but is very similar across gender for the AR interface. Users preferred the combined AR+Map condition, and felt that there were significant problems with using the AR view alone for navigation. [Copyright &y& Elsevier]

Abstract: In augmented reality, consistent illumination is an important aspect of integrating a virtual object into a video of a real scene. In this paper, we propose a novel image-based framework to estimate online the dynamically changing illumination parameters of outdoor video sequences captured by a fixed or pure rotation camera. Unlike previous approaches, which either require knowledge of the scene geometry or significant storage to preserve time- and view-dependent basis images or statistical parameters, our approach requires a very simple interaction at the initialization stage: a few brushes are used to select areas with a specified normal surface, and these areas are used to calculate the sunlight parameters. Additional areas with the same normal parameters will be detected automatically, and an optimization procedure is applied to ensure the robustness and precision of our estimation. Our new approach adapts to videos of the same scene with varied sun positions and camera panning. The experimental results demonstrate the effectiveness and flexibility of our proposed approach. [Copyright &y& Elsevier]