Search

Sketching is a common way to depict design ideas freely. Augmented reality technology extends the human–computer interaction space from the screen to the real world, providing a new application space for 3D sketching. However, due to the difficulty of depth perception, it is difficult for sketching strokes to maintain good planar and spatial consistency, making drawing 3D sketches in AR challenging. In this article, we propose FMASketch—a method of drawing mid-air sketches similar to building blocks. The auxiliary surfaces can ensure that the strokes drawn by users can always stay on the same plane as drawing in 2D space. Gestures are used to stitch together multiple drawing planes, and the strokes on the planes form a 3D sketch, which is very similar to how 3D modeling is done on the desktop by drawing different views and assembling them. Experiments have verified the effectiveness of the auxiliary surface generation method and gesture operations in this article. The mid-air sketching method is easy to learn and provides a new idea for rapid modeling in AR scenarios. [ABSTRACT FROM AUTHOR]

Gamified and immersive educational interventions have proven to be powerful motivators to encourage participation in unattractive activities, and human–computer interaction (HCI) is crucial for the successful development of such educational interventions. However, existing interaction methods suffer from either visuo-tactile inconsistency or unfriendly interaction methods, resulting in an unenjoyable user experience. Hence, in this article, the visuo-tactile interaction design based on Norman's design principles (visibility, feedback, constraint, consistency, affordance, and mapping) and spatial augmented reality (SAR) for intuitive cultural education is proposed for the first time, and an application for Chinese chess, SARChess, is implemented as a case study. Players interact with tangible chess pieces rather than through a digital interface, and the visual output and tangible input are integrated seamlessly to achieve high-level visuo-tactile consistency. Furthermore, a mixed investigation method based on pretest–posttest design, subjective questionnaires, and theme-based interviews was designed and performed to evaluate the user experience, learning performance, and player motivation of SARChess from both objective and subjective perspectives. The results show that SARChess has a significantly better user experience, learning performance, and player motivation than conventional chess and mouse-interaction-based chess, and the unique gameplay and visual cues in SARChess make it more engaging to participants. Additionally, this study finds and summarizes some design implications for applying design principles to traditional cultural education applications. [ABSTRACT FROM AUTHOR]

Background: Cognitive load during AR use has been measured conventionally by performance tests and subjective rating. With the growing interest in physiological measurement using non‐invasive biometric sensors, unbiased real‐time detection of cognitive load in AR is expected. However, a range of sensors and parameters are used in various subject fields, and reported results are fragmented. Objectives: The aim of this review is to analyse systematically how physiological methods have been used to measure cognitive load and what the implications are for the future research on AR‐based tools. Methods: This paper took the systematic review approach. Through screening with 10 exclusion criteria, 23 studies, that contain 3 key elements: AR‐based intervention, cognitive state examination and physiological methods, were identified, analysed and synthesised. Results: Physiological methods in their current form require reference to provide meaningful interpretations and suggestions. Therefore, they are often combined with conventional methods. Many studies investigate the effect of wearable devices in comparison with non‐AR stimuli, which has been controversial, but detection of different causes of cognitive load are on the horizon. Eye‐tracking is the method most used and most consistent in the use of its parameters. Conclusions: A multi‐method approach combining two or more evaluation instruments is essential for the validation of users' cognitive state. In addition to the AR stimuli in question, having another independent variable such as task difficulty in experiment design is useful. Statistical approaches with more data input could help establish a reliable scale. The future research should attempt to dissociate cognitive load caused by different effects such as device, instruction, and other AR techniques as well as intrinsic and extraneous aspects, in a better experimental setup with multiple parameters. Lay Description: What is already known about this topic: The efficacy of AR‐based instructional tools can be evaluated using the concept of cognitive load.Cognitive load has been conventionally measured by self‐reporting and performance.There is a growing interest in physiological measurement of cognitive load with non‐invasive biometric sensors. What this paper adds: This paper systematically analyses how physiological methods are used to measure cognitive load in AR.It synthesises the fragmented results of AR studies conducted with a range of physiological methods. Implications for practice: Combining two or more evaluation instruments is essential for the validation of users' cognitive state.Eye‐tracking is the method most used, and fixation duration is the most consistent parameter.A statistical approach with more biometric data input could help establish a reliable cognitive load scale.Future research should attempt to separate cognitive load caused by different effects and different cognitive aspects. [ABSTRACT FROM AUTHOR]

Many forms of Intangible Culture Heritage (ICH) are fast disappearing. The Lantern Festival is an important part of Chinese tradition and a critical ICH of Taiwan. However, few young people attend ICH activities. Given the importance of ICH education, this study describes what participants learned after lantern exhibitions that combined Augmented Reality (AR), mobile websites, and Paper-based Exhibition Brochures. This study evaluated what a cohort of college students (n = 115) learned about ICH through experiments and questionnaires. The results show that the public understood ICH better through digital technology. All participants thought that applying AR to ICH was good for the public to gain more information on lantern culture, although it took a lot of time to learn the new technology. The results also encouraged the application of AR to engage in ICH education. Therefore, a large number of digital technologies that are popular among young people should be used as part of ICH education in order to improve their cultural awareness and learning motivation. [ABSTRACT FROM AUTHOR]

A satisfactory graphic design and good-looking 3D models and environments are the backbones of a positive user experience, especially in Augmented Reality (AR) / Virtual Reality (VR) app development. Where these technologies is seen as the an excellent realm of human-computer interaction. The purpose is to fool the viewer by the seamless incorporation of simulated features. Every AR system relies on true interaction and three-dimensional registration to function properly. In this research, we present a strategy for real-world 3D image registration and tracking. The primary foci of this study are the first three stages: initial registrations and matrix acquisitions, road scene feature extraction, and virtual information registration. At initial registration, a rough virtual plane is estimated onto which the objects will be projected. To this, we propose YoloV3 for transferring features from a virtual to a real-world setting. The projection process concludes with a guess at the camera's posture matrix. This tech is used in the vehicle's head-up display to augment reality. The average time required to register a virtual item is 43 seconds. The final step in making augmented reality content is to merge the computer-generated images of virtual objects with real-world photographs in full colour. Our results indicate that this method is effective and precise for 3D photo registration but has the potential to dramatically increase the verisimilitude of AR systems. [ABSTRACT FROM AUTHOR]

Hand reconstruction is the foundation of virtual reality and human–computer interaction, but currently it still faces challenges such as blurred interaction edge and inter‐hand occlusion. To solve these challenges, in this letter, the authors propose a framework called Multilevel Two Hand Interactive Former (MTHI‐Former) which considers the vertices of a hand model as graph structures and learn the connectivity relationships information between vertices. In this framework, the authors introduce two novel modules. The first is the Multi‐branch Image Feature Extraction Module , which is utilized to obtain accurate hand features. The second is the Multilevel Two Hand Interaction Module , which is utilized to fuse interactive hand information to enhance the attention features of interactive edges, and fuse vertex relationships to determine the occlusion relationship between interactive hands. The authors compare their method with recent methods on the InterHand 2.6M dataset, and the experimental results show that their method outperforms other representative methods, achieving a result of 13.15 mm and an improvement in performance of about 18%. [ABSTRACT FROM AUTHOR]

Augmented Reality (AR) provides new ways for situated visualization and human‐computer interaction in physical environments. Current evaluation procedures for AR applications rely primarily on questionnaires and interviews, providing qualitative means to assess usability and task solution strategies. Eye tracking extends these existing evaluation methodologies by providing indicators for visual attention to virtual and real elements in the environment. However, the analysis of viewing behavior, especially the comparison of multiple participants, is difficult to achieve in AR. Specifically, the definition of areas of interest (AOIs), which is often a prerequisite for such analysis, is cumbersome and tedious with existing approaches. To address this issue, we present a new visualization approach to define AOIs, label fixations, and investigate the resulting annotated scanpaths. Our approach utilizes automatic annotation of gaze on virtual objects and an image‐based approach that also considers spatial context for the manual annotation of objects in the real world. Our results show, that with our approach, eye tracking data from AR scenes can be annotated and analyzed flexibly with respect to data aspects and annotation strategies. [ABSTRACT FROM AUTHOR]

Electrodeposition is an old and effective method for the fabrication of organic films. Though electrodeposited organic films have been widely used in various applications, highly luminescent films have been a great challenge because the electrochemically doped state may strongly quench the fluorescence. In the first part of this review, the organic electrodeposition techniques, along with general electropolymerization and other special electrodepositions are introduced. In the second part of the review, we describe how to electrochemically fabricate luminescent films for organic light-emitting diodes (OLEDs). With the rational molecular design and well-controlled electrodeposition process, we have not only demonstrated high-performance OLEDs, but also paved a promising way to practice active-matrix OLEDs (AMOLEDs) and super-resolution OLEDs. In particular, RGB 3 × 3 array OLEDs based on active-matrix substrates, RGB passive-matrix OLEDs (PMOLEDs) with a resolution of 210 ppi, and monochromatic OLEDs with a super-resolution of 2822 ppi have been successfully fabricated. It is highly anticipated that the organic electrodeposition technology is of comparable or perhaps even higher contenders in manufacturing and downscaling OLEDs and AMOLEDs with low-cost and high-resolution for the human-computer interaction fields such as augmented reality (AR), virtual reality (VR), etc. [ABSTRACT FROM AUTHOR]

Aiming at the problem that it is difficult to evaluate the interaction efficiency of new interaction means such as voice and gesture in the application of command and control system, based on the analysis of the interaction efficiency evaluation of traditional 2D graphical user interface GOMS, a human-computer interaction GOMS model oriented to augmented reality command is proposed. The model combines the features of 3D interface and multi-modal interaction in the augmented reality environment to improve the design. By analyzing the use time of different interaction modes and the influence of different interaction scenarios on the interaction operation time, the model predicts the time required by users to complete a certain task, quantifies the performance of different interaction modes, and provides basis and reference for the optimization design of the interaction system. [ABSTRACT FROM AUTHOR]

With innovations in the field of gaze and eye tracking, a new concentration of research in the area of gaze-tracked systems and user interfaces has formed in the field of Extended Reality (XR). Eye trackers are being used to explore novel forms of spatial human-computer interaction, to understand human attention and behavior, and to test expectations and human responses. In this article, we review gaze interaction and eye tracking research related to XR that has been published since 1985, which includes a total of 215 publications. We outline efforts to apply eye gaze for direct interaction with virtual content and design of attentive interfaces that adapt the presented content based on eye gaze behavior and discuss how eye gaze has been utilized to improve collaboration in XR. We outline trends and novel directions and discuss representative high-impact papers in detail. [ABSTRACT FROM AUTHOR]

Human action recognition systems use data collected from a wide range of sensors to accurately identify and interpret human actions. One of the most challenging issues for computer vision is the automatic and precise identification of human activities. A significant increase in feature learning-based representations for action recognition has emerged in recent years, due to the widespread use of deep learning-based features. This study presents an in-depth analysis of human activity recognition that investigates recent developments in computer vision. Augmented reality, human–computer interaction, cybersecurity, home monitoring, and surveillance cameras are all examples of computer vision applications that often go in conjunction with human action detection. We give a taxonomy-based, rigorous study of human activity recognition techniques, discussing the best ways to acquire human action features, derived using RGB and depth data, as well as the latest research on deep learning and hand-crafted techniques. We also explain a generic architecture to recognize human actions in the real world and its current prominent research topic. At long last, we are able to offer some study analysis concepts and proposals for academics. In-depth researchers of human action recognition will find this review an effective tool. [ABSTRACT FROM AUTHOR]

Human hand gestures are the most important tools for interacting with the real environment. Capturing hand motion is critical for a wide range of applications in Augmented Reality (AR)/Virtual Reality (VR), Human-computer Interface (HCI), and many other disciplines. This paper presents a 3 module pipeline for effective hand gesture detection in real-time at the speed of 100 frames per second (fps).Various hand gestures can be captured by simple RGB camera and then processed to first detect the palm and then find essential 3D landmarks, which helps in creating skeletal representation of hand. In order to form a 3D mesh around the skeletal hand 2D and 3D annotations of Hand gestures are merged and in the final module 3D animated hand gestures are presented using advanced neural network. 3D representation of hand gestures ensures greater understanding of depth ambiguity problem in monocular pose estimations and can be effectively used in computer vision and graphics applications. The proposed design is compared with several benchmarks to highlight improvements in the results achieved over conventional methods. [ABSTRACT FROM AUTHOR]

Augmented reality (AR) applications have gainedmuch research and industry attention.Moreover, themobile counterpart--mobile augmented reality (MAR) is one of the most explosive growth areas for AR applications in the mobile environment (e.g., smartphones). The technical improvements in the hardware of smartphones, tablets, and smart-glasses provide an advantage for thewide use ofmobile AR in the realworld and experience these AR applications anywhere. However, themobile nature ofMAR applications can limit users' interaction capabilities, such as input and haptic feedback. In this survey, we analyze current research issues in the area of human-computer interaction for haptic technologies in MAR scenarios. The survey irst presents human sensing capabilities and their applicability in AR applications. We classify haptic devices into two groups according to the triggered sense: cutaneous/tactile: touch, active surfaces, and mid-air; kinesthetic: manipulandum, grasp, and exoskeleton. Due to MAR applications' mobile capabilities, we mainly focus our study on wearable haptic devices for each category and their AR possibilities. To conclude, we discuss the future paths that haptic feedback should follow for MAR applications and their challenges. [ABSTRACT FROM AUTHOR]

Mobile applications on smartphones and tablets have become part of our everyday lives. The number of augmented reality (AR) technology applications is also increasing. Augmented reality has proven to be effective in various areas of human life, from education, marketing, and training to navigation. All people have the right to access information and use available technologies, but not everyone has the same opportunities. To contribute to the digital inclusion of people who are often disadvantaged in this regard, we should think about the accessibility of digital technologies, including mobile augmented reality (MAR). The specificity of MAR is a new way of human–computer interaction compared to traditional mobile solutions. The objective of this review paper is to analyze the handheld AR solutions developed for people with different disabilities to identify accessibility challenges related to interaction when performing different tasks in AR. It also explores and presents accessibility features and other accessibility best practices, as well as potential future research directions related to the personalization and customization of such solutions for individuals. The results of this literature review can contribute to the creation of accessibility guidelines in the field of handheld AR and encourage the development of accessible AR solutions that can benefit not only people with disabilities but also people without disabilities. [ABSTRACT FROM AUTHOR]

A sociological revision of Aron Gurwitsch provides a helpful layered theory of conscious experience as a four-domain structure: the theme, the thematic field, the halo, and the social horizon. The social horizon—the totality of the social world that is unknown, vaguely known, taken for granted, or ignored by the subject despite objectively influencing the thoughts and actions of the subject—, helps conceptualize how everyday human–computer interaction (HCI) can obscure social structures. Two examples illustrate the usefulness of this framework: (1) illuminating new forms of social control known as "surveillance capitalism" that influence the other domains of consciousness despite being invisible to the subject and (2) explaining how computer use tends to "hide" the ecological impacts of digital-technological systems from the attention of subjects. A sociological four-domain theory of consciousness supplements ethnomethodological HCI research by preserving the field's long-standing focus on micro-level interactions and user experience while simultaneously drawing attention to a "commonsense ignorance of social structure"—in this case, the invisibility of the social structures behind personal computer use. Social structure can be reproduced even when actors are unaware of engaging in its reproduction. [ABSTRACT FROM AUTHOR]

Summary: Operators want to maintain awareness of the structure being tested while observing sensor data. Normally, the human's gaze shifts to a separate device or screen during the experiment for data information, missing the structure's physical response. The human–computer interaction provides valuable data and information but separates the human from the reality. The sensor data does not collect experiment safety, quality, and other contextual information of critical value to the operator. To solve this problem, this research provides humans with real‐time information about vibrations using an augmented reality (AR) application. An application is developed to augment sensor data on top of the area of interest, which allows the user to perceive real‐time changes that the data may not warn of. This paper presents the results of an experiment that show how AR can provide a channel for direct sensor feedback while increasing awareness of reality. In the experiment a researcher attempts to closely follow a moving sensor with their own sensor while observing the moving sensor's data with and without AR. The results of the reported experiment indicate that augmenting the information collected from sensors in real time narrows the operator's focus to the structure of interest for more efficient and informed experimentation. [ABSTRACT FROM AUTHOR]

How do researchers search for knowledge? What are their behaviors and habits, and what technologies do they use? This article proposes that three design shifts - involving more human senses, enabling comparative and simultaneous viewing, and allowing immediate access to full content - will create a more fruitful research process for scholars who conduct a literature review, learn about a new topic in a related discipline, or catch up on advances in their field. The three proposed designed shifts were tested with prototyping. The systematic prototyping procedure is a method that can be employed by others to advance this field. [ABSTRACT FROM AUTHOR]

As lightweight, low-cost EEG headsets emerge, the feasibility of consumer-oriented brain–computer interfaces (BCI) increases. The combination of portable smartphones and easy-to-use EEG dry electrode headbands offers intriguing new applications and methods of human–computer interaction. In previous research, augmented reality (AR) scenarios have been identified to profit from additional user state information—such as that provided by a BCI. In this work, we implemented a system that integrates user attentional state awareness into a smartphone application for an AR written language translator. The attentional state of the user is classified in terms of internally and externally directed attention by using the Muse 2 electroencephalography headband with four frontal electrodes. The classification results are used to adapt the behavior of the translation app, which uses the smartphone's camera to display translated text as augmented reality elements. We present the first mobile BCI system that uses a smartphone and a low-cost EEG device with few electrodes to provide attention awareness to an AR application. Our case study with 12 participants did not fully support the assumption that the BCI improves usability. However, we are able to show that the classification accuracy and ease of setup are promising paths toward mobile consumer-oriented BCI usage. For future studies, other use cases, applications, and adaptations will be tested for this setup to explore the usability. [ABSTRACT FROM AUTHOR]

Communication is an essential part of most professional activities. For complex industrial products such as ships, some design problems can only be found during the construction process. However, it is time-consuming for on-site assemblers to provide feedback on problems through traditional methods such as phone calls and photos. In recent years, adding structured visual cues through augmented reality (AR) has become an important method to assist remote collaborative tasks. However, previous studies and commercial solutions often had limited annotation types and relied on specific devices for tracking and reconstruction, which is unsatisfactory for deployment in industrial equipment and large scenes. The presented work tackles issues with feedback and collaborative decision-making of design problems in the pipe outfitting stage and proposes a cross-platform AR annotation solution on desktop computers, industrial tablets, and AR head-mounted devices (HMD). The novelty of the current work lies in a cross-functional annotation taxonomy for real-time collaboration, device-cloud integrative context awareness methods based on RGB-D sensors compatible with more devices, 2D–3D annotation mapping algorithms, and visual enhancement strategies for large scenes. A prototype system is developed and verified on a ship unit model and a physical pipe platform. The results indicate that the solution provides good performance in real-time capability and reconstruction precision, allows rapid localization of faulty parts, and creates annotations flexibly and reliably. This solution is also expected to play an active role in collaborative quality inspection and intelligent management in shipbuilding and other heavy industries. [ABSTRACT FROM AUTHOR]

Fire safety training is crucial to increase building occupants' chances of surviving a fire emergency. Traditional training methods, such as lectures using video presentations, have limitations that can hinder learning performance. This article describes the development and testing of an alternative training solution using Augmented Reality (AR) technology. Through a controlled between-subject experiment, the AR-based fire safety training method was compared against a conventional video-based one based on participants' knowledge acquisition and retention, intrinsic motivation, and self-efficacy. Results suggest that the AR system was well-designed and as effective as the traditional method in terms of knowledge acquisition and retention and overall learning experience. However, it was found to be superior in terms of intrinsic motivation gain and self-efficacy retention. These findings demonstrate the potential of AR-based training methods to enhance building occupants' safety and provide directions for future developments and research in the field. • An AR fire safety training system is developed and compared to video training. • Training effectiveness is assessed through learning performance measures. • AR training outperformed video training in intrinsic motivation and self-efficacy. • The developed AR training system may enhance building occupants' safety. [ABSTRACT FROM AUTHOR]

Distance education theories indicate the importance of learner–content interaction and discuss it in conjunction with learner–instructor and learner–learner interaction. The measures of learner–content interaction are inconsistent and often measure learner–learner and learner–instructor interaction only. Using content analysis, this study examined 107 manuscripts to understand how learner–content interaction is currently measured to identify what constructs are being considered and the measurement tools being used. Results report 113 different measurement tools in use, showing a large variety of measurement items. Most tools were designed for specific cases and not built upon other relevant tools. A universal learner–content interaction measurement tool must consider being proactive; having strong pedagogical base, learner agency, motivation, and engagement; and measure a variety of learning experiences. [ABSTRACT FROM AUTHOR]

This study investigates the use of augmented reality technology (AR) in the field of maritime navigation and how researchers and designers have addressed AR data visualisation. The paper presents a systematic review analysing the publication type, the AR device, which information elements are visualised and how, the validation method and technological readiness. Eleven AR maritime solutions identified from scientific papers are studied and discussed in relation to previous navigation tools. It is found that primitive information such as course, compass degrees, boat speed and geographic coordinates continue to be fundamental information to be represented even with AR maritime solutions. [ABSTRACT FROM AUTHOR]

This research aims to define and demonstrate using dynamically loaded content (DLC) as interactive 3D objects in an augmented reality environment and explores how humans will interact with the DLC in an augmented reality spatial environment. Several vital elements are investigated in this research, including how the interface provides an icon (or 3D marker) to activate the DLC, what features and capabilities the viewing HUD would need for the user, and what functionality can be transferred along with the DLC. The primary hardware for this research is the HoloLens 1 and 2. This research project is divided into three phases: 1) Designing an interface system to allow users to view, select, and adjust DLC as if the content were a standalone application. 2) Loading the DLC and interacting with its specific functionality. 3) Standardizing a language to transfer DLC from the web to augmented reality devices without installing a separate program. The conventional method of creating dynamically loaded content is to source their properties and functionality from data stored within programs that are installed into the device. The issue with this approach is needing to install separate programs onto the device to configure different content dynamically. The novel approach in this research sources DLC, properties, and functionalities from data stored on the web. The DLC will then be represented to the user as 3D objects in the 3D space. [ABSTRACT FROM AUTHOR]

With the development of motion capture technologies, 3D action recognition has become a popular task that finds great applicability in many areas, such as augmented reality, human–computer interaction, sports, or healthcare. On the other hand, the acquisition of 3D human skeleton data is an expensive and time-consuming process, mainly due to the high costs of capturing technologies and the absence of suitable actors. We overcome these issues by focusing on the 2D skeleton modality that can be easily extracted from ordinary videos. The objective of this work is to demonstrate a high descriptive power of such a 2D skeleton modality by achieving accuracy on the task of daily action recognition competitive to 3D skeleton data. More importantly, we thoroughly analyze the factors that significantly influence the 2D recognition accuracy, such as the sensitivity towards data normalization, scaling, quantization, and 3D-to-2D distortions in skeleton orientations and sizes, which are caused by the loss of depth dimension and fixed-angle camera view. We also provide valuable insights on how to mitigate these problems to increase recognition accuracy significantly. The experimental evaluation is conducted on three datasets different in nature. The ability to learn different types of actions better using either 2D or 3D skeletons is also reported. Throughout experiments, a generic light-weight LSTM network is used, whose architecture can be easily tuned to achieve the desired trade-off between its accuracy and efficiency. We show that the proposed approach achieves not only the state-of-the-art results in 2D skeleton action recognition but is also highly competitive to the best-performing methods classifying 3D skeleton sequences or the visual content extracted from ordinary videos. [ABSTRACT FROM AUTHOR]

10 years ago, Castellani et al. (Journal of Computer Supported Cooperative Work, vol. 18, no. 2–3, 2009, pp. 199–227, 2009) showed that using just an audio channel for remote troubleshooting can lead to a range of problems and already envisioned a future in which augmented reality (AR) could solve many of these issues. In the meantime, AR technologies have found their way into our everyday lives and using such technologies to support remote collaboration has been widely studied within the fields of Human-Computer Interaction and Computer-Supported Cooperative Work. In this paper, we contribute to this body of research by reporting on an extensive empirical study within a Fab Lab of troubleshooting and expertise sharing and the potential relevance of articulation work to their realization. Based on the findings of this study, we derived design challenges that led to an AR-based concept, implemented as a HoloLens application, called shARe-it. This application is designed to support remote troubleshooting and expertise sharing through different communication channels and AR-based interaction modalities. Early testing of the application revealed that novel interaction modalities such as AR-based markers and drawings play only a minor role in remote collaboration due to various limiting factors. Instead, the transmission of a shared view and especially arriving at a shared understanding of the situation as a prerequisite for articulation work continue to be the decisive factors in remote troubleshooting. [ABSTRACT FROM AUTHOR]

Perceived workload and usability are crucial components of human-computer interactions. Currently, there is a gap in research comparing Augmented Reality (AR) and Virtual Reality (VR) systems for workload and usability. This study attempts to bridge that gap through the comparison of the HP Windows Mixed Reality system and the Meta 2 system for a ball-sorting task. Subjective questionnaires on workload and usability were implemented as comparative measures for three game scenarios of increasing difficulty. Forty-one participants were recruited from the University of Central Florida and its surrounding communities. Results showed significantly lower cumulative total workload and greater usability (for the subscale of ease of use) for the HP Windows Mixed Reality system when compared to the Meta 2 system. There were no statistically significant differences reported for the other usability subscales between the two systems. Also, there were no statistically significant differences in total workload within the three scenarios for both systems. The findings could be attributed to differences in control schemes (i.e., native handheld controllers versus hand gestures), user experience with AR and VR systems, and difficulty of task scenarios. [ABSTRACT FROM AUTHOR]

• Using Geospatial Technologies to understand Mixed Reality better. • Utilizing the Internet of Things to communicate between Mixed Reality Environments and the Real Worlds. • Defining a novel Virtual-to-Real interaction method for Mixed Reality Interactions. • Demonstrating a case study with an application prototype, where a virtual man executes a physical task. As Mixed Reality applications penetrate people's daily activities, strong synergies with Geospatial technologies and the Internet of Things are revealed. The first is due to the significance of the spatial reference of all involved actors of a Mixed Reality environment: the end-user, the real and the virtual objects. The second is due to the ever-increasing participation of sensors controlling devices and machines always and everywhere. This paper highlights these synergies and proposes a case proving they crucially empower a Mixed Reality experience. Through this synergy, a novel interaction method for Mixed Reality Environments emerges, the "physical task execution" interaction method. An open-source application prototype was developed to support the case study where we assign a physical task to a virtual man to execute, utilising the Internet of Things in a Mixed Reality Environment. [ABSTRACT FROM AUTHOR]

Shape-from-focus depth measurement is widely used in equipment manufacturing, biomedicine, heritage conservation, engineering measurement, etc. However, the acquisition of multi-focus image sequence is not very flexible and time-consuming in conventional shape from focus. Here, a novel depth measurement from shape from focus method based on ultrafast optical axial scanning was proposed. An electronically tunable lens was used to achieve high-speed focus shift, by synchronizing a CCD with the electronically tunable lens, multi-focus image sequence can be acquired at high speed (up to 2.5 ms). The focusing area of each multi-focus image was extracted by the energy of image gradient operator. Finally, the motionless depth measurement can be achieved. A tilted marker pen was used to test the performance of the depth measurement, the results show that the depth resolution of the proposed method is measured to be ∼ 3.2 cm in a depth range of ∼ 24 cm. The feasibility of the proposed method was verified using misplaced portrait sculptures. This method will further expand the application of depth measurement from shape from focus in human–computer interaction, 3D shape measurement, and augmented reality. [ABSTRACT FROM AUTHOR]

Augmented reality (AR) interactions have been associated with increased biomechanical loads on the neck and shoulders. To provide a better understanding of the factors that may impact such biomechanical loads, this repeated-measures laboratory study evaluated the effects of error rates and target sizes on neck and shoulder biomechanical loads during two standardized AR tasks (omni-directional pointing and cube placing). Twenty participants performed the two AR tasks with different error rates and target sizes. During the tasks, angles, moments, and muscle activity in the neck and shoulders were measured. The results showed that the target sizes and error rates significantly affected angles, moments, and muscle activity in the neck and shoulder regions. Specifically, the presence of errors increased neck extension, shoulder flexion angles and associated moments. Muscle activity in the neck (splenius capitis) and shoulder (anterior and medial deltoids) also increased when the errors were introduced. Moreover, interacting with larger targets resulted in greater neck extension moments and shoulder abduction angles along with higher muscle activity in the splenius capitis and upper trapezius muscles. These findings indicate the importance of reducing errors and incorporating appropriate target sizes in the AR interfaces to minimize risks of musculoskeletal discomfort and injuries in the neck and shoulders. • Biomechanical loads during augmented reality (AR) interactions were evaluated. • Errors and larger targets increased neck and shoulder biomechanical loads. • Error reduction and appropriate target sizes in AR may help reduce musculoskeletal risks. [ABSTRACT FROM AUTHOR]

Mixed reality (MR) technology development is now gaining momentum due to advances in computer vision, sensor fusion, and realistic display technologies. With most of the research and development focused on delivering the promise of MR, the privacy and security implications of this technology are yet to be thoroughly investigated. This survey article aims to put in to light these risks and to look into the latest security and privacy work on MR. Specifically, we list and review the different protection approaches that have been proposed to ensure user and data security and privacy in MR. We extend the scope to include work on related technologies such as augmented reality, virtual reality, and human-computer interaction as crucial components, if not the origins, of MR, as well as numerous related work from the larger area of mobile devices, wearables, and Internet-of-Things. We highlight the lack of investigation, implementation, and evaluation of data protection approaches in MR. Further challenges and directions on MR security and privacy are also discussed. [ABSTRACT FROM AUTHOR]

• A gaze-based method that enables pure hands-free object positioning in a virtual world for head-mounted displays. • Our method can function with both pure head gaze and pure eye gaze. • Experimental results show that eye and head gaze-based control with our method can achieve 78% and 96% success rates, respectively, when compared to a hand-held controller. In virtual and augmented reality, gaze-based methods have been explored for decades as effective user interfaces for hands-free interaction. Though several well-known gaze-based methods exist for simple interactions such as selection, no solutions exist for 3D manipulation tasks requiring a higher degree of freedom (DoF). In this paper, we introduce OrthoGaze, a novel user interface that allows users to intuitively manipulate the three-dimensional position of a virtual object using only their eye or head gaze. Our approach makes use of three selectable, orthogonal planes, where each plane not only helps guide the user's gaze in an arbitrary virtual space, but also allows for 2-DoF manipulations of object position. To evaluate our method, we conducted two user studies involving aiming and docking tasks in virtual reality to evaluate the fundamental characteristics of sustained gaze aiming and to determine which type of gaze-based control performs best when combined with OrthoGaze. Results showed that eye gaze was more accurate than head gaze for sustained aiming. Additionally, eye and head gaze-based control for 3D manipulations achieved 78% and 96% performance, respectively, in comparison with a hand-held controller. Subjective results also suggest that gaze-based manipulation can comprehensively cause more fatigue than controller-based. From the experimental results, we expect OrthoGaze to become an effective method for pure hands-free object manipulation in head-mounted displays. [ABSTRACT FROM AUTHOR]

Conventionally, the interaction between a user and augmented reality (AR) application is limited. Mostly, it allows virtual information browsing on the AR marker or basic manipulation such as moving and resizing of the 3D model. This study presents a rich interaction that allows users to mimic physical action such as shaking and pouring in chemistry experiment using card-based or box-based marker design. The user is required to follow the instruction to select the correct AR marker that represents apparatus or solutions and performs the physical action to conduct an experiment. Usability's result showed the respondents were satisfied with the application. [ABSTRACT FROM AUTHOR]

Floating three‐dimensional (3D) display implements direct interaction between human hands and virtual 3D images, which offers natural and effective augmented reality interaction. In this study, we propose a novel floating autostereoscopic display, combining head tracking lenticular display with an image projection system, to offer the observers with an accurate 3D image floating in midair without any optical elements between observers and the virtual 3D image. Combined with a gesture recognition device, the proposed system can achieve in situ augmented reality interaction with the floating 3D image. A distortion correction method is developed to achieve 3D display with accurate spatial information. Moreover, a coordinate calibration method is designed to improve the accuracy in the in situ interaction. Experiments were performed to prove the feasibility of the proposed system, and the good results show the potential of human‐computer interaction in medicine and life sciences. [ABSTRACT FROM AUTHOR]

• A novel augmented reality (AR) based choreographing interface is presented. • It is tested against a virtual reality (VR), PC and mobile based interfaces. • We evaluate the user experience and usability of different interactive techniques. • Augmented Reality is effective in training and education applications. Choreography is a creative art of defining sequences of movement. The choreographing process benefits a lot from the use of digital applications. In this study, we investigate the effects of different interactive techniques on user experience for a choreography generator interface. We develop a virtual reality-based and an augmented reality-based interface and compare them with 1) a personal computer-based choreography generator interface and 2) a touch-based mobile application for choreography generation. We evaluate user performance and user experience on the interfaces in terms of task completion time, as well as subjective criteria, like mental stress, physical stress, and pleasure. Our research contributes to the study of how different interaction methods of the same application affect user experience. This research also contributes to the study of virtual and augmented reality in education and training applications. The results verify the effectiveness of augmented reality in developing training and design applications, particularly choreography. The augmented reality interface emerged as the preferred choice for users. Furthermore, the results demonstrate the effectiveness of natural user interaction approaches in interface design. [ABSTRACT FROM AUTHOR]

Medical imaging technologies have become an essential component in different areas related to health care. Volume visualization (VV) of medical data is an invaluable support in tasks such as clinical diagnosis, treatment planning, surgery rehearsal, education, and research. Several algorithms and systems have been developed to enable the visualization and interaction with volumetric data. Slice-based visualization methods dominate the field of medical volumes scanning since they allow more detailed analysis of the data. However, an intensive training is usually required for the user to be able to effectively explore the data. In this paper, we present novel a slice-based methodology which objective is to facilitate the exploration of medical volumetric data. The proposed method consist of the use of augmented reality principles to determine the spatial position and orientation of rigid planar objects within a defined space in the real-world which represents the medical volumetric information. The results obtained by a usability study indicate the feasibility of employing this technique for a natural human-computer interaction with the medical data, having the potential of making the process of medical volume exploration more easy and intuitive. [ABSTRACT FROM AUTHOR]

• Reviewing recent trends in markerless vision-based 3D hand motion tracking • Proposing a new taxonomy based on hand-object interaction constraints • Distinguishing between tracking in isolation and in real/virtual interaction • Studying some datasets in the context of hand-object interaction Unlabelled Image Vision-based 3D hand tracking is a key and popular component for interaction studies in a broad range of domains such as virtual reality (VR), augmented reality (AR) and natural human-computer interaction (HCI). While this research field has been well studied in the last decades, most approaches have considered the human hand in isolation and not in action or in interaction with the surrounding environment. Even the common collaborative and strong interactions with the other hand have been ignored. However, many of today's computer applications require more and more hand-object interactions. Furthermore, employing contextual information about the object in the hand (e.g. the shape, the texture, and the pose) can remarkably constrain the tracking problem. The most studied contextual constraints involve interaction with real objects and not with virtual objects which is still a very big challenge. The goal of this survey is to develop an up-to-date taxonomy of the state-of-the-art vision-based hand pose estimation and tracking methods with a new classification scheme: hand-object interaction constraints. This taxonomy allows us to examine the strengths and weaknesses of the current state of the art and to highlight future trends in the domain. [ABSTRACT FROM AUTHOR]

In recent years, augmented reality games (ARGs) such as Pokémon Go have become increasingly popular. These games not only afford a novel gaming experience but also have the potential to alter how players view their physical realities. In addition to the common experiences and gratifications people derive from games, (location-based) ARGs can afford, for example outdoor adventures, communal activities, and health benefits, but also create problems stemming from, for example privacy concerns and poor usability. This raises some important research questions as to what drives people to use these new applications, and why they may be willing to spend money on the content sold within them. In this study, we investigate the various gratifications people derive from ARGs (Pokémon Go) and the relationship of these gratifications with the players' intentions to continue playing and spending money on them. We employ data drawn from players of Pokémon Go (N = 1190) gathered through an online survey. The results indicate that game enjoyment, outdoor activity, ease of use, challenge, and nostalgia are positively associated with intentions to reuse (ITR), meanwhile outdoor activity, challenge, competition, socializing, nostalgia and ITR are associated with in-app purchase intentions (IPI). In contrast with our expectations, privacy concerns or trendiness were not associated with reuse intentions or IPI. [ABSTRACT FROM AUTHOR]

As smoking has emerged as a health-risk behavior, communication scholars and practitioners have put many efforts into finding out ways to achieve smoking cessation. In this research, participants (N = 57) were randomly assigned to a spatial augmented reality (SAR) condition (3D projection mapping) and 2D flat screen to be exposed to an anti-smoking message. This research provides insightful evidence that the effects of SAR on people's behavioral intention to spread anti-smoking messages online could be explained by spatial presence and negative emotions. Implications for research on the potential of SAR in terms of emotions and online viral behavioral intentions are discussed. [ABSTRACT FROM AUTHOR]

Recent advancements in life-science instrumentation and automation enable entirely new modes of human interaction with microbiological processes and corresponding applications for science and education through biology cloud laboratories. A critical barrier for remote and on-site life-science experimentation (for both experts and nonexperts alike) is the absence of suitable abstractions and interfaces for programming living matter. To this end we conceptualize a programming paradigm that provides stimulus and sensor control functions for real-time manipulation of physical biological matter. Additionally, a simulation mode facilitates higher user throughput, program debugging, and biophysical modeling. To evaluate this paradigm, we implemented a JavaScript-based web toolkit, "Bioty," that supports real-time interaction with swarms of phototactic Euglena cells hosted on a cloud laboratory. Studies with remote and on-site users demonstrate that individuals with little to no biology knowledge and intermediate programming knowledge were able to successfully create and use scientific applications and games. This work informs the design of programming environments for controlling living matter in general, for living material microfabrication and swarm robotics applications, and for lowering the access barriers to the life sciences for professional and citizen scientists, learners, and the lay public. [ABSTRACT FROM AUTHOR]

An experiment is reported that studied the effects of spatial embodiment in augmented reality on medical attitudes about the self. College students (N = 90) viewed public service announcements (PSAs) with overlaid virtual fetuses and X-rayed images of lungs on various interfaces representing embodiment—a two-dimensional screen, a three-dimensional (3D) mannequin, and the participants' bodies (3D). Results indicated that PSA messages with richer embodied interfaces increase the sense of "being there," also known as spatial presence (SP), in sequential order; this leads to increased negative emotion regarding smoking cigarettes and an increased willingness to engage with a cigarette cessation campaign. When the SP mediates the dual model process, only affective attitudes increase the behavioral intention to engage with the campaign. [ABSTRACT FROM AUTHOR]

Highlights • The effectiveness of Intelligent Tutoring Systems in psychomotor domains can potentially be enhanced by combining them with Augmented Reality paradigms. • Due to ITS complexity and design, work on the use of ITSs with AR is significantly limited and how these paradigms work together is not well understood. • Existing work shows that combining ITSs with AR enhances significantly enhances performance and learning. • Combining AR with ITS requires a gateway interface and networking capabilities in both the ITS and the AR application, due to non-standardised mechanisms used in these systems. • Additional research on the design of the interface; authoring systems for ITSs in AR and their learning effectiveness is required. Graphical abstract Abstract Augmented Reality overlays virtual objects on the real world in real-time and has the potential to enhance education, however, few AR training systems provide personalised learning support. Combining AR with intelligent tutoring systems has the potential to improve training outcomes by providing personalised learner support, such as feedback on the AR environment. This paper reviews the current state of AR training systems combined with ITSs and proposes a series of requirements for combining the two paradigms. In addition, this paper identifies a growing need to provide more research in the context of design and implementation of adaptive augmented reality tutors (ARATs). These include possibilities of evaluating the user interfaces of ARAT and potential domains where an ARAT might be considered effective. [ABSTRACT FROM AUTHOR]

The Virtual Jumpcube is a virtual reality setup from 2015 that allows for jumping and flying in audiovisual virtual environments. Recently, we have included several haptic and olfactory stimuli that should further increase the degree of immersion in the experienced virtuality. These additional media channels were tested by the participants of several events and the feedback of 196 jumpers was gathered in a questionnaire. In this paper, we describe the stimulation hardware and software as well as the performed experiment and we present the major findings of the evaluation. It shows that if employed correctly, haptic and olfactory stimuli can enhance immersion and user experience significantly. Major success factors appear to be the amplitude and frequency of stimulation as well as the temporal synchronization with the other media channels, in particular the visual stimuli. [ABSTRACT FROM AUTHOR]

This paper outlines the results of a project to develop and test an augmented reality smart phone application ('app') to aid the safety of navigation of leisure sailors with limited navigational skills. The app works in two modes: in the 'turned off mode' the app gives an alarm 30 seconds before the boat is predicted to ground. The navigator then immediately stops the boat, picks up the phone and, uses the app to view the surrounding water where NoGo areas less than 3 metres in depth are outlined in red. By panning the smart phone around, safe escape routes with deep water are visible. The prototype app was tested on a group of boat owners in western Norway with very good results, both from technical and usability perspectives. This report outlines the concept of operation of the app, details some of the difficulties encountered in its development and testing, specifies the issues that remain to be resolved to turn the concept into an effective system, and outlines future development plans. [ABSTRACT FROM AUTHOR]

The aim of this study was to evaluate a passive upper-limb exoskeleton as an ergonomic control to reduce the musculoskeletal load in the shoulders associated with augmented reality (AR) interactions. In a repeated-measures laboratory study, each of the 20 participants performed a series of AR tasks with and without a commercially-available upper-limb exoskeleton. During the AR tasks, muscle activity (anterior, middle, posterior deltoid, and upper trapezius), shoulder joint postures/moment, and self-reported discomfort were collected. The results showed that the exoskeleton significantly reduced muscle activity in the upper trapezius and deltoid muscle groups and self-reported discomfort. However, the shoulder postures and task performance measures were not affected by the exoskeleton during the AR interactions. Given the significant decrease in muscle activity and discomfort without compromising task performance, a passive exoskeleton can be an effective ergonomic control measure to reduce the risks of developing musculoskeletal discomfort or injuries in the shoulder regions. • Musculoskeletal load associated with AR interactions were characterized. • The effects of an exoskeleton on musculoskeletal load were evaluated during AR tasks. • The results indicate an exoskeleton can reduce shoulder strain during AR interactions. [ABSTRACT FROM AUTHOR]

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human–computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine. [Display omitted] • We review HoloLens 1 usage in medicine from its release in 2016 until 2021. • 217 studies are analyzed according to our novel taxonomy. • We highlight promising applications and most popular research directions. • Major limitations impeding clinical adoption are discussed. • The HoloLens emerges as major driving force in medical AR research in the past years. [ABSTRACT FROM AUTHOR]