Search

The fourth industrial revolution is promoting the Operator 4.0 paradigm, originating from a renovated attention towards human factors, growingly involved in the design of modern, human-centered processes. New technologies, such as augmented reality or collaborative robotics are thus increasingly studied and progressively applied to solve the modern operators’ needs. Human-centered design approaches can help to identify user’s needs and functional requirements, solving usability issues, or reducing cognitive or physical stress. The paper reviews the recent literature on augmented reality-supported collaborative robotics from a human-centered perspective. To this end, the study analyzed 21 papers selected after a quality assessment procedure and remarks the poor adoption of user-centered approaches and methodologies to drive the development of human-centered augmented reality applications to promote an efficient collaboration between humans and robots. To remedy this deficiency, the paper ultimately proposes a structured framework driven by User eXperience approaches to design augmented reality interfaces by encompassing previous research works. Future developments are discussed, stimulating fruitful reflections and a decisive standardization process.

With the wave of the Fourth Industrial Revolution, the construction industry is also witnessing the application of numerous state-of-the-art technologies. Among these, augmented reality (AR) technology has the advantage of utilizing existing 3D models and BIM data and is thus an area of active research. However, the main area of research to date has either been in visualizing information during the design phase, where architects and project stakeholders can share viewings, or in confirming the required information for construction management through visualization during the construction phase. As such, more research is required in the application of AR during the facility management (FM) phase. Research utilizing BIM in the FM phase, which constitutes the longest period during the lifecycle of a building, has been continuously carried out but has faced challenges with regard to on-site application. The reason for this is that information required for BIM during the design, construction and FM phases is different, and the reproduced information is vast, so identifying the required BIM data for FM and interfacing with other systems is difficult. As a measure to overcome this limitation, advanced countries such as the US and UK have developed and are using Construction Operations Building information exchange (COBie), which is an open-source BIM-based information exchange system. In order to effectively convert open-source BIM data to AR data, this research defined COBie data for windows and doors, converted them to a system and validated that it could actually be applied for on-site FM. The results of this system’s creation and validation showed that the proposed AR-based smart FMS demonstrated faster and easier access to information compared with existing 2D blueprint-based FM work, while information obtained through AR allowed for immediate, more visual and easier means to express the information when integrated with actual objects.

Today’s user experience (UX) educators and designers can no longer just focus on creating more usable systems, but must also rise to the level of strategists, using design thinking and human–computer interaction (HCI) solutions to improve academic and business outcomes. Both psychological, designer, and engineering approaches are adopted in this study. An invited program review committee met to review progress of the UX program at the Beijing Normal University (BNUX). They considered issues and challenges facing the program today, and the steps that it could make to develop further. During a recent augmented reality (AR) project on designing future life experience on smart home and wearables, several experiential concepts and prototypes were generated to demonstrate HCI and UX research directions. The committee was impressed by BNUX with its energy, enthusiasm, and a sense of purpose on practicing transdisciplinary teaching and learning activities. Recommendations on the current organization of education, the relation between project-based learning and research, and opportunities for exposure and visibility are provided.

In this work, we propose a convenient method for evaluating levels of angular jerk in augmented reality (AR) and virtual reality (VR). Jerk is a rarely analyzed metric in usability studies, although it can be measured and calculated easily with most head-worn displays and can yield highly relevant information to designers. Here, we developed and implemented a system capable of calculating and analyzing jerk in real-time based on orientation data from an off-the-shelf head-worn display. An experiment was then carried out to determine whether the presence of AR user interface annotations results in changes to users’ angular head jerk when conducting a time-pressured visual search task. Analysis of the data indicates that a decrease in jerk is significantly associated with the use of AR augmentations. As noted in the limitations section, however, the conclusions drawn from this work should be limited, as this analysis method is novel in the VR/AR space and because of methodological limitations that limited the reliability of the jerk data. The work presented herein considerably facilitates the use of jerk as a quick component measure of usability and serves as an initial point off which future research involving jerk in VR and AR can be performed. Published version

Augmented reality assisted assembly training (ARAAT) is an effective and affordable technique for labor training in the automobile and electronic industry. In general, most tasks of ARAAT are conducted by real-time hand operations. In this paper, we propose an algorithm of dynamic gesture recognition and prediction that aims to evaluate the standard and achievement of the hand operations for a given task in ARAAT. We consider that the given task can be decomposed into a series of hand operations and furthermore each hand operation into several continuous actions. Then, each action is related with a standard gesture based on the practical assembly task such that the standard and achievement of the actions included in the operations can be identified and predicted by the sequences of gestures instead of the performance throughout the whole task. Based on the practical industrial assembly, we specified five typical tasks, three typical operations, and six standard actions. We used Zernike moments combined histogram of oriented gradient and linear interpolation motion trajectories to represent 2D static and 3D dynamic features of standard gestures, respectively, and chose the directional pulse-coupled neural network as the classifier to recognize the gestures. In addition, we defined an action unit to reduce the dimensions of features and computational cost. During gesture recognition, we optimized the gesture boundaries iteratively by calculating the score probability density distribution to reduce interferences of invalid gestures and improve precision. The proposed algorithm was evaluated on four datasets and proved to increase recognition accuracy and reduce the computational cost from the experimental results.

Indoor localization is a basic element in location-based services (LBSs), including seamless indoor and outdoor navigation, location-based precision marketing, spatial recognition in robotics, augmented reality, and mixed reality. The popularity of LBSs in the augmented reality and mixed reality fields has increased the demand for a stable and efficient indoor positioning method. However, the problem of indoor visual localization has not been appropriately addressed, owing to the strict trade-off between accuracy and cost. Therefore, we use point cloud and RGB characteristic information for the accurate acquisition of three-dimensional indoor space. The proposed method is a novel visual positioning system (VPS) capable of determining the user’s position by matching the pose information of the object estimated by the improved point-graph neural network (GNN) with the pose information label of a voxel database object addressed in predefined voxel units. We evaluated the performance of the proposed system considering a stationary object in indoor space. The results verify that high positioning accuracy and direction estimation can be efficiently achieved. Thus, spatial information of indoor space estimated using the proposed novel VPS can aid in indoor navigation.

AudienceMR is designed as a multi-user mixed reality space that seamlessly extends the local user space to become a large, shared classroom where some of the audience members are seen seated in a real space, and more members are seen through an extended portal. AudienceMR can provide a sense of the presence of a large-scale crowd/audience with the associated spatial context. In contrast to virtual reality (VR), however, with mixed reality (MR), a lecturer can deliver content or conduct a performance from a real, actual, comfortable, and familiar local space, while interacting directly with real nearby objects, such as a desk, podium, educational props, instruments, and office materials. Such a design will elicit a realistic user experience closer to an actual classroom, which is currently prohibitive owing to the COVID-19 pandemic. This paper validated our hypothesis by conducting a comparative experiment assessing the lecturer’s experience with two independent variables: (1) an online classroom platform type, i.e., a 2D desktop video teleconference, a 2D video screen grid in VR, 3D VR, and AudienceMR, and (2) a student depiction, i.e., a 2D upper-body video screen and a 3D full-body avatar. Our experiment validated that AudienceMR exhibits a level of anxiety and fear of public speaking closer to that of a real classroom situation, and a higher social and spatial presence than 2D video grid-based solutions and even 3D VR. Compared to 3D VR, AudienceMR offers a more natural and easily usable real object-based interaction. Most subjects preferred AudienceMR over the alternatives despite the nuisance of having to wear a video see-through headset. Such qualities will result in information conveyance and an educational efficacy comparable to those of a real classroom, and better than those achieved through popular 2D desktop teleconferencing or immersive 3D VR solutions.

Mobile Augmented Reality (MAR) is designed to keep pace with high-end mobile computing and their powerful sensors. This evolution excludes users with low-end devices and network constraints. This article presents ModAR, a hybrid Android prototype that expands the MAR experience to the aforementioned target group. It combines feature-based image matching and pose estimation with fast rendering of 3D textured models. Planar objects of the real environment are used as pattern images for overlaying users’ meshes or the app’s default ones. Since ModAR is based on the OpenCV C++ library at Android NDK and OpenGL ES 2.0 graphics API, there are no dependencies on additional software, operating system version or model-specific hardware. The developed 3D graphics engine implements optimized vertex-data rendering with a combination of data grouping, synchronization, sub-texture compression and instancing for limited CPU/GPU resources and a single-threaded approach. It achieves up to 3× speed-up compared to standard index rendering, and AR overlay of a 50 K vertices 3D model in less than 30 s. Several deployment scenarios on pose estimation demonstrate that the oriented FAST detector with an upper threshold of features per frame combined with the ORB descriptor yield best results in terms of robustness and efficiency, achieving a 90% reduction of image matching time compared to the time required by the AGAST detector and the BRISK descriptor, corresponding to pattern recognition accuracy of above 90% for a wide range of scale changes, regardless of any in-plane rotations and partial occlusions of the pattern.

The construction of high-rise building projects is a dangerous vocation due to the unique-ness and nature of the activities, as well as the complexity of the working environment, yet safety issues remain crucial in the construction industry. Digital technologies, such as building information modeling (BIM), have been identified as valuable tools for increasing construction productivity, efficiency, and safety. This research aimed to mitigate the accident safety factors in high-rise building projects via integrating BIM with emerging digital technologies in the construction industry, such as photogrammetry, GPS, RFID, augmented reality, (AR), virtual reality (VR), and drone technology. Qualitative research was conceived in the ground theory approach. Forty-five online interviews with construction stakeholders and qualitative data analysis were carried out using the NVivo 11 software package. According to the findings, interviewees were more motivated to use photogrammetry and drone technologies in high-rise building projects in order to increase construction safety. Positive, negative, and neutral attitudes about BIM integration with emerging digital technologies were dis-covered. Furthermore, a research framework was developed by consolidating research findings that articulate the measures and future needs of BIM integration with other digital technologies to mitigate construction accidents in high-rise building projects. The framework also renders practical references for industry practitioners towards effective and safer construction. © 2021 by the authors. Licensee MDPI, Basel, Switzerland.

Recently, the development of 3D graphics technology has led to various technologies being combined with reality, where a new reality is defined or studied, they are typically named by combining the name of the technology with “reality”. Representative “reality” includes Augmented Reality, Virtual Reality, Mixed Reality, and eXtended Reality (XR). In particular, research on XR in the web environment is actively being conducted. The Web eXtended Reality Device Application Programming Interface (WebXR Device API), released in 2018, allows instant deployment of XR services to any XR platform requiring only an active web browser. However, the currently released tentative version has poor stability. Therefore, in this study, the performance evaluation of WebXR Device API is performed using three experiments. A camera trajectory experiment is analyzed using ground truth, we checked the standard deviation between the ground truth and WebXR for the X, Y, and Z axes. The difference image experiment is conducted for the front, left, and right directions, which resulted in a visible difference image for each image of ground truth and WebXR, small mean absolute error, and high match rate. In the experiment for measuring the 3D rendering speed, a frame rate similar to that of real-time is obtained.

Currently, the route planning functions in 2D/3D campus navigation systems in the market are unable to process indoor and outdoor localization information simultaneously, and the UI experiences are not optimal because they are limited by the service platforms. An ARCore-based augmented reality campus navigation system is designed in this paper in order to solve the relevant problems. Firstly, the proposed campus navigation system uses ARCore to enhance reality by presenting 3D information in real scenes. Secondly, a visual inertial ranging algorithm is proposed for real-time locating and map generating in mobile devices. Finally, rich Unity3D scripts are designed in order to enhance users’ autonomy and enjoyment during navigation experience. In this paper, indoor navigation and outdoor navigation experiments are carried out at the Lingang campus of Shanghai University of Electric Power. Compared with the AR outdoor navigation system of Gaode, the proposed AR system can achieve increased precise outdoor localization by deploying the visual inertia odometer on the mobile phone and realizes the augmented reality function of 3D information and real scene, thus enriching the user’s interactive experience. Furthermore, four groups of students have been selected for system testing and evaluation. Compared with traditional systems, such as Gaode map or Internet media, experimental results show that our system could facilitate the effectiveness and usability of learning on campus.

Several efforts have been made to use virtual reality (VR) and augmented reality (AR) for medical and dental education and surgical support. The current methods still require users to wear devices such as a head-mounted display (HMD) and smart glasses, which pose challenges in hygiene management and long-term use. Additionally, it is necessary to measure the user’s inter-pupillary distance and to reflect it in the device settings each time to accurately display 3D images. This setting is difficult for daily use. We developed and implemented a novel anatomy education method using a spatial reality display capable of stereoscopic viewing with the naked eye without an HMD or smart glasses. In this study, we developed two new applications: (1) a head and neck anatomy education application, which can display 3D-CG models of the skeleton and blood vessels of the head and neck region using 3D human body data available free of charge from public research institutes, and (2) a DICOM image autostereoscopic viewer, which can automatically convert 2D CT/MRI/CBCT image data into 3D-CG models. In total, 104 students at the School of Dentistry experienced and evaluated the system, and the results suggest its usefulness. A stereoscopic display without a head-mounted display is highly useful and promising for anatomy education.

Warning pedestrians of oncoming vehicles is critical to improving pedestrian safety. Due to the limitations of a pedestrian’s carrying capacity, it is crucial to find an effective solution to provide warnings to pedestrians in real-time. Limited numbers of studies focused on warning pedestrians of oncoming vehicles. Few studies focused on developing visual warning systems for pedestrians through wearable devices. In this study, various real-time projection algorithms were developed to provide accurate warning information in a timely way. A pilot study was completed to test the algorithm and the user interface design. The projection algorithms can update the warning information and correctly fit it into an easy-to-understand interface. By using this system, timely warning information can be sent to those pedestrians who have lower situational awareness or obstructed view to protect them from potential collisions. It can work well when the sightline is blocked by obstructions.

We propose a novel wearable haptic device that can provide kinesthetic haptic feedback for stiffness rendering of virtual objects in augmented reality (AR). Rendering stiffness of objects using haptic feedback is crucial for realistic finger-based object manipulation, yet challenging particularly in AR due to the co-presence of a real hand, haptic device, and rendered AR objects in the scenes. By adopting passive actuation with a tendon-based transmission mechanism, the proposed haptic device can generate kinesthetic feedback strong enough for immersive manipulation and prevention of inter-penetration in a small-form-factor, while maximizing the wearability and minimizing the occlusion in AR usage. A selective locking module is adopted in the device to allow for the rendering of the elasticity of objects. We perform an experimental study of two-finger grasping to verify the efficacy of the proposed haptic device for finger-based manipulation in AR. We also quantitatively compare/articulate the effects of different types of feedbacks across haptic and visual sense (i.e., kinesthetic haptic feedback, vibrotactile haptic feedback, and visuo-haptic feedback) for stiffness rendering of virtual objects in AR for the first time.

Virtual and augmented reality (VR, AR) systems present 3D images that users can interact with using controllers or gestures. The design of the user input process is crucial and determines the interactive efficiency, comfort, and adoption. Gesture-based input provides a device-free interaction that may improve safety and creativity compared to using a hand controller while allowing the hands to perform other tasks. Microgestures with small finger and hand motions may have an advantage over the larger forearm and upper arm gestures by reducing distraction, reducing fatigue, and increasing privacy during the interaction. The design of microgestures should consider user experience, ergonomic principles, and interface design to optimize productivity and comfort while minimizing errors. Forty VR/AR or smart device users evaluated a set of 33 microgestures, designed by ergonomists, and linked them to 20 common AR/VR commands based on usability, comfort, and preference. Based primarily on preference, a set of microgestures linked to specific commands is proposed for VR or AR systems. The proposed microgesture set will likely minimize fatigue and optimize usability. Furthermore, the methodology presented for selecting microgestures and assigning them to commands can be applied to the design of other gesture sets.

Intuitive and safe interfaces for robots are challenging issues in robotics. Robo-HUD is a gadget-less interaction concept for contactless operation of industrial systems. We use virtual collision detection based on time-of-flight sensor data, combined with augmented reality and audio feedback, allowing the operators to navigate a virtual menu by “hover and hold” gestures. When incorporated with virtual safety barriers, the collision detection also functions as a safety feature, slowing or stopping the robot if a barrier is breached. Additionally, a user focus recognition module monitors the awareness, enabling the interaction only when intended. Early case studies show that these features present good use-cases for inspection tasks and operation in difficult environments, where contactless operation is needed.

New technologies, such as cloud computing, the Internet of Things, wireless communications, etc., have already become part of our daily lives. This paper provides an insight into one of the new technologies, i.e., augmented reality (AR), as part of the manufacturing paradigm Industry 4.0 (I4.0). The aim of this paper is to contribute to the current state in the field of AR by assessing the main areas of the application of AR, the used devices and the tracking methods in support of the digitalization of the industry. Searches via Science Direct, Google Scholar and the Internet in general have resulted in the collection of a large number of papers. The examined works are classified according to several criteria and the most important data resulting from them are presented here. A comprehensive analysis of the literature has indicated the main areas of application of AR in I4.0 and, among these, those that stand out are maintenance, assembly and human robot collaboration. Finally, a roadmap for the application of AR in companies is proposed and the most promising future areas of research are listed.

Augmented reality is one of the fastest growing fields, receiving increased funding for the last few years as people realise the potential benefits of rendering virtual information in the real world. Most of today’s augmented reality marker-based applications use local feature detection and tracking techniques. The disadvantage of applying these techniques is that the markers must be modified to match the unique classified algorithms or they suffer from low detection accuracy. Machine learning is an ideal solution to overcome the current drawbacks of image processing in augmented reality applications. However, traditional data annotation requires extensive time and labour, as it is usually done manually. This study incorporates machine learning to detect and track augmented reality marker targets in an application using deep neural networks. We firstly implement the auto-generated dataset tool, which is used for the machine learning dataset preparation. The final iOS prototype application incorporates object detection, object tracking and augmented reality. The machine learning model is trained to recognise the differences between targets using one of YOLO’s most well-known object detection methods. The final product makes use of a valuable toolkit for developing augmented reality applications called ARKit.

With the rise of information technology and digitization, education has been faced with the need to adopt new learning models using technology to create innovative educational methodologies. In addition, due to pandemic restrictions and in order to help contain the spread of the virus (COVID-19), all educational institutions have been forced to switch immediately to online education. The application of augmented reality (AR) in education provides important benefits, such as increased engagement and interactivity, and can help to minimize the negative effects of the disruption of face-to-face education. Therefore, this paper focuses on describing the effect of an augmented reality mobile application (NetAR) that was developed for engineering students as a complement to traditional education. To achieve this objective, an experimental group and a control group were established to work with the application for three weeks for three hours a day. Moreover, there are a number of usability issues with AR that may impact learning effectiveness and motivation. Therefore, the usability of the application was evaluated with the IBM Computer System Usability Questionnaire (CSUQ) tool. The usability results show that users are satisfied with NetAR, and the statistical data from the control group indicate that the application positively affects learning. This work was supported by the EduTech project (609785-EPP-1-2019-1-ES-EPPKA2-CBHEJP) co-funded by the Erasmus+ Programme of the European Union.

In tourist areas, it is necessary to prepare a method that supports tourists’ activities by providing information concerning disaster support facilities during normal times, in addition to sightseeing spots and tourism-related facilities, because there is a risk of confusion during disasters, as tourists are not aware of the locations of disaster support facilities. The present study aims to develop a navigation system that supports the activities of users during both normal times and disasters by integrating augmented reality (AR) and web geographic information systems (Web-GISs), as well as by using pictograms. The system can not only effectively provide users with information concerning sightseeing spots and tourism-related facilities but also information concerning disaster support facilities. The system was operated over a period of 6 weeks in Chofu City, Tokyo Metropolis, Japan. Based on the results of a questionnaire survey for 60 users, the system was highly evaluated for its originality in terms of displays and functions using pictograms, navigation using AR, and obtaining information during disasters. Additionally, based on the results of access log analysis, the system was continuously utilized by users during the operation period. Therefore, by continuously operating the system, it can be expected that users will further utilize each function of the system.

A smart helmet is a wearable device that has attracted attention in various fields, especially in applied sciences, where extensive studies have been conducted in the past decade. In this study, the current status and trends of smart helmet research were systematically reviewed. Five research questions were set to investigate the research status of smart helmets according to the year and application field, as well as the trend of smart helmet development in terms of types of sensors, microcontrollers, and wireless communication technology. A total of 103 academic research articles published in the past 11 years (2009–2020) were analyzed to address the research questions. The results showed that the number of smart helmet applications reported in literature has been increasing rapidly since 2018. The applications have focused mostly on ensuring the safety of motorcyclists. A single-board-based modular concept unit, such as the Arduino board, and sensor for monitoring human health have been used the most for developing smart helmets. Approximately 85% of smart helmets have been developed to date using wireless communication technology to transmit data obtained from smart helmets to other smart devices or cloud servers.

This study proposes a pupil-tracking method applicable to drivers both with and without sunglasses on, which has greater compatibility with augmented reality (AR) three-dimensional (3D) head-up displays (HUDs). Performing real-time pupil localization and tracking is complicated by drivers wearing facial accessories such as masks, caps, or sunglasses. The proposed method fulfills two key requirements: low complexity and algorithm performance. Our system assesses both bare and sunglasses-wearing faces by first classifying images according to these modes and then assigning the appropriate eye tracker. For bare faces with unobstructed eyes, we applied our previous regression-algorithm-based method that uses scale-invariant feature transform features. For eyes occluded by sunglasses, we propose an eye position estimation method: our eye tracker uses nonoccluded face area tracking and a supervised regression-based pupil position estimation method to locate pupil centers. Experiments showed that the proposed method achieved high accuracy and speed, with a precision error of &lt, 10 mm in &lt, 5 ms for bare and sunglasses-wearing faces for both a 2.5 GHz CPU and a commercial 2.0 GHz CPU vehicle-embedded system. Coupled with its performance, the low CPU consumption (10%) demonstrated by the proposed algorithm highlights its promise for implementation in AR 3D HUD systems.

Realistic interactions with real objects (e.g., animals, toys, robots) in an augmented reality (AR) environment enhances the user experience. The common AR apps on the market achieve realistic interactions by superimposing pre-modeled virtual proxies on the real objects in the AR environment. This way user perceives the interaction with virtual proxies as interaction with real objects. However, catering to environment change, shape deformation, and view update is not a trivial task. Our proposed method uses the dynamic silhouette of a real object to enable realistic interactions. Our approach is practical, lightweight, and requires no additional hardware besides the device camera. For a case study, we designed a mobile AR application to interact with real animal dolls. Our scenario included a virtual human performing four types of realistic interactions. Results demonstrated our method’s stability that does not require pre-modeled virtual proxies in case of shape deformation and view update. We also conducted a pilot study using our approach and reported significant improvements in user perception of spatial awareness and presence for realistic interactions with a virtual human.

In the last few decades, several initiatives and approaches are set up to support maintenance procedures for the railway industry in adopting the principles of Industry 4.0. Contextualized maintenance technologies such as Augmented Reality (AR) overlay can integrate virtual information on physical objects to improve decision-making and action-taking processes. Operators work in a dynamic working environment requiring both high adaptive capabilities and expert knowledge. There is a need to support the operators with tailor-based information that is customized and contextualized to their expertise and experience. It calls for AR tools and approaches that combine complex methodologies with high usability requirements. The development of these AR tools could benefit from a structured approach. Therefore, the objective of this paper is to propose an adaptive architectural framework aimed at shaping and structuring the process that provides operators with tailored support when using an AR tool. Case study research is applied within a revelatory railway industry setting. It was found that the framework ensures that self-explanatory AR systems can capture the knowledge of the operator, support the operator during maintenance activities, conduct failure analysis, provide problem-solving strategies, and improve learning capabilities. This study contributes to the necessity of having a human-centered approach for the successful adaption of AR technology tools for the railway industry.

Modal analysis provides the dynamic behavior of an object or structure, and is often undertaken using the Finite Element Method (FEM) due to its ability to deal with arbitrary geometries. This article investigates the use of Augmented Reality (AR) to provide the in situ visualization of a modal analysis for an aluminum impeller. Finite Element Analysis (FEA) software packages regularly use heat maps and shape deformation to visualize the outcomes of a given simulation. AR allows the superimposition of digital information on a view of the real-world environment, and provides the opportunity to overlay such simulation results onto real-world objects and environments. The presented modal analysis undertaken herein provides natural frequencies and the corresponding deformation of an aluminum impeller. The results indicate the ability for the design part and finite element analysis results to be viewed on the physical part. A mobile AR-FEA-based system was developed for Modal Analysis result visualization. This study offers designers and engineers a new way to visualize such simulation results.

When it concerns dissemination, the relationship between cultural heritage and technology has not always been easy. On the one hand, the emotions involved in knowing, enjoying, and feeling a real heritage remain should not be eclipsed by the technological experience. On the other hand, technology cannot be relegated to the point where its use is so superficial as to make it irrelevant. Hence, in the search of good practices in the dissemination of cultural heritage, it is essential that interdisciplinary work teams, composed of humanists and technicians, design new experiences that try to achieve a balance between what must be shown and the way in which it is done. This paper aims to show the potential of augmented reality (AR) for the dissemination of cultural heritage, since it allows the perceiving of real remains while offering virtual reconstructions and complementary information, sounds, images, etc. A bibliometric analysis focusing on its use for the preservation of the memory of vernacular architecture is made. Then, from the understanding and interpretation of the ideas exposed in literature, a decalogue of good practices in the use of AR for architectural heritage is established considering the holistic perspectives that any dissemination initiative must have. These initiatives can attract people to rural areas and contribute to the improvement of their social and economic situation. As an example, a proof of concept of an AR experience is presented at the end of the paper.

NASA’s human spaceflight efforts are moving towards long-duration exploration missions requiring asynchronous communication between onboard crew and an increasingly remote ground support. In current missions aboard the International Space Station, there is a near real-time communication loop between Mission Control Center and astronauts. This communication is essential today to support operations, maintenance, and science requirements onboard, without which many tasks would no longer be feasible. As NASA takes the next leap into a new era of human space exploration, new methods and tools compensating for the lack of continuous, real-time communication must be explored. The Human-Computer Interaction Group at NASA Ames Research Center has been investigating emerging technologies and their applicability to increase crew autonomy in missions beyond low Earth orbit. Interactions using augmented reality and the Internet of Things have been researched as possibilities to facilitate usability within procedure execution operations. This paper outlines four research efforts that included technology demonstrations and usability studies with prototype procedure tools implementing emerging technologies. The studies address habitat feedback integration, analogous procedure testing, task completion management, and crew training. Through these technology demonstrations and usability studies, we find that low- to medium-fidelity prototypes, evaluated early in the design process, are both effective for garnering stakeholder buy-in and developing requirements for future systems. In this paper, we present the findings of the usability studies for each project and discuss ways in which these emerging technologies can be integrated into future human spaceflight operations.

Simulation for surgical training is increasingly being considered a valuable addition to traditional teaching methods. 3D-printed physical simulators can be used for preoperative planning and rehearsal in spine surgery to improve surgical workflows and postoperative patient outcomes. This paper proposes an innovative strategy to build a hybrid simulation platform for training of pedicle screws fixation: the proposed method combines 3D-printed patient-specific spine models with augmented reality functionalities and virtual X-ray visualization, thus avoiding any exposure to harmful radiation during the simulation. Software functionalities are implemented by using a low-cost tracking strategy based on fiducial marker detection. Quantitative tests demonstrate the accuracy of the method to track the vertebral model and surgical tools, and to coherently visualize them in either the augmented reality or virtual fluoroscopic modalities. The obtained results encourage further research and clinical validation towards the use of the simulator as an effective tool for training in pedicle screws insertion in lumbar vertebrae.

Traditional in-app virtual reality (VR)/augmented reality (AR) applications pose a challenge of reaching users due to their dependency on operating systems (Android, iOS). Besides, it is difficult for general users to create their own VR/AR applications and foster their creative ideas without advanced programming skills. This paper addresses these issues by proposing an interactive extended reality toolkit, named BlocklyXR. The objective of this research is to provide general users with a visual programming environment to build an extended reality application for digital storytelling. The contextual design was generated from real-world map data retrieved from Mapbox GL. ThreeJS was used for setting up, rendering 3D environments, and controlling animations. A block-based programming approach was adapted to let users design their own story. The capability of BlocklyXR was illustrated with a use case where users were able to replicate the existing PalmitoAR utilizing the block-based authoring toolkit with fewer efforts in programming. The technology acceptance model was used to evaluate the adoption and use of the interactive extended reality toolkit. The findings showed that visual design and task technology fit had significantly positive effects on user motivation factors (perceived ease of use and perceived usefulness). In turn, perceived usefulness had statistically significant and positive effects on intention to use, while there was no significant impact of perceived ease of use on intention to use. Study implications and future research directions are discussed.

Reaching the underwater world is undoubtedly an incomparable adventure. Impressive geological structures, flourishing ecosystems, shipwrecks, and submerged landscapes lie beneath the sea surface in wait for discovery. However, this world is accessible only to those who have the chance to dive or to scientists conducting underwater research. By means of a dynamically developing sector of informatics utilizing Virtual (VR) and Augmented Reality (AR) practices, the VIRTUALDiver project intends to provide access to all the aforementioned hidden &ldquo, treasures&rdquo, through the creation of an innovative platform providing unique interactive experiences. More specifically, specialized guided tours in natural and virtual environments covering areas of touristic, cultural, and environmental interest. VIRTUALDiver is an experience, design and content presentation platform, a custom-made add-on environment within the Unity 3D authoring tool, offering the ability to manage multimedia content in a simplified way. No specialized programming knowledge is required, enabling the project&rsquo, s interdisciplinary consortium to easily collaborate and exchange ideas. The expected result is the establishment of a successful educational and entertaining cultural product to support businesses and professionals operating in the field of culture-tourism. Above all, VIRTUALDiver aspires to become a novel form of storytelling, immersing the user into unique experiences under the waves.

Although Augmented Reality (AR) technology has entered many market and knowledge domains such as games and leisure activities, it remains rather limited in digital heritage. After studying the potentiality of using modern AR elements in a museum context, this paper proposes the use of additional game and educational elements in the core AR application in order to enhance the overall on-the-spot museum visitor&rsquo, s experience. An agile AR application design methodology was followed by taking into account the needs of small-to-medium sized real-world museums. Moreover, a heuristic evaluation protocol was applied by a group of experts in order to test the proof-of-concept AR application, in which some novel elements were proposed such as the AR quiz game. The main findings indicate that enhanced AR experiences in museum settings can make a nice fit with the user environment, physical and perceptual abilities, known metaphors, and user position and motion in 3D space. Moreover, AR services can be provided under a minimum distraction and physical effort. As a conclusion, AR technologies are mature enough to be standardized for museum usage, while the audience seems to be ready to take advantage of the related enhanced museum experiences to maximize both user satisfaction and learning outcomes.

Three-dimensional (3D) visualization technology, such as augmented reality (AR), hasserved as the display for building information modeling (BIM)-based architectural design collaboration to provide more effective design observation and communication for stakeholders. That said, AR has several technical limitations in terms of personal device issues, user experience, and visualization quality. A new form of AR called spatial augmented reality (SAR) has been introduced to address these issues, which uses a digital projector to present graphics on physical objects for augmenting real-world objects. Therefore, SAR has great benefits and potentials to combine with BIM for design collaboration. This paper introduces a BIM-based SAR operational framework, where 3D building models generated from BIM software are imported to projection mapping tools to display building surface textures on physical white building models. A case study using Revit and 3ds Max as the BIM software, and Mad Mapper as the projection mapping tool, was conducted to demonstrate the feasibility of the proposed framework and to evaluate the projection performance of SAR. The case study showed that the texture of BIM models could be projected on the objects clearly and realistically. Additionally, the proposed SAR method potentially offers intuitive observation of building models and comfortable wear-free experience for collaborative design, and the qualitative analysis by changing the parameters was conducted to test the different projection conditions. Since itis expected that the use of SAR can be promoted by overcoming the discussed technical limitations and possible solution application, this study aims to traceability provide the whole process of BIM-based SAR for architectural design collaboration. Refereed/Peer-reviewed

Augmented reality (AR) has received increasing attention in the research literature as a fundamental pedagogical tool that can enhance learning at most educational levels. In academic contexts, this technology permits the superimposing of three-dimensional images onto the real environment. Although AR has been found to enhance learning in the academic environment, no systematic review of it has been conducted to identify, evaluate, and summarize empirical findings on its advantages and challenges in e-learning contexts. Hence, a systematic review of the research literature was conducted on the use of AR in e-learning contexts, with a focus on the key benefits and challenges related to its adoption and implementation. Electronic searches on databases, including Springer, Science Direct, EBSCO, and Google Scholar, were performed to retrieve relevant journal articles; 28 studies were included after they were screened using the inclusion and exclusion criteria. The key benefits of using AR in e-learning included support of kinesthetic (tactile) learning, collaborative learning, distance/remote learning, learner-centered learning, and creative learning. Studies also reported that AR enhanced students’ engagement, motivation, attention/focus, interactivity, verbal participation, concentration, knowledge retention, and spatial abilities, as well as information accessibility. The findings suggest that challenges associated with AR in e-learning include information and cognitive overload, lack of experience in using the technology, resistance from teachers, complex technology, costly technology, and technical issues, such as connectivity problems.

The nature of industrial manufacturing processes and the continuous need to adapt production systems to new demands require tools to support workers during transitions to new processes. At the early stage of transitions, human error rate is often high and the impact in quality and production loss can be significant. Over the past years, eXtended Reality (XR) technologies (such as virtual, augmented, immersive, and mixed reality) have become a popular approach to enhance operators&rsquo, capabilities in the Industry 4.0 paradigm. The purpose of this research is to explore the usability of dialogue-based XR enhancement to ease the cognitive burden associated with manufacturing tasks, through the augmentation of linked multi-modal information available to support operators. The proposed Interactive XR architecture, using the Spoken Dialogue Systems&rsquo, modular and user-centred architecture as a basis, was tested in two use case scenarios: the maintenance of a robotic gripper and as a shop-floor assistant for electric panel assembly. In both cases, we have confirmed a high user acceptance rate with an efficient knowledge communication and distribution even for operators without prior experience or with cognitive impairments, therefore demonstrating the suitability of the solution for assisting human workers in industrial manufacturing processes. The results endorse an initial validation of the Interactive XR architecture to achieve a multi-device and user-friendly experience to solve industrial processes, which is flexible enough to encompass multiple tasks.

We propose a see-through near-eye display featuring an exit pupil expander (EPE), which is composed of two multiplexed slanted gratings. Via a two-dimensional expansion, the exit pupil (EP) is able to be enlarged up to 10 &times, 8 mm2. Besides, the prescription for correcting the refractive errors can be integrated as well. The design rules are set forth in detail, followed by the results and discussion regarding the efficiency, field of view (FOV), exit pupil, angular resolution (AR), modulation transfer function (MTF), contrast ratio (CR), distortion, and simulated imaging.

Augmented reality (AR) is a fairly new technology enabling human machine interaction by superimposing virtual information on a real environment. Potential applications can be found in many areas of research from recent years. This study presents a systematic review of existing AR systems in the automotive field, synthesizing 55 studies from 2002 to 2019. The main research questions are: where AR technology has been applied within the automotive industry, what is the purpose of its application, what are the general characteristics of these systems, and what are the emphasized benefits and challenges of using AR in this field? The aim of this paper is to provide an insight into the AR applications and technologies in the automotive field.

The Dougong structure is an ancient architectural innovation of the East. Its construction method is complex and challenging to understand from drawings. Scale models were developed to preserve this culturally-unique architectural technique by learning through their assembly process. In this work, augmented reality (AR)-based systems that support the manual assembly of the Dougong models with instant interactions were developed. The first objective was to design new AR-assisted functions that overcome existing limitations of paper-based assembly instructions. The second one was to clarify whether or not and how AR can improve the operational efficiency or quality of the manual assembly process through experiments. The experimental data were analyzed with both qualitative and quantitative measures to evaluate the assembly efficiency, accuracy, and workload of these functions. The results revealed essential requirements for improving the functional design of the systems. They also showed the potential of AR as an effective human interfacing technology for assisting the manual assembly of complex objects.

Augmented Reality (AR) functionalities may be effectively leveraged in collaborative service scenarios (e.g., remote maintenance, on-site building, street gaming, etc.). Standard development cycles for collaborative AR require to code for each specific visualization platform and implement the necessary control mechanisms over the shared assets. in order to face this challenge, this paper describes SARA, an architecture to support cross-platform collaborative Augmented Reality applications based on microservices. The architecture is designed to work over the concept of collaboration models which regulate the interaction and permissions of each user over the AR assets. Five of these collaboration models were initially integrated in SARA (turn, layer, ownership, hierarchy-based and unconstrained examples) and the platform enables the definition of new ones. Thanks to the reusability of its components, during the development of an application, SARA enables focusing on the application logic while avoiding the implementation of the communication protocol, data model handling and orchestration between the different, possibly heterogeneous, devices involved in the collaboration (i.e., mobile or wearable AR devices using different operating systems). to describe how to build an application based on SARA, a prototype for HoloLens and iOS devices has been implemented. the prototype is a collaborative voxel-based game in which several players work real time together on a piece of land, adding or eliminating cubes in a collaborative manner to create buildings and landscapes. Turn-based and unconstrained collaboration models are applied to regulate the interaction. the development workflow for this case study shows how the architecture serves as a framework to support the deployment of collaborative AR services, enabling the reuse of collaboration model components, agnostically handling client technologies.

Users are attracted by augmented reality games to fulfil their needs. Two objectives are proposed: (1) to research the motivations of those using augmented reality mobile games, (2) to define a structural model based on Uses and Gratifications Theory for the adoption of augmented reality mobile games. The present study examines the case of Pok&eacute, mon Go. The model is composed of eight constructs: enjoyment, fantasy, escapism, social interaction, social presence, achievement, self-presentation and continuance intention. The SEM model was empirically assessed based on 1183 responses from Pok&eacute, mon Go users around the world. Results clearly confirmed the positive influence of almost all the proposed constructs on continuance intention for Pok&eacute, mon Go. First, these findings may be helpful for the online gaming industry in identifying the game functions that retain more gamers and improve the user experience. Second, the online gaming industry might use these results in order to classify those players with behaviours that favour the use of online games.

About 250 million people in the world suffer from color vision deficiency (CVD). Contact lenses and glasses with a color filter are available to partially improve the vision of people with CVD. Tinted glasses uniformly change the colors in a user&rsquo, s field of view (FoV), which can improve the contrast of certain colors while making others hard to identify. On the other hand, an optical see-through head-mounted display (OST-HMD) provides a new alternative by applying a controllable overlay to a user&rsquo, s FoV. The method of color calibration for people with CVD, such as the Daltonization process, needs to make the calibrated color darker, which has not yet been featured on recent commercial OST-HMDs. We propose a new approach to realize light subtraction on OST-HMDs using a transmissive LCD panel, a prototype system, named ALCC-glasses, to validate and demonstrate the new arriving light chroma controllable augmented reality technology for CVD compensation.

The learning of programming is a field of research with relevant studies and publications for more than 25 years. Since its inception, it has been shown that its difficulty lies in the high level of abstraction required to understand certain programming concepts. However, this level can be reduced by using tools and graphic representations that motivate students and facilitate their understanding, associating real-world elements with specific programming concepts. Thus, this paper proposes the use of an intelligent tutoring system (ITS) that helps during the learning of programming by using a notation based on a metaphor of roads and traffic signs represented by 3D graphics in an augmented reality (AR) environment. These graphic visualizations can be generated automatically from the source code of the programs thanks to the modular and scalable design of the system. Students can use them by leveraging the available feedback system, and teachers can also use them in order to explain programming concepts during the classes. This work highlights the flexibility and extensibility of the proposal through its application in different use cases that we have selected as examples to show how the system could be exploited in a multitude of real learning scenarios.

Personalized production is moving the progress of industrial automation forward, and demanding new tools for improving the decision-making of the operators. This paper presents a new, projection-based augmented reality system for assisting operators during electronic component assembly processes. The paper describes both the hardware and software solutions, and depicts the results obtained during a usability test with the new system.

Technical documentation is evolving from static contents presented on paper or via digital publishing to real-time on-demand contents displayed via virtual and augmented reality (AR) devices. However, how best to provide personalized and context-relevant presentation of technical information is still an open field of research. In particular, the systems described in the literature can manage a limited number of modalities to convey technical information, and do not consider the &lsquo, people&rsquo, factor. Then, in this work, we present a Context-Aware Technical Information Management (CATIM) system, that dynamically manages (1) what information as well as (2) how information is presented in an augmented reality interface. The system was successfully implemented, and we made a first evaluation in the real industrial scenario of the maintenance of a hydraulic valve. We also measured the time performance of the system, and results revealed that CATIM performs fast enough to support interactive AR.

s eye position. Compared to classical SPAAM techniques that still rely on the human element and to other camera based calibration procedures, the proposed technique is flexible and easy to replicate in both laboratory environments and real-world settings., s eye must be placed within the eye-motion-box of the see-through display. The method exploits standard camera calibration and computer vision techniques to estimate the projection parameters of the display model for a generic position of the camera. At execution time, the projection parameters can then be refined through a planar homography that encapsulates the shift and scaling effect associated with the estimated relative translation from the old camera position to the current user&rsquo, In recent years, the entry into the market of self contained optical see-through headsets with integrated multi-sensor capabilities has led the way to innovative and technology driven augmented reality applications and has encouraged the adoption of these devices also across highly challenging medical and industrial settings. Despite this, the display calibration process of consumer level systems is still sub-optimal, particularly for those applications that require high accuracy in the spatial alignment between computer generated elements and a real-world scene. State-of-the-art manual and automated calibration procedures designed to estimate all the projection parameters are too complex for real application cases outside laboratory environments. This paper describes an off-line fast calibration procedure that only requires a camera to observe a planar pattern displayed on the see-through display. The camera that replaces the user&rsquo

This paper proposes a framework that allows 3D freeform manipulation of a face in live video. Unlike existing approaches, the proposed framework provides natural 3D manipulation of a face without background distortion and interactive face editing by a user&rsquo, s input, which leads to freeform manipulation without any limitation of range or shape. To achieve these features, a 3D morphable face model is fitted to a face region in a video frame and is deformed by the user&rsquo, s input. The video frame is then mapped as a texture to the deformed model, and the model is rendered on the video frame. Because of the high computational cost, parallelization and acceleration schemes are also adopted for real-time performance. Performance evaluation and comparison results show that the proposed framework is promising for 3D face editing in live video.

Feature Pyramid Network (FPN) builds a high-level semantic feature pyramid and detects objects of different scales in corresponding pyramid levels. Usually, features within the same pyramid levels have the same weight for subsequent object detection, which ignores the feature requirements of different scale objects. As we know, for most detection networks, it is hard to detect small objects and occluded objects because there is little information to exploit. To solve the above problems, we propose an Enhanced Feature Pyramid Object Detection Network (EFPN), which innovatively constructs an enhanced feature extraction subnet and adaptive parallel detection subnet. Enhanced feature extraction subnet introduces Feature Weight Module (FWM) to enhance pyramid features by weighting the fusion feature map. Adaptive parallel detection subnet introduces Adaptive Context Expansion (ACE) and Parallel Detection Branch (PDB). ACE aims to generate the features of adaptively enlarged object context region and original region. PDB predicts classification and regression results separately with the two features. Experiments showed that EFPN outperforms FPN in detection accuracy on Pascal VOC and KITTI datasets. Furthermore, the performance of EFPN meets the real-time requirements of autonomous driving systems.

Industrial augmented reality (AR) applications demand high on the visual consistency of virtual-real registration. To present, the marker-based registration method is most popular because it is fast, robust, and convenient to obtain the registration matrix. In practice, the registration matrix should multiply an offset matrix that describes the transformation between the attaching position and the initial position of the marker relative to the object. However, the offset matrix is usually measured, calculated, and set manually, which is not accurate and convenient. This paper proposes an accurate and automatic marker&ndash, object offset matrix calibration method. First, the normal direction of the target object is obtained by searching and matching the top surface of the CAD model. Then, the spatial translation is estimated by aligning the projected and the imaged top surface. Finally, all six parameters of the offset matrix are iteratively optimized using a 3D image alignment framework. Experiments were performed on the publicity monocular rigid 3D tracking dataset and an automobile gearbox. The average translation and rotation errors of the optimized offset matrix are 2.10 mm and 1.56 degree respectively. The results validate that the proposed method is accurate and automatic, which contributes to a universal offset matrix calibration tool for marker-based industrial AR applications.

Virtual commerce applies immersive technology such as augmented reality and virtual reality into e-commerce to shift consumer perception from 2D product catalogs to 3D immersive virtual spaces. In virtual commerce, the alignment of application design paradigms and the factors influencing consumer behavior is paramount to promote purchase of products and services. The question of their relation needs to be answered, together with the possible improvement of application design. This paper used a systematic literature review approach to synthesize research on virtual commerce from both application design and consumer behavior research, considering the promotion of purchase in virtual commerce settings. Throughout the review, influential factors to purchase and preeminent design artifacts were identified. Then, the research gaps were discovered by mapping the design artifacts to the influential factors, which can inspire future research opportunities on the synergy of these two research directions. Moreover, the evolution of virtual commerce research along with multiple directions were discussed, including the suggestion of meta-commerce as a future trend.

Recently, the JPEG working group (ISO/IEC JTC1 SC29 WG1) developed an international standard, JPEG 360, that specifies the metadata and functionalities for saving and sharing 360-degree images efficiently to create a more realistic environment in various virtual reality services. We surveyed the metadata formats of existing 360-degree images and compared them to the JPEG 360 metadata format. We found that existing omnidirectional cameras and stitching software packages use formats that are incompatible with the JPEG 360 standard to embed metadata in JPEG image files. This paper proposes an easy-to-use tool for embedding JPEG 360 standard metadata for 360-degree images in JPEG image files using a JPEG-defined box format: the JPEG universal metadata box format. The proposed implementation will help 360-degree cameras and software vendors provide immersive services to users in a standardized manner for various markets, such as entertainment, education, professional training, navigation, and virtual and augmented reality applications. We also propose and develop an economical JPEG 360 standard compatible panoramic image acquisition system from a single PTZ camera with a special-use case of a wide field of view image of a conference or meeting. A remote attendee of the conference/meeting can see the realistic and immersive environment through our PTZ panorama in virtual reality.

Augmented reality medical training simulators can provide a realistic and immersive experience by overlapping the virtual scene on to the real world. Latency in augmented reality (AR) medical training simulators is an important issue as it can lead to motion sickness for users. This paper proposes a framework that can achieve real-time rendering of the 3D scene aligned to the real world using a head-mounted display (HMD). Model deformation in the 3D scene is categorised into local deformation derived from user interaction and global deformation determined by the simulation scenario. Target shapes are predefined by a simulation scenario, and control points are placed to embed the predefined shapes. Free-form deformation (FFD) is applied to multiple shapes to efficiently transfer the simulated model to the HMD. Global deformation is computed by blending a mapping matrix of each FFD with an assigned weighting value. The local and global deformation are then transferred through the control points updated from a deformed surface mesh and its corresponding weighting value. The proposed framework is verified in terms of latency caused by data transmission and the accuracy of a transmitted surface mesh in a vaginal examination (VE) training simulation. The average latency is reduced to 7 ms, less than the latency causing motion sickness in virtual reality simulations. The maximum relative error is less than 3%. Our framework allows seamless rendering of a virtual scene to the real world with substantially reduced latency and without the need for an external tracking system.