Search

The architecture, engineering, and construction (AEC) industry is undergoing a substantial change, and the technological advancements are shaping the future of this industry. Over years, numerous tools have been developed based on building information modeling (BIM) and the Internet of Things (IoT) to manage a variety of tasks and activities within different construction and operational phases in the AEC industry. Recently, with the advent of novel internet-enabled technologies in providing the ability of real-time connectivity, researchers have started to develop virtual design and construction (VDC) and digital twin approaches to remotely monitor, control, and optimize different work progresses and activities in real-time at any work level in the AEC industry. This paper presents a reference model to illustrates the main advantages and applications of VDC and digital twin, and to demonstrate their trends and anticipated cost savings in the AEC industry at a global level. These technological advancements along with augmented reality (AR) and virtual reality (VR) can offer the unique benefit of real-time monitoring of the current status as well as predicting the future of any physical structure at any level of work. The global demand and usage of these technological advancements will lead to great cost savings in the AEC industry which would be US$950 million in design and construction and US$400 million in operation and maintenance phases in the non-residential AEC industry in 2025. This paper also discusses the future challenges of VDC and digital twin in the industry. As a main line of future research, these technologies can simultaneously be utilized with artificial intelligence to enable human-centered decision making to automate and optimize work progresses. Such integrations of VDC, digital twin, artificial intelligence, and human-centered approaches will shape the future of the AEC industry and provide numerous research opportunities in this domain.

The performance of smart vehicle (SV) applications like autonomous driving and in-vehicle augmented reality based traffic information system depends on the Field of View (FoV) and the timely processing of the SVs sensor data. Vehicular networking (VN) technology can enhance the performance of these applications by enabling a SV to access the sensing and processing capabilities of other neighbouring SVs. The processing and storage capacity of a SV is limited compared to cloud servers and the communication link between two SVs is unreliable due to their mobility and the nature of wireless channels. Hence, developing efficient processing and sensing schemes for SVs and VNs can help in optimizing the performance of SV applications. In this paper, we propose Contextual Bandits (CB), Markov decision process (MDP) and deep Q-network (DQN) based sensing and processing schemes for VNs. Simulation results show that the proposed schemes outperform the baseline schemes in a variety of scenarios.

Industry 4.0 is a revolutionizing concept which has a more significant implication to numerous industries of various sectors worldwide. Industry 4.0 acts as a technical catalyst in accelerated growth through systematic adaptation and new-generation cutting-edge engineering tools. IoT, Blockchain, Artificial Intelligence, Augmented Reality, 3D printing, Big-data analytics, cloud-computing are some of the primary technologies which work in join hand to make the vision of Industry 4.0 successful. In addition, it promotes autonomous industry, which in turn considerably reduces human intervention, thus resulting in efficient working and safeguard companies' interests. Due to the faster adaption of Industry 4.0, authors have attempted to define each and every concept related to it and to explore a futuristic roadmap, special focusing on its implication in the contemporary market. In the 21st century Industry, 4.0 is the need of the hour, so it has become essential to understand the state-of-the-art technologies involved.

Estimating the depth of virtual content has proven to be a challenging task in Augmented Reality (AR) applications. Existing studies have shown that the visual system makes use of multiple depth cues to infer the distance of objects, occlusion being one of the most important ones. The ability to generate appropriate occlusions becomes particularly important for AR applications that require the visualization of augmented objects placed below a real surface. Examples of these applications are medical scenarios in which the visualization of anatomical information needs to be observed within the patient's body. In this regard, existing works have proposed several focus and context (F+C) approaches to aid users in visualizing this content using Video See-Through (VST) Head-Mounted Displays (HMDs). However, the implementation of these approaches in Optical See-Through (OST) HMDs remains an open question due to the additive characteristics of the display technology. In this article, we, for the first time, design and conduct a user study that compares depth estimation between VST and OST HMDs using existing in-situ visualization methods. Our results show that these visualizations cannot be directly transferred to OST displays without increasing error in depth perception tasks. To tackle this gap, we perform a structured decomposition of the visual properties of AR F+C methods to find best-performing combinations. We propose the use of chromatic shadows and hatching approaches transferred from computer graphics. In a second study, we perform a factorized analysis of these combinations, showing that varying the shading type and using colored shadows can lead to better depth estimation when using OST HMDs.

The state-of-the-art optical see-through head-mounted displays (OST-HMD) for augmented reality applications lack the ability to render correct light interaction behavior between digital and physical objects, known as mutual occlusion capability. This paper presents a novel optical architecture for enabling a compact, high performance, occlusion-capable optical see-through head-mounted display (OCOST-HMD) with correct, pupil-matched viewing perspective. The proposed design utilizes a single-layer, double-pass architecture, offering a compact OCOST-HMD solution that is capable of rendering per-pixel mutual occlusion, a correctly pupil-matched viewing between virtual and real views, and a very wide see-through field of view (FOV). Based on this architecture, we demonstrate a design embodiment and a compact prototype implementation. The prototype offers a virtual display with an FOV of 34 by 22, an angular resolution of 1.06 arc minutes per pixel, and an average image contrast greater than 40% at the Nyquist frequency of 53 cycles/mm. Further, the prototype system affords a wide see-though FOV of 90 by 50, within which about 40 diagonally is occlusion-enabled, along with an angular resolution of 1.0 arc minutes, comparable to a 20/20 vision and a dynamic range greater than 100:1. Lastly, we composed a quantitative color study that compares the effects of occlusion between a conventional HMD system and our OCOST-HMD system and the resulting response exhibited in different studies.

In spatial augmented reality, a widely dynamic projection mapping system has been developed as a novel approach to graphics presentation for widely moving objects in dynamic situations. However, this method necessitates a novel tracking marker design that is resistant to random and complex occlusion and out-of-focus blurring, which conventional markers have not achieved. This article presents a uniform circumferential marker that becomes an ellipse in perspective projection and expresses geometric information. It can track the relative posture of a dynamically moving sphere with high speed, high accuracy, and robustness owing to continuous contour lines, thereby supporting both wide-range movement in the depth direction and human interaction. Moreover, a biased circumferential marker is proposed to embed unique coding, where the absolute posture is decoded with a novel recognition algorithm. Moreover, rough initialization using the geometry of multiple ellipses is proposed for both markers to start the automatic and precise tracking. Real-time rotation visualization onto the surface of a moving sphere is made possible with the high-speed, widely dynamic projection mapping system. The tracking performance is demonstrated to exhibit sufficient basic tracking performance as well as robustness against blurring and occlusion compared to conventional dot-based markers.

Military technologies are typically the product of long-term development efforts. These technologies are often adapted late in the development process for use by other users, including other members of the defense enterprise, foreign militaries, or the commercial sector. The need to adapt a technology can arise from cost over-runs or changing operational requirements. This article describes a value-based methodology for developing a transition plan early in the development process, when the system is still in the conceptual phase. Performing this analysis early in the design phase allows for design choices that will support adapting the system for other users. Moreover, it provides a risk mitigation strategy against cost over-run and changes in operational requirements. The process begins by capturing the functional architecture of the technology and analyzing them against a set of tasks associated with different military positions. From there, the adapted technology is evaluated for its projected adequacy based on the value added and the effort required for adaptation. The end-state of this analysis is a value-based model that identifies a portfolio of possible alternative markets and/or uses, which can be comparatively analyzed. A case study is presented for adapting a military augmented reality system for a different market.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Current robotic endoluminal interventions require surgeons to hold a proximal joystick to control the distal flexible robot under 2-D X-ray guidance. However, the 2-D X-ray image is not intuitive, which not only increases the risk of surgical misoperation but also the workload of surgeons. Moreover, contact teleoperation exposes surgeons to the potentially infectious environment. To address it, this article proposes an augmented-reality-assisted touchless teleoperated robot for endoluminal intervention, called ARei. It aims to provide immersive experiences with augmented information. The robot integrates perceptual information obtained from an electromagnetic (EM) sensor, a shape sensor, and the virtual anatomy into a head-mounted display (HMD). The touchless teleoperation is used to control the robot, with the assistance of gesture recognition technology. Results show that the mean error of the calibration between the HMD and the EM tracking system is 4.67 mm, and the mean distance error between the points measured by the EM sensor and the points obtained by shape reconstruction with calibration is 5.19 mm (3.05%).

Multi-user mobile Augmented Reality (AR) has been successfully used in various fields as a novel visual interaction technology. But current mainstream wearable device-based and app-based solutions are still facing cross-platform, real-time communication, and intensive computing requirements. Mobile Web technology is envisioned to be a promising supporting technology for cross-platform application of mobile AR especially in 5G networks, which provide pervasive communication and computing resources thereby forming a formidable framework for the practical application of multi-user mobile Web AR. However, the problem of how to use these new techniques properly to achieve efficient communication and computing collaboration is obviously paramount in order for multi-user mobile Web AR to be realized in 5G networks. In this article, we propose the first edge-assisted multi-user collaborative framework for mobile Web AR. Firstly, we propose a heuristic mechanism for efficient communication planning, which allows multi-user interaction synchronization to be achieved. Secondly, we introduce a motion-aware key frame selection mechanism to optimize the computational efficiency of the edge system, and simultaneously alleviate the initialization problem by collaborating with nearby mobile devices using the Device-to-Device communication technique. Experiments are conducted in a real-world 5G network, and the results demonstrate the superiority of our proposed collaborative framework.

The COVID-19 has led to home quarantine in global scale, which increased people's need for indoor fitness activities in order to relieve stress/anxiety and learn knowledge well during the prevention. In this paper, an augmented reality (AR) indoor exergame is developed, mainly a kind of moving target hitting task with Lomachenko's headwear speedball as the prototype. Then, a comparative experiment is performed to test the effect of the exergame in relieving anxiety, as well as the differences between AR and VR modes (using HTC Vive Pro) on user experience (flow experience and motion sickness) and interaction efficiency. Results from twenty-eight participants show that the anxiety level of participants significantly reduced after using the exergame. Although the difference on flow experience is not significant, there is a marginally significant difference on nausea level, a dimension of motion sickness. Moreover, interaction efficiency between the two modes is significantly different. In the task of moving target hitting, AR mode performed better in avoiding motion sickness and improving interaction performance. IEEE

The internet-of-things (IoT) refers to the growing field of interconnected pervasive computing devices and the networking that supports smart, embedded applications. The IoT has multiple human-computer interaction challenges due to its many formats and interlinked components, and central to these is the need to provide sensory information and situational context pertaining to users in a more human-friendly, easily understandable format. This work addresses this by applying mixed reality toward expressing the underlying behaviors and states internal to IoT devices and IoT-enabled objects. It extends the authors’ previous research on IoT Avatars (mixed reality character representations of physical IoT devices), presenting a new head-mounted display framework and interconnection architecture. This contributes i) an exploration of mixed reality for smart spaces, ii) an approach toward expressive avatar behaviors using fuzzy inference, and iii) an early functional prototype of a hybrid physical and mixed reality IoT-enabled object. This approach is a step toward new information presentation, interaction, and engagement capabilities for smart devices and environments.

With the advent of digital technology, emergent media such as smart tablets, immersive multi-projector installations, augmented reality, and virtual reality are increasingly integrated into museums...

Mobile edge computing (MEC) is a promising paradigm to meet the low latency requirement of applications like virtual/augmented reality, which moves computation work-loads from remote cloud to network edge. Task offloading is a crucial problem in a MEC system, which significantly concerns with the achieved latency of tasks and the energy consumption on edge nodes. Although many efforts have been paid on the problem, most of them ignore the hierarchical architecture of MEC and the heterogeneity of tasks generated by different applications. In this paper, we consider a three-layer MEC system with tasks of various types stochastically arrive in real time, in which each task can be executed on mobile smart devices, offloaded to the edge server, or offloaded to the remote cloud. The task offloading problem in such a MEC system with the objective to minimize the averaged latency over time is NP-hard. To solve it, we employ Lyapunov optimization and duality theory to reformulate the problem and decompose it into a set of subproblems. Each subproblem can be distributedly solved by a mobile device or the edge server individually. Extensive simulations are also performed to verify the feasibility and superiority of our approach.

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this paper proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average F_{measure}. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR.

Background In recent years, the development of digital imaging technology has had a significant influence in liver surgery. The ability to obtain a 3-dimensional (3D) visualization of the liver anatomy has provided surgery with virtual reality of simulation 3D computer models, 3D printing models and more recently holograms and augmented reality (when virtual reality knowledge is superimposed onto reality). In addition, the utilization of real-time fluorescent imaging techniques based on indocyanine green (ICG) uptake allows clinicians to precisely delineate the liver anatomy and/or tumors within the parenchyma, applying the knowledge obtained preoperatively through digital imaging. The combination of both has transformed the abstract thinking until now based on 2D imaging into a 3D preoperative conception (virtual reality), enhanced with real-time visualization of the fluorescent liver structures, effectively facilitating intraoperative navigated liver surgery (augmented reality). Data sources A literature search was performed from inception until January 2021 in MEDLINE (PubMed), Embase, Cochrane library and database for systematic reviews (CDSR), Google Scholar, and National Institute for Health and Clinical Excellence (NICE) databases. Results Fifty-one pertinent articles were retrieved and included. The different types of digital imaging technologies and the real-time navigated liver surgery were estimated and compared. Conclusions ICG fluorescent imaging techniques can contribute essentially to the real-time definition of liver segments; as a result, precise hepatic resection can be guided by the presence of fluorescence. Furthermore, 3D models can help essentially to further advancing of precision in hepatic surgery by permitting estimation of liver volume and functional liver remnant, delineation of resection lines along the liver segments and evaluation of tumor margins. In liver transplantation and especially in living donor liver transplantation (LDLT), 3D printed models of the donor's liver and models of the recipient's hilar anatomy can contribute further to improving the results. In particular, pediatric LDLT abdominal cavity models can help to manage the largest challenge of this procedure, namely large-for-size syndrome.

Background Augmented reality (AR), virtual reality (VR), and mixed reality (MR) are emerging technologies that are starting to be translated into clinical practice. Limited data are available regarding these tools in use during live surgery of the spine. Our objective was to systematically collect, analyze, and interpret the existing data regarding AR, VR, and MR use in spine surgery on living people. Methods A systematic review was conducted using the PRISMA (preferred reporting items for systematic reviews and meta-analyses) guidelines. The PubMed, PubMed Central, Cochrane Reviews, and Embase databases were searched. Combinations and variations of the phrases “augmented reality,” “virtual reality,” and spine surgery using both “AND” and “OR” configurations were used to find relevant studies. The references of the included reports from the systematic review were also screened for possible inclusion as a part of a manual review. The included studies were full-text publications written in English that had included any spine surgery on live persons with the use of VR or AR. Results A total of 1566 unique reports were found, and 15 full-text publications met the criteria for the present study. The total number of patients from all studies was 241, with a weighted average age of 50.37 years. Surgical procedures using AR, VR, and/or MR were diverse and spanned from simple discectomy to intradural spinal tumor resection. All patients experienced improvement in their symptoms present at clinical presentation. The highest complication rate reported in the studies was 6.1% and was for suboptimal pedicle screw placement. No complications led to clinical sequelae. Conclusions The systematically collected, analyzed, and interpreted data of existing peer-reviewed full-text articles showed favorable metrics regarding surgical efficacy, pedicle screw target accuracy, radiation exposure, clinical outcomes, and disability and pain for patients with spinal pathology treated with the help of AR, VR, and/or MR.

In this interview Andreas Hinterhuber and Michael Nilles, CDIO of Henkel, discuss the game-changing opportunities that the digital transformation opens to companies that embed a digital core into their business models. Michael Nilles sees digital transformation as the Holy Grail: a force that is not easy to find, not easy to capture and that has the potential to dramatically improve the customer experience. In B2C, the Holy Grail for Henkel’s Beauty Care business is for instance the beauty tech ecosystem, a series of connected devices harnessing big data and augmented reality allowing to create meaningful, personalized and direct relationships with consumers. In B2B, the Holy Grail is Henkel’s digital twin along the digital thread. Henkel builds a digital twin along the entire value chain of the customer – starting with the initial customer request via deployment, until and after sales service – in order to sell outcomes to customers. In B2B the digital twin enables outcome-based servitization. Digital technologies are thus more than enabling technologies: digital technologies allow the creation of fundamentally new, disruptive business models. The digital transformation needs small, agile teams with end-to-end responsibility for project delivery, start-up mentality and customer obsession. Finally, key metrics of the digital transformation are digital efficiency/productivity, digital revenues and digital growth (net of cannibalization).

Malfunctions are a major challenge in partially automated driving. During such malfunctions, the driver must be able to adequately take over vehicle guidance without being requested to intervene. This may be particularly difficult in urban areas due to their high complexity and information density. Augmented Reality Head-Up Displays (ARHUDs) may have the potential to support the driver during the monitoring task by providing driving-related information at its required location in the primary field of view. The effects of an ARHUD compared to a Baseline concept in case of malfunctions were investigated in a driving simulation experiment with 52 participants. In a partially automated urban drive, participants experienced a longitudinal and a lateral malfunction in permuted order. The concepts -ARHUD or Baseline- were presented as a between-subject factor. The results showed significantly shorter take-over times when using the ARHUD, resulting in fewer crashes. For those who were able to avoid the crash, no differences in the take-over quality between both concepts were found. There was one difference in visual attention: the attention ratio on the instrument cluster was lower for the ARHUD. In addition, the ARHUD revealed a significantly higher trust and usability rating. However, there were no differences in acceptance and subjective workload between the two concepts. The results showed that the ARHUD has more potential to prevent crashes in the event of malfunctions compared to the Baseline. Nevertheless, the high number of crashes, regardless of the concept, showed the importance of a fallback level for partially automated urban driving.

Applications of virtual reality (VR) and augmented reality (AR) assist both health care providers and patients in cardiovascular education, complementing traditional learning methods. Interventionalists have successfully used VR to plan difficult procedures and AR to facilitate complex interventions. VR/AR has already been used to treat patients, during interventions in rehabilitation programs and in immobilized intensive care patients. There are numerous additional potential applications in the catheterization laboratory. By using AR, interventionalists could combine visual fluoroscopy information projected and registered on the patient body with data derived from preprocedural imaging and live fusion of different imaging modalities such as fluoroscopy with echocardiography. Persistent technical challenges to overcome include the integration of different imaging modalities into VR/AR and the harmonization of data flow and interfaces. Cybersickness might exclude some patients and users from the potential benefits of VR/AR. Critical ethical considerations arise in the application of VR/AR in vulnerable patients. In addition, digital applications must not distract physicians from the patient. It is our duty as physicians to participate in the development of these innovations to ensure a virtual health reality benefit for our patients in a real-world setting. The purpose of this review is to summarize the current and future role of VR and AR in different fields within cardiology, its challenges, and perspectives.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

A variety of information of the real-time scenes is carried by the images and videos. Processing these images and videos in an intelligent way helps in many domains such as computer vision, object detection, deep learning and 3D reconstruction leaving its large usage in applications such as auto pilots, augmented reality, smart vehicles, etc. The quality of image and videos plays a vital role in case of real-time systems. One such scenario is where the images are captured without sufficient illumination. Images captured in cameras where sufficient light is not present, leads to noisy and information-loss images. It is a fact that, dark images have mainly two aspects which make its study a difficult task. They are at its low dynamic range and high propensity for generating high noise levels. Hence, an approach based on deep learning based system is adopted. For this purpose, Generative Adversarial Network (GAN) based Extremely Dark Video Enhancement Network (GEVE) model is proposed. The main objective of GEVE is to team the model with low /normal- light image pairs. Thus, GAN network learns the translation from light feeble images and images captured under normal illumination and automatically translate original images taken under extremely low light conditions into images of quality. It is clearly observed that the proposed GEVE outperforms the known state-of-art techniques. We are the view that the proposed system is an ideal candidate to handle dark image/video frames.

During crisis situations, emergency operators require fast information access to achieve situation awareness and make the best possible decisions. Augmented reality could be used to visualize the wealth of user-generated content available on social media and enable context-adaptive functions for emergency operators. Although emergency operators agree that social media analytics will be important for their future work, it poses a challenge to filter and visualize large amounts of social media data. We conducted a goal-directed task analysis to identify the situation awareness requirements of emergency operators. By collecting tweets during two storms in Germany we evaluated the usefulness of Twitter data for achieving situation awareness and conducted interviews with emergency operators to derive filter strategies for social media data. We synthesized the results by discussing how the unique interface of augmented reality can be used to integrate social media data into emergency control rooms for situation awareness support.

© 2018 by the authors. Collaboration has been common in workplaces in various engineering settings and in our daily activities. However, how to effectively engage collaborators with collaborative tasks has long been an issue due to various situational and technical constraints. The research in this paper addresses the issue in a specific scenario, which is how to enable users to interact with public information from their own perspective. We describe a 3D mobile interaction technique that allows users to collaborate with other people by creating a symmetric and collaborative ambience. This in turn can increase their engagement with public displays. In order to better understand the benefits and limitations of this technique, we conducted a usability study with a total of 40 participants. The results indicate that the 3D mobile interaction technique promotes collaboration between users and also improves their engagement with the public displays.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Augmented reality (AR) applications that overlay a user's perception of the real world with digitally generated information are on the cusp of commercial viability. AR has appeared in several commercial platforms like Microsoft HoloLens and smartphones. They extend the user experience beyond two dimensions and supplement a user's normal 3D world. A typical location-based multi-player AR application works through a three-step process, wherein the system collects sensory data from the real world, identifies objects based on their context, and finally, renders information on top of the user's senses. However, because these AR applications frequently exchange data with users, they have exposed new individual and public safety issues. In this paper, we develop ARSpy, a user location tracking system solely based on network traffic information of the user, and we test it on location-based multi-player AR applications. We demonstrate the effectiveness of the proposed scheme via real-world experiments on 12 volunteers and show that we could obtain the geolocation of any target with high accuracy. We also propose three defense methods to mitigate these side channel attacks. Our results reveal a potential security threat in location-based multi-player AR applications and serve as a critical security reminder to AR users.

Augmented reality (AR) has huge potential as a science outreach tool, especially around palaeontology where it is possible to bring extinct animals to life. This paper shares our experiences as academic geoscientists in developing a series of AR applications during a three-year period. We do not focus on the technical issues of app design and coding but on the lessons learnt during this time which may be of value to others, whether academics, event coordinators or museum curators interested in using digital technology to enhance user experience or convey educational messages. We identify eight issues that are worth keeping in mind when commissioning, developing, and deploying AR apps for use as a science communication tool. The issues are: (1) download barriers; (2) ageing novelty of AR; (3) the ‘stickiness’ of an app tends to be moment-locations specific; (4) maintenance of bespoke apps; (5) ‘boom and lapse’ cycles in funding; (6) difficulty of embedding educational content; (7) priority conflict between users and event organisers; and (8) the quality of the animation is key.

Ultrasound offers a real-time 2-D view of structures within the human body. While many medical education programs have already dedicated a portion of their curriculum to ultrasound, others are concerned about cost, accessibility and limits to student practice. Student benefit may be affected by cognitive errors, which are in part owing to the mental heuristics required to visualize a 3-D structure by interpreting a 2-D image. A possible solution to eliminating subjectivity in ultrasound interpretation is the use of 3-D models to augment the traditional 2-D ultrasound experience. PubMed, Embase and Web of Science were searched for primary literature exploring relationships between 3-D modeling applications and their use in ultrasound education. The search and review process was guided by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist. Overall, 14 of the included 16 studies indicated a significant improvement in medical education of ultrasound with the intervention of 3-D modeling applications. This systematic review confirms that 3-D modeling applications benefit student learning in ultrasound education while illuminating the need for more research in this field.

Refereed/Peer-reviewed This paper presents an Augmented Reality (AR)-based network cabling tutoring system that trains users how to interconnect cables within a Virtual Local Area Network (VLAN) on a physical switching rack. AR arrows combined with text-based instruction and a checklist provided assistance during practical learning. When learners made a mistake, an Intelligent Tutoring System (ITS) identified the source of the mistake and provided real-time feedback using text, AR and text-to-speech mechanisms. The design was motivated by the human-cognitive architecture and its five evolutionary principles (proposed by Sweller and Sweller (2006)). Users performed four consecutive network cabling training tasks with assistance from our ITS. We found that users made fewer errors when the AR cues, text-based instruction and checklist solutions were replaced with summarised information and then removed completely in the final task compared to those who used an identical system with fixed instruction. Cognitive Load Theory (CLT) explains our results by suggesting that the instructional mechanisms become redundant as knowledge increases. Implications of the study are discussed as well as how AR can help facilitate knowledge transfer in the network cabling domain.

As a hyper-natural interaction technique in 3D user interfaces, non-isomorphic rotation has been considered an effective approach for rotation tasks, where a static or dynamic control-display gain can be applied to amplify or attenuate a rotation. However, it is not clear whether non-isomorphic rotation can benefit 6-degree-of-freedom (6-DOF) manipulation tasks in AR and VR. In this article, we extended the usability studies of non-isomorphic rotation from rotation-only tasks to 6-DOF manipulation tasks and analyzed the collected data using a 2-component model. Using a mixed reality (MR) simulation approach, we also investigated whether environment (AR or VR) had an impact on 3D manipulation tasks. The results reveal that although both static and dynamic non-isomorphic rotation techniques could save time and effort in ballistic phases, only dynamic non-isomorphic rotation was significantly faster than isomorphic rotation. Interestingly, while environment had no significant impact on overall user performance, we found evidence that it could affect fine-tuning in correction phases. We also found that most participants preferred AR over VR, indicating that environmental visual realism could be helpful to improve user experience.

Background This review aims to examine the existing literature to address currently used virtual, augmented, and mixed reality modalities in the areas of preoperative surgical planning, intraoperative guidance, and postoperative management in the field of cardiothoracic surgery. In addition, this innovative technology provides future perspectives and potential benefits for cardiothoracic surgeons, trainees, and patients. Methods A targeted, non-systematic literature assessment was performed within the Medline and Google Scholar databases to help identify current trends and to provide better understanding of the current state-of-the-art extended reality (XR) modalities in cardiothoracic surgery. Related articles published up to July 2020, are included in the review. Results XR is a novel technique gaining increasing application in cardiothoracic surgery. It provides a three-dimensional (3D) and realistic view of structures and environments and offers the user the ability to interact with digital projections of surgical targets. Recent studies showed the validity and benefits of XR applications in cardiothoracic surgery. Examples include XR-guided pre-operative planning, intraoperative guidance and navigation, post-operative pain and rehabilitation management, surgical simulation, and patient education. Conclusions XR is gaining interest in the field of cardiothoracic surgery. In particular, there are promising roles for XR applications in televirtuality, surgical planning, surgical simulation, and perioperative management. However, future refinement and research is needed to further implement XR in the aforementioned settings within cardiothoracic surgery.

The furniture business has developed quite well every year, but the promotions used by the company still use the usual methods, such as using brochures that only display 2D information and images. With so much competition, consumers want something unique and different so they can find out more detailed and real product information offered. Augmented reality is the right technology to meet these consumer needs. Augmented reality is a technology that combines the virtual world with the real world directly. Therefore, an application was designed that functions as a company profile as well as displays 3D rattan chair products using Augmented reality technology on android devices. The method used in this research is the Multimedia Development Life Cycle (MDLC) which includes Concept, Design, Material Collecting, Assembly, Testing, and Distribution. This Augmented reality application requires streaming video taken from a smartphone camera as an input source, then this application will track and detect markers using a tracking system, after the marker is detected, the 3D chair model in the catalog will appear above the marker as if it were a chair model. it's real. the results of the hypothesis test with a significant level of 0.139, which is greater than 0.05, then what is accepted is H1. It can be stated that an augmented reality-based electronic catalog application can increase buying interest in potential consumers of Raffi cushion rotan

The development of Smart Home Controllers has seen rapid growth in recent years, especially for smart devices, that can utilize the Internet of Things (IoT). However, a large portion of the household devices and appliances already in use, are not IoT enabled, and therefore, requires their default control mechanisms for the devices to operate. This paper proposes a smart appliance controller that uses Augmented Reality, MQTT, and other up-to-date platforms to control aftermarket home appliances in the most efficient manner. The proposed work integrates mobile AR with IoT, to control household appliances, with the help of infrared (IR) signals. The characteristics of the system are evaluated through a series of tests and performance measures. The results of the test highlight th quick response time of MQTT for the implementation of a Home Automation System, when compared to the request-reply protocol: CoAP (4 times as fast).

OBJECTIVES Extended reality (XR), encompassing both virtual reality (VR) and augmented reality, allows the user to interact with a computer-generated environment based on reality. In essence, the immersive nature of VR and augmented reality technology has been warmly welcomed in all aspects of medicine, gradually becoming increasingly feasible to incorporate into everyday practice. In recent years, XR has become increasingly adopted in thoracic surgery, although the extent of its applications is unclear. Here, we aim to review the current applications of XR in thoracic surgery. METHODS A systematic database search was conducted of original articles that explored the use of VR and/or augmented reality in thoracic surgery in EMBASE, MEDLINE, Cochrane database and Google Scholar, from inception to December 2020. RESULTS Our search yielded 1494 citations, of which 21 studies published from 2007 to 2019 were included in this review. Three main areas were identified: (i) the application of XR in thoracic surgery training; (ii) preoperative planning of thoracic procedures; and (iii) intraoperative assistance. Overall, XR could produce progression along the learning curve, enabling trainees to reach acceptable standards before performing in the operating theatre. Preoperatively, through the generation of 3D-renderings of the thoracic cavity and lung anatomy, VR increases procedural accuracy and surgical confidence through familiarization of the patient’s anatomy. XR-assisted surgery may have therapeutic use particularly for complex cases, where conventional methods would yield inadequate outcomes due to inferior accuracy. CONCLUSION XR represents a salient step towards improving thoracic surgical training, as well as enhancing preoperative planning and intraoperative guidance., Extended reality (XR), encompassing both virtual (VR) and augmented reality (AR), allows the user to interact with a computer-generated environment based on reality.

Augmented reality is a device in combining objects between 2D and 3D virtual worlds, there's information that has not been implemented directly to the user, then by responding into scope that is 3D on the object with the resulting time at the same time to read in real-time, related to learning media where the teaching and learning process generally uses a book, then develops using projectors and laptops as information technology, besides the development of mobile (cellular) or android-based information technology that can make it easier for learning, especially the introduction of the solar system, the beginning of the solar system introduced various space objects both planets and surrounding space objects, in class VI who received the material as a basis before going to the next level, but the teaching and learning process there are several factors that influence i if it is examined that there are elderly teachers who have difficulty keeping up with technological developments, from learning media support tools such as projectors and teaching aids are still very limited, and not all teachers use information technology as a learning medium, the purpose of which is to make an introduction to AR learning media applications android-based solar system for class VI, based on these 3 problems as a solution an educational application is needed that supports learning activities using android augmented reality for the introduction of the solar system with the marker based tracking method, becomes a marker of 3D objects to bring up from the virtual world to the real which contains information on each of these objects, in other words visualizing the results of the marker directly into the real world, the final result is the creation of an AR application for learning the introduction of the solar system.

Visual odometry (VO) is a fundamental building block for many robotics and augmented reality (AR) applications. However, most existing RGB-D VO systems could suffer from large performance degradation when large occlusions are present or a large portion of depth values are invalid due to the limited range of an RGB-D camera, prohibiting the usage of most systems in practical applications. In this paper, we present RGB-D DSO, a RGB-D direct sparse odometry which can achieve state-of-theart accuracy, robustness and efficiency in the indoor scenes. Our RGB-D DSO is a significant extension of monocular DSO [1], with the core part being sliding-window optimization with occlusion removal and a depth refinement module. Occlusion removal excludes negative effects arising from occluded objects when minimizing the final energy function for camera pose tracking. Depth refinement ensures sufficient valid depth values uniformly distributed for the depth map of a keyframe. Experimental results on three public datasets [2], [3], [4] demonstrate that our method achieves smaller absolute trajectory errors (ATE) and relative pose error (RPE) than the state-of-the-art methods DVO [5], ORB-SLAM2 [6], CVO[7] and ACVO[8]. Meanwhile, our system takes only 21.93ms to track a frame, which is 3.7X faster than DVO (80.27ms/frame) and 1.4X faster than ORB-SLAM2 (29.91ms/frame).

We present the case of a 74-year-old man with a history of a squamous cell carcinoma in the left axilla. The patient underwent a multidisciplinary surgical resection through an extended forequarter amputation with thoracic wall resection and reconstruction. With regard to the complexity of the case, three-dimensional virtual reality–based patient-specific reconstructions were used as a supplemental tool to conventional computed tomography imaging to plan the procedure. With this report, we aim to stimulate further research to improve and automate the workflow and to bring virtual and augmented reality reconstructions into the surgical theater of the future.

The programming of manipulators is a common task in robotics, for which numerous solutions exist. In this work, a new programming method related to the common master-slave approach is introduced, in which the master is replaced by a digital twin created through haptic and visual rendering. To achieve this, we present an algorithm that enables the haptic rendering of any programmed robot with a serial manipulator on a general-purpose haptic interface. The results show that the proposed haptic rendering reproduces the kinematic properties of the programmed robot and directly provides the desired joint space trajectories. In addition to a stand-alone usage, we demonstrate that the proposed algorithm can be easily paired with existing visual technology for virtual and augmented reality to facilitate a highly immersive programming experience.

This study aims at developing and evaluating an affordable and user-friendly motion capture system for human upper limbs' joint kinematics estimation. The objective is to provide quantitative assessments during the clinical evaluation of poststroke patients performing daily living activities. The proposed system is based on the simultaneous use of affordable inertial measurement units, and a set of augmented reality markers tracked with an affordable RGB camera. Two practical calibration processes were developed to calibrate the sensors modules and then to determine their location on body segments. Then, all measured quantities were fused into a constrained extended Kalman filter based on an upper limbs' biomechanical model. The proposed system was validated with nine healthy volunteers performing five daily living activities. Joint angles estimated using the proposed affordable system were compared with a gold standard stereophotogrammetric system. The results showed a low average rms difference (2.7°) along with a high average correlation (0.87).

Coronavirus disease 2019 created unintended but significant experiential barriers for surgical learners to interact at the bedside for teaching/case presentations. We hypothesized that an international grand rounds using the Microsoft HoloLens 2 extended reality (XR) headset would create an improved bedside-learning experience compared to traditional grand rounds formats.From December 2020 to March 2021, the world's first 2 international mixed reality grand rounds events using the HoloLens 2 headset were held, broadcasting transatlantically (between the University of Michigan and the Imperial College of London) bedside rounding experiences on 5 complex surgical patients to an international audience of 325 faculty, residents, and medical trainees. Participants completed pre- and post-event surveys to assess their experience.Of the 325 participants, 267 (80%) completed pre-surveys, and 95 (29%) completed both the pre- and post-surveys. Respondents (average age, 38 y; 44% women, 56% men; 211 US, 56 UK) included 92 (34%) medical students and residents and 175 faculty and staff. In the pre-event survey, 76% had little or no earlier experience with XR devices, and 94% thought implementation of XR into medical curricula was valuable. In the post-survey, 96% thought telerounding using XR technology was important for the current era, and 99% thought the ability to visualize the examination, imaging, and laboratory results at bedside via XR rounding was highly valuable and that this format was superior to traditional grand rounds.Almost all of the participants in the mixed reality international grand rounds felt the immersive XR experiences-allowing visualization of clinical findings, imaging, and laboratory results at the patient's bedside-were superior to a traditional grand rounds format, and that it could be a valuable tool for surgical teaching and telerounding.

International audience; Situated visualization is an emerging concept within visualization, in which data is visualized in situ, where it is relevant to people. The concept has gained interest from multiple research communities, including visualization, human-computer interaction (HCI) and augmented reality. This has led to a range of explorations and applications of the concept, however, this early work has focused on the operational aspect of situatedness leading to inconsistent adoption of the concept and terminology. First, we contribute a literature survey in which we analyze 44 papers that explicitly use the term "situated visualization" to provide an overview of the research area, how it defines situated visualization, common application areas and technology used, as well as type of data and type of visualizations. Our survey shows that research on situated visualization has focused on technology-centric approaches that foreground a spatial understanding of situatedness. Secondly, we contribute five perspectives on situatedness (space, time, place, activity, and community) that together expand on the prevalent notion of situatedness in the corpus. We draw from six case studies and prior theoretical developments in HCI. Each perspective develops a generative way of looking at and working with situatedness in design and research. We outline future directions, including considering technology, material and aesthetics, leveraging the perspectives for design, and methods for stronger engagement with target audiences. We conclude with opportunities to consolidate situated visualization research.

Camera-based multi-class (1D and 2D) barcode detectors that can help in decoding barcodes in different complex scenes have huge potential applications in situations where Internet of Things (IoT) is combined with Artificial Intelligence (AI) and Augmented Reality (AR). The decoding rate in such applications under real-life complex scenes is greatly affected by two major factors: First, we cannot accurately localize the barcodes, and secondly, we cannot decode the blur samples. In this paper, we firstly propose a barcode localization algorithm that is capable of regressing 4 vertices of barcodes accurately. Our localization method comprises an anchor-free approach that outputs multi-scale output prediction maps. These segmentation-like maps of each scale are then further divided into three types of maps (classification, centerness, and 8-D regression). 8-D localization result of barcodes along-with classification result is then obtained after post-processing. Secondly, we propose a conditional generative adversarial network-based model for deblurring blur QR codes. Extensive decoding experiments on a challenging complex scene dataset show that our localization and deblurring methods can contribute to improving the decoding rate of existing barcode decoders.

Purpose The purpose of this paper is to introduce a digitalization model (DM) for building construction courses in architectural education as a response to the recent emerging technologies in the era of digital transformations. This DM is developed and applied through augmented reality (AR) technologies to boost perception, understandings and ability to solve building construction details. Design/methodology/approach Based on a thorough review of recent technologies like AR, virtual reality (VR), building information modelling (BIM) and their applications in architectural education, the methodology involves the generation of a model, its application and evaluation. The model is based on the integration of BIM and AR which is applied into a third year “Building Construction Project” course. Each student has designed a residential building and accordingly prepared the construction drawings by adapting the DM. An online survey –based on Technology Acceptance Model – is conducted to evaluate the DM by quantitative data analysis from SPSS and Excel. Findings The key findings of the study include the following items: determination of the proper digital tools, the definition of the steps and workflow based on the building project phases to develop construction drawings and define precise details effectively. By the help of this, the DM is generated and applied. According to the survey and results, the DM which involves BIM-based AR is considered as beneficial, highly motivating and providing better perception on construction details. Originality/value Implication of AR/VR technologies is frequently seen in design studios, whereas building construction courses state its traditional approach. However, there is a huge potential in the digitalization of building construction education by increasing the perception of students together with the increased level of communication. The study aims to close the gap of digitalization by proposing a DM, which brings a systematic approach considering each phase of building construction project as conceptual, schematic, design development and construction documents by using BIM integrated AR. Moreover, the novel model specifically brings a new approach by generation of QR codes for construction details to embed videos or simulations into the two-dimensional drawing sheets. Furthermore, the DM proposes a new approach to satisfy emerging needs and requirements of the Architecture Engineering Construction industry.

Mutual occlusion is an essential feature for augmented reality (AR) displays for allowing the virtual content to be clearly perceived under an excessively illuminated environment. Although a few works have been done to facilitate the performance of occlusion-capable optical see-through augmented reality (OC-OST-AR) displays, the realization of mutual occlusion in a wide field-of-view (FOV) is still challenging. Divergent from typical hard-edge occlusion and soft edge-occlusion designs, we propose the paired-ellipsoidal-mirror (PEM) structure. The proposed system is allowed to support either hard-edge occlusion or enhanced soft-edge occlusion in a wide FOV by optionally fixing a spatial light modulator (SLM) before the entrance pupil or at an inner focal plane. The numerical aperture (NA) of the system is efficiently increased by the combination of paired ellipsoidal mirror imaging and aperture stop restriction. With proof-of-concept prototypes built, virtual display in a FOV of H160°×V74° and mutual occlusion in a FOV of H122°×V74° are demonstrated with a basic design, respectively. Furthermore, a mixed FOV of H95.3°×V52.9° is demonstrated by an optimized design with vertical parallax reduction and virtual display improvement.

In augmented reality, it is important to achieve visual consistency between inserted virtual objects and the real scene. As specular and transparent objects can produce caustics, which affect the appearance of inserted virtual objects, we herein propose a framework for differential rendering beyond the Lambertian-world assumption. Our key idea is to jointly optimize illumination and parameters of specular and transparent objects. To estimate the parameters of transparent objects efficiently, the psychophysical scaling method is introduced while considering visual characteristics of the human eye to obtain the step size for estimating the refractive index. We verify our technique on multiple real scenes, and the experimental results show that the fusion effects are visually consistent.

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle's three-dimensional surrounding represented as an object-level environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as object-level 3D reconstruction, is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of in-vehicle object-level 3D reconstruction in two specific use cases: (i) augmented reality and (ii) automated parking. For augmented reality, we propose a cost-efficient solution called monocular 3D Shaping that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a Latent Shape Space, where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low- to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner.