Search

The architecture, engineering, and construction (AEC) industry is undergoing a substantial change, and the technological advancements are shaping the future of this industry. Over years, numerous tools have been developed based on building information modeling (BIM) and the Internet of Things (IoT) to manage a variety of tasks and activities within different construction and operational phases in the AEC industry. Recently, with the advent of novel internet-enabled technologies in providing the ability of real-time connectivity, researchers have started to develop virtual design and construction (VDC) and digital twin approaches to remotely monitor, control, and optimize different work progresses and activities in real-time at any work level in the AEC industry. This paper presents a reference model to illustrates the main advantages and applications of VDC and digital twin, and to demonstrate their trends and anticipated cost savings in the AEC industry at a global level. These technological advancements along with augmented reality (AR) and virtual reality (VR) can offer the unique benefit of real-time monitoring of the current status as well as predicting the future of any physical structure at any level of work. The global demand and usage of these technological advancements will lead to great cost savings in the AEC industry which would be US$950 million in design and construction and US$400 million in operation and maintenance phases in the non-residential AEC industry in 2025. This paper also discusses the future challenges of VDC and digital twin in the industry. As a main line of future research, these technologies can simultaneously be utilized with artificial intelligence to enable human-centered decision making to automate and optimize work progresses. Such integrations of VDC, digital twin, artificial intelligence, and human-centered approaches will shape the future of the AEC industry and provide numerous research opportunities in this domain.

The performance of smart vehicle (SV) applications like autonomous driving and in-vehicle augmented reality based traffic information system depends on the Field of View (FoV) and the timely processing of the SVs sensor data. Vehicular networking (VN) technology can enhance the performance of these applications by enabling a SV to access the sensing and processing capabilities of other neighbouring SVs. The processing and storage capacity of a SV is limited compared to cloud servers and the communication link between two SVs is unreliable due to their mobility and the nature of wireless channels. Hence, developing efficient processing and sensing schemes for SVs and VNs can help in optimizing the performance of SV applications. In this paper, we propose Contextual Bandits (CB), Markov decision process (MDP) and deep Q-network (DQN) based sensing and processing schemes for VNs. Simulation results show that the proposed schemes outperform the baseline schemes in a variety of scenarios.

Estimating the depth of virtual content has proven to be a challenging task in Augmented Reality (AR) applications. Existing studies have shown that the visual system makes use of multiple depth cues to infer the distance of objects, occlusion being one of the most important ones. The ability to generate appropriate occlusions becomes particularly important for AR applications that require the visualization of augmented objects placed below a real surface. Examples of these applications are medical scenarios in which the visualization of anatomical information needs to be observed within the patient's body. In this regard, existing works have proposed several focus and context (F+C) approaches to aid users in visualizing this content using Video See-Through (VST) Head-Mounted Displays (HMDs). However, the implementation of these approaches in Optical See-Through (OST) HMDs remains an open question due to the additive characteristics of the display technology. In this article, we, for the first time, design and conduct a user study that compares depth estimation between VST and OST HMDs using existing in-situ visualization methods. Our results show that these visualizations cannot be directly transferred to OST displays without increasing error in depth perception tasks. To tackle this gap, we perform a structured decomposition of the visual properties of AR F+C methods to find best-performing combinations. We propose the use of chromatic shadows and hatching approaches transferred from computer graphics. In a second study, we perform a factorized analysis of these combinations, showing that varying the shading type and using colored shadows can lead to better depth estimation when using OST HMDs.

The state-of-the-art optical see-through head-mounted displays (OST-HMD) for augmented reality applications lack the ability to render correct light interaction behavior between digital and physical objects, known as mutual occlusion capability. This paper presents a novel optical architecture for enabling a compact, high performance, occlusion-capable optical see-through head-mounted display (OCOST-HMD) with correct, pupil-matched viewing perspective. The proposed design utilizes a single-layer, double-pass architecture, offering a compact OCOST-HMD solution that is capable of rendering per-pixel mutual occlusion, a correctly pupil-matched viewing between virtual and real views, and a very wide see-through field of view (FOV). Based on this architecture, we demonstrate a design embodiment and a compact prototype implementation. The prototype offers a virtual display with an FOV of 34 by 22, an angular resolution of 1.06 arc minutes per pixel, and an average image contrast greater than 40% at the Nyquist frequency of 53 cycles/mm. Further, the prototype system affords a wide see-though FOV of 90 by 50, within which about 40 diagonally is occlusion-enabled, along with an angular resolution of 1.0 arc minutes, comparable to a 20/20 vision and a dynamic range greater than 100:1. Lastly, we composed a quantitative color study that compares the effects of occlusion between a conventional HMD system and our OCOST-HMD system and the resulting response exhibited in different studies.

In spatial augmented reality, a widely dynamic projection mapping system has been developed as a novel approach to graphics presentation for widely moving objects in dynamic situations. However, this method necessitates a novel tracking marker design that is resistant to random and complex occlusion and out-of-focus blurring, which conventional markers have not achieved. This article presents a uniform circumferential marker that becomes an ellipse in perspective projection and expresses geometric information. It can track the relative posture of a dynamically moving sphere with high speed, high accuracy, and robustness owing to continuous contour lines, thereby supporting both wide-range movement in the depth direction and human interaction. Moreover, a biased circumferential marker is proposed to embed unique coding, where the absolute posture is decoded with a novel recognition algorithm. Moreover, rough initialization using the geometry of multiple ellipses is proposed for both markers to start the automatic and precise tracking. Real-time rotation visualization onto the surface of a moving sphere is made possible with the high-speed, widely dynamic projection mapping system. The tracking performance is demonstrated to exhibit sufficient basic tracking performance as well as robustness against blurring and occlusion compared to conventional dot-based markers.

Military technologies are typically the product of long-term development efforts. These technologies are often adapted late in the development process for use by other users, including other members of the defense enterprise, foreign militaries, or the commercial sector. The need to adapt a technology can arise from cost over-runs or changing operational requirements. This article describes a value-based methodology for developing a transition plan early in the development process, when the system is still in the conceptual phase. Performing this analysis early in the design phase allows for design choices that will support adapting the system for other users. Moreover, it provides a risk mitigation strategy against cost over-run and changes in operational requirements. The process begins by capturing the functional architecture of the technology and analyzing them against a set of tasks associated with different military positions. From there, the adapted technology is evaluated for its projected adequacy based on the value added and the effort required for adaptation. The end-state of this analysis is a value-based model that identifies a portfolio of possible alternative markets and/or uses, which can be comparatively analyzed. A case study is presented for adapting a military augmented reality system for a different market.

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Current robotic endoluminal interventions require surgeons to hold a proximal joystick to control the distal flexible robot under 2-D X-ray guidance. However, the 2-D X-ray image is not intuitive, which not only increases the risk of surgical misoperation but also the workload of surgeons. Moreover, contact teleoperation exposes surgeons to the potentially infectious environment. To address it, this article proposes an augmented-reality-assisted touchless teleoperated robot for endoluminal intervention, called ARei. It aims to provide immersive experiences with augmented information. The robot integrates perceptual information obtained from an electromagnetic (EM) sensor, a shape sensor, and the virtual anatomy into a head-mounted display (HMD). The touchless teleoperation is used to control the robot, with the assistance of gesture recognition technology. Results show that the mean error of the calibration between the HMD and the EM tracking system is 4.67 mm, and the mean distance error between the points measured by the EM sensor and the points obtained by shape reconstruction with calibration is 5.19 mm (3.05%).

Multi-user mobile Augmented Reality (AR) has been successfully used in various fields as a novel visual interaction technology. But current mainstream wearable device-based and app-based solutions are still facing cross-platform, real-time communication, and intensive computing requirements. Mobile Web technology is envisioned to be a promising supporting technology for cross-platform application of mobile AR especially in 5G networks, which provide pervasive communication and computing resources thereby forming a formidable framework for the practical application of multi-user mobile Web AR. However, the problem of how to use these new techniques properly to achieve efficient communication and computing collaboration is obviously paramount in order for multi-user mobile Web AR to be realized in 5G networks. In this article, we propose the first edge-assisted multi-user collaborative framework for mobile Web AR. Firstly, we propose a heuristic mechanism for efficient communication planning, which allows multi-user interaction synchronization to be achieved. Secondly, we introduce a motion-aware key frame selection mechanism to optimize the computational efficiency of the edge system, and simultaneously alleviate the initialization problem by collaborating with nearby mobile devices using the Device-to-Device communication technique. Experiments are conducted in a real-world 5G network, and the results demonstrate the superiority of our proposed collaborative framework.

The COVID-19 has led to home quarantine in global scale, which increased people's need for indoor fitness activities in order to relieve stress/anxiety and learn knowledge well during the prevention. In this paper, an augmented reality (AR) indoor exergame is developed, mainly a kind of moving target hitting task with Lomachenko's headwear speedball as the prototype. Then, a comparative experiment is performed to test the effect of the exergame in relieving anxiety, as well as the differences between AR and VR modes (using HTC Vive Pro) on user experience (flow experience and motion sickness) and interaction efficiency. Results from twenty-eight participants show that the anxiety level of participants significantly reduced after using the exergame. Although the difference on flow experience is not significant, there is a marginally significant difference on nausea level, a dimension of motion sickness. Moreover, interaction efficiency between the two modes is significantly different. In the task of moving target hitting, AR mode performed better in avoiding motion sickness and improving interaction performance. IEEE

Mobile edge computing (MEC) is a promising paradigm to meet the low latency requirement of applications like virtual/augmented reality, which moves computation work-loads from remote cloud to network edge. Task offloading is a crucial problem in a MEC system, which significantly concerns with the achieved latency of tasks and the energy consumption on edge nodes. Although many efforts have been paid on the problem, most of them ignore the hierarchical architecture of MEC and the heterogeneity of tasks generated by different applications. In this paper, we consider a three-layer MEC system with tasks of various types stochastically arrive in real time, in which each task can be executed on mobile smart devices, offloaded to the edge server, or offloaded to the remote cloud. The task offloading problem in such a MEC system with the objective to minimize the averaged latency over time is NP-hard. To solve it, we employ Lyapunov optimization and duality theory to reformulate the problem and decompose it into a set of subproblems. Each subproblem can be distributedly solved by a mobile device or the edge server individually. Extensive simulations are also performed to verify the feasibility and superiority of our approach.

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this paper proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average F_{measure}. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR.

The internet-of-things (IoT) refers to the growing field of interconnected pervasive computing devices and the networking that supports smart, embedded applications. The IoT has multiple human-computer interaction challenges due to its many formats and interlinked components, and central to these is the need to provide sensory information and situational context pertaining to users in a more human-friendly, easily understandable format. This work addresses this by applying mixed reality toward expressing the underlying behaviors and states internal to IoT devices and IoT-enabled objects. It extends the authors’ previous research on IoT Avatars (mixed reality character representations of physical IoT devices), presenting a new head-mounted display framework and interconnection architecture. This contributes i) an exploration of mixed reality for smart spaces, ii) an approach toward expressive avatar behaviors using fuzzy inference, and iii) an early functional prototype of a hybrid physical and mixed reality IoT-enabled object. This approach is a step toward new information presentation, interaction, and engagement capabilities for smart devices and environments.

In this interview Andreas Hinterhuber and Michael Nilles, CDIO of Henkel, discuss the game-changing opportunities that the digital transformation opens to companies that embed a digital core into their business models. Michael Nilles sees digital transformation as the Holy Grail: a force that is not easy to find, not easy to capture and that has the potential to dramatically improve the customer experience. In B2C, the Holy Grail for Henkel’s Beauty Care business is for instance the beauty tech ecosystem, a series of connected devices harnessing big data and augmented reality allowing to create meaningful, personalized and direct relationships with consumers. In B2B, the Holy Grail is Henkel’s digital twin along the digital thread. Henkel builds a digital twin along the entire value chain of the customer – starting with the initial customer request via deployment, until and after sales service – in order to sell outcomes to customers. In B2B the digital twin enables outcome-based servitization. Digital technologies are thus more than enabling technologies: digital technologies allow the creation of fundamentally new, disruptive business models. The digital transformation needs small, agile teams with end-to-end responsibility for project delivery, start-up mentality and customer obsession. Finally, key metrics of the digital transformation are digital efficiency/productivity, digital revenues and digital growth (net of cannibalization).

Malfunctions are a major challenge in partially automated driving. During such malfunctions, the driver must be able to adequately take over vehicle guidance without being requested to intervene. This may be particularly difficult in urban areas due to their high complexity and information density. Augmented Reality Head-Up Displays (ARHUDs) may have the potential to support the driver during the monitoring task by providing driving-related information at its required location in the primary field of view. The effects of an ARHUD compared to a Baseline concept in case of malfunctions were investigated in a driving simulation experiment with 52 participants. In a partially automated urban drive, participants experienced a longitudinal and a lateral malfunction in permuted order. The concepts -ARHUD or Baseline- were presented as a between-subject factor. The results showed significantly shorter take-over times when using the ARHUD, resulting in fewer crashes. For those who were able to avoid the crash, no differences in the take-over quality between both concepts were found. There was one difference in visual attention: the attention ratio on the instrument cluster was lower for the ARHUD. In addition, the ARHUD revealed a significantly higher trust and usability rating. However, there were no differences in acceptance and subjective workload between the two concepts. The results showed that the ARHUD has more potential to prevent crashes in the event of malfunctions compared to the Baseline. Nevertheless, the high number of crashes, regardless of the concept, showed the importance of a fallback level for partially automated urban driving.

During crisis situations, emergency operators require fast information access to achieve situation awareness and make the best possible decisions. Augmented reality could be used to visualize the wealth of user-generated content available on social media and enable context-adaptive functions for emergency operators. Although emergency operators agree that social media analytics will be important for their future work, it poses a challenge to filter and visualize large amounts of social media data. We conducted a goal-directed task analysis to identify the situation awareness requirements of emergency operators. By collecting tweets during two storms in Germany we evaluated the usefulness of Twitter data for achieving situation awareness and conducted interviews with emergency operators to derive filter strategies for social media data. We synthesized the results by discussing how the unique interface of augmented reality can be used to integrate social media data into emergency control rooms for situation awareness support.

© 2018 by the authors. Collaboration has been common in workplaces in various engineering settings and in our daily activities. However, how to effectively engage collaborators with collaborative tasks has long been an issue due to various situational and technical constraints. The research in this paper addresses the issue in a specific scenario, which is how to enable users to interact with public information from their own perspective. We describe a 3D mobile interaction technique that allows users to collaborate with other people by creating a symmetric and collaborative ambience. This in turn can increase their engagement with public displays. In order to better understand the benefits and limitations of this technique, we conducted a usability study with a total of 40 participants. The results indicate that the 3D mobile interaction technique promotes collaboration between users and also improves their engagement with the public displays.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

A variety of information of the real-time scenes is carried by the images and videos. Processing these images and videos in an intelligent way helps in many domains such as computer vision, object detection, deep learning and 3D reconstruction leaving its large usage in applications such as auto pilots, augmented reality, smart vehicles, etc. The quality of image and videos plays a vital role in case of real-time systems. One such scenario is where the images are captured without sufficient illumination. Images captured in cameras where sufficient light is not present, leads to noisy and information-loss images. It is a fact that, dark images have mainly two aspects which make its study a difficult task. They are at its low dynamic range and high propensity for generating high noise levels. Hence, an approach based on deep learning based system is adopted. For this purpose, Generative Adversarial Network (GAN) based Extremely Dark Video Enhancement Network (GEVE) model is proposed. The main objective of GEVE is to team the model with low /normal- light image pairs. Thus, GAN network learns the translation from light feeble images and images captured under normal illumination and automatically translate original images taken under extremely low light conditions into images of quality. It is clearly observed that the proposed GEVE outperforms the known state-of-art techniques. We are the view that the proposed system is an ideal candidate to handle dark image/video frames.

One of the main targets of criticism of head-mounted displays (HMDs) is the field of view (FOV) size, whether in virtual or augmented reality. This limitation is prominent with optical see-through head-mounted displays (OST-HMD), as those with narrow overlay FOV (OFOV) sizes only provide a small window to view virtual objects. We investigated if restricting this OFOV negatively affects a user’s ability to memorize spatial locations in a simulation of a work environment, and consequently, long-term memory transfer to an equivalent scenario in the real world two days later. To find empirical evidence, we conducted a within-subjects experiment with 18 participants performing in three phases with an OST-HMD, simulated on an immersive HMD. For each phase, they viewed the training scenario with a different OFOV size of the augmentable area (30°, 70°, 110° diagonal). Results from recall tests showed that smaller OFOV size did not significantly affect user’s performance on both short-term and transfer tests, but HMD data revealed that users rotated their heads less with a 110° OFOV. We also found that proximity of objects to memorize had an interaction effect with smaller OFOV sizes. Our findings could have implications on the design and HMD choices of augmented training.

Augmented reality (AR) applications that overlay a user's perception of the real world with digitally generated information are on the cusp of commercial viability. AR has appeared in several commercial platforms like Microsoft HoloLens and smartphones. They extend the user experience beyond two dimensions and supplement a user's normal 3D world. A typical location-based multi-player AR application works through a three-step process, wherein the system collects sensory data from the real world, identifies objects based on their context, and finally, renders information on top of the user's senses. However, because these AR applications frequently exchange data with users, they have exposed new individual and public safety issues. In this paper, we develop ARSpy, a user location tracking system solely based on network traffic information of the user, and we test it on location-based multi-player AR applications. We demonstrate the effectiveness of the proposed scheme via real-world experiments on 12 volunteers and show that we could obtain the geolocation of any target with high accuracy. We also propose three defense methods to mitigate these side channel attacks. Our results reveal a potential security threat in location-based multi-player AR applications and serve as a critical security reminder to AR users.

Augmented reality (AR) has huge potential as a science outreach tool, especially around palaeontology where it is possible to bring extinct animals to life. This paper shares our experiences as academic geoscientists in developing a series of AR applications during a three-year period. We do not focus on the technical issues of app design and coding but on the lessons learnt during this time which may be of value to others, whether academics, event coordinators or museum curators interested in using digital technology to enhance user experience or convey educational messages. We identify eight issues that are worth keeping in mind when commissioning, developing, and deploying AR apps for use as a science communication tool. The issues are: (1) download barriers; (2) ageing novelty of AR; (3) the ‘stickiness’ of an app tends to be moment-locations specific; (4) maintenance of bespoke apps; (5) ‘boom and lapse’ cycles in funding; (6) difficulty of embedding educational content; (7) priority conflict between users and event organisers; and (8) the quality of the animation is key.

Ultrasound offers a real-time 2-D view of structures within the human body. While many medical education programs have already dedicated a portion of their curriculum to ultrasound, others are concerned about cost, accessibility and limits to student practice. Student benefit may be affected by cognitive errors, which are in part owing to the mental heuristics required to visualize a 3-D structure by interpreting a 2-D image. A possible solution to eliminating subjectivity in ultrasound interpretation is the use of 3-D models to augment the traditional 2-D ultrasound experience. PubMed, Embase and Web of Science were searched for primary literature exploring relationships between 3-D modeling applications and their use in ultrasound education. The search and review process was guided by the Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) checklist. Overall, 14 of the included 16 studies indicated a significant improvement in medical education of ultrasound with the intervention of 3-D modeling applications. This systematic review confirms that 3-D modeling applications benefit student learning in ultrasound education while illuminating the need for more research in this field.

Refereed/Peer-reviewed This paper presents an Augmented Reality (AR)-based network cabling tutoring system that trains users how to interconnect cables within a Virtual Local Area Network (VLAN) on a physical switching rack. AR arrows combined with text-based instruction and a checklist provided assistance during practical learning. When learners made a mistake, an Intelligent Tutoring System (ITS) identified the source of the mistake and provided real-time feedback using text, AR and text-to-speech mechanisms. The design was motivated by the human-cognitive architecture and its five evolutionary principles (proposed by Sweller and Sweller (2006)). Users performed four consecutive network cabling training tasks with assistance from our ITS. We found that users made fewer errors when the AR cues, text-based instruction and checklist solutions were replaced with summarised information and then removed completely in the final task compared to those who used an identical system with fixed instruction. Cognitive Load Theory (CLT) explains our results by suggesting that the instructional mechanisms become redundant as knowledge increases. Implications of the study are discussed as well as how AR can help facilitate knowledge transfer in the network cabling domain.

As a hyper-natural interaction technique in 3D user interfaces, non-isomorphic rotation has been considered an effective approach for rotation tasks, where a static or dynamic control-display gain can be applied to amplify or attenuate a rotation. However, it is not clear whether non-isomorphic rotation can benefit 6-degree-of-freedom (6-DOF) manipulation tasks in AR and VR. In this article, we extended the usability studies of non-isomorphic rotation from rotation-only tasks to 6-DOF manipulation tasks and analyzed the collected data using a 2-component model. Using a mixed reality (MR) simulation approach, we also investigated whether environment (AR or VR) had an impact on 3D manipulation tasks. The results reveal that although both static and dynamic non-isomorphic rotation techniques could save time and effort in ballistic phases, only dynamic non-isomorphic rotation was significantly faster than isomorphic rotation. Interestingly, while environment had no significant impact on overall user performance, we found evidence that it could affect fine-tuning in correction phases. We also found that most participants preferred AR over VR, indicating that environmental visual realism could be helpful to improve user experience.

The furniture business has developed quite well every year, but the promotions used by the company still use the usual methods, such as using brochures that only display 2D information and images. With so much competition, consumers want something unique and different so they can find out more detailed and real product information offered. Augmented reality is the right technology to meet these consumer needs. Augmented reality is a technology that combines the virtual world with the real world directly. Therefore, an application was designed that functions as a company profile as well as displays 3D rattan chair products using Augmented reality technology on android devices. The method used in this research is the Multimedia Development Life Cycle (MDLC) which includes Concept, Design, Material Collecting, Assembly, Testing, and Distribution. This Augmented reality application requires streaming video taken from a smartphone camera as an input source, then this application will track and detect markers using a tracking system, after the marker is detected, the 3D chair model in the catalog will appear above the marker as if it were a chair model. it's real. the results of the hypothesis test with a significant level of 0.139, which is greater than 0.05, then what is accepted is H1. It can be stated that an augmented reality-based electronic catalog application can increase buying interest in potential consumers of Raffi cushion rotan

The development of Smart Home Controllers has seen rapid growth in recent years, especially for smart devices, that can utilize the Internet of Things (IoT). However, a large portion of the household devices and appliances already in use, are not IoT enabled, and therefore, requires their default control mechanisms for the devices to operate. This paper proposes a smart appliance controller that uses Augmented Reality, MQTT, and other up-to-date platforms to control aftermarket home appliances in the most efficient manner. The proposed work integrates mobile AR with IoT, to control household appliances, with the help of infrared (IR) signals. The characteristics of the system are evaluated through a series of tests and performance measures. The results of the test highlight th quick response time of MQTT for the implementation of a Home Automation System, when compared to the request-reply protocol: CoAP (4 times as fast).

Augmented reality is a device in combining objects between 2D and 3D virtual worlds, there's information that has not been implemented directly to the user, then by responding into scope that is 3D on the object with the resulting time at the same time to read in real-time, related to learning media where the teaching and learning process generally uses a book, then develops using projectors and laptops as information technology, besides the development of mobile (cellular) or android-based information technology that can make it easier for learning, especially the introduction of the solar system, the beginning of the solar system introduced various space objects both planets and surrounding space objects, in class VI who received the material as a basis before going to the next level, but the teaching and learning process there are several factors that influence i if it is examined that there are elderly teachers who have difficulty keeping up with technological developments, from learning media support tools such as projectors and teaching aids are still very limited, and not all teachers use information technology as a learning medium, the purpose of which is to make an introduction to AR learning media applications android-based solar system for class VI, based on these 3 problems as a solution an educational application is needed that supports learning activities using android augmented reality for the introduction of the solar system with the marker based tracking method, becomes a marker of 3D objects to bring up from the virtual world to the real which contains information on each of these objects, in other words visualizing the results of the marker directly into the real world, the final result is the creation of an AR application for learning the introduction of the solar system.

Visual odometry (VO) is a fundamental building block for many robotics and augmented reality (AR) applications. However, most existing RGB-D VO systems could suffer from large performance degradation when large occlusions are present or a large portion of depth values are invalid due to the limited range of an RGB-D camera, prohibiting the usage of most systems in practical applications. In this paper, we present RGB-D DSO, a RGB-D direct sparse odometry which can achieve state-of-theart accuracy, robustness and efficiency in the indoor scenes. Our RGB-D DSO is a significant extension of monocular DSO [1], with the core part being sliding-window optimization with occlusion removal and a depth refinement module. Occlusion removal excludes negative effects arising from occluded objects when minimizing the final energy function for camera pose tracking. Depth refinement ensures sufficient valid depth values uniformly distributed for the depth map of a keyframe. Experimental results on three public datasets [2], [3], [4] demonstrate that our method achieves smaller absolute trajectory errors (ATE) and relative pose error (RPE) than the state-of-the-art methods DVO [5], ORB-SLAM2 [6], CVO[7] and ACVO[8]. Meanwhile, our system takes only 21.93ms to track a frame, which is 3.7X faster than DVO (80.27ms/frame) and 1.4X faster than ORB-SLAM2 (29.91ms/frame).

The programming of manipulators is a common task in robotics, for which numerous solutions exist. In this work, a new programming method related to the common master-slave approach is introduced, in which the master is replaced by a digital twin created through haptic and visual rendering. To achieve this, we present an algorithm that enables the haptic rendering of any programmed robot with a serial manipulator on a general-purpose haptic interface. The results show that the proposed haptic rendering reproduces the kinematic properties of the programmed robot and directly provides the desired joint space trajectories. In addition to a stand-alone usage, we demonstrate that the proposed algorithm can be easily paired with existing visual technology for virtual and augmented reality to facilitate a highly immersive programming experience.

This study aims at developing and evaluating an affordable and user-friendly motion capture system for human upper limbs' joint kinematics estimation. The objective is to provide quantitative assessments during the clinical evaluation of poststroke patients performing daily living activities. The proposed system is based on the simultaneous use of affordable inertial measurement units, and a set of augmented reality markers tracked with an affordable RGB camera. Two practical calibration processes were developed to calibrate the sensors modules and then to determine their location on body segments. Then, all measured quantities were fused into a constrained extended Kalman filter based on an upper limbs' biomechanical model. The proposed system was validated with nine healthy volunteers performing five daily living activities. Joint angles estimated using the proposed affordable system were compared with a gold standard stereophotogrammetric system. The results showed a low average rms difference (2.7°) along with a high average correlation (0.87).

International audience; Situated visualization is an emerging concept within visualization, in which data is visualized in situ, where it is relevant to people. The concept has gained interest from multiple research communities, including visualization, human-computer interaction (HCI) and augmented reality. This has led to a range of explorations and applications of the concept, however, this early work has focused on the operational aspect of situatedness leading to inconsistent adoption of the concept and terminology. First, we contribute a literature survey in which we analyze 44 papers that explicitly use the term "situated visualization" to provide an overview of the research area, how it defines situated visualization, common application areas and technology used, as well as type of data and type of visualizations. Our survey shows that research on situated visualization has focused on technology-centric approaches that foreground a spatial understanding of situatedness. Secondly, we contribute five perspectives on situatedness (space, time, place, activity, and community) that together expand on the prevalent notion of situatedness in the corpus. We draw from six case studies and prior theoretical developments in HCI. Each perspective develops a generative way of looking at and working with situatedness in design and research. We outline future directions, including considering technology, material and aesthetics, leveraging the perspectives for design, and methods for stronger engagement with target audiences. We conclude with opportunities to consolidate situated visualization research.

Camera-based multi-class (1D and 2D) barcode detectors that can help in decoding barcodes in different complex scenes have huge potential applications in situations where Internet of Things (IoT) is combined with Artificial Intelligence (AI) and Augmented Reality (AR). The decoding rate in such applications under real-life complex scenes is greatly affected by two major factors: First, we cannot accurately localize the barcodes, and secondly, we cannot decode the blur samples. In this paper, we firstly propose a barcode localization algorithm that is capable of regressing 4 vertices of barcodes accurately. Our localization method comprises an anchor-free approach that outputs multi-scale output prediction maps. These segmentation-like maps of each scale are then further divided into three types of maps (classification, centerness, and 8-D regression). 8-D localization result of barcodes along-with classification result is then obtained after post-processing. Secondly, we propose a conditional generative adversarial network-based model for deblurring blur QR codes. Extensive decoding experiments on a challenging complex scene dataset show that our localization and deblurring methods can contribute to improving the decoding rate of existing barcode decoders.

Purpose The purpose of this paper is to introduce a digitalization model (DM) for building construction courses in architectural education as a response to the recent emerging technologies in the era of digital transformations. This DM is developed and applied through augmented reality (AR) technologies to boost perception, understandings and ability to solve building construction details. Design/methodology/approach Based on a thorough review of recent technologies like AR, virtual reality (VR), building information modelling (BIM) and their applications in architectural education, the methodology involves the generation of a model, its application and evaluation. The model is based on the integration of BIM and AR which is applied into a third year “Building Construction Project” course. Each student has designed a residential building and accordingly prepared the construction drawings by adapting the DM. An online survey –based on Technology Acceptance Model – is conducted to evaluate the DM by quantitative data analysis from SPSS and Excel. Findings The key findings of the study include the following items: determination of the proper digital tools, the definition of the steps and workflow based on the building project phases to develop construction drawings and define precise details effectively. By the help of this, the DM is generated and applied. According to the survey and results, the DM which involves BIM-based AR is considered as beneficial, highly motivating and providing better perception on construction details. Originality/value Implication of AR/VR technologies is frequently seen in design studios, whereas building construction courses state its traditional approach. However, there is a huge potential in the digitalization of building construction education by increasing the perception of students together with the increased level of communication. The study aims to close the gap of digitalization by proposing a DM, which brings a systematic approach considering each phase of building construction project as conceptual, schematic, design development and construction documents by using BIM integrated AR. Moreover, the novel model specifically brings a new approach by generation of QR codes for construction details to embed videos or simulations into the two-dimensional drawing sheets. Furthermore, the DM proposes a new approach to satisfy emerging needs and requirements of the Architecture Engineering Construction industry.

Mutual occlusion is an essential feature for augmented reality (AR) displays for allowing the virtual content to be clearly perceived under an excessively illuminated environment. Although a few works have been done to facilitate the performance of occlusion-capable optical see-through augmented reality (OC-OST-AR) displays, the realization of mutual occlusion in a wide field-of-view (FOV) is still challenging. Divergent from typical hard-edge occlusion and soft edge-occlusion designs, we propose the paired-ellipsoidal-mirror (PEM) structure. The proposed system is allowed to support either hard-edge occlusion or enhanced soft-edge occlusion in a wide FOV by optionally fixing a spatial light modulator (SLM) before the entrance pupil or at an inner focal plane. The numerical aperture (NA) of the system is efficiently increased by the combination of paired ellipsoidal mirror imaging and aperture stop restriction. With proof-of-concept prototypes built, virtual display in a FOV of H160°×V74° and mutual occlusion in a FOV of H122°×V74° are demonstrated with a basic design, respectively. Furthermore, a mixed FOV of H95.3°×V52.9° is demonstrated by an optimized design with vertical parallax reduction and virtual display improvement.

In augmented reality, it is important to achieve visual consistency between inserted virtual objects and the real scene. As specular and transparent objects can produce caustics, which affect the appearance of inserted virtual objects, we herein propose a framework for differential rendering beyond the Lambertian-world assumption. Our key idea is to jointly optimize illumination and parameters of specular and transparent objects. To estimate the parameters of transparent objects efficiently, the psychophysical scaling method is introduced while considering visual characteristics of the human eye to obtain the step size for estimating the refractive index. We verify our technique on multiple real scenes, and the experimental results show that the fusion effects are visually consistent.

Emerging automotive applications such as in-vehicle Augmented Reality (AR) and fully automated parking require a comprehensive understanding of the vehicle's three-dimensional surrounding represented as an object-level environmental model. In this model, not only 3D poses (positions and orientations) and 3D sizes of detected objects are registered, but 3D shapes (geometries) need to be reconstructed precisely. A combination of 3D object detection and 3D surface reconstruction techniques, referred to as object-level 3D reconstruction, is fundamental to building such environmental models. However, the possibilities to incorporate object-level 3D reconstruction in a car have not been sufficiently explored either in academic research or in the industry. This primarily stems from the cost and resource constraints associated with the automotive domain. In this paper, we address these constraints by proposing implementations of in-vehicle object-level 3D reconstruction in two specific use cases: (i) augmented reality and (ii) automated parking. For augmented reality, we propose a cost-efficient solution called monocular 3D Shaping that requires only a single frame from a monocular camera as input. For automated parking, we propose a resource-efficient alternative that generates more precise 3D reconstruction results by taking advantage of additional 3D sensors (such as Lidars). The crux of our proposed approaches lies in the use of a Latent Shape Space, where various 3D shapes are represented using only two parameters. As a result, highly complex 3D shapes can now be transmitted using a low- to medium-bandwidth in-vehicle communication infrastructure in a cost-effective manner.

360° video is becoming an increasingly popular technology on commercial social platforms and vital part of emerging Virtual Reality/Augmented Reality (VR/AR) applications. However, the delivery of 360° video content in mobile networks is challenging because of its size. The encoding of 360° video into multiple quality layers and tiles and edge cache-assisted video delivery have been proposed as a remedy to the excess bandwidth requirements of 360° video delivery systems. Existing works using the above tools have shown promising performance for Video-on-Demand (VoD) 360° delivery, but they cannot be straightforwardly extended in a live-streaming setup. Motivated by the above, we study edge cache-assisted 360° live video streaming to increase the overall quality of the delivered 360° videos to users and reduce the service cost. We employ Long Short-Term Memory (LSTM) networks to forecast the evolution of the content requests and prefetch content to caches. To further enhance the delivered video quality, users located in the overlap of the coverage areas of multiple Small Base Stations (SBSs) are allowed to receive data from any of these SBSs. We evaluate and compare the performance of our algorithm with Least Frequently Used (LFU), Least Recently Used (LRU), and First In First Out (FIFO) algorithms. The results show the superiority of the proposed approach in terms of delivered video quality, cache-hit-ratio and backhaul link usage.

Registering the 2D images (2D space) with the 3D model of the environment (3D space) provides a promising solution to outdoor Augmented Reality (AR) virtual-real registration. In this work, we use the position and orientation of the ground camera image to synthesize a corresponding rendered image from the outdoor large-scale 3D image-based point cloud. To achieve the virtual-real registration, we indirectly establish the spatial relationship between 2D and 3D space by matching the above two kinds (2D/3D space) of cross-domain images. However, matching cross-domain images goes beyond the capability of handcrafted descriptors and existing deep neural networks . To address this issue, we propose an end-to-end network, Y-Net, to learn Domain Robust Feature Representations (DRFRs) for the cross-domain images. Besides, we introduce a cross-domain-constrained loss function that balances the loss in image content and cross-domain consistency of the feature representations. Experimental results show that the DRFRs simultaneously preserve the representation of image content and suppress the influence of independent domains. Furthermore, Y-Net outperforms the existing algorithms on extracting feature representations and achieves state-of-the-art performance in cross-domain image retrieval . Finally, we validate the Y-Net-based registration approach on campus to demonstrate its possible applicability.

The amount of arable land is limited, yet the demand for agricultural food products is increasing. This issue has led to the notion of precision farming, where smart city-based technologies (e.g., Internet of Things, digital twins, artificial intelligence) are employed in combination to cater for increased production with fewer resources. Widely used in manufacturing, augmented reality has demonstrated impactful solutions for information communication, remote monitoring and increased interaction. Yet, the technology has only recently begun to find a footing alongside precision farming solutions, despite the many benefits possible to farmers through augmenting the physical world with digital objects. Therefore, this article reflects on literature discussing current applied solutions within agriculture, where augmented realty has demonstrated a significant impact for monitoring and production. The findings discuss that augmented reality must be coupled with other technologies (e.g., simultaneous localization and mapping algorithms, global positioning systems, and sensors), specifically 9 are identified across 2 application domains (livestock and crop farming) to be beneficial. Attention is also provided on how augmented reality should be employed within agriculture, where related-work examples are drawn from in order to discuss suitable hardware approaches and constraints (e.g., mobility).

Cultural-heritage research has created a vast amount of information regarding heritage objects and sites. At the same time, recent efforts on the digitization of cultural heritage have provided novel means of documenting tangible cultural resources including digital images, videos, audio testimonies, and 3D reconstructions. Furthermore, ethnographic research is nowadays combined with advanced capturing technologies such as motion capture to record the intangible dimensions of heritage as these are manifested through human expression in dance, heritage crafts, etc. This amount of information is now available and should be used to create novel forms of experiential access to cultural heritage powered by the web and mobile technologies mixed with novel interaction paradigms such as virtual and augmented reality. In this article, a platform is presented that facilitates a cloud-based web application to support location-based narratives on cultural-heritage resources provided through map-based or story-based representation approaches. At the same time, the platform through the power of modern mobile devices can provide these experiences on the move using location-based and image recognition-based augmented reality to facilitate multiple usage contexts. The platform was implemented to support the goal of the project CuRe, in the context of the “Greece-Germany” bilateral collaboration action.

In the aviation industry, cable bracket is one of the most common parts. The traditional assembly state inspection method of cable bracket is to manually compare by viewing 3D models. The purpose of this paper is to address the problem of inefficiency of traditional inspection method. In order to solve the problem that machine learning algorithm requires large dataset and manually labeling of dataset is a laborious and time-consuming task, a simulation platform is developed to automatically generate synthetic realistic brackets images with pixel-level annotations based on 3D digital mock-up. In order to obtain accurate shapes of brackets from 2D image, a brackets recognizer based on Mask R-CNN is trained. In addition, a semi-automatic cable bracket inspection method is proposed. With this method, the inspector can easily obtain the inspection result only by taking a picture with a portable device, such as augmented reality (AR) glasses. The inspection task will be automatically executed via bracket recognition and matching. The experimental result shows that the proposed method for automatically labeling dataset is valid and the proposed cable bracket inspection method can effectively inspect cable bracket in the aircraft. Finally, a prototype system based on client-server framework has been developed for validation purpose.

Background Although many cervical cytology diagnostic support systems have been developed, it is challenging to classify overlapping cell clusters with a variety of patterns in the same way that humans do. In this study, we developed a fast and accurate system for the detection and classification of atypical cell clusters by using a two‐step algorithm based on two different deep learning algorithms. Methods We created 919 cell images from liquid‐based cervical cytological samples collected at Sapporo Medical University and annotated them based on the Bethesda system as a dataset for machine learning. Most of the images captured overlapping and crowded cells, and images were oversampled by digital processing. The detection system consists of two steps: (1) detection of atypical cells using You Only Look Once v4 (YOLOv4) and (2) classification of the detected cells using ResNeSt. A label smoothing algorithm was used for the dataset in the second classification step. This method annotates multiple correct classes from a single cell image with a smooth probability distribution. Results The first step, cell detection by YOLOv4, was able to detect all atypical cells above ASC‐US without any observed false negatives. The detected cell images were then analyzed in the second step, cell classification by the ResNeSt algorithm, which exhibited average accuracy and F‐measure values of 90.5% and 70.5%, respectively. The oversampling of the training image and label smoothing algorithm contributed to the improvement of the system's accuracy. Conclusion This system combines two deep learning algorithms to enable accurate detection and classification of cell clusters based on the Bethesda system, which has been difficult to achieve in the past. We will conduct further research and development of this system as a platform for augmented reality microscopes for cytological diagnosis., Through two different types of CNNs, we were able to exclusively detect and determine atypical cell clusters with high sensitivity. The deep learning approach was also extremely useful for cervical Pap Smears with diverse images.

With the progress of human society, more and more scientific and technological achievements are applied to people’s development activities, and virtual reality technology is one of them. This case is a study on the path, form, and role of virtual reality and augmented reality technology in public art design. Through combining the public art form and current situation, we put forward the problem of increasing public art creation methods through the intervention of virtual reality technology, change the display method of public art with innovative technical solutions, and explore innovative ways of public art design application. In the research process, the author carried out detailed case research, technical demonstration, and creative practice on virtual reality technology; combined and studied a lot of advanced technical means that can be applied in public art creation and display design; and explored the collaborative innovation of technology and art, the public art design methods, and application approaches in China. On the basis of a comprehensive understanding of the artistic effects that can be achieved by related software and the future directions studied by technical personnel, analyzing the demand and feasibility of the technology in the application of public art, and exploring and summarizing virtual reality technology and augmented reality technology under the background of rapid development and popularization, public art design uses these technologies to solve practical problems and promotes the process of art design innovation and publicization by means of innovative media and experience forms. It increases the importance of virtual technology in public art and changes the traditional business model of public art. Experimental results show that the public’s attention to public equipment based on sensors and virtual reality technology has increased by more than 60%, and modern technology is used to make up for the development of public art and culture. The digital sustainable development of interactive installations of public art in my country provides new ideas.

Purpose Augmented reality (AR) integrates the digital world with the real world and thus, provides a real-time experience to the users. With AR, the immediate surroundings become a learning platform for the users. The perception of the products has been enhanced many times with AR; thus, enriching user experience and responsiveness. The purpose of this paper is to bring forth the basics of AR and provide an overview of the research work carried out by researchers in the implementation of AR in different sectors. Design/methodology/approach This paper summarizes the usefulness of AR in different industries. The authors have identified the peer-reviewed research publications from Web of Science, Scopus, Google Scholar, etc. The selection of literature has been made based upon the significance of AR in recent times. The industries/sectors where AR has been implemented successfully have been considered for this paper. The paper has been divided into various sections and subsections to bring more clarity to the readers. Findings This paper presents a brief and a precise information on Industry 4.0 and AR. The basic working of AR system and its implications have also been discussed. The preference of AR over virtual reality (VR) has also been deliberated in this paper. The authors have presented the usefulness of AR in different sectors such as smart factories, ship yard building, online shopping, surgery and education. This paper discusses the AR-ready procedures being followed in these sectors. Originality/value AR has been an add-on to VR systems. The processes in industries have become very handy and informative with AR. Because the application of AR in different sectors has not been discussed in a single paper; thus, this work presents a systematic literature review on the applications of AR in different sectors/industries.

PURPOSE The purpose of this study was to develop and preliminarily test a radiotherapy system for patient posture correction and alignment using mixed reality (MixR) visualization. The write-up of this work also provides an opportunity to introduce the concepts and technology of MixR for a medical physics audience who may be unfamiliar with the topic. METHODS A MixR application was developed for on optical-see-through head-mounted display (HoloLens 2) allowing a user to simultaneously and directly view a patient and a reference hologram derived from their simulation CT scan. The hologram provides a visual reference for the exact posture needed during treatment and is initialized in relation to the origin of a radiotherapy device using marker-based tracking. The system further provides marker-less tracking that allows the user tofreely navigate the room as they view and align the patient from various angles. The system was preliminarily tested using both a rigid (pelvis) and nonrigid (female mannequin) anthropomorphic phantom. Each phantom was aligned via hologram and accuracy quantified using CBCT and CT. RESULTS A fully realized system was developed. Rigid registration accuracy was on the order of 3.0 ± 1.5 mm based on the performance of three users repeating alignment five times each. The lateral direction showed the most variability among users and was associated with the largest off-sets (approximately 2.0 mm). For nonrigid alignment, the MixR setup outperformed a setup based on three-point alignment and setup photos, the latter of which showed a difference in arm position of 2 cm and a torso roll of 6-7°. CONCLUSIONS MixR visualization is a rapidly emerging domain that has the potential to significantly impact the field of medicine. The current application is an illustration of this and highlights the advantages of MixR for patient setup in radiation oncology. The key feature of the system is the way in which it transforms nonrigid registration into rigid registration by providing an efficient, portable, and cost-effective mechanism for reproducing patient posture without the use of ionizing radiation. Preliminary estimates of registration accuracy indicate clinical viability and form the foundation for further development and clinical testing.

The application of augmented reality technology to the partial hepatectomy procedure has high practical significance, but the existing augmented reality navigation system has major drawbacks in the display and registration methods, which result in low precision. The augmented reality surgical navigation system proposed in this study has been improved in the above two aspects, which can significantly improve the surgical accuracy. The use of optical see-through head-mounted displays for imaging displays can prevent doctors from reconstructing the patient's two-dimensional image information in their minds and reduce the psychological burden of doctors. In the registration process, the biomechanical properties of the liver are introduced, and a non-rigid registration method based on biomechanics is proposed and realized by a meshless algorithm. In addition, this study uses the moving grid algorithm to carry out clinical experiments on ex-vivo pig liver for experimental verification. The mark-based interactive registration error is 4.21 ± 1.6 mm, and the registration error is reduced after taking the biomechanical properties of the liver into account, which is − 0.153 ± 0.398 mm. The cutting error of the liver model is 0.159 ± 0.292 mm. In addition, with the aid of the navigation system proposed in this paper, the experiment of ex-vivo pig liver cutting was completed with an error of − 1.164 ± 0.576 mm. As a proof-of-concept study, the augmented reality navigation system proposed in this study improves the traditional image-guided surgery in terms of display and registration methods, and the feasibility of the system is verified by ex-vivo pig liver experiments. Therefore, the navigation system has a certain guiding significance for clinical surgery.