Search

Augmented reality applications use object tracking to estimate the pose of a camera and to superimpose virtual content onto the observed object. Today, a number of tracking systems are available, ready to be used in industrial applications. However, such systems are hard to handle for a service maintenance engineer, due to obscure configuration procedures. In this article, we investigate options towards replacing the manual configuration process with a machine learning approach based on automatically synthesized data. We present an automated process of creating object tracker facilities exclusively from synthetic data. The data is highly enhanced to train a convolutional neural network, while still being able to receive reliable and robust results during real world applications only from simple RGB cameras. Comparison against related work using the LINEMOD dataset showed that we are able to outperform similar approaches. For our intended industrial applications with high accuracy demands, its performance is still lower than common object tracking methods with manual configuration. Yet, it can greatly support those as an add-on during initialization, due to its higher reliability.

Current robotic endoluminal interventions require surgeons to hold a proximal joystick to control the distal flexible robot under 2-D X-ray guidance. However, the 2-D X-ray image is not intuitive, which not only increases the risk of surgical misoperation but also the workload of surgeons. Moreover, contact teleoperation exposes surgeons to the potentially infectious environment. To address it, this article proposes an augmented-reality-assisted touchless teleoperated robot for endoluminal intervention, called ARei. It aims to provide immersive experiences with augmented information. The robot integrates perceptual information obtained from an electromagnetic (EM) sensor, a shape sensor, and the virtual anatomy into a head-mounted display (HMD). The touchless teleoperation is used to control the robot, with the assistance of gesture recognition technology. Results show that the mean error of the calibration between the HMD and the EM tracking system is 4.67 mm, and the mean distance error between the points measured by the EM sensor and the points obtained by shape reconstruction with calibration is 5.19 mm (3.05%).

Head-mounted loupes can increase the user's visual acuity to observe the details of an object. On the other hand, optical see-through head-mounted displays (OST-HMD) are able to provide virtual augmentations registered with real objects. In this article, we propose AR-Loupe, combining the advantages of loupes and OST-HMDs, to offer augmented reality in the user's magnified field-of-vision. Specifically, AR-Loupe integrates a commercial OST-HMD, Magic Leap One, and binocular Galilean magnifying loupes, with customized 3D-printed attachments. We model the combination of user's eye, screen of OST-HMD, and the optical loupe as a pinhole camera. The calibration of AR-Loupe involves interactive view segmentation and an adapted version of stereo single point active alignment method (Stereo-SPAAM). We conducted a two-phase multi-user study to evaluate AR-Loupe. The users were able to achieve sub-millimeter accuracy ( 0.82 mm) on average, which is significantly ( ) smaller compared to normal AR guidance ( 1.49 mm). The mean calibration time was 268.46 s. With the increased size of real objects through optical magnification and the registered augmentation, AR-Loupe can aid users in high-precision tasks with better visual acuity and higher accuracy.

Simulating shadow interactions between real and virtual objects is important for augmented reality (AR), in which accurately and efficiently detecting real shadows from live videos is a crucial step. Most of the existing methods are capable of processing only scenes captured under a fixed viewpoint. In contrast, this paper proposes a new framework for shadow detection in live outdoor videos captured under moving viewpoints. The framework splits each frame into a tracked region, which is the region tracked from the previous video frame through optical flow analysis, and an emerging region, which is newly introduced into the scene due to the moving viewpoint. The framework subsequently extracts features based on the intensity profiles surrounding the boundaries of candidate shadow regions. These features are then utilized to both correct erroneous shadow boundaries for the tracked region and to detect shadow boundaries for the emerging region by a Bayesian learning module. To remove spurious shadows, spatial layout constraints are further considered for emerging regions. The experimental results demonstrate that the proposed framework outperforms the state-of-the-art shadow tracking and detection algorithms on a variety of challenging cases in real time, including shadows on backgrounds with complex textures, nonplanar shadows, fast-moving shadows with changing typologies, and shadows cast by nonrigid objects. The quantitative experiments show that our method outperforms the best existing method, achieving a 33.3% increase in the average F_{measure}. Coupled with an image-based shadow-casting method, the proposed framework generates realistic shadow interaction results. This capability will be particularly beneficial for supporting AR.

Egocentric augmented reality (AR) interfaces are quickly becoming a key asset for assisting high precision activities in the peripersonal space in several application fields. In these applications, accurate and robust registration of computer-generated information to the real scene is hard to achieve with traditional Optical See-Through (OST) displays given that it relies on the accurate calibration of the combined eye-display projection model. The calibration is required to efficiently estimate the projection parameters of the pinhole model that encapsulate the optical features of the display and whose values vary according to the position of the user's eye. In this article, we describe an approach that prevents any parallax-related AR misregistration at a pre-defined working distance in OST displays with infinity focus; our strategy relies on the use of a magnifier placed in front of the OST display, and features a proper parameterization of the virtual rendering camera achieved through a dedicated calibration procedure that accounts for the contribution of the magnifier. We model the registration error due to the viewpoint parallax outside the ideal working distance. Finally, we validate our strategy on a OST display, and we show that sub-millimetric registration accuracy can be achieved for working distances of ±100 mm around the focal length of the magnifier.

The application of augmented reality technology to the partial hepatectomy procedure has high practical significance, but the existing augmented reality navigation system has major drawbacks in the display and registration methods, which result in low precision. The augmented reality surgical navigation system proposed in this study has been improved in the above two aspects, which can significantly improve the surgical accuracy. The use of optical see-through head-mounted displays for imaging displays can prevent doctors from reconstructing the patient's two-dimensional image information in their minds and reduce the psychological burden of doctors. In the registration process, the biomechanical properties of the liver are introduced, and a non-rigid registration method based on biomechanics is proposed and realized by a meshless algorithm. In addition, this study uses the moving grid algorithm to carry out clinical experiments on ex-vivo pig liver for experimental verification. The mark-based interactive registration error is 4.21 ± 1.6 mm, and the registration error is reduced after taking the biomechanical properties of the liver into account, which is − 0.153 ± 0.398 mm. The cutting error of the liver model is 0.159 ± 0.292 mm. In addition, with the aid of the navigation system proposed in this paper, the experiment of ex-vivo pig liver cutting was completed with an error of − 1.164 ± 0.576 mm. As a proof-of-concept study, the augmented reality navigation system proposed in this study improves the traditional image-guided surgery in terms of display and registration methods, and the feasibility of the system is verified by ex-vivo pig liver experiments. Therefore, the navigation system has a certain guiding significance for clinical surgery.

As augmented reality technologies develop, real-time interactions between objects present in the real world and virtual space are required. Generally, recognition and location estimation in augmented reality are carried out using tracking techniques, typically markers. However, using markers creates spatial constraints in simultaneous tracking of space and objects. Therefore, we propose a system that enables camera tracking in the real world and visualizes virtual visual information through the recognition and positioning of objects. We scanned the space using an RGB-D camera. A three-dimensional (3D) dense point cloud map is created using point clouds generated through video images. Among the generated point cloud information, objects are detected and retrieved based on the pre-learned data. Finally, using the predicted pose of the detected objects, other information may be augmented. Our system estimates object recognition and 3D pose based on simple camera information, enabling the viewing of virtual visual information based on object location.

BackgroundOral and maxillofacial surgery currently relies on virtual surgery planning based on image data (CT, MM). Three-dimensional (3D) visualizations are typically used to plan and predict the outcome of complex surgical procedures. To translate the virtual surgical plan to the operating room, it is either converted into physical 3D-printed guides or directly translated using real-time navigation systems.PurposeThis study aims to improve the translation of the virtual surgery plan to a surgical procedure, such as oncologic or trauma surgery, in terms of accuracy and speed. Here we report an augmented reality visualization technique for image-guided surgery. It describes how surgeons can visualize and interact with the virtual surgery plan and navigation data while in the operating room. The user friendliness and usability is objectified by a formal user study that compared our augmented reality assisted technique to the gold standard setup of a perioperative navigation system (Brainlab). Moreover, accuracy of typical navigation tasks as reaching landmarks and following trajectories is compared.ResultsOverall completion time of navigation tasks was 1.71 times faster using augmented reality (P = .034). Accuracy improved significantly using augmented reality (P < .001), for reaching physical landmarks a less strong correlation was found (P = .087). Although the participants were relatively unfamiliar with VR/AR (rated 2.25/5) and gesture-based interaction (rated 2/5), they reported that navigation tasks become easier to perform using augmented reality (difficulty Brainlab rated 3.25/5, HoloLens 2.4/5).ConclusionThe proposed workflow can be used in a wide range of image-guided surgery procedures as an addition to existing verified image guidance systems. Results of this user study imply that our technique enables typical navigation tasks to be performed faster and more accurately compared to the current gold standard. In addition, qualitative feedback on our augmented reality assisted technique was more positive compared to the standard setup. (C) 2021 The Author. Published by Elsevier Inc. on behalf of The American Association of Oral and Maxillofacial Surgeons.

BACKGROUND Augmented reality (AR) is an imaging technology encompassing an interactive experience of a real-world environment enhanced by computer-generated perceptual information. It has been introduced to current medical practice to help the preoperative planning in many surgical fields. METHODS The authors applied AR to the computed tomography angiography of 8 patient's legs. Computed tomography angiography images were processed into Digital Imaging and Communications in Medicine files to make a prefabricated cutting guide and customized titanium plate. Also, three-dimensional reconstruction of the arterial supply of the leg was performed to identify the perforators. RESULTS Followed by preoperative marking of operative details on patient's skins in antero-posterior view, lateral view, and combination of both views. Inaccuracy of measurement was confirmed by duplex ultrasound which average error of the combination of antero-posterior and lateral viewed of both legs was lowest (0.7 ± 0.2 cm). Followed by lateral view (1.0 ± 0.3 cm) and antero-posterior view (1.2 ± 0.4 cm), respectively. CONCLUSIONS Augmented reality can improve patient's safety by directly locate the perforator and easily to design the skin paddle. Followed by satisfaction and confidence in patients and their relatives. Augmented reality also promoted understanding of operative steps for related assistants, residents, or fellows. Augmented reality can perform with existing equipment, mobile phone application, and can save the cost for preoperative planning. Distortion in the depth view can be more accurate by combining of AR in antero-posterior and lateral view.

Image-guided therapies have been on the rise in recent years as they can achieve higher accuracy and are less invasive than traditional methods. By combining augmented reality technology with image-guided therapy, more organs, and tissues can be observed by surgeons to improve surgical accuracy. In this review, 233 publications (dated from 2015 to 2020) on the design and application of augmented reality-based systems for image-guided therapy, including both research prototypes and commercial products, were considered for review. Based on their functions and applications. Sixteen studies were selected. The engineering specifications and applications were analyzed and summarized for each study. Finally, future directions and existing challenges in the field were summarized and discussed.

In augmented reality, it is important to achieve visual consistency between inserted virtual objects and the real scene. As specular and transparent objects can produce caustics, which affect the appearance of inserted virtual objects, we herein propose a framework for differential rendering beyond the Lambertian-world assumption. Our key idea is to jointly optimize illumination and parameters of specular and transparent objects. To estimate the parameters of transparent objects efficiently, the psychophysical scaling method is introduced while considering visual characteristics of the human eye to obtain the step size for estimating the refractive index. We verify our technique on multiple real scenes, and the experimental results show that the fusion effects are visually consistent.

Proper occlusion based rendering is very important to achieve realism in all indoor and outdoor Augmented Reality (AR) applications. This paper addresses the problem of fast and accurate dynamic occlusion reasoning by real objects in the scene for large scale outdoor AR applications. Conceptually, proper occlusion reasoning requires an estimate of depth for every point in augmented scene which is technically hard to achieve for outdoor scenarios, especially in the presence of moving objects. We propose a method to detect and automatically infer the depth for real objects in the scene without explicit detailed scene modeling and depth sensing (e.g. without using sensors such as 3D-LiDAR). Specifically, we employ instance segmentation of color image data to detect real dynamic objects in the scene and use either a top-down terrain elevation model or deep learning based monocular depth estimation model to infer their metric distance from the camera for proper occlusion reasoning in real time. The realized solution is implemented in a low latency real-time framework for video-see-though AR and is directly extendable to optical-see-through AR. We minimize latency in depth reasoning and occlusion rendering by doing semantic object tracking and prediction in video frames.

Interaction is one of the important topics to be discussed since it includes the interface where the end-user communicates with the augmented reality (AR) system. In handheld AR interface, the traditional interaction techniques are not suitable for some AR applications due to the different attributes of handheld devices that always refer to smartphones and tablets. Currently interaction techniques in handheld AR are known as touch-based technique, mid-air gesture-based technique and device-based technique that can led to a wide discussion in related research areas. However, this paper will focus to discover the device-based interaction technique because it has proven in the previous studies to be more suitable and robust in several aspects. A novel device-based 3D object rotation technique is proposed to solve the current problem in performing 3DOF rotation of 3D object. The goal is to produce a precise and faster 3D object rotation. Therefore, the determination of the rotation amplitudes per second is required before the fully implementation. This paper discusses the implementation in depth and provides a guideline for those who works in related to device-based interaction.

Multimodal interaction has become a recent research focus since it offers better user experience in augmented reality (AR) systems. However, most existing works only combine two modalities at a time, e.g., gesture and speech. Multimodal interactive system integrating gaze cue has rarely been investigated. In this article, we propose a multimodal interactive system that integrates gaze, gesture, and speech in a flexibly configurable AR system. Our lightweight head-mounted device supports accurate gaze tracking, hand gesture recognition, and speech recognition simultaneously. The system can be easily configured into various modality combinations, which enables us to investigate the effects of different interaction techniques. We evaluate the efficiency of these modalities using two tasks: the lamp brightness adjustment task and the cube manipulation task. We also collect subjective feedback when using such systems. The experimental results demonstrate that the Gaze+Gesture+Speech modality is superior in terms of efficiency, and the Gesture+Speech modality is more preferred by users. Our system opens the pathway toward a multimodal interactive AR system that enables flexible configuration.

Pada saat ini diseluruh bagian dunia sedang terlanda sebuah virus yaitu COVID-19 yang membuat setiap orang tidak dapat berpergian ke luar negri dengan mudah. Namun di era sudah canggih ini semua informasi dapat disajikan secara digitalisasi mulai dalam bentuk tulisan maupun gambar dan dapat juga ditampilkan berbentuk objek 3 dimensi dibantu dengan Augmented Reality untuk menampilkannya. Peneleitian ini memanfaatkan teknologi Augmented Reality digunakan untuk alat pengenalan landmark dari negara Asia Tenggara agar tampilan objek akan menjadi lebih atraktif dan dapat mengetahui informasi tentang landmark dari setiap negara dengan mudah. Dalam penelitian ini aplikasi Augmented Reality dibuat menggunakan FAST Corner Detection (FCD) algoritma dan Natural Feature Tracking (NFT) untuk mendeteksi marker. Pengujian dari aplikasi ini dilakukan pada tiga perangkat smartphone android, pada sudut kemiringan 20° - 90° ketiga smartpone dapat mendeteksi marker menggunakan kameranya dan menampilkan objek 3 dimensi sesuai dengan landmark yang dipilih pada layar smartphone. Untuk pengukuran jarak maximal yang dapat dibaca oleh handphone rata-rata ±75cm dan jarak minimun pendeteksian yaitu ±10cm.

Advancements in imaging techniques are key forces of progress in neurosurgery. The importance of accurate visualization of intraoperative anatomy cannot be overemphasized and is commonly delivered through traditional neuronavigation. Augmented Reality (AR) technology has been tested and applied widely in various neurosurgical subspecialties in intraoperative, clinical use and shows promise for the future. This systematic review of the literature explores the ways in which AR technology has been successfully brought into the operating room (OR) and incorporated into clinical practice. A comprehensive literature search was performed in the following databases from inception-April 2020: Ovid MEDLINE, Ovid EMBASE, and The Cochrane Library. Studies retrieved were then screened for eligibility against predefined inclusion/exclusion criteria. A total of 54 articles were included in this systematic review. The studies were sub- grouped into brain and spine subspecialties and analyzed for their incorporation of AR in the neurosurgical clinical setting. AR technology has the potential to greatly enhance intraoperative visualization and guidance in neurosurgery beyond the traditional neuronavigation systems. However, there are several key challenges to scaling the use of this technology and bringing it into standard operative practice including accurate and efficient brain segmentation of magnetic resonance imaging (MRI) scans, accounting for brain shift, reducing coregistration errors, and improving the AR device hardware. There is also an exciting potential for future work combining AR with multimodal imaging techniques and artificial intelligence to further enhance its impact in neurosurgery.

Traditional neurosurgical craniotomy primarily uses two‐dimensional cranial medical images to estimate the location of a patient’s intracranial lesions. Such work relies on the experience and skills of the doctor and may result in accidental injury to important intracranial physiological tissues. To help doctors more intuitively determine patient lesion information and improve the accuracy of surgical route formulation and craniotomy safety, an augmented reality method for displaying neurosurgery craniotomy lesions based on feature contour matching is proposed. This method uses threshold segmentation and region growing algorithms to reconstruct a 3‐D Computed tomography image of the patient’s head. The augmented reality engine is used to adjust the reconstruction model’s relevant parameters to meet the doctor’s requirements and determine the augmented reality matching method for feature contour matching. By using the mobile terminal to align the real skull model, the virtual lesion model is displayed. Using the designed user interface, doctors can view the patient’s personal information and can zoom in, zoom out, and rotate the virtual model. Therefore, the patient’s lesions information can be visualized accurately, which provides a visual basis for preoperative preparation.

Aiming at the problems of large data volume, long calculation time, and information feedback speed in traditional virtual augmented reality-based scenes, this paper constructed a virtual augmented reality scene based on 5G technology. First, based on the research of camera models and related calibration techniques, this article discusses the coordinate conversion and registration between virtual scenes and real scenes in augmented reality systems based on natural feature points. Secondly, the control mechanism of data synchronization and transmission are realized by using 5G communication technology. Then, through the approximate nearest neighbour search of the descriptor, the feature points are quickly matched. Finally, by tracking the QR-code logo black frame, the global holography matrix virtual real registration method relative to the initial image is used to calculate the camera’s posture and the fast tracking registration of the scene is realized. The experimental results prove that the scene constructed in this paper has the characteristics of real-time, wide field of view, high immersion and flexible expansion, which not only meets the requirements of matching accuracy, but also has the advantages of small calculation amount and fast calculation speed, and has good robustness and real-time.

A smartphone augmented reality (AR) application (app) was explored for clinical use in presurgical planning and lesion scalp localization. We programmed an AR App on a smartphone. The accuracy of the AR app was tested on a 3D-printed head model, using the Euclidean distance of displacement of virtual objects. For clinical validation, 14 patients with brain tumors were included in the study. Preoperative MRI images were used to generate 3D models for AR contents. The 3D models were then transferred to the smartphone AR app. Tumor scalp localization was marked, and a surgical corridor was planned on the patient’s head by viewing AR images on the smartphone screen. Standard neuronavigation was applied to evaluate the accuracy of the smartphone. Max-margin distance (MMD) and area overlap ratio (AOR) were measured to quantitatively validate the clinical accuracy of the smartphone AR technique. In model validation, the total mean Euclidean distance of virtual object displacement using the smartphone AR app was 4.7 ± 2.3 mm. In clinical validation, the mean duration of AR app usage was 168.5 ± 73.9 s. The total mean MMD was 6.7 ± 3.7 mm, and total mean AOR was 79%. The smartphone AR app provides a new way of experience to observe intracranial anatomy in situ, and it makes surgical planning more intuitive and efficient. Localization accuracy is satisfactory with lesions larger than 15 mm.

Aiming at the problem of real-time augmented reality tracking registration of motion blurred template matching image construction model, this paper proposes a template matching target tracking algorithm based on improved efficient second-order minimization (ESM). Based on ESM algorithm, an improved efficient second-order minimization algorithm is used to track the motion template. Firstly, the image acquisition and noise reduction preprocessing are carried out, and the image acquisition model of high-speed moving target and the edge feature detection algorithm of moving feature points are designed. The gray-scale histogram denoising method is used to suppress the edge fuzzy set of dynamic feature points in the image of high-speed moving target. The feature extraction and matching are modeled, and the selection of feature points and the search algorithm are optimized. Then the block detection method is used to reconstruct the region of high-speed moving target, and the edge contour detection and feature point matching of the moving target are carried out in the affine invariant region. The image block template matching method is used to realize the real-time augmented reality tracking registration of high-speed moving target. The simulation results show that the proposed method has good effect and superior performance in real-time augmented reality tracking registration of moving targets.

Background To achieve an optimal clinical outcome in reverse total shoulder arthroplasty (RSA), accurate placement of the components is essential. The recently introduced navigation technology of augmented reality (AR) through head-mounted displays (HMD) offers a promising new approach to visualize the anatomy and navigate component positioning in various orthopedic surgeries. We hypothesized that AR through HMD is feasible, reliable, and accurate for guidewire placement in RSA baseplate positioning. Methods Twelve human cadaver shoulders were scanned with computed tomography (CT) and RSA baseplate positioning was 3-D planned using dedicated software. The shoulders were prepared through a deltopectoral approach and an augmented reality hologram was superimposed using the HMD Microsoft HoloLense. The central guidewire was then navigated through the HMD to achieve the planned entry point and trajectory. Postoperatively, the shoulders were CT-scanned a second time and the deviation from the planning was calculated. Results The mean deviation of the entry point was 3.5 mm ± 1.7 mm (95% CI 2.4 mm; 4.6 mm). The mean deviation of the planned trajectory was 3.8° ± 1.7° (95% CI 2.6°; 4.9°). Conclusion Augmented reality seems feasible and reliable for baseplate guidewire positioning in reverse total shoulder arthroplasty. The achieved values were accurate.

The present study reviews the augmented reality application in the education area. Justification of the use is to improve the learning motivation and outcome of students through the use of a deep learning approach. A deep learning approach is associated with better academic performances. Augmented reality provides a tool to motivate learning and understanding in the student. In the medical domain, augmented reality may serve as replacement of cadavers especially during online learning. The main aim of the current review is to reflect the use of augmented reality as a teaching and learning tool to motivate and promote the deep learning approach process which includes life-long learning and possible research areas that warrant and assess the impact of augmented reality on academic performance. Augmented reality may be a tool in complementing the learning aids of the conventional methods of teaching and learning.

Kemajuan teknologi dimasa sekarang sangatlah pesat, Salah satunya dalam bidang media pembelajaran atau edukasi.Teknologi Augmented Reality (AR) Merupakan sebuah penggabungan dari digital konten yang telah di buat di komputer dari dengan dunia nyata.Penelitian ini bertujuan untuk memberi pengenalan tentang tumbuhan sehingga dapat mengetahui informasi tentang tanaman tersebut. Penelitian ini menggunakan sebuah metode yaitu natural feature tracking serta menggunakan sebuah sistem pengembangan yaitu MDLC( Multimedia Development Life Cycle) dan memakai algoritma Fast Corner Detection dan Natural Feature Tracking, Sehingga menghasilkan sebuah aplikasi yang dapat menampilkan objek 3D tentang tumbuhan, dan terdapat penjelasan singkat mengenai tumbuhan, Aplikasi ini dapat menggunakan sistem operasi Android, Pengujian aplikasi menggunakan 3 handphone android yaitu Samsung A2018 dengan versi andorid 9,Oppo A9 2019 dan Samsung A51 2020 bahwa pada saat kemiringan

Augmented Reality (AR) is an emerging technology capable of integrating visual technology with the physical world in novel ways. AR uses may range from education delivery to the operating room. This study sought to evaluate cost-effective AR glasses compared to a traditional computer screen when identifying anatomical structures in a chest radiograph (X-ray). The purpose of this experiment was to determine if the Vuzix Blade AR technology is feasible for use in the medical field to identify anatomical structures. Novice undergraduate and graduate students (n = 14) were recruited to participate in this feasibility study. Radio-graphic images were compiled, and subjects were asked to identify 12 anatomical structures on each image. Images were randomly assigned to identify structures using either the AR glasses or a traditional computer screen for each set. Images were viewed randomly by each subject using a crossover design to compare traditional computer screen vs. AR screen. Subjects identified statistically fewer anatomical structures using the AR glasses compared to using the traditional computer screen (p

Augmented Reality (AR) applications are one of the current mobile app trends. Its ability to display information as if to make the user merge with the information provided. The specialty of the application is in the marker. One marker that is in great demand is known as markerless, such as 3D Object Tracking. In making this 3D Object-based AR application, one must pass a stage where a real object is taken at points or known as points to be used as markers. The application used at this stage of making markers is known as the vuforia object scanner. From the experience of making AR applications, it was found conditions where markers with a large number of points were not able to display information properly, a small number of points were able to display information well and sometimes the two conditions were opposite or comparable, i.e. a lot of points makes a good marker and a few points make a bad marker. This inconsistent condition becomes a problem in this study. The test results show that the points do not affect the marker's ability to display information, but are influenced by the characteristics of the object used as the marker. Tests conducted on randomly selected objects resulted in 2 parameters to be tested in this study, namely the texture and pattern of the object. Markers with good quality to be used as markers for 3D objects are markers with a hard texture, with a detection angle of 90° and 135° and with a regular pattern

Clinical use of augmented reality, mixed reality, three-dimensional-navigation and artificial intelligence in liver surgery

In response to extreme traffic congestion in metropolitan areas that causes unnecessarily long travel times, high fuel consumption, and excessive greenhouse gas emissions, transportation agencies have implemented various strategies to mitigate traffic congestion. Managed lanes—one of the measures applied worldwide—provide benefits to road users and operating agencies by integrating advanced technologies such as electronic and dynamic tolling systems. However, those agencies already implementing or considering implementing the managed lane strategy are seeking a solution to effectively and properly charge toll rates based on vehicle occupancy and penalize violating vehicles. Vehicle passenger detection systems (VPDSs) have been developed and evaluated worldwide, but limitations still inhibit their full implementation. This study confirms that the performance of the deep learning algorithm, a core VPDS technology, declines under certain adverse weather conditions because of lack of training data sets. The performance of the “you only look once” (YOLOv3) model trained with a normal weather data set decreased by as much as 8.5% when it was tested for adverse weather conditions. In this study, augmented reality (AR) models are developed to enhance the accuracy of vehicle passenger detection (VPDA) by the VPDS by training the algorithm with AR images representing virtual adverse weather conditions. Models trained with AR image sets of various weather categories (fog, rain, and snow) attained VPDA enhanced by up to 7.9%. The final model significantly improves VPDA under adverse weather conditions. The proposed models could be considered for implementation with road weather information systems under adverse weather conditions.

One key predictor of team performance and productivity is the teams’ ability to coordinate its subtasks as precisely as possible over time. Thereby, the quality of the temporal coordination in teams is highly dependent on several cognitive team skills, like for instance the ability to build a precise and stable mental model of the situation (shared situation awareness) and of the team task process (task state awareness/ TSA). To support these cognitive skills in teams, various methods and trainings already exist. However, considering teams that work de-located, the prerequisites and therefore the requirements for team supportive methods change. This work presents the results of an empirical usability and user experience (UX) study of an interface for an Augmented Reality-based assistance system for spatially dispersed teams, called Ambient Awareness Tool (AAT). The AAT interface consists of graphical information about the teamwork process and aims at enhancing the TSA of the team, thereby supporting its temporal coordination. Within the framework of a participatory design process we conducted a two-part expert-user-study, comprising a laboratory experiment for the evaluation of the interface usability and a follow-up online survey to additionally investigate the UX. By means of the usability scores we first inferred three interface configuration clusters. A subsequent UX evaluation then allowed us to designate the configuration cluster with the highest usability and UX scores. As a result, we defined one interface configuration that will be investigated concerning its actual impact on the temporal coordination of spatially dispersed teams in a further study.

International audience; Objective: A craniotomy is the removal of a part of the skull to allow surgeons to have access to the brain and treat tumors. When accessing the brain, a tissue deformation occurs and can negatively influence the surgical procedure outcome. In this work, we present a novel Augmented Reality neurosurgical system to superimpose pre-operative 3D meshes derived from MRI onto a view of the brain surface acquired during surgery. Methods: Our method uses cortical vessels as main features to drive a rigid then non-rigid 3D/2D registration. We first use a feature extractor network to produce probability maps that are fed to a pose estimator network to infer the 6-DoF rigid pose. Then, to account for brain deformation, we add a nonrigid refinement step formulated as a Shape-from-Template problem using physics-based constraints that helps propagate the deformation to sub-cortical level and update tumor location. Results: We tested our method retrospectively on 6 clinical datasets and obtained low pose error, and showed using synthetic dataset that considerable brain shift compensation and low TRE can be achieved at cortical and sub-cortical levels. Conclusion: The results show that our solution achieved accuracy below the actual clinical errors demonstrating the feasibility of practical use of our system. Significance: This work shows that we can provide coherent Augmented Reality visualization of 3D cortical vessels observed through the craniotomy using a single camera view and that cortical vessels provide strong features for performing both rigid and non-rigid registration.

Kemajuan teknologi di era Globalisasi, banyak dimanfaatkan dalam berbagai bidang, salah satunya dalam bidang bisnis. Sebagai contoh pada media promosi, yaitu hadirnya teknologi augmented reality yang merupakan sebuah teknologi yang membawa objek virtual 3D yang dibuat menggunakan komputer, ke dalam dunia nyata 3D. Cara kerja augmented reality sendiri yaitu dengan melacak marker (penanda) melalui kamera yang bertindak sebagai mata yang dimana dapat mengenali objek dan kemudian diproyeksikan dalam waktu nyata. Untuk itu teknologi tersebut akan diterapkan pada penelitian ini, yang membahas tentang penerapan augmented reality pada brosur smartphone dengan algoritma FAST Corner Detection (FCD). Tujuan dari penggunaan teknologi augmented reality pada brosur ini, untuk membuat pembaca dapat melihat objek 3D pada brosur menggunakan aplikasi augmented reality smartphone yang dapat diinstal pada smartphone pembaca. Dan fungsi dari penggunaan algoritma FCD sendiri yaitu untuk mengenali marker dengan cara melacak titik-titik (sudut) pusat pada gambar marker. Hasil pengujian dari peneltian ini yaitu aplikasi dapat digunakan dengan baik untuk scan marker pada intensitas cukup cahaya sampai banyak cahaya dan dengan jarak ≤50 cm

Zusammenfassung In dieser Arbeit stellen wir einen neuen Ansatz zur geometrischen Inspektion großer mechanischer Bauteile vor. Dazu benutzen wir ein Portal, das auf einer Linearachse manuell verschoben werden kann. An dem Portal sind ein frei beweglicher 3D-Streifenlichtsensor und eine zusätzliche Augmented-Reality-Kamera angebracht. Insgesamt deckt das Portal einen Arbeitsraum von 8 , 0 × 3 , 0 × 0 , 8 m 3 8,0\times 3,0\times 0,8\hspace{0.1667em}{\text{m}^{3}} ab, wobei die Position und Orientierung der Sensoren in einem globalen Koordinatensystem durch ein optisches Motion-Capturing-System bestimmt werden. Es wird gezeigt, dass Nutzer mittels Augmented Reality sehr schnell und intuitiv grobe Geometrieabweichungen erkennen können. Zur detaillierten, metrischen Vermessung können 3D-Punktewolken aufgenommen und automatisiert ausgewertet werden.

Augmented reality ( AR ) is important for training complex tasks, such as navigation, assembly, and medical procedures. The effectiveness of such training may depend on accurate spatial localization of AR objects in the environment. This article presents two experiments that test egocentric distance perception in augmented reality within and at the boundaries of action space (up to 35 m) in comparison with distance perception in a matched real-world ( RW ) environment. Using the Microsoft HoloLens, in Experiment 1, participants in two different RW settings judged egocentric distances (ranging from 10 to 35 m) to an AR avatar or a real person using a visual matching measure. Distances to augmented targets were underestimated compared to real targets in the two indoor, RW contexts. Experiment 2 aimed to generalize the results to an absolute distance measure using verbal reports in one of the indoor environments. Similar to Experiment 1, distances to augmented targets were underestimated compared to real targets. We discuss these findings with respect to the importance of methodologies that directly compare performance in real and mediated environments, as well as the inherent differences present in mediated environments that are “matched” to the real world.

With the continuous advancement of science and technology, the improvement of mobile terminal hardware performance and the large-scale popularization of smart phones have brought new experiences and methods to surveying and mapping work. This article mainly studies the application of remote sensing image processing technology based on mobile augmented reality technology in surveying and mapping engineering. First, perform grayscale processing on the image in the experiment, then remove the noise in the image and smooth the image through the median filter method and finally use the Canny operator to perform edge detection to obtain a binarized image containing only the target object, and this is done by image feature extraction. After using three-dimensional scanning modeling to extract the image feature points, the target manager is used for sample analysis. Obtain the projection matrix through the interface, and then perform coordinate conversion to complete the positioning of the target scene. In this paper, the BRISK feature point detection algorithm with fast speed and small calculation is used to detect the target, and SVM is used for remote sensing feature classification. Experimental data show that the recognition success rate of the algorithm is 84%. The results show that mobile augmented reality technology and remote sensing image processing technology can improve the efficiency and accuracy of surveying and mapping engineering, and have strong ease of use and stability.

Purpose Resection site repair during laparoscopic oncological surgery (e.g. laparoscopic partial nephrectomy) poses some unique challenges and opportunities for augmented reality (AR) navigation support. This work introduces an AR registration workflow that addresses the time pressure that is present during resection site repair. Methods We propose a two-step registration process: the AR content is registered as accurately as possible prior to the tumour resection (the primary registration). This accurate registration is used to apply artificial fiducials to the physical organ and the virtual model. After the resection, these fiducials can be used for rapid re-registration (the secondary registration). We tested this pipeline in a simulated-use study with $$N=18$$ N = 18 participants. We compared the registration accuracy and speed for our method and for landmark-based registration as a reference. Results Acquisition of and, thereby, registration with the artificial fiducials were significantly faster than the initial use of anatomical landmarks. Our method also had a trend to be more accurate in cases in which the primary registration was successful. The accuracy loss between the elaborate primary registration and the rapid secondary registration could be quantified with a mean target registration error increase of 2.35 mm. Conclusion This work introduces a registration pipeline for AR navigation support during laparoscopic resection site repair and provides a successful proof-of-concept evaluation thereof. Our results indicate that the concept is better suited than landmark-based registration during this phase, but further work is required to demonstrate clinical suitability and applicability.

Introduction: Augmented reality can improve planning and execution of surgical procedures. Head-mounted devices such as the HoloLens® (Microsoft, Redmond, WA, USA) are particularly suitable to achieve these aims because they are controlled by hand gestures and enable contactless handling in a sterile environment. Objectives: So far, these systems have not yet found their way into the operating room for surgery of the parotid gland. This study explored the feasibility and accuracy of augmented reality-assisted parotid surgery. Methods: 2D MRI holographic images were created, and 3D holograms were reconstructed from MRI DICOM files and made visible via the HoloLens. 2D MRI slices were scrolled through, 3D images were rotated, and 3D structures were shown and hidden only using hand gestures. The 3D model and the patient were aligned manually. Results: The use of augmented reality with the HoloLens in parotic surgery was feasible. Gestures were recognized correctly. Mean accuracy of superimposition of the holographic model and patient’s anatomy was 1.3 cm. Highly significant differences were seen in position error of registration between central and peripheral structures (p = 0.0059), with a least deviation of 10.9 mm (centrally) and highest deviation for the peripheral parts (19.6-mm deviation). Conclusion: This pilot study offers a first proof of concept of the clinical feasibility of the HoloLens for parotid tumor surgery. Workflow is not affected, but additional information is provided. The surgical performance could become safer through the navigation-like application of reality-fused 3D holograms, and it improves ergonomics without compromising sterility. Superimposition of the 3D holograms with the surgical field was possible, but further invention is necessary to improve the accuracy.

Item does not contain fulltext BACKGROUND: During a deep inferior epigastric perforator (DIEP) flap harvest, the identification and localization of the epigastric arteries and its perforators are crucial. Holographic augmented reality is an innovative technique that can be used to visualize this patient-specific anatomy extracted from a computed tomographic scan directly on the patient. This study describes an innovative workflow to achieve this. METHODS: A software application for the Microsoft HoloLens was developed to visualize the anatomy as a hologram. By using abdominal nevi as natural landmarks, the anatomy hologram is registered to the patient. To ensure that the anatomy hologram remains correctly positioned when the patient or the user moves, real-time patient tracking is obtained with a quick response marker attached to the patient. RESULTS: Holographic augmented reality can be used to visualize the epigastric arteries and its perforators in preparation for a deep inferior epigastric perforator flap harvest. CONCLUSIONS: Potentially, this workflow can be used visualize the vessels intraoperatively. Furthermore, this workflow is intuitive to use and could be applied for other flaps or other types of surgery.

[EN] This research develops a novel teleoperation for robot manipulators based on augmented reality. The proposed interface is equipped with full capabilities in order to replace the classical teach pendant of the robot for carrying out teleoperation tasks. The proposed interface is based on an augmented reality headset for projecting computer-generated graphics onto the real environment and a gamepad to interact with the computer-generated graphics and provide robot commands. In order to demonstrate the benefits of the proposed method, several usability tests were conducted using a 6R industrial robot manipulator in order to compare the proposed interface and the conventional teach pendant interface for teleoperation tasks. In particular,the results ofthese usabilitytests show thattheproposedapproach is more intuitive, ergonomic,and easy to use. Furthermore, the comparison results also show that the proposed method clearly improves the velocity of the teleoperation task, regardless of the user's previous experience in robotics and augmented reality technology., This work was supported by the Spanish Government under the project DPI2017-87656-C2-1-R.

Adding attention-awareness to an Augmented Reality setting by using a Brain-Computer Interface promises many interesting new applications and improved usability. The possibly complicated setup and relatively long training period of EEG-based BCIs however, reduce this positive effect immensely. In this study, we aim at finding solutions for person-independent, training-free BCI integration into AR to classify internally and externally directed attention. We assessed several different classifier settings on a dataset of 14 participants consisting of simultaneously recorded EEG and eye tracking data. For this, we compared the classification accuracies of a linear algorithm, a non-linear algorithm, and a neural net that were trained on a specifically generated feature set, as well as a shallow neural net for raw EEG data. With a real-time system in mind, we also tested different window lengths of the data aiming at the best payoff between short window length and high classification accuracy. Our results showed that the shallow neural net based on 4-second raw EEG data windows was best suited for real-time person-independent classification. The accuracy for the binary classification of internal and external attention periods reached up to 88% accuracy with a model that was trained on a set of selected participants. On average, the person-independent classification rate reached 60%. Overall, the high individual differences could be seen in the results. In the future, further datasets are necessary to compare these results before optimizing a real-time person-independent attention classifier for AR.

Sports video is loved by the audience because of its unique charm, so it has high research value and application value to analyze and study the video data of competition. Based on the background of football match, this paper studies the football detection and tracking algorithm in football game video and analyzes the real-time image of real-time mobile devices in sports video augmented reality. Firstly, the image is preprocessed by image graying, image denoising, image binarization, and so on. Secondly, Hough transform is used to locate and detect football, and according to the characteristics of football, Hough transform is improved. Based on the good performance of SIFT algorithm in feature matching, a football tracking algorithm based on SIFT feature matching is proposed, which matches the detected football with the sample football. The simulation results show that the improved Hough transform can effectively detect football and has good antijamming performance. And the designed football tracking algorithm based on SIFT feature matching can accurately track the football trajectory; therefore, the football detection and tracking algorithm designed in this paper is suitable for real-time football monitoring and tracking.

Virtual Reality (VR), Augmented Reality (AR) and Artificial Intelligence (AI) are modern days technological phenomenon that are equipped with powerful ability to revolutionize education and research. The present insight not only outlines the caveats of traditional teaching, but explores the current and potential applications of VR, AR, and AI in the field of orthodontic teaching and research. The digital technology world immersed in artificial intelligence not only enhances the ability of both student and educator in assimilating and imparting knowledge, respectively, but also, helps to create a symbiotic relationship with each other. Also, emphasis is laid on understanding and implementing the interaction of strategic framework, professionalism and patient care delivery in the era of AI, VR, and AR in order to support dynamic engagements in a pedagogical, ethically-appropriate, community-centred, culturally-sensitive and economically-feasible teaching and research milieu

Accurately placing virtual objects in a scene is a challenging tasks in handheld augmented reality (HAR). To add and arrange virtual objects in HAR, users must manipulate 6 degrees of freedom (DoFs) of the virtual object, namely: position (3) and orientation (3). However, it is difficult to manipulate all DoFs with the two-dimensional display of the handheld device. We present SlidAR+, a method for controlling the position and orientation of objects in HAR. SlidAR+ is an extension of SlidAR [1], a technique that allows users to control the position of a virtual object by manipulating only 1 DoF. We use the direction of gravity as a constraint to improve the user’s control and reduce the time it takes to adjust the orientation. Upon comparing it with a state-of-the-art object manipulation method, using SlidAR+, user were able to complete the tasks faster under our expected conditions and were also preferred by most participants.

The popularity of augmented reality in medical application is rising exponentially over time, especially in the medical sector. It possesses a greater possibility in the reduction of surgical risks by raising visual awareness during the operation. Incorrect representation or inadequate detail on target region or delay in processing time may result in serious consequences. Also, the absence of a proper mechanism to handle occlusions like nerves, vessels or medical equipment may affect the performance. Therefore, this research aims to improve the visualization accuracy of bowel region and reduce the processing time. a novel enhanced energy function using augmented reality for bowel is proposed. The proposed system targets the modified region and weighted factor that encompasses the power of the combined region and dense cue with longterm and accurate augmented reality display mechanism. The system is capable of providing detailed visual output by precisely placing the model developed using the CT images of the target object, over the live video. Also, by applying the least square approach, the system is capable of addressing larger deformation and occlusion that appears during the surgical procedure providing the most accurate display. The feature tracking and tracking recovery components help the entire visualization procedure to stay on track by automatically registering and re-registering the surface whenever required. The proposed system capable of running image registration without human involvement and it can even decide when to trigger the reregistration process whenever required. The results from the proposed system has minimized the overlay error by a larger number. We validated the system with different sets of samples from endoscopy. The dataset included the samples from the bowel region from people with three different age groups. The overlay error accuracy was 0.24777px, and the performance was 44fps. The proposed system is concentrated on the overlay accuracy and the processing time. This study has addressed the shortcoming of the previous systems regarding manual registration and rigid assumptions.

In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality, a markerless augmented reality tracking registration method based on multi-modal template matching and point clouds is proposed. The method first adapts the linear parallel multi-modal LineMod template matching method with scale invariance to identify the texture-less target and obtain the reference image as the key frame that is most similar to the current perspective. Then, we can obtain the initial pose of the camera and solve the problem of re-initialization because of tracking registration interruption. A point cloud-based method is used to calculate the precise pose of the camera in real time. In order to solve the problem that the traditional iterative closest point (ICP) algorithm cannot meet the real-time requirements of the system, Kd-tree (k-dimensional tree) is used under the graphics processing unit (GPU) to replace the part of finding the nearest points in the original ICP algorithm to improve the speed of tracking registration. At the same time, the random sample consensus (RANSAC) algorithm is used to remove the error point pairs to improve the accuracy of the algorithm. The results show that the proposed tracking registration method has good real-time performance and robustness.

In spatial augmented reality applications, incorrect projection mapping may occur when projecting images onto moving non-rigid surfaces. This may detract from the user experience, as the image may not be perceived as originally intended. This is especially apparent when using low-cost projectors and cameras or when surfaces are moving quickly. In this paper, an algorithm is developed which predicts the motion of a non-rigid surface, so that when an image is being projected onto the surface, the projection “matches” the surface shape, while using low-cost equipment. Using an interconnected mass–spring–damper system to model the surface, the surface position is predicted using a Kalman filter-based algorithm, which also compensates for the processing delays and fast-moving surfaces. To accurately model real-world materials, the mass–spring system parameters are found using a system identification approach. When the prediction algorithm is implemented experimentally, in real-time, the results show convergent results in the sense that the surface predictions converge to the measured position of a non-rigid surface. The error results show that the algorithm is both accurate and robust and can currently be applied in spatial augmented reality applications.

Eye pupil tracking is important for augmented reality (AR) three-dimensional (3D) head-up displays (HUDs). Accurate and fast eye tracking is still challenging due to multiple driving conditions with eye occlusions, such as wearing sunglasses. In this paper, we propose a system for commercial use that can handle practical driving conditions. Our system classifies human faces into bare faces and sunglasses faces, which are treated differently. For bare faces, our eye tracker regresses the pupil area in a coarse-to-fine manner based on a revised Supervised Descent Method based eye-nose alignment. For sunglasses faces, because the eyes are occluded, our eye tracker uses whole face alignment with a revised Practical Facial Landmark Detector for pupil center tracking. Furthermore, we propose a structural inference-based re-weight network to predict eye position from non-occluded areas, such as the nose and mouth. The proposed re-weight sub-network revises the importance of different feature map positions and predicts the occluded eye positions by non-occluded parts. The proposed eye tracker is robust via a tracker-checker and a small model size. Experiments show that our method achieves high accuracy and speed, approximately 1.5 and 6.5 mm error for bare and sunglasses faces, respectively, at less than 10 ms on a 2.0GHz CPU. The evaluation dataset was captured indoors and outdoors to reflect multiple sunlight conditions. Our proposed method, combined with AR 3D HUDs, shows promising results for commercialization with low crosstalk 3D images.