Search

Purpose: Optical-see-through head-mounted displays have the ability to seamlessly integrate virtual content with the real world through a transparent lens and an optical combiner. Although their potential for use in surgical settings has been explored, their clinical translation is sparse in the current literature, largely due to their limited tracking capabilities and the need for manual alignment of virtual representations of objects with their real-world counterparts., Methods: We propose a simple and robust hand-eye calibration process for the depth camera of the Microsoft HoloLens 2, utilizing a tracked surgical stylus fitted with infrared reflective spheres as the calibration tool., Results: Using a Monte Carlo simulation and a paired-fiducial registration algorithm, we show that a calibration accuracy of 1.65 mm can be achieved with as little as 6 fiducial points. We also present heuristics for optimizing the accuracy of the calibration. The ability to use our calibration method in a clinical setting is validated through a user study, with users achieving a mean calibration accuracy of 1.67 mm in an average time of 42 s., Conclusion: This work enables real-time hand-eye calibration for the Microsoft HoloLens 2, without any need for a manual alignment process. Using this framework, existing surgical navigation systems employing optical or electromagnetic tracking can easily be incorporated into an augmented reality environment with a high degree of accuracy., (© 2024. CARS.)

Purpose: The operating microscope plays a central role in middle and inner ear procedures that involve working within tightly confined spaces under limited exposure. Augmented reality (AR) may improve surgical guidance by combining preoperative computed tomography (CT) imaging that can provide precise anatomical information, with intraoperative microscope video feed. With current technology, the operator must manually interact with the AR interface using a computer. The latter poses a disruption in the surgical flow and is suboptimal for maintaining the sterility of the operating environment. The purpose of this study was to implement and evaluate free-hand interaction concepts leveraging hand tracking and gesture recognition as an attempt to reduce the disruption during surgery and improve human-computer interaction., Methods: An electromagnetically tracked surgical microscope was calibrated using a custom 3D printed calibration board. This allowed the augmentation of the microscope feed with segmented preoperative CT-derived virtual models. Ultraleap's Leap Motion Controller 2 was coupled to the microscope and used to implement hand-tracking capabilities. End-user feedback was gathered from a surgeon during development. Finally, users were asked to complete tasks that involved interacting with the virtual models, aligning them to physical targets, and adjusting the AR visualization., Results: Following observations and user feedback, we upgraded the functionalities of the hand interaction system. User feedback showed the users' preference for the new interaction concepts that provided minimal disruption of the surgical workflow and more intuitive interaction with the virtual content., Conclusion: We integrated hand interaction concepts, typically used with head-mounted displays (HMDs), into a surgical stereo microscope system intended for AR in otologic microsurgery. The concepts presented in this study demonstrated a more favorable approach to human-computer interaction in a surgical context. They hold potential for a more efficient execution of surgical tasks under microscopic AR guidance., (© 2024. CARS.)

Introduction: Thermal imaging can be used for the non-invasive detection of blood vessels of the skin. However, mapping the results to the patient currently lacks user-friendliness. Augmented reality may provide a useful tool to superimpose thermal information on the patient., Methods: A system to support planning in reconstructive surgery using a thermal camera was designed. The obtained information was superimposed on the physical object using a Microsoft HoloLens. An RGB, depth, and thermal camera were combined to capture a scene of different modalities and reconstruct a virtual scene in real time. To register the different cameras and the AR device, an active calibration target was developed and evaluated. A Vuforia marker was used to register the hologram in the virtual space. The accuracy of the projected hologram was evaluated in a laboratory setting with participants by measuring the error between the physical object and the hologram., Results: The AR-based system was evaluated by 21 participants in a laboratory setting. The mean projection error is 10.3 ± 9.4 mm. The system is able to stream a three-dimensional scene with augmented thermal information in real time at 5 frames per second. The active calibration target can be used independently of the environment., Conclusion: The calibration target provides an easy-to-use method for the registration of cameras capturing the visible to long-infrared spectral range. The inside-out tracking of the HoloLens in combination with a Vuforia marker is not accurate enough for the intended clinical use., (© 2024. CARS.)

Competing Interests: Declaration of competing interest The authors declare no competing interests.

Purpose: Calibration of an optical see-through head-mounted display is critical for augmented reality-based surgical navigation. While conventional methods have advanced, calibration errors remain significant. Moreover, prior research has focused primarily on calibration accuracy and procedure, neglecting the impact on the overall surgical navigation system. Consequently, these enhancements do not necessarily translate to accurate augmented reality in the optical see-through head mount due to systemic errors, including those in calibration., Method: This study introduces a simulated augmented reality-based calibration to address these issues. By replicating the augmented reality that appeared in the optical see-through head mount, the method achieves calibration that compensates for augmented reality errors, thereby reducing them. The process involves two distinct calibration approaches, followed by adjusting the transformation matrix to minimize displacement in the simulated augmented reality., Results: The efficacy of this method was assessed through two accuracy evaluations: registration accuracy and augmented reality accuracy. Experimental results showed an average translational error of 2.14 mm and rotational error of 1.06° across axes in both approaches. Additionally, augmented reality accuracy, measured by the overlay regions' ratio, increased to approximately 95%. These findings confirm the enhancement in both calibration and augmented reality accuracy with the proposed method., Conclusion: The study presents a calibration method using simulated augmented reality, which minimizes augmented reality errors. This approach, requiring minimal manual intervention, offers a more robust and precise calibration technique for augmented reality applications in surgical navigation., (© 2024. CARS.)

Purpose: This study investigates the potential utility of augmented reality (AR) in the endoscopic transsphenoidal approach (TSA). While previous research has addressed technical challenges in AR for TSA, this paper explores how design factors can improve AR for neurosurgeons from a human-centred design perspective., Methods: Preliminary qualitative research involved observations of TSA procedures ( n = 2 ) and semi-structured interviews with neurosurgeons ( n = 4 ). These informed the design of an AR mockup, which was evaluated with neurosurgeons ( n = 3 ). An interactive low-fidelity prototype-the "AR-assisted Navigation for the TransSphenoidal Approach (ANTSA)"-was then developed in Unity 3D. A user study ( n = 4 ) evaluated the low-fidelity prototype of ANTSA through contextual interviews, providing feedback on design factors., Results: AR visualisations may be beneficial in streamlining the sellar phase and reducing intraoperative errors such as excessive or inadequate exposure. Key design recommendations include a lean mesh rendering, an intuitive colour palette, and optional structure highlighting., Conclusion: This research presents user-centred design guidelines to improve sensemaking and surgical workflow in the sellar phase of TSA, with the goal of improving clinical outcomes. The specific improvements that AR could bring to the workflow are discussed along with surgeons' reservations and its possible application towards training less experienced physicians., (© 2024. The Author(s).)

Purpose: We present a novel method for augmented reality in endoscopic endonasal surgery. Our method does not require the use of external tracking devices and can show hidden anatomical structures relevant to the surgical intervention., Methods: Our method registers a preoperative 3D model of the nasal cavity to an intraoperative 3D model by estimating a scaled-rigid transformation. Registration is based on a two-stage ICP approach on the reconstructed nasal cavity. The hidden structures are then transferred from the preoperative 3D model to the intraoperative one using the estimated transformation, projected and overlaid into the endoscopic images to obtain the augmented reality., Results: We performed qualitative and quantitative validation of our method on 12 clinical cases. Qualitative results were obtained from an ENT surgeon from visual inspection of the hidden structures in the augmented images. Quantitative results were obtained by measuring a target registration error using a novel transillumination-based approach. The results show that the hidden structures of interest are augmented at the expected locations in most cases., Conclusion: Our method was able to augment the endoscopic images in a sufficiently precise manner when the intraoperative nasal cavity did not deform considerably with respect to its preoperative state. This is a promising step towards trackerless augmented reality in endonasal surgery., (© 2024. CARS.)

Purpose: This research endeavors to improve tumor localization in minimally invasive surgeries, a challenging task primarily attributable to the absence of tactile feedback and limited visibility. The conventional solution uses laparoscopic ultrasound (LUS) which has a long learning curve and is operator-dependent., Methods: The proposed approach involves augmenting LUS images onto laparoscopic images to improve the surgeon's ability to estimate tumor and internal organ anatomy. This augmentation relies on LUS pose estimation and filtering., Results: Experiments conducted with clinical data exhibit successful outcomes in both the registration and augmentation of LUS images onto laparoscopic images. Additionally, noteworthy results are observed in filtering, leading to reduced flickering in augmentations., Conclusion: The outcomes reveal promising results, suggesting the potential of LUS augmentation in surgical images to assist surgeons and serve as a training tool. We have used the LUS probe's shaft to disambiguate the rotational symmetry. However, in the long run, it would be desirable to find more convenient solutions., (© 2024. CARS.)

Background: Laparoscopic left hemihepatectomy (LLH) has been shown to be an effective and safe method for treating hepatolithiasis primarily affecting the left hemiliver. However, this procedure still presents challenges. Due to pathological changes in intrahepatic duct stones, safely dissecting the hilar vessels and determining precise resection boundaries remains difficult, even with fluorescent imaging. Our team proposed a new method of augmented reality navigation (ARN) combined with Indocyanine green (ICG) fluorescence imaging for LLH in hepatolithiasis cases. This study aimed to investigate the feasibility of this combined approach in the procedure., Methods: Between May 2021 and September 2023, 16 patients with hepatolithiasis who underwent LLH were included. All patients underwent preoperative 3D evaluation and were then guided using ARN and ICG fluorescence imaging during the procedure. Perioperative and short-term postoperative outcomes were assessed to evaluate the safety and efficacy of the method., Results: All 16 patients successfully underwent LLH. The mean operation time was 380.31 ± 92.17 min, with a mean estimated blood loss of 116.25 ± 64.49 ml. ARN successfully aided in guiding hilar vessel dissection in all patients. ICG fluorescence imaging successfully identified liver resection boundaries in 11 patients (68.8%). In the remaining 5 patients (31.3%) where fluorescence imaging failed, virtual liver segment projection (VLSP) successfully identified their resection boundaries. No major complications occurred in any patients. Immediate stone residual rate, stone recurrence rate, and stone extraction rate through the T-tube sinus tract were 12.5%, 6.3%, and 6.3%, respectively., Conclusion: The combination of ARN and ICG fluorescence imaging enhances the safety and precision of LLH for hepatolithiasis. Moreover, ARN may serve as a safe and effective tool for identifying precise resection boundaries in cases where ICG fluorescence imaging fails., (© 2024. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Background: Retroperitoneal sarcomas (RPSs) present a surgical challenge, with complex anatomic relationships to organs and vascular structures. This pilot study investigated the role of three-dimensional (3D) augmented reality (3DAR) compared with standard imaging in preoperative planning and resection strategies., Methods: For the study, 13 patients who underwent surgical resection of their RPS were selected based on the location of their tumor (right, left, pelvis). From the patients' preoperative computed tomography (CT) scans, 3DAR models were created using a D2P program and projected by an augmented-reality (AR) glass (Hololens). The 3DAR models were evaluated by three experienced sarcoma surgeons and compared with the baseline two-dimensional (2D) contrast-enhanced CT scans., Results: Three members of the surgical team evaluated 13 models of retroperitoneal sarcomas, resulting in a total of 26 responses. When the surgical team was asked to evaluate whether the 3DAR better prepared the surgeon for planned surgical resection, 10 responses favored the 3DAR, 5 favored the 2D CT scans and 11 showed no difference (p = 0.074). According to 15 (57.6 %) of the 26 responses, the 3DAR offered additional value over standard imaging in the preoperative planning (median score of 4; range, 1-5). The median stated likelihood that the surgeons would consult the 3DAR was 5 (range, 2-5) for the preoperative setting and 3 (range, 1-5) for the intraoperative setting., Conclusions: This pilot study suggests that the use of 3DAR may provide additional value over current standard imaging in the preoperative planning for surgical resection of RPS, and the technology merits further study., (© 2024. Society of Surgical Oncology.)

Purpose: Proper visualization and interaction with complex anatomical data can improve understanding, allowing for more intuitive surgical planning. The goal of our work was to study what the most intuitive yet practical platforms for interacting with 3D medical data are in the context of surgical planning., Methods: We compared planning using a monitor and mouse, a monitor with a haptic device, and an augmented reality (AR) head-mounted display which uses a gesture-based interaction. To determine the most intuitive system, two user studies, one with novices and one with experts, were conducted. The studies involved planning of three scenarios: (1) heart valve repair, (2) hip tumor resection, and (3) pedicle screw placement. Task completion time, NASA Task Load Index and system-specific questionnaires were used for the evaluation., Results: Both novices and experts preferred the AR system for pedicle screw placement. Novices preferred the haptic system for hip tumor planning, while experts preferred the mouse and keyboard. In the case of heart valve planning, novices preferred the AR system but there was no clear preference for experts. Both groups reported that AR provides the best spatial depth perception., Conclusion: The results of the user studies suggest that different surgical cases may benefit from varying interaction and visualization methods. For example, for planning surgeries with implants and instrumentations, mixed reality could provide better 3D spatial perception, whereas using landmarks for delineating specific targets may be more effective using a traditional 2D interface., (© 2024. CARS.)

Purpose: The retroperitoneal nature of the pancreas, marked by minimal intraoperative organ shifts and deformations, makes augmented reality (AR)-based systems highly promising for pancreatic surgery. This study presents preliminary data from a prospective study aiming to develop the first wearable AR assistance system, ARAS, for pancreatic surgery and evaluating its usability, accuracy, and effectiveness in enhancing the perioperative outcomes of patients., Methods: We developed ARAS as a two-phase system for a wearable AR device to aid surgeons in planning and operation. This system was used to visualize and register patient-specific 3D anatomical models during the surgery. The location and precision of the registered 3D anatomy were evaluated by assessing the arterial pulse and employing Doppler and duplex ultrasonography. The usability, accuracy, and effectiveness of ARAS were assessed using a five-point Likert scale questionnaire., Results: Perioperative outcomes of five patients underwent various pancreatic resections with ARAS are presented. Surgeons rated ARAS as excellent for preoperative planning. All structures were accurately identified without any noteworthy errors. Only tumor identification decreased after the preparation phase, especially in patients who underwent pancreaticoduodenectomy because of the extensive mobilization of peripancreatic structures. No perioperative complications related to ARAS were observed., Conclusions: ARAS shows promise in enhancing surgical precision during pancreatic procedures. Its efficacy in preoperative planning and intraoperative vascular identification positions it as a valuable tool for pancreatic surgery and a potential educational resource for future surgical residents., (© 2024. The Author(s).)

Purpose: We aim to investigate the integration of augmented reality (AR) within the context of increasingly complex surgical procedures and instrument handling toward the transition to smart operating rooms (OR). In contrast to cumbersome paper-based surgical instrument manuals still used in the OR, we wish to provide surgical staff with an AR head-mounted display that provides in-situ visualization and guidance throughout the assembly process of surgical instruments. Our requirement analysis supports the development and provides guidelines for its transfer into surgical practice., Methods: A three-phase user-centered design approach was applied with online interviews, an observational study, and a workshop with two focus groups with scrub nurses, circulating nurses, surgeons, manufacturers, clinic IT staff, and members of the sterilization department. The requirement analysis was based on key criteria for usability. The data were analyzed via structured content analysis., Results: We identified twelve main problems with the current use of paper manuals. Major issues included sterile users' inability to directly handle non-sterile manuals, missing details, and excessive text information, potentially delaying procedure performance. Major requirements for AR-driven guidance fall into the categories of design, practicability, control, and integration into the current workflow. Additionally, further recommendations for technical development could be obtained., Conclusion: In conclusion, our insights have outlined a comprehensive spectrum of requirements that are essential for the successful implementation of an AI- and AR-driven guidance for assembling surgical instruments. The consistently appreciative evaluation by stakeholders underscores the profound potential of AR and AI technology as valuable assistance and guidance., (© 2024. The Author(s).)

Mixed-reality surgical simulators are seen more objective than conventional training. The simulators' utility in training must be established through validation studies. Establish face-, content-, and construct-validity of a novel mixed-reality surgical simulator developed by McGill University, CAE-Healthcare, and DePuy Synthes. This study, approved by a Research Ethics Board, examined a simulated L4-L5 oblique lateral lumbar interbody fusion (OLLIF) scenario. A 5-point Likert scale questionnaire was used. Chi-square test verified validity consensus. Construct validity investigated 276 surgical performance metrics across three groups, using ANOVA, Welch-ANOVA, or Kruskal-Wallis tests. A post-hoc Dunn's test with a Bonferroni correction was used for further analysis on significant metrics. Musculoskeletal Biomechanics Research Lab, McGill University, Montreal, Canada. DePuy Synthes, Johnson & Johnson Family of Companies, research lab. Thirty-four participants were recruited: spine surgeons, fellows, neurosurgical, and orthopedic residents. Only seven surgeons out of the 34 were recruited in a side-by-side cadaver trial, where participants completed an OLLIF surgery first on a cadaver and then immediately on the simulator. Participants were separated a priori into three groups: post-, senior-, and junior-residents. Post-residents rated validity, median > 3, for 13/20 face-validity and 9/25 content-validity statements. Seven face-validity and 12 content-validity statements were rated neutral. Chi-square test indicated agreeability between group responses. Construct validity found eight metrics with significant differences (p < 0.05) between the three groups. Validity was established. Most face-validity statements were positively rated, with few neutrally rated pertaining to the simulation's graphics. Although fewer content-validity statements were validated, most were rated neutral (only four were negatively rated). The findings underscored the importance of using realistic physics-based forces in surgical simulations. Construct validity demonstrated the simulator's capacity to differentiate surgical expertise., (© 2024. International Federation for Medical and Biological Engineering.)

Purpose: Targeting accuracy determines outcomes for percutaneous needle interventions. Augmented reality (AR) in IR may improve procedural guidance and facilitate access to complex locations. This study aimed to evaluate percutaneous needle placement accuracy using a goggle-based AR system compared to an ultrasound (US)-based fusion navigation system., Methods: Six interventional radiologists performed 24 independent needle placements in an anthropomorphic phantom (CIRS 057A) in four needle guidance cohorts (n = 6 each): (1) US-based fusion, (2) goggle-based AR with stereoscopically projected anatomy (AR-overlay), (3) goggle AR without the projection (AR-plain), and (4) CT-guided freehand. US-based fusion included US/CT registration with electromagnetic (EM) needle, transducer, and patient tracking. For AR-overlay, US, EM-tracked needle, stereoscopic anatomical structures and targets were superimposed over the phantom. Needle placement accuracy (distance from needle tip to target center), placement time (from skin puncture to final position), and procedure time (time to completion) were measured., Results: Mean needle placement accuracy using US-based fusion, AR-overlay, AR-plain, and freehand was 4.5 ± 1.7 mm, 7.0 ± 4.7 mm, 4.7 ± 1.7 mm, and 9.2 ± 5.8 mm, respectively. AR-plain demonstrated comparable accuracy to US-based fusion (p = 0.7) and AR-overlay (p = 0.06). Excluding two outliers, AR-overlay accuracy became 5.9 ± 2.6 mm. US-based fusion had the highest mean placement time (44.3 ± 27.7 s) compared to all navigation cohorts (p < 0.001). Longest procedure times were recorded with AR-overlay (34 ± 10.2 min) compared to AR-plain (22.7 ± 8.6 min, p = 0.09), US-based fusion (19.5 ± 5.6 min, p = 0.02), and freehand (14.8 ± 1.6 min, p = 0.002)., Conclusion: Goggle-based AR showed no difference in needle placement accuracy compared to the commercially available US-based fusion navigation platform. Differences in accuracy and procedure times were apparent with different display modes (with/without stereoscopic projections). The AR-based projection of the US and needle trajectory over the body may be a helpful tool to enhance visuospatial orientation. Thus, this study refines the potential role of AR for needle placements, which may serve as a catalyst for informed implementation of AR techniques in IR., (© 2024. This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply.)

Purpose: Numerous navigation devices for percutaneous, CT-guided interventions exist and are, due to their advantages, increasingly integrated into the clinical workflow. However, effective training methods to ensure safe usage are still lacking. This study compares the potential of an augmented reality (AR) training application with conventional instructions for the Cube Navigation System (CNS), hypothesizing enhanced training with AR, leading to safer clinical usage., Methods: An AR-tablet app was developed to train users puncturing with CNS. In a study, 34 medical students were divided into two groups: One trained with the AR-app, while the other used conventional instructions. After training, each participant executed 6 punctures on a phantom (204 in total) following a standardized protocol to identify and measure two potential CNS procedural user errors: (1) missing the coordinates specified and (2) altering the needle trajectory during puncture. Training performance based on train time and occurrence of procedural errors, as well as scores of User Experience Questionnaire (UEQ) for both groups, was compared., Results: Training duration was similar between the groups. However, the AR-trained participants showed a 55.1% reduced frequency of the first procedural error (p > 0.05) and a 35.1% reduced extent of the second procedural error (p < 0.01) compared to the conventionally trained participants. UEQ scores favored the AR-training in five of six categories (p < 0.05)., Conclusion: The AR-app enhanced training performance and user experience over traditional methods. This suggests the potential of AR-training for navigation devices like the CNS, potentially increasing their safety, ultimately improving outcomes in percutaneous needle placements., (© 2024. The Author(s).)

Educators are witnessing the unfolding of the era of artificial intelligence, raising the question of how to transfer the benefits of yesterday's pedagogy to the future of education. An interactive digital mind map of infectious diseases was developed for second-year medical students ( n  = 865). Analysis of user engagement showed global distribution with 498 QR scans on a single day. Student responses ( n  = 79, 9.1% response rate) indicated positive feedback on the resources of Extremely Satisfied (range 65-75%, n  = 59-51). The findings of the study support further expansion of MedMicroMaps to cross-platform interfaces with adaptations for diverse audiences within allied health fields., Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-024-02047-3., Competing Interests: Competing InterestsThe author of the article had no conflicting established partnerships of financial or non-financial support during the pilot study and held an academic position at St. George’s University. No funding agencies contributed to the pilot study. The author of the article currently holds an academic role at Rocky Vista University and has recently established a start-up company MedMicroMaps, LLC. to apply for grant funding with the National Science Foundation, Small Business Innovation Research program. MedMicroMaps, LLC. has no established financial relationships with external funding partners., (© The Author(s) 2024.)

Purpose: This bibliometric analysis of the top 100 cited articles on extended reality (XR) in neurosurgery aimed to reveal trends in this research field. Gender differences in authorship and global distribution of the most-cited articles were also addressed., Methods: A Web of Science electronic database search was conducted. The top 100 most-cited articles related to the scope of this review were retrieved and analyzed for trends in publications, journal characteristics, authorship, global distribution, study design, and focus areas. After a brief description of the top 100 publications, a comparative analysis between spinal and cranial publications was performed., Results: From 2005, there was a significant increase in spinal neurosurgery publications with a focus on pedicle screw placement. Most articles were original research studies, with an emphasis on augmented reality (AR). In cranial neurosurgery, there was no notable increase in publications. There was an increase in studies assessing both AR and virtual reality (VR) research, with a notable emphasis on VR compared to AR. Education, surgical skills assessment, and surgical planning were more common themes in cranial studies compared to spinal studies. Female authorship was notably low in both groups, with no significant increase over time. The USA and Canada contributed most of the publications in the research field., Conclusions: Research regarding the use of XR in neurosurgery increased significantly from 2005. Cranial research focused on VR and resident education while spinal research focused on AR and neuronavigation. Female authorship was underrepresented. North America provides most of the high-impact research in this area., (© 2024. The Author(s).)

Today, the global prevalence of autism is high and continues to increase dramatically. Effective support and interventions are therefore warranted. Augmented reality (AR), one of the recent modalities of immersive technology, is gaining traction in autism interventions. However, there is currently a lack of reviews on the use of AR in children with autism, and what is more, to the authors' knowledge, no systematic review to date has been carried out to exclusively examine the effects of AR on basic social skills in children with autism. Therefore, an evidence-based systematic review was conducted to fill the gap and answer the specific research question: Can AR support children with an autism diagnosis towards developing or promoting social skills, such as greetings? The initial search yielded a total of 416 records. After excluding duplicate articles and screening the abstract and full text, 13 studies were included for analysis. A narrative approach was employed to synthesise and evaluate the research findings to substantially explore the effects of AR-based social interventions. The favourable role of AR technology in fostering social skills in children diagnosed with autism was widely recognised in the included studies although multiple methodological limitations were identified in relation to the quality of the included studies. Overall, the promising findings may suggest the effectiveness of AR in improving social skills in children with autism. Nonetheless, this field of research still calls for more high-quality studies relying on rigorous methodologies., (© 2023. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Purpose: For tumor resection, surgeons need to localize the tumor. For this purpose, a magnetic seed can be inserted into the tumor by a radiologist and, during surgery, a magnetic detection probe informs the distance to the seed for localization. In this case, the surgeon still needs to mentally reconstruct the position of the tumor from the probe's information. The purpose of this study is to develop and assess a method for 3D localization and visualization of the seed, facilitating the localization of the tumor., Methods: We propose a method for 3D localization of the magnetic seed by extending the magnetic detection probe with a tracking-based localization. We attach a position sensor (QR-code or optical marker) to the probe in order to track its 3D pose (respectively, using a head-mounted display with a camera or optical tracker). Following an acquisition protocol, the 3D probe tip and seed position are subsequently obtained by solving a system of equations based on the distances and the 3D probe poses., Results: The method was evaluated with an optical tracking system. An experimental setup using QR-code tracking (resp. using an optical marker) achieves an average of 1.6 mm (resp. 0.8 mm) 3D distance between the localized seed and the ground truth. Using a breast phantom setup, the average 3D distance is 4.7 mm with a QR-code and 2.1 mm with an optical marker., Conclusion: Tracking the magnetic detection probe allows 3D localization of a magnetic seed, which opens doors for augmented reality target visualization during surgery. Such an approach should enhance the perception of the localized region of interest during the intervention, especially for breast tumor resection where magnetic seeds can already be used in the protocol., (© 2024. The Author(s).)

Purpose: This study evaluates the nnU-Net for segmenting brain, skin, tumors, and ventricles in contrast-enhanced T1 (T1CE) images, benchmarking it against an established mesh growing algorithm (MGA)., Methods: We used 67 retrospectively collected annotated single-center T1CE brain scans for training models for brain, skin, tumor, and ventricle segmentation. An additional 32 scans from two centers were used test performance compared to that of the MGA. The performance was measured using the Dice-Sørensen coefficient (DSC), intersection over union (IoU), 95th percentile Hausdorff distance (HD95), and average symmetric surface distance (ASSD) metrics, with time to segment also compared., Results: The nnU-Net models significantly outperformed the MGA (p < 0.0125) with a median brain segmentation DSC of 0.971 [95CI: 0.945-0.979], skin: 0.997 [95CI: 0.984-0.999], tumor: 0.926 [95CI: 0.508-0.968], and ventricles: 0.910 [95CI: 0.812-0.968]. Compared to the MGA's median DSC for brain: 0.936 [95CI: 0.890, 0.958], skin: 0.991 [95CI: 0.964, 0.996], tumor: 0.723 [95CI: 0.000-0.926], and ventricles: 0.856 [95CI: 0.216-0.916]. NnU-Net performance between centers did not significantly differ except for the skin segmentations Additionally, the nnU-Net models were faster (mean: 1139 s [95CI: 685.0-1616]) than the MGA (mean: 2851 s [95CI: 1482-6246])., Conclusions: The nnU-Net is a fast, reliable tool for creating automatic deep learning-based segmentation pipelines, reducing the need for extensive manual tuning and iteration. The models are able to achieve this performance despite a modestly sized training set. The ability to create high-quality segmentations in a short timespan can prove invaluable in neurosurgical settings., (© 2024. The Author(s).)

Background: In the last decade, there has been a great effort in developing new technologies to enhance surgical visualization and guidance. This comprehensive and narrative review aimed to provide a wide and extensive overview of the current state of the art on this topic and their near-future perspectives linked to the development of artificial intelligence (AI), by focusing on the most recent and relevant literature., Methods: A comprehensive and narrative review of the literature was performed by searching specific terms on PubMed/MEDLINE, Scopus, and Embase databases, assessing the current state of the art on this topic., Results: Fluorescence-guided surgery, contrast-enhanced ultrasound (CEUS), ultra-high frequency ultrasound (UHFUS), photoacoustic imaging (PAI), and augmented reality (AR) are boosting the field of image-guided techniques as the rapid development of AI in surgery is promising a more automated decision-making and surgical movements in the operating room., Conclusion: Fluorescence-guided surgery, CEUS, UHFUS, PAI, and AR are becoming crucial to give surgeons a new level of information during the intervention, with the right timing and sequence, and represent the future of surgery. As many more controlled studies are needed to validate the employment of these technologies, the next generation of surgeons must become more familiar with the basics of AI to better incorporate new tools into the daily surgical practice of the future., (Copyright © 2023 Society for Surgery of the Alimentary Tract. Published by Elsevier Inc. All rights reserved.)

Mixed reality technologies, such as virtual (VR) and augmented (AR) reality, present promising opportunities to advance education and professional training due to their adaptability to diverse contexts. Distortions in the perceived distance in such mediated conditions, however, are well documented and have imposed nontrivial challenges that complicate and limit transferring task performance in a virtual setting to the unmediated reality (UR). One potential source of the distance distortion is the vergence-accommodation conflict-the discrepancy between the depth specified by the eyes' accommodative state and the angle at which the eyes converge to fixate on a target. The present study involved the use of a manual pointing task in UR, VR, and AR to quantify the magnitude of the potential depth distortion in each modality. Conceptualizing the effect of vergence-accommodation offset as a constant offset to the vergence angle, a model was developed based on the stereoscopic viewing geometry. Different versions of the model were used to fit and predict the behavioral data for all modalities. Results confirmed the validity of the conceptualization of vergence-accommodation as a device-specific vergence offset, which predicted up to 66% of the variance in the data. The fitted parameters indicate that, due to the vergence-accommodation conflict, participants' vergence angle was driven outwards by approximately 0.2°, which disrupted the stereoscopic viewing geometry and produced distance distortion in VR and AR. The implications of this finding are discussed in the context of developing virtual environments that minimize the effect of depth distortion., Competing Interests: Conflict of interestThe authors have no conflicts of interest to declare that are relevant to the content of this article., (© The Author(s) 2024.)

Introduction: Sentinel lymph node biopsy for oral and oropharyngeal squamous cell carcinoma is a well-established staging method. One variation is to inject a radioactive tracer near the primary tumor of the patient. After a few minutes, audio feedback from an external hand-held [Formula: see text]-detection probe can monitor the uptake into the lymphatic system. Such probes place a high cognitive load on the surgeon during the biopsy, as they require the simultaneous use of both hands and the skills necessary to correlate the audio signal with the location of tracer accumulation in the lymph nodes. Therefore, an augmented reality (AR) approach to directly visualize and thus discriminate nearby lymph nodes would greatly reduce the surgeons' cognitive load., Materials and Methods: We present a proof of concept of an AR approach for sentinel lymph node biopsy by ex vivo experiments. The 3D position of the radioactive [Formula: see text]-sources is reconstructed from a single [Formula: see text]-image, acquired by a stationary table-attached multi-pinhole [Formula: see text]-detector. The position of the sources is then visualized using Microsoft's HoloLens. We further investigate the performance of our SLNF algorithm for a single source, two sources, and two sources with a hot background., Results: In our ex vivo experiments, a single [Formula: see text]-source and its AR representation show good correlation with known locations, with a maximum error of 4.47 mm. The SLNF algorithm performs well when only one source is reconstructed, with a maximum error of 7.77 mm. For the more challenging case to reconstruct two sources, the errors vary between 2.23 mm and 75.92 mm., Conclusion: This proof of concept shows promising results in reconstructing and displaying one [Formula: see text]-source. Two simultaneously recorded sources are more challenging and require further algorithmic optimization., (© 2023. The Author(s).)

Purpose: Concomitant with the significant advances in computing technology, the utilization of augmented reality-based navigation in clinical applications is being actively researched. In this light, we developed novel object tracking and depth realization technologies to apply augmented reality-based neuronavigation to brain surgery., Methods: We developed real-time inside-out tracking based on visual inertial odometry and a visual inertial simultaneous localization and mapping algorithm. The cube quick response marker and depth data obtained from light detection and ranging sensors are used for continuous tracking. For depth realization, order-independent transparency, clipping, and annotation and measurement functions were developed. In this study, the augmented reality model of a brain tumor patient was applied to its life-size three-dimensional (3D) printed model., Results: Using real-time inside-out tracking, we confirmed that the augmented reality model remained consistent with the 3D printed patient model without flutter, regardless of the movement of the visualization device. The coordination accuracy during real-time inside-out tracking was also validated. The average movement error of the X and Y axes was 0.34 ± 0.21 and 0.04 ± 0.08 mm, respectively. Further, the application of order-independent transparency with multilayer alpha blending and filtered alpha compositing improved the perception of overlapping internal brain structures. Clipping, and annotation and measurement functions were also developed to aid depth perception and worked perfectly during real-time coordination. We named this system METAMEDIP navigation., Conclusions: The results validate the efficacy of the real-time inside-out tracking and depth realization technology. With these novel technologies developed for continuous tracking and depth perception in augmented reality environments, we are able to overcome the critical obstacles in the development of clinically applicable augmented reality neuronavigation., (© 2023. CARS.)

Robotic-assisted orthopedic surgery (RAOS) is revolutionizing the field, offering the potential for increased accuracy and precision and improved patient outcomes. This comprehensive review explores the historical perspective, current robotic systems, advantages and limitations, clinical outcomes, patient satisfaction, future developments, and innovation in RAOS. Based on systematic reviews, meta-analyses, and recent studies, this article highlights the most significant findings and compares RAOS to conventional techniques. As robotic-assisted surgery continues to evolve, clinicians and researchers must stay informed and adapt their practices to provide optimal patient care. Evidence from published studies corroborates these claims, highlighting superior component positioning, decreased incidence of complications, and heightened patient satisfaction. However, challenges such as costs, learning curves, and technical issues must be resolved to fully capitalize on these advantages., (© 2023. The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature.)

Purpose: Middle and inner ear procedures target hearing loss, infections, and tumors of the temporal bone and lateral skull base. Despite the advances in surgical techniques, these procedures remain challenging due to limited haptic and visual feedback. Augmented reality (AR) may improve operative safety by allowing the 3D visualization of anatomical structures from preoperative computed tomography (CT) scans on real intraoperative microscope video feed. The purpose of this work was to develop a real-time CT-augmented stereo microscope system using camera calibration and electromagnetic (EM) tracking., Methods: A 3D printed and electromagnetically tracked calibration board was used to compute the intrinsic and extrinsic parameters of the surgical stereo microscope. These parameters were used to establish a transformation between the EM tracker coordinate system and the stereo microscope image space such that any tracked 3D point can be projected onto the left and right images of the microscope video stream. This allowed the augmentation of the microscope feed of a 3D printed temporal bone with its corresponding CT-derived virtual model. Finally, the calibration board was also used for evaluating the accuracy of the calibration., Results: We evaluated the accuracy of the system by calculating the registration error (RE) in 2D and 3D in a microsurgical laboratory setting. Our calibration workflow achieved a RE of 0.11 ± 0.06 mm in 2D and 0.98 ± 0.13 mm in 3D. In addition, we overlaid a 3D CT model on the microscope feed of a 3D resin printed model of a segmented temporal bone. The system exhibited small latency and good registration accuracy., Conclusion: We present the calibration of an electromagnetically tracked surgical stereo microscope for augmented reality visualization. The calibration method achieved accuracy within a range suitable for otologic procedures. The AR process introduces enhanced visualization of the surgical field while allowing depth perception., (© 2023. CARS.)

Purpose: Understanding the properties and aspects of the robotic system is essential to a successful medical intervention, as different capabilities and limits characterize each. Robot positioning is a crucial step in the surgical setup that ensures proper reachability to the desired port locations and facilitates docking procedures. This very demanding task requires much experience to master, especially with multiple trocars, increasing the barrier of entry for surgeons in training., Methods: Previously, we demonstrated an Augmented Reality-based system to visualize the rotational workspace of the robotic system and proved it helps the surgical staff to optimize patient positioning for single-port interventions. In this work, we implemented a new algorithm to allow for an automatic, real-time robotic arm positioning for multiple ports., Results: Our system, based on the rotational workspace data of the robotic arm and the set of trocar locations, can calculate the optimal position of the robotic arm in milliseconds for the positional and in seconds for the rotational workspace in virtual and augmented reality setups., Conclusions: Following the previous work, we extended our system to support multiple ports to cover a broader range of surgical procedures and introduced the automatic positioning component. Our solution can decrease the surgical setup time and eliminate the need to repositioning the robot mid-procedure and is suitable both for the preoperative planning step using VR and in the operating room-running on an AR headset., (© 2023. The Author(s).)

Purpose: Up to date, there has been a lack of software infrastructure to connect 3D Slicer to any augmented reality (AR) device. This work describes a novel connection approach using Microsoft HoloLens 2 and OpenIGTLink, with a demonstration in pedicle screw placement planning., Methods: We developed an AR application in Unity that is wirelessly rendered onto Microsoft HoloLens 2 using Holographic Remoting. Simultaneously, Unity connects to 3D Slicer using the OpenIGTLink communication protocol. Geometrical transform and image messages are transferred between both platforms in real time. Through the AR glasses, a user visualizes a patient's computed tomography overlaid onto virtual 3D models showing anatomical structures. We technically evaluated the system by measuring message transference latency between the platforms. Its functionality was assessed in pedicle screw placement planning. Six volunteers planned pedicle screws' position and orientation with the AR system and on a 2D desktop planner. We compared the placement accuracy of each screw with both methods. Finally, we administered a questionnaire to all participants to assess their experience with the AR system., Results: The latency in message exchange is sufficiently low to enable real-time communication between the platforms. The AR method was non-inferior to the 2D desktop planner, with a mean error of 2.1 ± 1.4 mm. Moreover, 98% of the screw placements performed with the AR system were successful, according to the Gertzbein-Robbins scale. The average questionnaire outcomes were 4.5/5., Conclusions: Real-time communication between Microsoft HoloLens 2 and 3D Slicer is feasible and supports accurate planning for pedicle screw placement., (© 2023. The Author(s).)

Background: Laparoscopic anatomical Segment 8 (S8) resection is a highly challenging hepatectomy. Augmented reality navigation (ARN), which could be combined with indocyanine green (ICG) fluorescence imaging, has been applied in various complex liver resections and may also be applied in laparoscopic anatomical S8 resection. However, no study has explored how to apply ARN plus ICG fluorescence imaging (ARN-FI) in laparoscopic anatomical S8 resection, or explored its accuracy., Patients and Methods: This study is a post hoc analysis that included 31 patients undergoing laparoscopic anatomical S8 resection from the clinical NaLLRFI trial, and the resected liver volume was measured in each patient. The perioperative parameters of safety and feasibility, as well as the accuracy analysis outcomes were compared., Results: There were 16 patients in the ARN-FI group and 15 patients underwent conventional laparoscopic hepatectomy without ARN or fluorescence imaging (non-ARN-FI group). There was no significant difference in baseline characteristics between the two groups. Compared with the non-ARN-FI group, the ARN-FI group had lower intraoperative bleeding (median 125 vs. 300 mL, P = 0.003). No significant difference was observed in other postoperative short-term outcomes. Accuracy analysis indicated that the actual resected liver volume (ARLV) in the ARN-FI group was more accurate., Conclusions: ARN-FI was associated with less intraoperative bleeding and more accurate resection volume. These techniques may address existing challenges and provide rational guidance for laparoscopic anatomical S8 resection., (© 2023. Society of Surgical Oncology.)

We present a novel method for intraoperative patient-to-image registration by learning Expected Appearances. Our method uses preoperative imaging to synthesize patient-specific expected views through a surgical microscope for a predicted range of transformations. Our method estimates the camera pose by minimizing the dissimilarity between the intraoperative 2D view through the optical microscope and the synthesized expected texture. In contrast to conventional methods, our approach transfers the processing tasks to the preoperative stage, reducing thereby the impact of low-resolution, distorted, and noisy intraoperative images, that often degrade the registration accuracy. We applied our method in the context of neuronavigation during brain surgery. We evaluated our approach on synthetic data and on retrospective data from 6 clinical cases. Our method outperformed state-of-the-art methods and achieved accuracies that met current clinical standards.

Introduction: The learning curve in minimally invasive surgery (MIS) is steep compared to open surgery. One of the reasons is that training in the operating room in MIS is mainly limited to verbal instructions. The iSurgeon telestration device with augmented reality (AR) enables visual instructions, guidance, and feedback during MIS. This study aims to compare the effects of the iSurgeon on the training of novices performing repeated laparoscopic cholecystectomy (LC) on a porcine liver compared to traditional verbal instruction methods., Methods: Forty medical students were randomized into the iSurgeon and the control group. The iSurgeon group performed 10 LCs receiving interactive visual guidance. The control group performed 10 LCs receiving conventional verbal guidance. The performance assessment using Objective Structured Assessments of Technical Skills (OSATS) and Global Operative Assessment of Laparoscopic Skills (GOALS) scores, the total operating time, and complications were compared between the two groups., Results: The iSurgeon group performed LCs significantly better (global GOALS 17.3 ± 2.6 vs. 16 ± 2.6, p ≤ 0.001, LC specific GOALS 7 ± 2 vs. 5.9 ± 2.1, p ≤ 0.001, global OSATS 25.3 ± 4.3 vs. 23.5 ± 3.9, p ≤ 0.001, LC specific OSATS scores 50.8 ± 11.1 vs. 41.2 ± 9.4, p ≤ 0.001) compared to the control group. The iSurgeon group had significantly fewer intraoperative complications in total (2.7 ± 2.0 vs. 3.6 ± 2.0, p ≤ 0.001) than the control group. There was no difference in operating time (79.6 ± 25.7 vs. 84.5 ± 33.2 min, p = 0.087)., Conclusion: Visual guidance using the telestration device with AR, iSurgeon, improves performance and lowers the complication rates in LCs in novices compared to conventional verbal expert guidance., (© 2023. The Author(s).)

Background: Right hemi-hepatectomy plus total caudate lobectomy is the appropriate procedure for type IIIa or partial type II pCCA. However, the laparoscopic implementation of this procedure remains technically challenging, especially hilar vascular dissection and en bloc resection of the total caudate lobe. Augmented reality navigation can provide intraoperative navigation to enhance visualization of invisible hilar blood vessels and guide the parenchymal transection plane., Methods: Eleven patients who underwent laparoscopic right hemi-hepatectomy plus total caudate lobectomy from January 2021 to January 2023 were enrolled in this study. Augmented reality navigation technology and the anterior approach were utilized in this operation. Routine operative and short-term postoperative outcomes were assessed to evaluate the feasibility of the novel navigation method in this operation., Results: Right hemi-hepatectomy plus total caudate lobectomy was successfully performed in all 11 enrolled patients. Among the 11 patients, the mean operation time was 454.5 ± 25.0 min and the mean estimated blood loss was 209.1 ± 56.1 ml. Negative surgical margins were achieved in all patients. The postoperative course of all the patients was uneventful, and the mean length of postoperative hospital stay was 10.5 ± 1.2 days., Conclusion: Laparoscopic right hemi-hepatectomy plus total caudate lobectomy via the anterior approach may be feasible and safe for pCCA with the assistance of augmented reality navigation., (© 2023. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Purpose: In "human teleoperation" (HT), mixed reality (MR) and haptics are used to tightly couple an expert leader to a human follower [1]. To determine the feasibility of HT for teleultrasound, we quantify the ability of humans to track a position and/or force trajectory via MR cues. The human response time, precision, frequency response, and step response were characterized, and several rendering methods were compared., Methods: Volunteers (n=11) performed a series of tasks as the follower in our HT system. The tasks involved tracking pre-recorded series of motions and forces while pose and force were recorded. The volunteers then performed frequency response tests and filled out a questionnaire., Results: Following force and pose simultaneously was more difficult but did not lead to significant performance degradation versus following one at a time. On average, subjects tracked positions, orientations, and forces with RMS tracking errors of [Formula: see text] mm, [Formula: see text], [Formula: see text] N, steady-state errors of [Formula: see text] mm, [Formula: see text] N, and lags of [Formula: see text] ms, respectively. Performance decreased with input frequency, depending on the input amplitude., Conclusion: Teleoperating a person through MR is a novel concept with many possible applications. However, it is unknown what performance is achievable and which approaches work best. This paper thus characterizes human tracking ability in MR HT for teleultrasound, which is important for designing future tightly coupled guidance and training systems using MR., (© 2023. CARS.)

Objective: Augmented reality (AR) and virtual reality (VR) facilitate motor learning by enabling the practice of task-specific activities in a rich environment. Therefore, AR and VR gait training may improve balance and gait in Parkinson's Disease (PD)., Methods: Thirty patients with PD were randomly divided into study (n = 15) and control (n = 15) groups. The study group was given AR and VR gait training combined with conventional training. The control group was given conventional training only. The training was applied to both groups 3 days a week for 6 weeks. Motor symptoms with the Unified Parkinson Disease Rating Scale-Motor Examination (UPDRS-III), balance with posturography and Berg Balance Scale (BBS), perceived balance confidence with Activity-Specific Balance Confidence Scale (ABC), gait with spatio-temporal gait analysis, and functional mobility with Timed Up and Go Test (TUG) were assessed., Results: At the end of the study; UPDRS-III, posturography measurements, BBS, ABC, spatio-temporal gait parameters, and TUG improved in the study group (p < 0.05), while BBS, ABC, and only spatial gait parameters (except for step width) improved in the control group (p < 0.05). There was no change in posturography measurement, temporal gait parameters, and TUG in control group (p > 0.05). When the developed parameters in both groups were compared, the amount of improvement in BBS and ABC was found similar (p > 0.05), while the improvement in the other parameters was found higher in the study group (p < 0.05)., Conclusion: It was concluded that AR and VR gait training provides the opportunity to practice walking with different tasks in increasingly difficult environments, thus improving balance and walking by facilitating motor learning., (© 2022. The Author(s) under exclusive licence to Belgian Neurological Society.)

The primary aim of this systematic review is to investigate the inclusion of autistic individuals in the design process of immersive technologies. This study follows the preferred reporting items for systematic reviews and meta-analyses standards for systematic literature reviews. To ensure the research questions and subsequent stages of the review incorporate pertinent parameters, the problem, interest, context framework has also been employed. Findings highlight that, while early proponents of immersive technology emphasized the importance of user involvement in design of new technology, immaturity of the technology often limited the implementation of direct user input to the design process. Nonetheless, analysis of the literature published between 2002-2022 identified 20 studies in which substantial influence of autistic individuals and stakeholders was found in the design process of immersive technologies. The roles of autistic individuals varied from active co-designers and co-creators to essential contributors in refining prototypes and providing critical feedback, ensuring the final products align with their needs and preferences. Results underscore the need to align research and design of immersive technologies more closely with the priorities and preferences of autistic individuals. Further is needed regarding actively involving autistic individuals in the design and implementation of immersive technology applications. On this basis, we maintain that more inclusive and effective deployment of immersive technologies is needed in order to ensure that resultant technologies are fit for purpose and address the actual needs of the autistic community., (© 2023. The Author(s).)

Purpose: In general minimally invasive surgical procedures, surgeons are tied to 2D visualization, leading to the loss of depth perception. This can lead to large mental load for the surgeons and may be responsible for the long learning curve. To restore the sense of depth, this study investigated the use and benefits of an autostereoscopic (3D) display during a simulated laparoscopic task., Methods: A mixed reality simulator was developed for comparing the performance of participants while using 2D and autostereoscopic 3D visualization. An electromagnetic sensor was mounted on a physical instrument, and its pose was mapped to the virtual instrument. The virtual scene was developed using Simulation Open Framework Architecture (SOFA). Finite element modeling was used to calculate interaction forces, which were then mapped to visual soft tissue deformation., Results: Ten non-expert participants completed a virtual laparoscopic task, where the subjects were asked to contact eighteen target areas distributed on the surface of the vagina, both in 2D and 3D. Results showed an improvement with 3D vision in task completion time (-16%), total traveled distance (-25%) and errors made (-14%). There was no difference in the average contact forces between the vagina and the instrument. Only the difference in time and forces were shown to be statistically significant., Conclusion: Overall, autostereoscopic 3D showed superiority over conventional 2D visualization. The traveled trajectory increased in 2D as the instrument was retracted more between the targets to avoid contact. The 2D and 3D deformation upon contact seems not to contribute differently to force perception. However, the participants only had visual feedback, but no haptic feedback. Therefore, it could be interesting to include haptic feedback in a future study., (© 2023. CARS.)

Purpose: The treatment of pelvic and acetabular fractures remains technically demanding, and traditional surgical navigation systems suffer from the hand-eye mis-coordination. This paper describes a multi-view interactive virtual-physical registration method to enhance the surgeon's depth perception and a mixed reality (MR)-based surgical navigation system for pelvic and acetabular fracture fixation., Methods: First, the pelvic structure is reconstructed by segmentation in a preoperative CT scan, and an insertion path for the percutaneous LC-II screw is computed. A custom hand-held registration cube is used for virtual-physical registration. Three strategies are proposed to improve the surgeon's depth perception: vertices alignment, tremble compensation and multi-view averaging. During navigation, distance and angular deviation visual cues are updated to help the surgeon with the guide wire insertion. The methods have been integrated into an MR module in a surgical navigation system., Results: Phantom experiments were conducted. Ablation experimental results demonstrated the effectiveness of each strategy in the virtual-physical registration method. The proposed method achieved the best accuracy in comparison with related works. For percutaneous guide wire placement, our system achieved a mean bony entry point error of 2.76 ± 1.31 mm, a mean bony exit point error of 4.13 ± 1.74 mm, and a mean angular deviation of 3.04 ± 1.22°., Conclusions: The proposed method can improve the virtual-physical fusion accuracy. The developed MR-based surgical navigation system has clinical application potential. Cadaver and clinical experiments will be conducted in future., (© 2023. CARS.)

Brain-computer interface (BCI), an emerging technology that facilitates communication between brain and computer, has attracted a great deal of research in recent years. Researchers provide experimental results demonstrating that BCI can restore the capabilities of physically challenged people, hence improving the quality of their lives. BCI has revolutionized and positively impacted several industries, including entertainment and gaming, automation and control, education, neuromarketing, and neuroergonomics. Notwithstanding its broad range of applications, the global trend of BCI remains lightly discussed in the literature. Understanding the trend may inform researchers and practitioners on the direction of the field, and on where they should invest their efforts more. Noting this significance, we have analyzed 25,336 metadata of BCI publications from Scopus to determine advancement of the field. The analysis shows an exponential growth of BCI publications in China from 2019 onwards, exceeding those from the United States that started to decline during the same period. Implications and reasons for this trend are discussed. Furthermore, we have extensively discussed challenges and threats limiting exploitation of BCI capabilities. A typical BCI architecture is hypothesized to address two prominent BCI threats, privacy and security, as an attempt to make the technology commercially viable to the society., (© 2023. Springer-Verlag GmbH Germany, part of Springer Nature.)

High-quality gynecologist and midwife training is particularly relevant to limit medical complications and reduce maternal and fetal morbimortalities. Physical and virtual training simulators have been developed. However, physical simulators offer a simplified model and limited visualization of the childbirth process, while virtual simulators still lack a realistic interactive system and are generally limited to imposed predefined gestures. Objective performance assessment based on the simulation numerical outcomes is still not at hand. In the present work, we developed a virtual childbirth simulator based on the Mixed-Reality (MR) technology coupled with HyperMSM (Hyperelastic Mass-Spring Model) formulation for real-time soft-tissue deformations, providing intuitive user interaction with the virtual physical model and a quantitative assessment to enhance the trainee's gestures. Microsoft HoloLens 2 was used and the MR simulator was developed including a complete holographic obstetric model. A maternal pelvis system model of a pregnant woman (including the pelvis bone, the pelvic floor muscles, the birth canal, the uterus, and the fetus) was generated, and HyperMSM formulation was applied to simulate the soft tissue deformations. To induce realistic reactions to free gestures, the virtual replicas of the user's detected hands were introduced into the physical simulation and were associated with a contact model between the hands and the HyperMSM models. The gesture of pulling any part of the virtual models with two hands was also implemented. Two labor scenarios were implemented within the MR childbirth simulator: physiological labor and forceps-assisted labor. A scoring system for the performance assessment was included based on real-time biofeedback. As results, our developed MR simulation application was developed in real-time with a refresh rate of 30-50 FPS on the HoloLens device. HyperMSM model was validated using FE outcomes: high correlation coefficients of [0.97-0.99] and weighted root mean square relative errors of 9.8% and 8.3% were obtained for the soft tissue displacement and energy density respectively. Experimental tests showed that the implemented free-user interaction system allows to apply the correct maneuvers (in particular the "Viennese" maneuvers) during the labor process, and is capable to induce a truthful reaction of the model. Obtained results confirm also the possibility of using our simulation's outcomes to objectively evaluate the trainee's performance with a reduction of 39% for the perineal strain energy density and 5.6 mm for the vertical vaginal diameter when the "Viennese" technique is applied. This present study provides, for the first time, an interactive childbirth simulator with an MR immersive experience with direct free-hand interaction, real-time soft-tissue deformation feedback, and an objective performance assessment based on numerical outcomes. This offers a new perspective for enhancing next-generation training-based obstetric teaching. The used models of the maternal pelvic system and the fetus will be enhanced, and more delivery scenarios (e.g. instrumental delivery, breech delivery, shoulder dystocia) will be designed and integrated. The third stage of labor will be also investigated to include the delivery of the placenta, and the clamping and cutting of the umbilical cord., (© 2023. International Federation for Medical and Biological Engineering.)

Purpose: Previous work has demonstrated the high accuracy of augmented reality (AR) head-mounted displays for pedicle screw placement in spinal fusion surgery. An important question that remains unanswered is how pedicle screw trajectories should be visualized in AR to best assist the surgeon., Methodology: We compared five AR visualizations displaying the drill trajectory via Microsoft HoloLens 2 with different configurations of abstraction level (abstract or anatomical), position (overlay or small offset), and dimensionality (2D or 3D) against standard navigation on an external screen. We tested these visualizations in a study with 4 expert surgeons and 10 novices (residents in orthopedic surgery) on lumbar spine models covered by Plasticine. We assessed trajectory deviations ([Formula: see text]) from the preoperative plan, dwell times (%) on areas of interest, and the user experience., Results: Two AR visualizations resulted in significantly lower trajectory deviations (mixed-effects ANOVA, p<0.0001 and p<0.05) compared to standard navigation, whereas no significant differences were found between participant groups. The best ratings for ease of use and cognitive load were obtained with an abstract visualization displayed peripherally around the entry point and with a 3D anatomical visualization displayed with some offset. For visualizations displayed with some offset, participants spent on average only 20% of their time examining the entry point area., Conclusion: Our results show that real-time feedback provided by navigation can level task performance between experts and novices, and that the design of a visualization has a significant impact on task performance, visual attention, and user experience. Both abstract and anatomical visualizations can be suitable for navigation when not directly occluding the execution area. Our results shed light on how AR visualizations guide visual attention and the benefits of anchoring information in the peripheral field around the entry point., (© 2023. The Author(s).)

Purpose: Laparoscopic liver resection is a minimal invasive surgery. Augmented reality can map preoperative anatomy information extracted from computed tomography to the intraoperative liver surface reconstructed from stereo 3D laparoscopy. However, liver surface registration is particularly challenging as the intraoperative surface is only partially visible and suffers from large liver deformations due to pneumoperitoneum. This study proposes a deep learning-based robust point cloud registration network., Methods: This study proposed a low overlap liver surface registration algorithm combining local mixed features and global features of point clouds. A learned overlap mask is used to filter the non-overlapping region of the point cloud, and a network is used to predict the overlapping region threshold to regulate the training process., Results: We validated the algorithm on the DePoLL (the Deformable Porcine Laparoscopic Liver) dataset. Compared with the baseline method and other state-of-the-art registration methods, our method achieves minimum target registration error (TRE) of 19.9 ± 2.7 mm., Conclusion: The proposed point cloud registration method uses the learned overlapping mask to filter the non-overlapping areas in the point cloud, then the extracted overlapping area point cloud is registered according to the mixed features and global features, and this method is robust and efficient in low-overlap liver surface registration., (© 2023. CARS.)

Background: Laparoscopic right anterior sectionectomy (LRAS) is an attractive surgical option for tumors in the right anterior section (RAS), which can remove tumor-bearing segments while sparing more normal liver tissue 1 . However, the definition of the resection plane, the guidance during the resection, and the protection of the right posterior hepatic duct are still the key points of this procedure 2 . Our center attempted to use augmented reality navigation system and indocyanine green fluorescence (ICG) imaging technology to solve these difficulties 3 , and reported this in LRAS for the first time., Methods: A 47-year-old female was admitted to our institution for a tumor in the RAS. Therefore, LRAS was performed. First, a virtual liver segment projection combined with the ischemic line caused by the occlusion of RAS blood flow was used to mark the RAS boundary, and it was confirmed using the ICG negative staining. Then, during the parenchymal transection, the precise resection plane was guided assisted by the ICG fluorescence imaging system. In addition, the right anterior Glissonean pedicle (RAGP) was divided using a linear stapler after confirming the spatial relationship of the bile duct using ICG fluorescence imaging., Results: The operation lasted 360 min with 100 mL of intraoperative blood loss. There were no postoperative complications, and the patient was discharged after 8 days., Conclusion: The augmented reality navigation system plus ICG imaging can make LRAS more precisely and safely., (© 2023. The Society for Surgery of the Alimentary Tract.)

To evaluate the feasibility and accuracy of AR-assisted pedicle screw placement using a new intraoperative rapid registration method of combining preoperative CT scanning and intraoperative C-arm 2D fluoroscopy in cadavers. Five cadavers with intact thoracolumbar spines were employed in this study. Intraoperative registration was performed using anteroposterior and lateral views of preoperative CT scanning and intraoperative 2D fluoroscopic images. Patient-specific targeting guides were used for pedicle screw placement from Th1-L5, totaling 166 screws. Instrumentation for each side was randomized (augmented reality surgical navigation (ARSN) vs. C-arm) with an equal distribution of 83 screws in each group. CT was performed to evaluate the accuracy of both techniques by assessing the screw positions and the deviations between the inserted screws and planned trajectories. Postoperative CT showed that 98.80% (82/83) screws in ARSN group and 72.29% (60/83) screws in C-arm group were within the 2-mm safe zone (p < 0.001). The mean time for instrumentation per level in ARSN group was significantly shorter than that in C-arm group (56.17 ± 3.33 s vs. 99.22 ± 9.03 s, p < 0.001). The overall intraoperative registration time was 17.2 ± 3.5 s per segment. AR-based navigation technology can provide surgeons with accurate guidance of pedicle screw insertion and save the operation time by using the intraoperative rapid registration method of combining preoperative CT scanning and intraoperative C-arm 2D fluoroscopy., (© 2023. The Author(s) under exclusive licence to Society for Imaging Informatics in Medicine.)