Search

Automatic surgical skill assessment in robotic surgery based on video data is essential in facilitating faster learning curves for trainees while relieving expert surgeons from the time- and cost-intensive feedback process. Recent years have shown several advancements in this area by utilizing deep learning. While current research focuses on novel architectures, the influence of video-preprocessing on their performance remains unknown. In this work, we present the first investigation on the influence of video-preprocessing on deep learning-based surgical skill assessment. Thus, we integrated four preprocessing modules, i.e. Deblurring, Segmentbased Sampling, Optical Flow and the Combination of all of them, into skill assessment on the JIGSAWS dataset using a well-established network architecture. Despite all single preprocessing steps showing no clear improvement, the Combination of all steps showed higher median performance and lower variance. Furthermore, we performed frame-wise investigations on the influence of optical flow artifacts and their reduction in the combined setting. Our results highlight the potential of well-calibrated video-preprocessing for automatic surgical skill assessment.

Ultrasound (US) imaging, in contrast to other image guidance techniques, offers the distinct advantage of providing volumetric image data in real-time (4D) without using ionizing radiation. The goal of this study was to perform the first quantitative comparison of three different 4D US systems with fast matrix array probes and real-time data streaming regarding their target tracking accuracy and system latency. Sinusoidal motion of varying amplitudes and frequencies was used to simulate breathing motion with a robotic arm and a static US phantom. US volumes and robot positions were acquired online and stored for retrospective analysis. A template matching approach was used for target localization in the US data. Target motion measured in US was compared to the reference trajectory performed by the robot to determine localization accuracy and system latency. Using the robotic setup, all investigated 4D US systems could detect a moving target with sub-millimeter accuracy. However, especially high system latency increased tracking errors substantially and should be compensated with prediction algorithms for respiratory motion compensation.

In radiation therapy of abdominal targets, optimal tumor irradiation can be challenging due to intrafractional motion. Current target localization methods are mainly indirect, surrogate-based and the patient is exposed to additional radiation due to X-ray imaging. In contrast, 4D ultrasound (4DUS) imaging provides volumetric images of soft tissue tumors in real-time without ionizing radiation, facilitating a non-invasive, direct tracking method. In this study, the target was defined by features located in its local neighborhood. Features were extracted using the FAST detector and the BRISK descriptor, which were extended to 3D. To account for anatomical variability, a feature library was generated that contains manually annotated target information and relative locations of the features. During tracking, features were extracted from the current 4DUS volume and compared to the feature library. Recognized features are used to estimate feature position and shape. The developed method was evaluated in 4DUS sequences of the liver of three healthy subjects. For each dataset, a target was defined and manually contoured in a training and a test sequence. Training was used for library creation, the test sequence for target tracking. The target estimations are compared to the annotations to quantify a tracking error. The results show that binary feature libraries can be used for robust target localization in 4DUS data of the liver and could potentially serve as a tracking method less sensitive to target deformation.

4D ultrasound (4D US) is gaining relevance as a tracking method in radiation therapy (RT) with modern matrix array probes offering new possibilities for real-time target detection. However, for clinical implementation of USguided RT, image quality, volumetric framerate and artifacts caused by the probe’s presence during planning and / or setup computed tomography (CT) must be quantified. We compared three diagnostic 4D US systems with matrix array probes using a commercial wire phantom to measure spatial resolution as well as a calibration and a torso phantom to assess different image quality metrics. CT artifacts were quantified in the torso phantom by calculating the total variation and percentage of affected voxels between a reference CT scan and CT scans with probes in place. We found that state-of-the-art 4D US systems with small probes can fit inside the CT bore and cause fewer metal artifacts than larger probes. US image quality varies between systems and is task-dependent. Volume sizes and framerates are much higher than the commercial guidance solution for US-guided RT, warranting further investigation regarding clinical performance for image guidance.

Ultrasound is being increasingly investigated for real-time target localization in image-guided interventions. Yet, in-vivo validation remains challenging due to the difficulty to obtain a reliable ground truth. For this purpose, real-time volumetric (4D) ultrasound imaging was performed simultaneously with electromagnetic localization of three wireless transponders implanted in the liver of a radiotherapy patient. 4D ultrasound and electromagnetic tracking were acquired at framerates of 12Hz and 8Hz, respectively, during free breathing over 8 min following treatment. The electromagnetic antenna was placed directly above and the ultrasound probe on the right side of the patient to visualize the liver transponders. It was possible to record 25.7 s of overlapping ultrasound and electromagnetic position data of one transponder. Good spatial alignment with 0.6 mm 3D root-mean-square error between both traces was achieved using a rigid landmark transform. However, data acquisition was impaired since the electromagnetic tracking highly influenced the ultrasound equipment and vice versa. High intensity noise streaks appeared in the ultrasound scan lines irrespective of the chosen frequency (1.7-3.3 MHz, 2/4 MHz harmonic). To allow for target visualization and tracking in the ultrasound volumes despite the artefacts, an online filter was designed where corrupted pixels in the newest ultrasound frame were replaced with non-corrupted pixels from preceding frames. Aside from these artefacts, the recorded electromagnetic tracking data was fragmented and only the transponder closest to the antenna could be detected over a limited period of six consecutive breathing cycles. This problem was most likely caused by interference from the metal holder of the ultrasound probe and was solved in a subsequent experiment using a 3D-printed non-metal probe fixation. Real-time wireless electromagnetic tracking was compared with 4D ultrasound imaging in-vivo for the first time. For stable tracking, large metal components need to be avoided during data acquisition and ultrasound filtering is required.

Real-time target localization with ultrasound holds high potential for image guidance and motion compensation in radiosurgery due to its non-invasive image acquisition free from ionizing radiation. However, a two-step localization has to be performed when integrating ultrasound into the existing radiosurgery workflow. In addition to target localization inside the ultrasound volume, the probe itself has to be localized in order to transform the target position into treatment room coordinates. By adapting existing camera calibration tools, we have developed a method to extend the stereoscopic X-ray tracking system of a radiosurgery platform in order to locate objects such as marker geometries with six degrees of freedom. The calibration was performed with 0.1 mm reprojection error. By using the full area of the flat-panel detectors without pre-processing the extended software increased the tracking volume and resolution by up to 80%, substantially improving patient localization and marker detectability. Furthermore, marker-tracking showed sub-millimeter accuracy and rotational errors below 0.1°. This demonstrates that the developed extension framework can accurately localize marker geometries using an integrated X-ray system, establishing the link for the integration of real-time ultrasound image guidance into the existing system.

Automation of the image acquisition process via robotic solutions offer a large leap towards resolving ultrasound’s user-dependency. This paper, as part of a larger project aimed to develop a multipurpose 4d-ultrasonic force-sensitive robot for medical applications, focuses on achieving real-time remote visualisation for 4d ultrasound image transfer. This was possible through implementing our software modification on a GE Vivid 7 Dimension workstation, which operates a matrix array probe controlled by a KUKA LBR iiwa 7 7-DOF robotic arm. With the help of robotic positioning and the matrix array probe, fast volumetric imaging of target regions was feasible. By testing ultrasound volumes, which were roughly 880 kB in size, while using gigabit Ethernet connection, a latency of ∼57 ms was achievable for volume transfer between the ultrasound station and a remote client application, which as a result allows a frame count of 17.4 fps. Our modification thus offers for the first time real-time remote visualization, recording and control of 4d ultrasound data, which can be implemented in teleoperation.

Ultrasound (US) guidance is a rapidly growing area in image-guided radiotherapy. For motion compensation, the therapy target needs to be visualized with the US probe to continuously determine its position and adapt for shifts. While US has obvious benefits such as real-time capability and proven safety, one of the main drawbacks to date is its user dependency - high quality results require long years of clinical experience. To provide positioning assistance for the setup of US equipment by non-experts, we developed a visual guidance tool combining real-time US volume and CT visualization in a geometrically calibrated setup. By using a 4D US station with real-time data access and an optical tracking system, we achieved a calibration accuracy of 1.2 mm and a mean 2D contour distance of 1.7 mm between organ boundaries identified in US and CT. With this low calibration error as well as the good visual alignment of the structures, the developed probe positioning tool could be a valuable aid for ultrasound-guided radiotherapy and other interventions by guiding the user to a suitable acoustic window while potentially improving setup reproducibility.