Search

Neural fields are neural networks which map coordinates to a desired signal. When a neural field should jointly model multiple signals, and not memorize only one, it needs to be conditioned on a latent code which describes the signal at hand. Despite being an important aspect, there has been little research on conditioning strategies for neural fields. In this work, we explore the use of neural fields as decoders for 2D semantic segmentation. For this task, we compare three conditioning methods, simple concatenation of the latent code, Feature Wise Linear Modulation (FiLM), and Cross-Attention, in conjunction with latent codes which either describe the full image or only a local region of the image. Our results show a considerable difference in performance between the examined conditioning strategies. Furthermore, we show that conditioning via Cross-Attention achieves the best results and is competitive with a CNN-based decoder for semantic segmentation., Comment: 13 pages, 4 figures, submitted to ICANN 2023

Explaining the behaviour of intelligent agents learned by reinforcement learning (RL) to humans is challenging yet crucial due to their incomprehensible proprioceptive states, variational intermediate goals, and resultant unpredictability. Moreover, one-step explanations for RL agents can be ambiguous as they fail to account for the agent's future behaviour at each transition, adding to the complexity of explaining robot actions. By leveraging abstracted actions that map to task-specific primitives, we avoid explanations on the movement level. To further improve the transparency and explainability of robotic systems, we propose an explainable Q-Map learning framework that combines reward decomposition (RD) with abstracted action spaces, allowing for non-ambiguous and high-level explanations based on object properties in the task. We demonstrate the effectiveness of our framework through quantitative and qualitative analysis of two robotic scenarios, showcasing visual and textual explanations, from output artefacts of RD explanations, that are easy for humans to comprehend. Additionally, we demonstrate the versatility of integrating these artefacts with large language models (LLMs) for reasoning and interactive querying., Comment: https://lukaswill.github.io

Collaborative robotic needle insertions have the potential to improve placement accuracy and safety, e.g., during epidural anesthesia. Epidural anesthesia provides effective regional pain management but can lead to serious complications, such as nerve injury or cerebrospinal fluid leakage. Robotic assistance might prevent inadvertent puncture by providing haptic feedback to the physician. Haptic feedback can be realized on the basis of force measurements at the needle. However, contact should be avoided for delicate structures. We propose a proximity-based method to provide feedback prior to contact. We measure the distance to boundary layers, visualize the proximity for the operator and further feedback it as a haptic resistance. We compare our approach to haptic feedback based on needle forces and visual feedback without haptics. Participants are asked to realize needle insertions with each of the three feedback modes. We use phantoms that mimic the structures punctured during epidural anesthesia. We show that visual feedback improves needle placement, but only proximity-based haptic feedback reduces accidental puncture. The puncture rate is 62% for force-based haptic feedback, 60% for visual feedback and 6% for proximity-based haptic feedback. Final needle placement inside the epidural space is achieved in 38%, 70% and 96% for force-based haptic, visual and proximity-based haptic feedback, respectively. Our results suggest that proximity-based haptic feedback could improve needle placement safety in the context of epidural anesthesia.

Understanding the underlying pathology in different tissues and organs is crucial when fighting pandemics like COVID-19. During conventional autopsy, large tissue sample sets of multiple organs can be collected from cadavers. However, direct contact with an infectious corpse is associated with the risk of disease transmission and relatives of the deceased might object to a conventional autopsy. To overcome these drawbacks, we consider minimally invasive autopsies with robotic needle placement as a practical alternative. One challenge in needle based biopsies is avoidance of dense obstacles, including bones or embedded medical devices such as pacemakers. We demonstrate an approach for automated planning and visualising suitable needle insertion points based on computed tomography (CT) scans. Needle paths are modeled by a line between insertion and target point and needle insertion path occlusion from obstacles is determined by using central projections from the biopsy target to the surface of the skin. We project the maximum and minimum CT attenuation, insertion depth, and standard deviation of CT attenuation along the needle path and create two-dimensional intensity-maps projected on the skin. A cost function considering these metrics is introduced and minimized to find an optimal biopsy needle path. Furthermore, we disregard insertion points without sufficient room for needle placement. For visualisation, we display the color-coded cost function so that suitable points for needle insertion become visible. We evaluate our system on 10 post mortem CTs with six biopsy targets in abdomen and thorax annotated by medical experts. For all patients and targets an optimal insertion path is found. The mean distance to the target ranges from (49.9 ± 12.9)mm for the spleen to (90.1 ± 25.8)mm for the pancreas.

In pathology and legal medicine, the histopathological and microbiological analysis of tissue samples from infected deceased is a valuable information for developing treatment strategies during a pandemic such as COVID-19. However, a conventional autopsy carries the risk of disease transmission and may be rejected by relatives. We propose minimally invasive biopsy with robot assistance under CT guidance to minimize the risk of disease transmission during tissue sampling and to improve accuracy. A flexible robotic system for biopsy sampling is presented, which is applied to human corpses placed inside protective body bags. An automatic planning and decision system estimates optimal insertion point. Heat maps projected onto the segmented skin visualize the distance and angle of insertions and estimate the minimum cost of a puncture while avoiding bone collisions. Further, we test multiple insertion paths concerning feasibility and collisions. A custom end effector is designed for inserting needles and extracting tissue samples under robotic guidance. Our robotic post-mortem biopsy (RPMB) system is evaluated in a study during the COVID-19 pandemic on 20 corpses and 10 tissue targets, 5 of them being infected with SARS-CoV-2. The mean planning time including robot path planning is (5.72+-1.67) s. Mean needle placement accuracy is (7.19+-4.22) mm.

Needles are key tools to realize minimally invasive interventions. Physicians commonly rely on subjectively perceived insertion forces at the distal end of the needle when advancing the needle tip to the desired target. However, detecting tissue transitions at the distal end of the needle is difficult since the sensed forces are dominated by shaft forces. Disentangling insertion forces has the potential to substantially improve needle placement accuracy.We propose a collaborative system for robotic needle insertion, relaying haptic information sensed directly at the needle tip to the physician by haptic feedback through a light weight robot. We integrate optical fibers into medical needles and use optical coherence tomography to image a moving surface at the tip of the needle. Using a convolutional neural network, we estimate forces acting on the needle tip from the optical coherence tomography data. We feed back forces estimated at the needle tip for real time haptic feedback and robot control. When inserting the needle at constant velocity, the force change estimated at the tip when penetrating tissue layers is up to 94% between deep tissue layers compared to the force change at the needle handle of 2.36 %. Collaborative needle insertion results in more sensible force change at tissue transitions with haptic feedback from the tip (49.79 ± 25.51)% compared to the conventional shaft feedback (15.17 ± 15.92) %. Tissue transitions are more prominent when utilizing forces estimated at the needle tip compared to the forces at the needle shaft, indicating that a more informed advancement of the needle is possible with our system.

In pathology and legal medicine, the histopathological and microbiological analysis of tissue samples from infected deceased is a valuable information for developing treatment strategies during a pandemic such as COVID-19. However, a conventional autopsy carries the risk of disease transmission and may be rejected by relatives. We propose minimally invasive biopsy with robot assistance under CT guidance to minimize the risk of disease transmission during tissue sampling and to improve accuracy. A flexible robotic system for biopsy sampling is presented, which is applied to human corpses placed inside protective body bags. An automatic planning and decision system estimates optimal insertion point. Heat maps projected onto the segmented skin visualize the distance and angle of insertions and estimate the minimum cost of a puncture while avoiding bone collisions. Further, we test multiple insertion paths concerning feasibility and collisions. A custom end effector is designed for inserting needles and extracting tissue samples under robotic guidance. Our robotic post-mortem biopsy (RPMB) system is evaluated in a study during the COVID-19 pandemic on 20 corpses and 10 tissue targets, 5 of them being infected with SARS-CoV-2. The mean planning time including robot path planning is (5.72+-1.67) s. Mean needle placement accuracy is (7.19+-4.22) mm.

Purpose: Needle placement is a challenging problem for applications such as biopsy or brachytherapy. Tip force sensing can provide valuable feedback for needle navigation inside the tissue. For this purpose, fiber-optical sensors can be directly integrated into the needle tip. Optical coherence tomography (OCT) can be used to image tissue. Here, we study how to calibrate OCT to sense forces, e.g., during robotic needle placement. Methods: We investigate whether using raw spectral OCT data without a typical image reconstruction can improve a deep learning-based calibration between optical signal and forces. For this purpose, we consider three different needles with a new, more robust design which are calibrated using convolutional neural networks (CNNs). We compare training the CNNs with the raw OCT signal and the reconstructed depth profiles. Results: We find that using raw data as an input for the largest CNN model outperforms the use of reconstructed data with a mean absolute error of 5.81 mN compared to 8.04 mN. Conclusions: We find that deep learning with raw spectral OCT data can improve learning for the task of force estimation. Our needle design and calibration approach constitute a very accurate fiber-optical sensor for measuring forces at the needle tip.

© 2020 Martin Gromniak et al., published by De Gruyter, Berlin/Boston 2020. Forensic autopsies include a thorough examination of the corpse to detect the source or alleged manner of death as well as to estimate the time since death. However, a full autopsy may be not feasible due to limited time, cost or ethical objections by relatives. Hence, we propose an automated minimal invasive needle biopsy system with a robotic arm, which does not require any online calibrations during a procedure. The proposed system can be easily integrated into the workflow of a forensic biopsy since the robot can be flexibly positioned relative to the corpse. With our proposed system, we performed needle insertions into wax phantoms and livers of two corpses and achieved an accuracy of 4.34 ± 1.27 mm and 10.81 ± 4.44 mm respectively.

Optical tracking systems are widely used, for example, to navigate medical interventions. Typically, they require the presence of known geometrical structures, the placement of artificial markers, or a prominent texture on the target’s surface. In this work, we propose a 6D tracking approach employing volumetric optical coherence tomography (OCT) images. OCT has a micrometer-scale resolution and employs near-infrared light to penetrate few millimeters into, for example, tissue. Thereby, it provides sub-surface information which we use to track arbitrary targets, even with poorly structured surfaces, without requiring markers. Our proposed system can shift the OCT’s field-of-view in space and uses an adaptive correlation filter to estimate the motion at multiple locations on the target. This allows one to estimate the target’s position and orientation. We show that our approach is able to track translational motion with root-mean-squared errors below 0.25 mm and in-plane rotations with errors below 0.3°. For out-of-plane rotations, our prototypical system can achieve errors around 0.6°.

Electroencephalography (EEG) allows for source measurement of electrical brain activity. Particularly for inverse localization, the electrode positions on the scalp need to be known. Often, systems such as optical digitizing scanners are used for accurate localization with a stylus. However, the approach is time-consuming as each electrode needs to be scanned manually and the scanning systems are expensive. We propose using an RGBD camera to directly track electrodes in the images using deep learning methods. Studying and evaluating deep learning methods requires large amounts of labeled data. To overcome the time-consuming data annotation, we generate a large number of ground-truth labels using a robotic setup. We demonstrate that deep learning-based electrode detection is feasible with a mean absolute error of 5.69 +- 6.1mm and that our annotation scheme provides a useful environment for studying deep learning methods for electrode detection., Comment: Accepted at the CURAC 2019 Conference

Automatic motion compensation and adjustment of an intraoperative imaging modality's field of view is a common problem during interventions. Optical coherence tomography (OCT) is an imaging modality which is used in interventions due to its high spatial resolution of few micrometers and its temporal resolution of potentially several hundred volumes per second. However, performing motion compensation with OCT is problematic due to its small field of view which might lead to tracked objects being lost quickly. We propose a novel deep learning-based approach that directly learns input parameters of motors that move the scan area for motion compensation from optical coherence tomography volumes. We design a two-path 3D convolutional neural network (CNN) architecture that takes two volumes with an object to be tracked as its input and predicts the necessary motor input parameters to compensate the object's movement. In this way, we learn the calibration between object movement and system parameters for motion compensation with arbitrary objects. Thus, we avoid error-prone hand-eye calibration and handcrafted feature tracking from classical approaches. We achieve an average correlation coefficient of 0.998 between predicted and ground-truth motor parameters which leads to sub-voxel accuracy. Furthermore, we show that our deep learning model is real-time capable for use with the system's high volume acquisition frequency., Comment: Accepted at SPIE: Medical Imaging 2019

Head pose estimation and tracking is useful in variety of medical applications. With the advent of RGBD cameras like Kinect, it has become feasible to do markerless tracking by estimating the head pose directly from the point clouds. One specific medical application is robot assisted transcranial magnetic stimulation (TMS) where any patient motion is compensated with the help of a robot. For increased patient comfort, it is important to track the head without markers. In this regard, we address the head pose estimation problem using two different approaches. In the first approach, we build upon the more traditional approach of model based head tracking, where a head model is morphed according to the particular head to be tracked and the morphed model is used to track the head in the point cloud streams. In the second approach, we propose a new multi-scale convolutional neural network architecture for more accurate pose regression. Additionally, we outline a systematic data set acquisition strategy using a head phantom mounted on the robot and ground-truth labels generated using a highly accurate tracking system., Comment: Presented at CURAC 2018 conference