Search

The recently introduced Segment-Anything Model (SAM) has the potential to greatly accelerate the development of segmentation models. However, directly applying SAM to surgical images has key limitations including (1) the requirement of image-specific prompts at test-time, thereby preventing fully automated segmentation, and (2) ineffectiveness due to substantial domain gap between natural and surgical images. In this work, we propose CycleSAM, an approach for one-shot surgical scene segmentation that uses the training image-mask pair at test-time to automatically identify points in the test images that correspond to each object class, which can then be used to prompt SAM to produce object masks. To produce high-fidelity matches, we introduce a novel spatial cycle-consistency constraint that enforces point proposals in the test image to rematch to points within the object foreground region in the training image. Then, to address the domain gap, rather than directly using the visual features from SAM, we employ a ResNet50 encoder pretrained on surgical images in a self-supervised fashion, thereby maintaining high label-efficiency. We evaluate CycleSAM for one-shot segmentation on two diverse surgical semantic segmentation datasets, comprehensively outperforming baseline approaches and reaching up to 50% of fully-supervised performance.

The Science Demilitarized Zone (Science DMZ) is a network environment optimized for scientific applications. A Science DMZ provides an environment mostly free from competing traffic flows and complex security middleware such as firewalls or intrusion detection systems that often impede data transfer performance. The Science DMZ model provides a reference set of network design patterns, tuned hosts and protocol stacks dedicated to large data transfers and streamlined security postures that significantly improve data transfer performance, accelerating scientific collaborations and discovery. Over the past decade, many universities and organizations have adopted this model for their research computing. Despite becoming increasingly popular, there is a lack of quantitative studies comparing such a specialized network to conventional production networks regarding network characteristics and data transfer performance. We strive to answer the following research questions in this study: Does a Science DMZ exhibit significantly different behavior than a general-purpose campus network? Does it improve application performance compared to such general-purpose networks? Through a two-year-long quantitative network measurement study, we find that a Science DMZ exhibits lower latency, higher throughput, and lower jitter behaviors. However, we also see several non-intuitive results. For example, a DMZ may take a longer route to external destinations and experience higher latency than the campus network. While the DMZ model benefits researchers, the benefits are not automatic - careful network tuning based on specific use cases is required to realize the full potential of such infrastructure.

Mechanical stresses and strains in the microstructure of cathode materials evolving during charge/discharge cycles can reduce the long-term stability of intercalation-type alkali-metal-ion batteries. In this context, crystalline compounds exhibiting zero-strain (ZS) behavior are of particular interest. Near-ZS sodiation was experimentally measured in the tetragonal tungsten bronze (TTB) type compound Na$_x$FeF$\mathrm{_3}$. Using a first-principles method based on density functional theory, we investigate the potential of iron-based fluoride compounds with tungsten bronze (TB) structures as ZS cathode materials. Simulations were conducted to study the intercalation of the alkali metal ions Li$\mathrm{^+}$, Na$\mathrm{^+}$, and K$\mathrm{^+}$ into the TTB and two related TB structures of the cubic perovskite (PTB) and hexagonal (HTB) types. We describe compensating local volume effects that can explain the experimentally measured low volume change of Na$_x$FeF$\mathrm{_3}$. We discuss the structural and chemical prerequisites of the host lattice for ZS insertion mechanism for alkali ions in TB structures and present a qualitative descriptor to predict the local volume change, that provides a way for faster screening and discovery of novel ZS battery materials., Comment: 10 pages, 6 figures

In this study, we demonstrated the direct use of an inherently narrowband backward wave in the mid-infrared for CO$_2$ and H$_2$O vapor spectroscopy in ambient air. This wave is generated in a backward wave optical parametric oscillator (BWOPO) pumped by a multi longitudinal mode laser at 1030 nm, eliminating the need for additional spectral narrowing or wavelength stabilization. A full characterization of the source is presented, revealing a central output at 2712 nm, showcasing temperature tuning of -1.77 GHz/K, and achieving an output pulse energy of 2.3 $\mu$J. Novel methods are introduced for measuring the linewidth and wavelength stability using lab air. These approaches demonstrate a narrow output of 43 pm and establish an upper limit of stability at 65 MHz, with no active means of stabilization. These findings underscore the potential of BWOPOs as a robust platform for future DIAL systems.

Self-supervised learning (SSL) approaches have achieved great success when the amount of labeled data is limited. Within SSL, models learn robust feature representations by solving pretext tasks. One such pretext task is contrastive learning, which involves forming pairs of similar and dissimilar input samples, guiding the model to distinguish between them. In this work, we investigate the application of contrastive learning to the domain of medical image analysis. Our findings reveal that MoCo v2, a state-of-the-art contrastive learning method, encounters dimensional collapse when applied to medical images. This is attributed to the high degree of inter-image similarity shared between the medical images. To address this, we propose two key contributions: local feature learning and feature decorrelation. Local feature learning improves the ability of the model to focus on the local regions of the image, while feature decorrelation removes the linear dependence among the features. Our experimental findings demonstrate that our contributions significantly enhance the model's performance in the downstream task of medical segmentation, both in the linear evaluation and full fine-tuning settings. This work illustrates the importance of effectively adapting SSL techniques to the characteristics of medical imaging tasks. The source code will be made publicly available at: https://github.com/CAMMA-public/med-moco, Comment: Accepted at at ISBI-2024 (https://biomedicalimaging.org/2024/). 4 pages, 2 figures, 2 tables

This study introduces and evaluates the PRECISE framework, utilizing OpenAI's GPT-4 to enhance patient engagement by providing clearer and more accessible chest X-ray reports at a sixth-grade reading level. The framework was tested on 500 reports, demonstrating significant improvements in readability, reliability, and understandability. Statistical analyses confirmed the effectiveness of the PRECISE approach, highlighting its potential to foster patient-centric care delivery in healthcare decision-making.

A backward wave optical parametric oscillator (BWOPO) waveguide in periodically poled Rb-doped KTP is presented. The waveguide exhibits low loss (0.16 dB/cm) and has an oscillation threshold, almost 20 times lower than the corresponding bulk device. The backward wave has a narrow linewidth of 21 GHz at 1514.6 nm while the forward wave at 1688.7 nm has a spectrum replicating the pump. The unique spectral features of the BWOPO will unlock novel opportunities in low-power nonlinear integrated optics. A conversion efficiency of 8.4% was obtained limited by the emergence of backward stimulated polariton scattering.

This technical report provides a detailed overview of Endoscapes, a dataset of laparoscopic cholecystectomy (LC) videos with highly intricate annotations targeted at automated assessment of the Critical View of Safety (CVS). Endoscapes comprises 201 LC videos with frames annotated sparsely but regularly with segmentation masks, bounding boxes, and CVS assessment by three different clinical experts. Altogether, there are 11090 frames annotated with CVS and 1933 frames annotated with tool and anatomy bounding boxes from the 201 videos, as well as an additional 422 frames from 50 of the 201 videos annotated with tool and anatomy segmentation masks. In this report, we provide detailed dataset statistics (size, class distribution, dataset splits, etc.) and a comprehensive performance benchmark for instance segmentation, object detection, and CVS prediction. The dataset and model checkpoints are publically available at https://github.com/CAMMA-public/Endoscapes., Comment: 7 pages; 3 figures

Surgical robotics holds much promise for improving patient safety and clinician experience in the Operating Room (OR). However, it also comes with new challenges, requiring strong team coordination and effective OR management. Automatic detection of surgical activities is a key requirement for developing AI-based intelligent tools to tackle these challenges. The current state-of-the-art surgical activity recognition methods however operate on image-based representations and depend on large-scale labeled datasets whose collection is time-consuming and resource-expensive. This work proposes a new sample-efficient and object-based approach for surgical activity recognition in the OR. Our method focuses on the geometric arrangements between clinicians and surgical devices, thus utilizing the significant object interaction dynamics in the OR. We conduct experiments in a low-data regime study for long video activity recognition. We also benchmark our method againstother object-centric approaches on clip-level action classification and show superior performance.

Most studies on surgical activity recognition utilizing Artificial intelligence (AI) have focused mainly on recognizing one type of activity from small and mono-centric surgical video datasets. It remains speculative whether those models would generalize to other centers. In this work, we introduce a large multi-centric multi-activity dataset consisting of 140 videos (MultiBypass140) of laparoscopic Roux-en-Y gastric bypass (LRYGB) surgeries performed at two medical centers: the University Hospital of Strasbourg (StrasBypass70) and Inselspital, Bern University Hospital (BernBypass70). The dataset has been fully annotated with phases and steps. Furthermore, we assess the generalizability and benchmark different deep learning models in 7 experimental studies: 1) Training and evaluation on BernBypass70; 2) Training and evaluation on StrasBypass70; 3) Training and evaluation on the MultiBypass140; 4) Training on BernBypass70, evaluation on StrasBypass70; 5) Training on StrasBypass70, evaluation on BernBypass70; Training on MultiBypass140, evaluation 6) on BernBypass70 and 7) on StrasBypass70. The model's performance is markedly influenced by the training data. The worst results were obtained in experiments 4) and 5) confirming the limited generalization capabilities of models trained on mono-centric data. The use of multi-centric training data, experiments 6) and 7), improves the generalization capabilities of the models, bringing them beyond the level of independent mono-centric training and validation (experiments 1) and 2)). MultiBypass140 shows considerable variation in surgical technique and workflow of LRYGB procedures between centers. Therefore, generalization experiments demonstrate a remarkable difference in model performance. These results highlight the importance of multi-centric datasets for AI model generalization to account for variance in surgical technique and workflows.

Recently, spatiotemporal graphs have emerged as a concise and elegant manner of representing video clips in an object-centric fashion, and have shown to be useful for downstream tasks such as action recognition. In this work, we investigate the use of latent spatiotemporal graphs to represent a surgical video in terms of the constituent anatomical structures and tools and their evolving properties over time. To build the graphs, we first predict frame-wise graphs using a pre-trained model, then add temporal edges between nodes based on spatial coherence and visual and semantic similarity. Unlike previous approaches, we incorporate long-term temporal edges in our graphs to better model the evolution of the surgical scene and increase robustness to temporary occlusions. We also introduce a novel graph-editing module that incorporates prior knowledge and temporal coherence to correct errors in the graph, enabling improved downstream task performance. Using our graph representations, we evaluate two downstream tasks, critical view of safety prediction and surgical phase recognition, obtaining strong results that demonstrate the quality and flexibility of the learned representations. Code is available at github.com/CAMMA-public/SurgLatentGraph., Comment: 13 pages, 2 figures, MICCAI 2023

Purpose: To study the effect of proton linear energy transfer (LET) on rib fracture in breast cancer patients treated with pencil-beam scanning proton therapy (PBS) using a novel tool of dose-LET volume histogram (DLVH). Methods: From a prospective registry of patients treated with post-mastectomy proton therapy to the chest wall and regional lymph nodes for breast cancer between 2015 and 2020, we retrospectively identified rib fracture cases detected after completing treatment. Contemporaneously treated control patients that did not develop rib fracture were matched to patients 2:1 considering prescription dose, boost location, reconstruction status, laterality, chest wall thickness, and treatment year. The DLVH index, V(d, l), defined as volume(V) of the structure with at least dose(d) and LET(l), was calculated. DLVH plots between the fracture and control group were compared. Conditional logistic regression (CLR) model was used to establish the relation of V(d, l) and the observed fracture at each combination of d and l. The p-value derived from CLR model shows the statistical difference between fracture patients and the matched control group. Using the 2D p-value map, the DLVH features associated with the patient outcomes were extracted. Results: Seven rib fracture patients were identified, and fourteen matched patients were selected for the control group. The median time from the completion of proton therapy to rib fracture diagnosis was 12 months (range 5 to 14 months). Two patients had grade 2 symptomatic rib fracture while the remaining 5 were grade 1 incidentally detected on imaging. The derived p-value map demonstrated larger V(0-36 Gy[RBE], 4.0-5.0 keV/um) in patients experiencing fracture (p<0.1). Conclusions: In breast cancer patients receiving PBS, a larger volume of chest wall receiving moderate dose and high LET may result in increased risk of rib fracture., Comment: 1 Table and 3 Figures

Inter-modal image registration (IMIR) and image segmentation with abdominal Ultrasound (US) data has many important clinical applications, including image-guided surgery, automatic organ measurement and robotic navigation. However, research is severely limited by the lack of public datasets. We propose TRUSTED (the Tridimensional Renal Ultra Sound TomodEnsitometrie Dataset), comprising paired transabdominal 3DUS and CT kidney images from 48 human patients (96 kidneys), including segmentation, and anatomical landmark annotations by two experienced radiographers. Inter-rater segmentation agreement was over 94 (Dice score), and gold-standard segmentations were generated using the STAPLE algorithm. Seven anatomical landmarks were annotated, important for IMIR systems development and evaluation. To validate the dataset's utility, 5 competitive Deep Learning models for automatic kidney segmentation were benchmarked, yielding average DICE scores from 83.2% to 89.1% for CT, and 61.9% to 79.4% for US images. Three IMIR methods were benchmarked, and Coherent Point Drift performed best with an average Target Registration Error of 4.53mm. The TRUSTED dataset may be used freely researchers to develop and validate new segmentation and IMIR methods., Comment: Alexandre Hostettler, and Toby Collins share last authorship

Surgical action triplets describe instrument-tissue interactions as (instrument, verb, target) combinations, thereby supporting a detailed analysis of surgical scene activities and workflow. This work focuses on surgical action triplet detection, which is challenging but more precise than the traditional triplet recognition task as it consists of joint (1) localization of surgical instruments and (2) recognition of the surgical action triplet associated with every localized instrument. Triplet detection is highly complex due to the lack of spatial triplet annotation. We analyze how the amount of instrument spatial annotations affects triplet detection and observe that accurate instrument localization does not guarantee better triplet detection due to the risk of erroneous associations with the verbs and targets. To solve the two tasks, we propose MCIT-IG, a two-stage network, that stands for Multi-Class Instrument-aware Transformer-Interaction Graph. The MCIT stage of our network models per class embedding of the targets as additional features to reduce the risk of misassociating triplets. Furthermore, the IG stage constructs a bipartite dynamic graph to model the interaction between the instruments and targets, cast as the verbs. We utilize a mixed-supervised learning strategy that combines weak target presence labels for MCIT and pseudo triplet labels for IG to train our network. We observed that complementing minimal instrument spatial annotations with target embeddings results in better triplet detection. We evaluate our model on the CholecT50 dataset and show improved performance on both instrument localization and triplet detection, topping the leaderboard of the CholecTriplet challenge in MICCAI 2022., Comment: Accepted at MICCAI, 2023. Project Page: https://github.com/CAMMA-public/mcit-ig

A main source of capacity fading in lithium-ion batteries is the degradation of the active cathode materials caused by the series of volume changes during charge and discharge cycles. The quaternary colquiriite-type fluorides Li$_x$CaFeF$\mathrm{_6}$ and Li$_x$CaCoF$\mathrm{_6}$ were reported to have negligible volume changes in specific Li concentration ranges, making the underlying colquiriite structure a promising candidate for so-called zero-strain behavior. Using first-principles electronic structure calculations based on density functional theory with a Hubbard-$U$ correlation correction on the transition-metal ions, we systematically investigate the equilibrium volumes of the colquiriite-type fluorides Li$_x$CaMF$\mathrm{_6}$ with M =Ti, V, Cr, Mn, Fe, Co, and Ni at the Li concentrations $x$=0, 1, and 2. We elucidate the connection between the total volume of the structures and the local volumes of fluorine coordinated octahedra around the cations, and we find trends along the series of the 3d transition-metal elements. In the lithiation step from $x$=1 to $x$=2 we find volume changes of about 10 %, and we discuss the discrepancy to the experimentally reported smaller value for Li$_x$CaFeF$\mathrm{_6}$. The suitability as cathode material was furhter investigated by calculating the theoretical voltages and capacities. From $x$=0 to $x$=1 we describe the compensating structural mechanisms that lead to an exceptionally small volume change of Li$_x$CaMnF$\mathrm{_6}$, which posseses a high theoretical voltage and moderate capacity. This compound is therefore a particularly promising zero-strain cathode material., Comment: 14 pages, 10 Figures

Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.

Formalizing surgical activities as triplets of the used instruments, actions performed, and target anatomies is becoming a gold standard approach for surgical activity modeling. The benefit is that this formalization helps to obtain a more detailed understanding of tool-tissue interaction which can be used to develop better Artificial Intelligence assistance for image-guided surgery. Earlier efforts and the CholecTriplet challenge introduced in 2021 have put together techniques aimed at recognizing these triplets from surgical footage. Estimating also the spatial locations of the triplets would offer a more precise intraoperative context-aware decision support for computer-assisted intervention. This paper presents the CholecTriplet2022 challenge, which extends surgical action triplet modeling from recognition to detection. It includes weakly-supervised bounding box localization of every visible surgical instrument (or tool), as the key actors, and the modeling of each tool-activity in the form of triplet. The paper describes a baseline method and 10 new deep learning algorithms presented at the challenge to solve the task. It also provides thorough methodological comparisons of the methods, an in-depth analysis of the obtained results across multiple metrics, visual and procedural challenges; their significance, and useful insights for future research directions and applications in surgery., Comment: MICCAI EndoVis CholecTriplet2022 challenge report. Published at Elsevier journal of Medical Image Analysis. 25 pages, 15 figures, 8 tables

We develop a theory describing the transient transmission through noisy qubit-resonator systems with quadratic interactions as are found in superconducting and nanomechanical resonators coupled to solid-state qubits. After generalizing the quantum Langevin equations to arbitrary qubit-resonator couplings, we show that only the cases of linear and quadratic couplings allow for an analytical treatment within standard input-output theory. Focussing for the first time on quadratic couplings and allowing for arbitrary initial qubit coherences, it is shown that noise characteristics can be extracted from input-output measurements by recording both the averaged fluctuations in the transmission probability and the averaged phase. Our results represent an extension to the field of transmission-based noise spectroscopy with immediate practical applications., Comment: 8 pages, 3 figures

Oxygen vacancy formation energies play a major role in the electric field assisted abnormal grain growth of technologically relevant polycrystalline perovskite phases. The underlying effect on the atomic scale is assumed to be a redistribution of cationic and anionic point defects between grain boundaries and the bulk interior regions of the grains due to different defect formation energies in the structurally different regions, accompanied by the formation of space charge zones. Using atomistic calculations based on classical interatomic potentials, we derive and discuss optimized structures of the symmetric tilt grain boundaries $\Sigma$5(210)[001] and $\Sigma$5(310)[001], and of the asymmetric tilt grain boundary (430)[001]||(100)[001] in the electroceramic perovskite materials SrTiO$_3$, BaTiO$_3$, and BaZrO$_3$. We present profiles of oxygen vacancy formation energies across those GBs and discuss their dependence on composition and grain boundary type., Comment: 21 pages, 8 figures

Objective: To develop and validate a deep learning model for the identification of out-of-body images in endoscopic videos. Background: Surgical video analysis facilitates education and research. However, video recordings of endoscopic surgeries can contain privacy-sensitive information, especially if out-of-body scenes are recorded. Therefore, identification of out-of-body scenes in endoscopic videos is of major importance to preserve the privacy of patients and operating room staff. Methods: A deep learning model was trained and evaluated on an internal dataset of 12 different types of laparoscopic and robotic surgeries. External validation was performed on two independent multicentric test datasets of laparoscopic gastric bypass and cholecystectomy surgeries. All images extracted from the video datasets were annotated as inside or out-of-body. Model performance was evaluated compared to human ground truth annotations measuring the receiver operating characteristic area under the curve (ROC AUC). Results: The internal dataset consisting of 356,267 images from 48 videos and the two multicentric test datasets consisting of 54,385 and 58,349 images from 10 and 20 videos, respectively, were annotated. Compared to ground truth annotations, the model identified out-of-body images with 99.97% ROC AUC on the internal test dataset. Mean $\pm$ standard deviation ROC AUC on the multicentric gastric bypass dataset was 99.94$\pm$0.07% and 99.71$\pm$0.40% on the multicentric cholecystectomy dataset, respectively. Conclusion: The proposed deep learning model can reliably identify out-of-body images in endoscopic videos. The trained model is publicly shared. This facilitates privacy preservation in surgical video analysis., Comment: Jo\"el L. Lavanchy and Armine Vardazaryan contributed equally and share first co-authorship

Artificial intelligence is set to be deployed in operating rooms to improve surgical care. This early-stage clinical evaluation shows the feasibility of concurrently attaining real-time, high-quality predictions from several deep neural networks for endoscopic video analysis deployed for assistance during three laparoscopic cholecystectomies., Comment: 12 pages, 1 figure

Assessing the critical view of safety in laparoscopic cholecystectomy requires accurate identification and localization of key anatomical structures, reasoning about their geometric relationships to one another, and determining the quality of their exposure. Prior works have approached this task by including semantic segmentation as an intermediate step, using predicted segmentation masks to then predict the CVS. While these methods are effective, they rely on extremely expensive ground-truth segmentation annotations and tend to fail when the predicted segmentation is incorrect, limiting generalization. In this work, we propose a method for CVS prediction wherein we first represent a surgical image using a disentangled latent scene graph, then process this representation using a graph neural network. Our graph representations explicitly encode semantic information - object location, class information, geometric relations - to improve anatomy-driven reasoning, as well as visual features to retain differentiability and thereby provide robustness to semantic errors. Finally, to address annotation cost, we propose to train our method using only bounding box annotations, incorporating an auxiliary image reconstruction objective to learn fine-grained object boundaries. We show that our method not only outperforms several baseline methods when trained with bounding box annotations, but also scales effectively when trained with segmentation masks, maintaining state-of-the-art performance., Comment: 12 pages, 4 figures

One of the recent advances in surgical AI is the recognition of surgical activities as triplets of (instrument, verb, target). Albeit providing detailed information for computer-assisted intervention, current triplet recognition approaches rely only on single frame features. Exploiting the temporal cues from earlier frames would improve the recognition of surgical action triplets from videos. In this paper, we propose Rendezvous in Time (RiT) - a deep learning model that extends the state-of-the-art model, Rendezvous, with temporal modeling. Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition. We validate our proposal on the challenging surgical triplet dataset, CholecT45, demonstrating an improved recognition of the verb and triplet along with other interactions involving the verb such as (instrument, verb). Qualitative results show that the RiT produces smoother predictions for most triplet instances than the state-of-the-arts. We present a novel attention-based approach that leverages the temporal fusion of video frames to model the evolution of surgical actions and exploit their benefits for surgical triplet recognition., Comment: Accepted at IPCAI, 2023. Project Page: https://github.com/CAMMA-public/rendezvous-in-time

Noise induced decoherence is one of the main threats to large-scale quantum computation. In an attempt to assess the noise affecting a qubit we go beyond the standard steady-state solution of the transmission through a qubit-coupled cavity in input-output theory by including dynamical noise in the description of the system. We solve the quantum Langevin equations exactly for a noise-free system and treat the noise as a perturbation. In the long-time limit the corrections may be written as a sum of convolutions of the noise power spectral density with an integration kernel that depends on external control parameters. Using the convolution theorem, we invert the corrections and obtain relations for the noise spectral density as an integral over measurable quantities. Additionally, we treat the noise exactly in the dispersive regime, and again find that noise characteristics are imprinted in the long-time transmission in convolutions containing the power spectral density., Comment: 18 pages, 4 figures