Search

Purpose: Vision-based robot tool segmentation plays a fundamental role in surgical robots and downstream tasks. CaRTS, based on a complementary causal model, has shown promising performance in unseen counterfactual surgical environments in the presence of smoke, blood, etc. However, CaRTS requires over 30 iterations of optimization to converge for a single image due to limited observability. Method: To address the above limitations, we take temporal relation into consideration and propose a temporal causal model for robot tool segmentation on video sequences. We design an architecture named Temporally Constrained CaRTS (TC-CaRTS). TC-CaRTS has three novel modules to complement CaRTS - temporal optimization pipeline, kinematics correction network, and spatial-temporal regularization. Results: Experiment results show that TC-CaRTS requires much fewer iterations to achieve the same or better performance as CaRTS. TC- CaRTS also has the same or better performance in different domains compared to CaRTS. All three modules are proven to be effective. Conclusion: We propose TC-CaRTS, which takes advantage of temporal constraints as additional observability. We show that TC-CaRTS outperforms prior work in the robot tool segmentation task with improved convergence speed on test datasets from different domains.

Traditional teleoperation (leader/follower) systems primarily focus on one operator controlling one remote robot, but as robots become ubiquitous, there is an increasing need for multiple operators, including autonomous agents, to collaboratively control multiple robots. However, existing teleoperation frameworks do not inherently support the variety of possible collaborations, such as multiple operators, each with an input device (leader), controlling a robot and camera or different degrees of freedom of a single robot (follower). The same concept applies to teleoperating robots in a simulation environment through physical input devices. In this letter, we extend our novel simulation framework that is capable of incorporating multiple input devices asynchronously with a real-time dynamic simulation to incorporate a customizable shared control. For this purpose, we have identified and implemented a sufficient set of coordinate frames to encapsulate the pairing of multiple leaders, followers and cameras in a shared asynchronous manner with force feedback. We demonstrate the utility of this framework in accelerating user training, ease of learning, and enhanced task completion times through shared control by a supervisor.

Multi- and cross-modal learning consolidates information from multiple data sources which may offer a holistic representation of complex scenarios. Cross-modal learning is particularly interesting, because synchronized data streams are immediately useful as self-supervisory signals. The prospect of achieving self-supervised continual learning in surgical robotics is exciting as it may enable lifelong learning that adapts to different surgeons and cases, ultimately leading to a more general machine understanding of surgical processes. We present a learning paradigm using synchronous video and kinematics from robot-mediated surgery. Our approach relies on an encoder–decoder network that maps optical flow to the corresponding kinematics sequence. Clustering on the latent representations reveals meaningful groupings for surgeon gesture and skill level. We demonstrate the generalizability of the representations on the JIGSAWS dataset by classifying skill and gestures on tasks not used for training. For tasks seen in training, we report a 59 to 70% accuracy in surgical gestures classification. On tasks beyond the training setup, we note a 45 to 65% accuracy. Qualitatively, we find that unseen gestures form clusters in the latent space of novice actions, which may enable the automatic identification of novel interactions in a lifelong learning scenario. From predicting the synchronous kinematics sequence, optical flow representations of surgical scenes emerge that separate well even for new tasks that the model had not seen before. While the representations are useful immediately for a variety of tasks, the self-supervised learning paradigm may enable research in lifelong and user-specific learning.

Purpose Surgical simulations play an increasingly important role in surgeon education and developing algorithms that enable robots to perform surgical subtasks. To model anatomy, Finite Element Method (FEM) simulations have been held as the gold standard for calculating accurate soft-tissue deformation. Unfortunately, their accuracy is highly dependent on the simulation parameters, which can be difficult to obtain. Methods In this work, we investigate how live data acquired during any robotic endoscopic surgical procedure may be used to correct for inaccurate FEM simulation results. Since FEMs are calculated from initial parameters and cannot directly incorporate observations, we propose to add a correction factor that accounts for the discrepancy between simulation and observations. We train a network to predict this correction factor. Results To evaluate our method, we use an open-source da Vinci Surgical System to probe a soft-tissue phantom and replay the interaction in simulation. We train the network to correct for the difference between the predicted mesh position and the measured point cloud. This results in 15-30% improvement in the mean distance, demonstrating the effectiveness of our approach across a large range of simulation parameters. Conclusion We show a first step towards a framework that synergistically combines the benefits of model-based simulation and real-time observations. It corrects discrepancies between simulation and the scene that results from inaccurate modeling parameters. This can provide a more accurate simulation environment for surgeons and better data with which to train algorithms., 12 pages, 4 figures, to be published in IJCARS IPCAI special edition 2020

Augmented reality (AR) and robotic-assisted surgery (RAS) are both rapidly evolving technologies in recent years. RAS systems, such as the da Vinci Surgical System, aim to improve surgical precision and dexterity, as well as access to minimally-invasive procedures, while AR provides an advanced interface to enhance user perception. Combining the features of both, AR-integrated RAS has become an appealing concept with increased interest among the academic community. In this paper, we review the existing literature about AR-integrated RAS. We discuss the hardware components, application paradigms and clinical relevance of the literature. The concept of AR-integrated RAS has been shown to be feasible for various procedures. Encouraging preliminary results include reduced sight diversion and improved situation awareness. Special techniques, e.g., activation-on-demand, are taken into consideration to address visual clutter of the AR interface and ensure that the system is fail-safe. Although AR-integrated RAS is not yet mature, we believe that if the current trend of development continues, it will soon demonstrate its clinical value.

Background As a common malignant cancer of the urinary system, the precise molecular mechanisms of bladder cancer remain to be illuminated. The purpose of this study was to identify core genes with prognostic value as potential oncogenes for the diagnosis, prognosis or novel therapeutic targets of bladder cancer. Methods The gene expression profiles GSE3167 and GSE7476 were available from the Gene Expression Omnibus (GEO) database. Next, PPI network was built to filter the hub gene through the STRING database and Cytoscape software and GEPIA and Kaplan-Meier plotter were implemented. Frequency and type of hub genes and sub groups analysis were performed in cBioportal and ULCAN database. Finally,We used RT-qPCR to confirm our results. Results Totally, 251 DEGs were excavated from two datasets in our study. We only founded high expression of SMC4, TYMS, CCNB1, CKS1B, NUSAP1 and KPNA2 was associated with worse outcomes in bladder cancer patients and no matter from the type of mutation or at the transcriptional level of hub genes, the tumor showed a high form of expression. However, only the expression of SMC4,CCNB1and CKS1B remained changed between the cancer and the normal samples in our results of RT-qPCR. Conclusion In conclusion,These findings indicate that the SMC4,CCNB1 and CKS1B may serve as critical biomarkers in the development and poor prognosis.

Objective To determine the combination of fasting blood glucose (FBG) with squamous cell carcinoma antigen (SCCA) assessments in the prediction of tumor responses to chemotherapy and pretreatment prognostication among patients receiving neoadjuvant chemotherapy (NACT) for locally advanced cervical cancer (LACC). Methods Data of 347 LACC patients were retrospectively reviewed. Receiver operating characteristic (ROC) curves were constructed, and areas under the curves (AUCs) were compared to evaluate the ability to predict complete response (CR) following NACT. Patients were stratified into groups with low and high levels of SCCA and FBG and combined into low‐ or high‐SCCA and low‐ or high‐FBG groups. Cox regression analysis was performed to identify determinants of recurrence‐free survival (RFS) and overall survival (OS). Results The AUCs were 0.70, 0.68, and 0.66 for SCCA, FBG, and a combination of SCCA and FBG for predicting CR following NACT, respectively; however, the differences among AUCs were not significant (P = .496). Pretreatment SCCA and FBG levels were identified as independent predictors of RFS and OS. The high‐SCCA/high‐FBG group showed significantly worse prognosis than the low‐SCCA/low‐FBG group. After adjusting for other variables, high‐SCCA/high‐FBG remained independently associated with an increased risk of tumor recurrence and death. Conclusion SCCA, FBG, and a combination of SCCA and FBG could acceptably predict CR following NACT. Pretreatment SCCA and FBG levels were independent prognostic factors. The combination of SCCA and FBG levels refined the prognostic stratification of LACC patients, which allowed the group of patients with the highest risk of recurrence and death to be identified.

Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recognition accuracies. In this regard, we propose a novel online approach of multi-modal relational graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information through interactive message propagation in the latent feature space. In specific, we first extract embeddings from video and kinematics sequences with temporal convolutional networks and LSTM units. Next, we identify multi-relations in these multi-modal embeddings and leverage them through a hierarchical relational graph learning module. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset, outperforming current uni-modal and multi-modal methods on both suturing and knot typing tasks. Furthermore, we validated our method on in-house visual-kinematics datasets collected with da Vinci Research Kit (dVRK) platforms in two centers, with consistent promising performance achieved., Comment: Accepted for ICRA 2021

The study was to compare the efficacy between IV peramivir and oral oseltamivir treatments in patients with influenza.The PubMed, EMBASE, Scopus, ClinicalTrials.gov, and Cochrane Library databases were searched for studies published before January 2020.The meta-analysis was conducted to calculate the pooled effect size by using a random-effects model. Seven randomized controlled trials (RCTs) including 1,138 patients were reviewed. The incidence of total complications revealed no significant difference between 600 mg IV peramivir (P600) and 75 mg oral oseltamivir (O75) treatments (2.8% vs. 4.1%; risk ratio [RR] = 0.70; 95% confidence interval [CI]: 0.36-1.38). The incidence of pneumonia was not significantly different between the P600 and O75 treatment groups (2.2% vs. 2.7%; RR = 0.74; 95% CI: 0.37-1.51). Regarding the time to the alleviation of symptoms, no difference was found in P600 and O75 treatment (MD = -3.00; 95% CI: -11.07 to 5.06). The rate of fever clearance in 24 h and the time to fever resolution were not statistically different between the IV peramivir and oral oseltamivir treatments (at different dosages) groups.The treatment of influenza with IV peramivir or oral oseltamivir had similar clinical efficacy.

Pelvic ring disruptions result from blunt injury mechanisms and are often found in patients with multi-system trauma. To grade pelvic fracture severity in trauma victims based on whole-body CT, the Tile AO/OTA classification is frequently used. Due to the high volume of whole-body trauma CTs generated in busy trauma centers, an automated approach to Tile classification would provide substantial value, e. g., to prioritize the reading queue of the attending trauma radiologist. In such scenario, an automated method should perform grading based on a transparent process and based on interpretable features to enable interaction with human readers and lower their workload by offering insights from a first automated read of the scan. This paper introduces an automated yet interpretable pelvic trauma decision support system to assist radiologists in fracture detection and Tile grade classification. The method operates similarly to human interpretation of CT scans and first detects distinct pelvic fractures on CT with high specificity using a Faster-RCNN model that are then interpreted using a structural causal model based on clinical best practices to infer an initial Tile grade. The Bayesian causal model and finally, the object detector are then queried for likely co-occurring fractures that may have been rejected initially due to the highly specific operating point of the detector, resulting in an updated list of detected fractures and corresponding final Tile grade. Our method is transparent in that it provides finding location and type using the object detector, as well as information on important counterfactuals that would invalidate the system’s recommendation and achieves an AUC of 83.3%/85.1% for translational/rotational instability. Despite being designed for human-machine teaming, our approach does not compromise on performance compared to previous black-box approaches.

Background: The chemotherapy response score (CRS) system is a reproducible prognostic tool for patients receiving neoadjuvant chemotherapy (NACT) for tubo-ovarian high-grade serous carcinoma (HGSC). Achieving CRS 3 following NACT can be used as a surrogate for progression-free survival (PFS) and overall survival (OS). This study aimed to identify predictors of CRS 3 and develop a predictive nomogram.Methods: Data were extracted from 106 HGSC patients receiving NACT. Logistic regression was used to identify independent predictors for CRS 3. A nomogram was established based on the multivariate regression model.Results: All patients received three cycles of NACT, and CRS 3 was observed in 24 (22.6%) patients. Compared with patients in the CRS 1–2 group, patients in the CRS 3 groups had significantly improved PFS (log-rank test P < 0.0001). The multivariate regression analysis identified post-NACT CA125, percent decrease in CA125, post-NACT human epididymis protein 4 (HE4), and post-NACT hemoglobin level as independent predictors of CRS 3. The Hosmer-Lemeshow test showed goodness-of-fit of this regression model (P = 0.272). The nomogram including these factors presented good discrimination (area under the curve = 0.82), good calibration (mean absolute error = 0.039), and a net benefit within the threshold probabilities of CRS 3 > 5%.Conclusions: We validated the prognostic role of the CRS system and developed a nomogram that predicts the possibility of CRS 3 following NACT. The nomogram helps to identify patients who would benefit the most from NACT. More studies are warranted to validate this model.

In response to the rapid spread of the novel coronavirus, SARS-CoV-2, the U.S. has largely delegated implementation and rollback of non-pharmaceutical interventions (NPIs) to local governments on the state and county level. This asynchronous response combined with the heterogeneity of the U.S. complicates quantification of the effect of NPIs on the reproductive ratio of SARS-CoV-2 on a national level. We describe a data-driven approach to quantify the effect of NPIs that relies on county-level similarities to specialize a Bayesian mechanistic model based on observed fatalities. Using this approach, we estimate the effect of NPIs on the reproductive ratio R_t in 1,904 U.S. counties incorporating implementation, subsequent rollback, and mask mandate efficacy. We estimate that at some point before Aug 2nd, 2020, 1,808 out of the considered 1,904 U.S. counties had reduced the reproductive ratio of SARS-CoV-2 to below 1.0. However, on Aug 2nd, the reproductive ration remained below that threshold for only 702 counties. The estimated effect of any individual NPI is different across counties. Public school closings were estimated to be effective in metropolitan, urban, and suburban counties, while advisory NPIs were estimated to be effective in more rural counties. The cumulative prevalence predicted by the model ranges from 0 to 58.6% across the counties examined. The median is 2.6% while the 25th and 75th percentile are 1.3% and 44.6% respectively, indicating that most counties are far from herd immunity. Our results suggest that local conditions, including socioeconomic, demographic and infrastructural factors, in addition to the cumulative prevalence are pertinent to containment and re-opening decisions.

In response to the rapid spread of the novel coronavirus, SARS-CoV-2, the U.S. has largely delegated implementation of non-pharmaceutical interventions (NPIs) to local governments on the state and county level. This staggered implementation combined with the heterogeneity of the U.S. complicates quantification the effect of NPIs on the reproductive rate of SARS-CoV-2.We describe a data-driven approach to quantify the effect of NPIs that relies on county-level similarities to specialize a Bayesian mechanistic model based on observed fatalities. Using this approach, we estimate change in reproductive rate, Rt, due to implementation of NPIs in 1,417 U.S. counties.We estimate that as of May 28th, 2020 1,177 out of the considered 1,417 U.S. counties have reduced the reproductive rate of SARS-CoV-2 to below 1.0. The estimated effect of any individual NPI, however, is different across counties. Stay-at-home orders were estimated as the only effective NPI in metropolitan and urban counties, while advisory NPIs were estimated to be effective in more rural counties. The expected level of infection predicted by the model ranges from 0 to 28.7% and is far from herd immunity even in counties with advanced spread.Our results suggest that local conditions are pertinent to containment and re-opening decisions.