Search

Introduction: Automated skills assessment can provide surgical trainees with objective, personalized feedback during training. Here, we measure the efficacy of artificial intelligence (AI)-based feedback on a robotic suturing task. Materials and Methods: Forty-two participants with no robotic surgical experience were randomized to a control or feedback group and video-recorded while completing two rounds (R1 and R2) of suturing tasks on a da Vinci surgical robot. Participants were assessed on needle handling and needle driving, and feedback was provided via a visual interface after R1. For feedback group, participants were informed of their AI-based skill assessment and presented with specific video clips from R1. For control group, participants were presented with randomly selected video clips from R1 as a placebo. Participants from each group were further labeled as underperformers or innate-performers based on a median split of their technical skill scores from R1. Results: Demographic features were similar between the control (n = 20) and feedback group (n = 22) (p > 0.05). Observing the improvement from R1 to R2, the feedback group had a significantly larger improvement in needle handling score (0.30 vs −0.02, p = 0.018) when compared with the control group, although the improvement of needle driving score was not significant when compared with the control group (0.17 vs −0.40, p = 0.074). All innate-performers exhibited similar improvements across rounds, regardless of feedback (p > 0.05). In contrast, underperformers in the feedback group improved more than the control group in needle handling (p = 0.02). Conclusion: AI-based feedback facilitates surgical trainees' acquisition of robotic technical skills, especially underperformers. Future research will extend AI-based feedback to additional suturing skills, surgical tasks, and experience groups. [ABSTRACT FROM AUTHOR]

Introduction: Robotic surgical performance, in particular suturing, has been linked to postoperative clinical outcomes. Before attempting live surgery, virtual reality (VR) simulators afford opportunities for training surgeons to learn fundamental technical skills. Herein, we evaluate the association of suturing technical skill assessments between VR simulation and live surgery, and functional clinical outcomes. Materials and Methods: Twenty surgeons completed a VR suturing exercise on the Mimic™ Flex VR simulator and the anterior vesicourethral anastomosis during robot-assisted radical prostatectomy (RARP). Three independent and blinded graders provided technical skill scores using a validated assessment tool. Correlations between VR and live scores were assessed by Spearman's correlation coefficients (ρ). In addition, 117 historic RARP cases from participating surgeons were extracted, and the association between VR technical skill scores and urinary continence recovery was assessed by a multilevel mixed-effects model. Results: A total of 20 (6 training and 14 expert) surgeons participated. Statistically significant correlations for scores provided between VR simulation and live surgery were found for overall and needle driving scores (ρ = 0.555, p = 0.011; ρ = 0.570, p = 0.009, respectively). A subanalysis performed on training surgeons found significant correlations for overall scores between VR simulation and live surgery (ρ = 0.828, p = 0.042). Expert cases with high VR needle driving scores had significantly greater continence recovery rates at 24 months after RARP (98.5% vs 84.9%, p = 0.028). Conclusions: Our study found significant correlations in technical scores between VR and live surgery, especially among training surgeons. In addition, we found that VR needle driving scores were associated with continence recovery after RARP. Our data support the association of skill assessments between VR simulation and live surgery and potential implications for clinical outcomes. [ABSTRACT FROM AUTHOR]

Purpose: Previously, we identified 8 objective suturing performance metrics highly predictive of urinary continence recovery after robotic-assisted radical prostatectomy. Here, we aimed to test the feasibility of providing tailored feedback based upon these clinically relevant metrics and explore the impact on the acquisition of robotic suturing skills. Materials and Methods: Training surgeons were recruited and randomized to a feedback group or a control group. Both groups completed a baseline, midterm and final dry laboratory vesicourethral anastomosis (VUA) and underwent 4 intervening training sessions each, consisting of 3 suturing exercises. Eight performance metrics were recorded during each exercise: 4 automated performance metrics (derived from kinematic and system events data of the da Vinci-Robotic System) representing efficiency and console manipulation competency, and 4 suturing technical skill scores. The feedback group received tailored feedback (a visual diagramDverbal instructionsDvideo examples) based on these metrics after each session. Generalized linear mixed model was used to compare metric improvement (D) from baseline to the midterm and final VUA. Results: Twenty-three participants were randomized to the feedback group (11) or the control group (12). Demographic data and baseline VUA metrics were comparable between groups. The feedback group showed greater improvement than the control group in aggregate suturing scores at midterm (mean D feedback group 4.5 vs D control group 1.1) and final VUA (D feedback group 5.3 vs D control group 4.9). The feedback group also showed greater improvement in the majority of the included metrics at midterm and final VUA. Conclusions: Tailored feedback based on specific, clinically relevant performance metrics is feasible and may expedite the acquisition of robotic suturing skills. [ABSTRACT FROM AUTHOR]

Introduction: Robotic surgical performance, in particular suturing, has been associated with postoperative clinical outcomes. Suturing can be deconstructed into substep components (needle positioning, needle entry angle, needle driving, and needle withdrawal) allowing for the provision of more specific feedback while teaching suturing and more precision when evaluating suturing technical skill and prediction of clinical outcomes. This study evaluates if the technical skill required for particular substeps of the suturing process is associated with the execution of subsequent substeps in terms of technical skill, accuracy, and efficiency. Materials and Methods: Training and expert surgeons completed standardized sutures on the Mimic™ Flex virtual reality robotic simulator. Video recordings were deidentified, time annotated, and provided technical skill scores for each of the four suturing substeps. Hierarchical Poisson regression with generalized estimating equation was used to examine the association of technical skill rating categories between substeps. Results: Twenty-two surgeons completed 428 suturing attempts with 1669 individual technical skill assessments made. Technical skill scores between substeps of the suturing process were found to be significantly associated. When needle positioning was ideal, needle entry angle was associated with a significantly greater chance of being ideal (risk ratio [RR] = 1.12, p = 0.05). In addition, ideal needle entry angle and needle driving technical skill scores were each significantly associated with ideal needle withdrawal technical skill scores (RR = 1.27, p = 0.03; RR = 1.3, p = 0.03, respectively). Our study determined that ideal technical skill was associated with increased accuracy and efficiency of select substeps. Conclusions: Our study found significant associations in the technical skill required for completing substeps of suturing, demonstrating inter-relationships within the suturing process. Together with the known association between technical skill and clinical outcomes, training surgeons should focus on mastering not just the overall suturing process, but also each substep involved. Future machine learning efforts can better evaluate suturing, knowing that these inter-relationships exist. [ABSTRACT FROM AUTHOR]

Background: This study compares surgical performance during analogous vesico-urethral anastomosis (VUA) tasks in two robotic training environments, virtual reality (VR) and dry laboratory (DL), to investigate transferability of skill assessment across the two platforms. Utilizing computer-generated performance metrics and pupillary data, we evaluated the two environments to distinguish surgical expertise and ultimately whether performance in the VR simulation correlates with performance in live robotic surgery in the DL. Materials and Methods: Experts (≥300 cases) and trainees (<300 cases) performed analogous VUAs during VR and DL sessions on a da Vinci robotic console following an Institutional Review Board (IRB) approved protocol (HS-16-00318). Twenty-two metrics were generated in each environment (kinematic metrics, tissue metrics, and biometrics). The DL included 18 previously validated automated performance metrics (APMs) (kinematics and event metrics) captured by an Intuitive system data recorder. In both settings, Tobii Pro Glasses 2 recorded the task-evoked pupillary response (reported as Index of Cognitive Activity [ICA]) to indicate cognitive workload, analyzed by EyeTracking cognitive workload software. Pearson correlation, Mann–Whitney, and independent t-tests were used for the comparative analyses. Results: Our study included six experts (median caseload 1300 [interquartile range 400–3000]) and 11 trainees (25 [0–250]). A total of 8/9 metrics directly comparable between VR and DL showed significant positive correlation (r ≥ 0.554, p ≤ 0.032); 5/22 VR metrics distinguished expertise, including task time (p = 0.031), clutch usage (p = 0.040), unnecessary needle piercing (p = 0.026), and suspected injury to the endopelvic fascia (p = 0.040). This contrasts with 14/22 APMs in DL (p ≤ 0.038), including linear velocities of all three instruments (p ≤ 0.038) and dominant-hand instrument wrist articulation (p = 0.013). Trainees experienced higher cognitive workload (ICA) in both environments when compared with experts (p < 0.036). Conclusions: Most performance metrics between VR and DL exhibited moderate to strong correlations, showing transferability of skills across the platforms. Comparing training environments, APMs during DL tasks are better able to distinguish expertise than VR-generated metrics. [ABSTRACT FROM AUTHOR]

Objectives: To evaluate automated performance metrics (APMs) and clinical data of experts and super‐experts for four cardinal steps of robot‐assisted radical prostatectomy (RARP): bladder neck dissection; pedicle dissection; prostate apex dissection; and vesico‐urethral anastomosis. Subjects and Methods: We captured APMs (motion tracking and system events data) and synchronized surgical video during RARP. APMs were compared between two experience levels: experts (100–750 cases) and super‐experts (2100–3500 cases). Clinical outcomes (peri‐operative, oncological and functional) were then compared between the two groups. APMs and outcomes were analysed for 125 RARPs using multi‐level mixed‐effect modelling. Results: For the four cardinal steps selected, super‐experts showed differences in select APMs compared with experts (P < 0.05). Despite similar PSA and Gleason scores, super‐experts outperformed experts clinically with regard to peri‐operative outcomes, with a greater lymph node yield of 22.6 vs 14.9 nodes, respectively (P < 0.01), less blood loss (125 vs 130 mL, respectively; P < 0.01), and fewer readmissions at 30 days (1% vs 13%, respectively; P = 0.02). A similar but nonsignificant trend was seen for oncological and functional outcomes, with super‐experts having a lower rate of biochemical recurrence compared with experts (5% vs 15%, respectively; P = 0.13) and a higher continence rate at 3 months (36% vs 18%, respectively; P = 0.14). Conclusion: We found that experts and super‐experts differed significantly in select APMs for the four cardinal steps of RARP, indicating that surgeons do continue to improve in performance even after achieving expertise. We hope ultimately to identify associations between APMs and clinical outcomes to tailor interventions to surgeons and optimize patient outcomes. [ABSTRACT FROM AUTHOR]

Purpose We sought to develop and validate automated performance metrics to measure surgeon performance of vesicourethral anastomosis during robotic assisted radical prostatectomy. Furthermore, we sought to methodically develop a standardized training tutorial for robotic vesicourethral anastomosis. Materials and Methods We captured automated performance metrics for motion tracking and system events data, and synchronized surgical video during robotic assisted radical prostatectomy. Nonautomated performance metrics were manually annotated by video review. Automated and nonautomated performance metrics were compared between experts with 100 or more console cases and novices with fewer than 100 cases. Needle driving gestures were classified and compared. We then applied task deconstruction, cognitive task analysis and Delphi methodology to develop a standardized robotic vesicourethral anastomosis tutorial. Results We analyzed 70 vesicourethral anastomoses with a total of 1,745 stitches. For automated performance metrics experts outperformed novices in completion time (p <0.01), EndoWrist® articulation (p <0.03), instrument movement efficiency (p <0.02) and camera manipulation (p <0.01). For nonautomated performance metrics experts had more optimal needle to needle driver positioning, fewer needle driving attempts, a more optimal needle entry angle and less tissue trauma (each p <0.01). We identified 14 common robotic needle driving gestures. Random gestures were associated with lower efficiency (p <0.01), more attempts (p <0.04) and more trauma (p <0.01). The finalized tutorial contained 66 statements and figures. Consensus among 8 expert surgeons was achieved after 2 rounds, including among 58 (88%) after round 1 and 8 (12%) after round 2. Conclusions Automated performance metrics can distinguish surgeon expertise during vesicourethral anastomosis. The expert vesicourethral anastomosis technique was associated with more efficient movement and less tissue trauma. Standardizing robotic vesicourethral anastomosis and using a methodically developed tutorial may help improve robotic surgical training. [ABSTRACT FROM AUTHOR]

Purpose: Surgical performance is critical for clinical outcomes. We present a novel machine learning (ML) method of processing automated performance metrics (APMs) to evaluate surgical performance and predict clinical outcomes after robot-assisted radical prostatectomy (RARP). Materials and Methods: We trained three ML algorithms utilizing APMs directly from robot system data (training material) and hospital length of stay (LOS; training label) (≤2 days and >2 days) from 78 RARP cases, and selected the algorithm with the best performance. The selected algorithm categorized the cases as “Predicted as expected LOS (pExp-LOS)” and “Predicted as extended LOS (pExt-LOS).” We compared postoperative outcomes of the two groups (Kruskal–Wallis/Fisher's exact tests). The algorithm then predicted individual clinical outcomes, which we compared with actual outcomes (Spearman's correlation/Fisher's exact tests). Finally, we identified five most relevant APMs adopted by the algorithm during predicting. Results: The “Random Forest-50” (RF-50) algorithm had the best performance, reaching 87.2% accuracy in predicting LOS (73 cases as “pExp-LOS” and 5 cases as “pExt-LOS”). The “pExp-LOS” cases outperformed the “pExt-LOS” cases in surgery time (3.7 hours vs 4.6 hours, p = 0.007), LOS (2 days vs 4 days, p = 0.02), and Foley duration (9 days vs 14 days, p = 0.02). Patient outcomes predicted by the algorithm had significant association with the “ground truth” in surgery time (p < 0.001, r = 0.73), LOS (p = 0.05, r = 0.52), and Foley duration (p < 0.001, r = 0.45). The five most relevant APMs, adopted by the RF-50 algorithm in predicting, were largely related to camera manipulation. Conclusion: To our knowledge, ours is the first study to show that APMs and ML algorithms may help assess surgical RARP performance and predict clinical outcomes. With further accrual of clinical data (oncologic and functional data), this process will become increasingly relevant and valuable in surgical assessment and training. [ABSTRACT FROM AUTHOR]

Purpose We explore and validate objective surgeon performance metrics using a novel recorder (“dVLogger”) to directly capture surgeon manipulations on the da Vinci® Surgical System. We present the initial construct and concurrent validation study of objective metrics during preselected steps of robot-assisted radical prostatectomy. Materials and Methods Kinematic and events data were recorded for expert (100 or more cases) and novice (less than 100 cases) surgeons performing bladder mobilization, seminal vesicle dissection, anterior vesicourethral anastomosis and right pelvic lymphadenectomy. Expert/novice metrics were compared using mixed effect statistical modeling (construct validation). Expert reviewers blindly rated seminal vesicle dissection and anterior vesicourethral anastomosis using GEARS (Global Evaluative Assessment of Robotic Skills). Intraclass correlation measured inter-rater variability. Objective metrics were correlated to corresponding GEARS metrics using Spearman’s test (concurrent validation). Results The performance of 10 experts (mean 810 cases, range 100 to 2,000) and 10 novices (mean 35 cases, range 5 to 80) was evaluated in 100 robot-assisted radical prostatectomy cases. For construct validation the experts completed operative steps faster (p <0.001) with less instrument travel distance (p <0.01), less aggregate instrument idle time (p <0.001), shorter camera path length (p <0.001) and more frequent camera movements (p <0.03). Experts had a greater ratio of dominant-to-nondominant instrument path distance for all steps (p <0.04) except anterior vesicourethral anastomosis. For concurrent validation the median experience of 3 expert reviewers was 300 cases (range 200 to 500). Intraclass correlation among reviewers was 0.6-0.7. For anterior vesicourethral anastomosis and seminal vesicle dissection, kinematic metrics had low associations with GEARS metrics. Conclusions Objective metrics revealed experts to be more efficient and directed during preselected steps of robot-assisted radical prostatectomy. Objective metrics had limited associations to GEARS. These findings lay the foundation for developing standardized metrics for surgeon training and assessment. [ABSTRACT FROM AUTHOR]

Purpose Our group has previously reported the development and validation of FIRST (Fundamental Inanimate Robotic Skills Tasks), a series of 4 inanimate robotic skills tasks. Expanding on the initial validation, we now report face, content and construct validity of FIRST in a large multi-institutional cohort of experts and trainees. Materials and Methods A total of 96 residents, fellows and attending surgeons completed the FIRST exercises at participating institutions. Participants were classified based on previous robotic experience and task performance was compared across groups to establish construct validity. Face and content validity was assessed from participant ratings of the tasks on a 5-point Likert scale. Results A total of 51 novice, 22 intermediate and 23 expert participants with a median previous robotic experience of 0 (range 0 to 3), 10 (range 5 to 30) and 200 cases (range 55 to 2,000), respectively (p <0.001), were assessed across all 4 inanimate robotic skills tasks. Expert and intermediate groups reliably outperformed novices (p <0.01). Experts also performed better than intermediates on all exercises (p <0.01). A survey of participants on their perceptions of the tasks yielded excellent face and content validity. Conclusions We confirm robust face, content and construct validity of 4 inanimate robotic training tasks in a large multi-institutional cohort. FIRST tasks are reliably able to discern among expert, intermediate and novice robotic surgeons. Validation data from this large multi-institutional cohort is useful as we incorporate these tasks into a comprehensive robotic training curriculum. [ABSTRACT FROM AUTHOR]

Objectives: The objectives of this review are to summarize the current training modalities and assessment tools used in urological robotic surgery and to propose principles to guide the formation of a comprehensive robotics curriculum. Materials and Methods: The PUBMED database was systematically searched for relevant articles and their citations utilized to broaden our search. These articles were reviewed and summarized with a focus on novel developments. Results: A multitude of training modalities including didactic, dry lab, wet lab, and virtual reality have been developed. The use of these modalities can be divided into basic skills-based exercises and more advanced procedure-based exercises. Clinical training has largely followed traditional methods of surgical teaching with the exception of the unique development of tele-mentoring for the da Vinci interface. Tools to assess both real-life and simulator performance have been developed, including adaptions from Fundamentals of Laparoscopic Surgery and Objective Structured Assessment of Technical Skill, and novel tools such as Global Evaluative Assessment of Robotic Skills. Conclusions: The use of these different entities to create a standardized curriculum for robotic surgery remains elusive. Selection of training modalities and assessment tools should be based upon performance data-based validity and practical feasibility. Comparative assessment of different modalities (cross-modality validity) can help strengthen the development of common skill sets. Constant data collection must occur to guide continuing curriculum improvement. [ABSTRACT FROM AUTHOR]

Objectives To evaluate three standardized robotic surgery training methods, inanimate, virtual reality and in vivo, for their construct validity., To explore the concept of cross-method validity, where the relative performance of each method is compared., Materials and Methods Robotic surgical skills were prospectively assessed in 49 participating surgeons who were classified as follows: 'novice/trainee': urology residents, previous experience <30 cases ( n = 38) and 'experts': faculty surgeons, previous experience ≥30 cases ( n = 11)., Three standardized, validated training methods were used: (i) structured inanimate tasks; (ii) virtual reality exercises on the da Vinci Skills Simulator (Intuitive Surgical, Sunnyvale, CA, USA); and (iii) a standardized robotic surgical task in a live porcine model with performance graded by the Global Evaluative Assessment of Robotic Skills ( GEARS) tool., A Kruskal- Wallis test was used to evaluate performance differences between novices and experts (construct validity)., Spearman's correlation coefficient (ρ) was used to measure the association of performance across inanimate, simulation and in vivo methods (cross-method validity)., Results Novice and expert surgeons had previously performed a median (range) of 0 (0-20) and 300 (30-2000) robotic cases, respectively ( P < 0.001)., Construct validity: experts consistently outperformed residents with all three methods ( P < 0.001)., Cross-method validity: overall performance of inanimate tasks significantly correlated with virtual reality robotic performance (ρ = −0.7, P < 0.001) and in vivo robotic performance based on GEARS (ρ = −0.8, P < 0.0001)., Virtual reality performance and in vivo tissue performance were also found to be strongly correlated (ρ = 0.6, P < 0.001)., Conclusions We propose the novel concept of cross-method validity, which may provide a method of evaluating the relative value of various forms of skills education and assessment., We externally confirmed the construct validity of each featured training tool. [ABSTRACT FROM AUTHOR]

Purpose: We evaluated the concurrent and predictive validity of a novel robotic surgery simulator in a prospective, randomized study. Materials and Methods: A total of 24 robotic surgery trainees performed virtual reality exercises on the da Vinci® Skills Simulator using the da Vinci Si™ surgeon console. Baseline simulator performance was captured. Baseline live robotic performance on ex vivo animal tissue exercises was evaluated by 3 expert robotic surgeons using validated laparoscopic assessment metrics. Trainees were then randomized to group 1—simulator training and group 2—no training while matched for baseline tissue scores. Group 1 trainees underwent a 10-week simulator curriculum. Repeat tissue exercises were done at study conclusion to assess performance improvement. Spearman''s analysis was used to correlate baseline simulator performance with baseline ex vivo tissue performance (concurrent validity) and final tissue performance (predictive validity). The Kruskal-Wallis test was used to compare group performance. Results: Groups 1 and 2 were comparable in pre-study surgical experience and had similar baseline scores on simulator and tissue exercises (p >0.05). Overall baseline simulator performance significantly correlated with baseline and final tissue performance (concurrent and predictive validity each r = 0.7, p <0.0001). Simulator training significantly improved tissue performance on key metrics for group 1 subjects with lower baseline tissue scores (below the 50th percentile) than their group 2 counterparts (p <0.05). Group 1 tended to outperform group 2 on final tissue performance, although the difference was not significant (p >0.05). Conclusions: Our study documents the concurrent and predictive validity of the Skills Simulator. The benefit of simulator training appears to be most substantial for trainees with low baseline robotic skills. [ABSTRACT FROM AUTHOR]