Search

Introduction: Between 2011 and 2014, the Combat Casualty Training Consortium research study sought to evaluate all aspects of combat casualty care, including mortality, with a special focus on the incidence and causes of potentially preventable deaths among American combat fatalities. This study identified a major training gap in critical airway management. Because of the high rate of morbidity and mortality associated with poor or incorrect airway management, an effort to address this training gap was necessary., Materials and Methods: This experiment compared the training using the Advanced Joint Airway Management System, a novel high-fidelity airway simulator, relative to one of the most utilized simulators for endotracheal intubation (ETI) via a parallel group randomized control trial design. Before training, participants (n = 19) attempted an ETI on a cadaver using direct laryngoscopy. Performance during the attempt was recorded and scored by trained blinded raters. Participants were then randomly allocated to either novel or conventional training. Post-training, participants completed a second ETI under the same parameters. Analysis was completed via 2 × 2 mixed analysis of variance for (1) ETI Score, (2) Errors, (3) Critical Failures, and (4) Duration, across both the pre- and post-training tests and between the two groups., Results: A priori power analysis required a total sample size of 84 participants in this experimental design. Consequently, this study is under-powered to reach statistical significance. Scores for all trainees did improve with training, yet analyses did not reveal a difference in overall ETI score between the novel and conventional training group, at baseline or at the post-training test (P = .249)., Conclusion: The Advanced Joint Airway Management System simulator presents a training effect that is comparable to the conventional training model. However, given this study's small sample size, these results must be considered preliminary and further research is merited to draw firm conclusions about its impact on trainee performance. Future studies engaging larger cohorts of trainees and exploring the other capabilities of the Advanced Joint Airway Management System (cricothyroidotomy, needle chess decompression) are needed to further examine the educational potential of this novel airway management training system., (© The Association of Military Surgeons of the United States 2024. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site–for further information please contact journals.permissions@oup.com.)

Background: Our aim was assess whether an integrated Advanced Modular Manikin (AMM) provides improved participant experience compared with use of peripheral simulators alone during a standardized trauma team scenario. Simulation-based team training has been shown to improve team performance. To address limitations of existing manikin simulators, the AMM platform was created that enables interconnectedness, interoperability, and integration of multiple simulators ("peripherals") into an adaptable, comprehensive training system., Methods: A randomized single-blinded, crossover study with 2 conditions was used to assess learner experience differences when using the integrated AMM platform vs peripheral simulators. First responders, anesthesiologists, and surgeons rated their experience and workload with the conditions in a 3-scene standardized trauma scenario. Participant ratings were compared and focus groups conducted to obtain insight into participant experience., Results: Fourteen teams (n = 42) participated. Team experience ratings were higher for the integrated AMM condition compared with peripherals (Cohen's d = .25, p = 0.016). Participant experience varied by background with surgeons and first responders rating their experience significantly higher compared with anesthesiologists (p < 0.001). Higher workload ratings were observed with the integrated AMM condition (Cohen's d = .35, p = 0.014) driven primarily by anesthesiologist ratings. Focus groups revealed that participants preferred the integrated AMM condition based on its increased realism, physiologic responsiveness, and feedback provided on their interventions., Conclusions: This first comprehensive evaluation suggests that integration with the AMM platform provides benefits over individual peripheral simulators and has the potential to expand simulation-based learning opportunities and enhance learner experience, especially for surgeons., (Copyright © 2021 American College of Surgeons. All rights reserved.)

Purpose: Minimally invasive surgery requires objective methods for skill evaluation and training. This work presents the minimally acceptable classification (MAC) criterion for computational surgery: Given an obvious novice and an obvious expert, a surgical skill evaluation classifier must yield 100% accuracy. We propose that a rigorous motion analysis algorithm must meet this minimal benchmark in order to justify its cost and use., Methods: We use this benchmark to investigate two concepts: First, how separable is raw, multidimensional dry laboratory laparoscopic motion data between obvious novices and obvious experts? We utilized information theoretic techniques to analytically address this. Second, we examined the use of intent vectors to classify surgical skill using three FLS tasks., Results: We found that raw motion data alone are not sufficient to classify skill level; however, the intent vector approach is successful in classifying surgical skill level for certain tasks according to the MAC criterion. For a pattern cutting task, this approach yields 100% accuracy in leave-one-user-out cross-validation., Conclusion: Compared to prior art, the intent vector approach provides a generalized method to assess laparoscopic surgical skill using basic motion segments and passes the MAC criterion for some but not all FLS tasks.

Purpose: Most evaluations of surgical workflow or surgeon skill use simple, descriptive statistics (e.g., time) across whole procedures, thereby deemphasizing critical steps and potentially obscuring critical inefficiencies or skill deficiencies. In this work, we examine off-line, temporal clustering methods that chunk training procedures into clinically relevant surgical tasks or steps during robot-assisted surgery., Methods: Features calculated from the isogony principle are used to train four common machine learning algorithms from dry-lab laparoscopic data gathered from three common training exercises. These models are used to predict the binary or ternary skill level of a surgeon. K-fold and leave-one-user-out cross-validation are used to assess the accuracy of the generated models., Results: It is shown that the proposed scalar features can be trained to create 2-class and 3-class classification models that map to fundamentals of laparoscopic surgery skill level with median 85 and 63% accuracy in cross-validation, respectively, for the targeted dataset. Also, it is shown that the 2-class models can discern class at 90% of best-case mean accuracy with only 8 s of data from the start of the task., Conclusion: Novice and expert skill levels of unobserved trials can be discerned using a state vector machine trained with parameters based on the isogony principle. The accuracy of this classification comes within 90% of the classification accuracy from observing the full trial within 10 s of task initiation on average.

Background: The challenges of training and assessing endourologic skill have driven the development of new training systems. The Center for Research in Education and Simulation Technologies (CREST) has developed a team and a methodology to facilitate this development process., Methods: Backwards design principles were applied. A panel of experts first defined desired clinical and educational outcomes. Outcomes were subsequently linked to learning objectives. Gross task deconstruction was performed, and the primary domain was classified as primarily involving decision-making, psychomotor skill, or communication. A more detailed cognitive task analysis was performed to elicit and prioritize relevant anatomy/tissues, metrics, and errors. Reference anatomy was created using a digital anatomist and clinician working off of a clinical data set. Three dimensional printing can facilitate this process. When possible, synthetic or virtual tissue behavior and textures were recreated using data derived from human tissue. Embedded sensors/markers and/or computer-based systems were used to facilitate the collection of objective metrics. A learning Verification and validation occurred throughout the engineering development process., Results: Nine endourology-relevant training systems were created by CREST with this approach. Systems include basic laparoscopic skills (BLUS), vesicourethral anastomosis, pyeloplasty, cystoscopic procedures, stent placement, rigid and flexible ureteroscopy, GreenLight PVP (GL Sim), Percutaneous access with C-arm (CAT), Nephrolithotomy (NLM), and a vascular injury model. Mixed modalities have been used, including "smart" physical models, virtual reality, augmented reality, and video. Substantial validity evidence for training and assessment has been collected on systems. An open source manikin-based modular platform is under development by CREST with the Department of Defense that will unify these and other commercial task trainers through the common physiology engine, learning management system, standard data connectors, and standards., Conclusion: Using the CREST process has and will ensure that the systems we create meet the needs of training and assessing endourologic skills.

Background: Major vessel injury (MVI) is a dangerous complication associated with laparoscopic surgery that leads, if not properly handled, to blood loss, conversion to open surgery, and eventually death. In this paper, we describe the preliminary evaluation of the SimPORTAL MVI model, created with the goal of simulating an intra-corporeal injury to a large vessel., Methods: For this study, we created MVI models for 17 residents (PGY 1-4). Each resident was asked to perform an intracorporeal knot on a penrose drain within a maximum time limit of 6 min (in accordance with European basic laparoscopic urological skills rules) and then to subsequently repair a vessel injury on the MVI model, which was perfused with synthetic blood, within a maximum blood loss of 3 L. During the vessel repair, low lights and pulse sounds were used to simulate the operating room environment. All participants filled out a survey pre- and post-task to score various aspects of the model., Results: We successfully created a model that simulates a critical surgical event. None of the participants reported having previous experience repairing a MVI. Six participants were able to perform the intracorporeal knot, and 12 residents (70.5%) were able to repair the MVI model under the given time and blood loss limits. Eleven participants agreed that the MVI model behaves like a real vessel, and six felt to be capable of performing the task prior to attempting it. Sixteen participants thought that the MVI model should be part of laparoscopic curriculums during residency., Conclusions: The SimPORTAL MVI model is a feasible low-cost model that would be well appreciated as a part of laparoscopic curriculum for residents. Minor improvements, including pressure measurement in the vessel for task assessment, will be made in the future, and further studies are necessary to definitively validate this model.

Purpose: Assessing construct, face and content validity of the camera handling trainer (CHT), a novel low-fidelity training device for 30° laparoscope navigation skills., Methods: We developed a custom-designed box trainer with clinically based graphic targets. A total of 117 participants, stratified according to their previous experience (novice, competent, expert), took part to a CHT session and subsequently were asked to fill out a survey to assess the impact of the CHT on their 30° laparoscope navigation skills. Sixty of them were also studied for task performance during a 1-h session, with multiple time measurements., Results: All participants, regardless of the previous experience, significantly improved their performance after the CHT session. Regarding construct validity, the mean task performance on the last measurement for novice group was found to be comparable to the mean first attempt of both competent (p = 0.12) and expert (p = 0.24) participants. All participants agreed that "the CHT is a valid training tool" and that "the CHT should be part of the regular dry laboratory training sessions", assessing both face and content validity. Limitations include the need for assessment of predictive validity., Conclusions: The CHT is a valid training tool for 30° laparoscope navigation and thus should be considered as one of the fundamental exercises during basic laparoscopic hands-on training sessions for urologists.

Purpose: We evaluated the internal and construct validity of an assessment tool for cystoscopic and ureteroscopic cognitive and psychomotor skills at a multi-institutional level., Materials and Methods: Subjects included a total of 30 urology residents at Ohio State University, Columbus, Ohio; Penn Presbyterian Medical Center, Philadelphia, Pennsylvania; and Mayo Clinic, Rochester, Minnesota. A single external blinded reviewer evaluated cognitive and psychomotor skills associated with cystoscopic and ureteroscopic surgery using high fidelity bench models. Exercises included navigation, basketing and relocation; holmium laser lithotripsy; and cystoscope assembly. Each resident received a total cognitive score, checklist score and global psychomotor skills score. Construct validity was assessed by calculating correlations between training year and performance scores (both cognitive and psychomotor). Internal validity was confirmed by calculating correlations between test components., Results: The median total cognitive score was 91 (IQR 86.25, 97). For psychomotor performance residents had a median total checklist score of 7 (IQR 5, 8) and a median global psychomotor skills score of 21 (IQR 18, 24.5). Construct validity was supported by the positive and statistically significant correlations between training year and total cognitive score (r = 0.66, 95% CI 0.39-0.82, p = 0.01), checklist scores (r = 0.66, 95% CI 0.35-0.84, p = 0.32) and global psychomotor skills score (r = 0.76, 95% CI 0.55-0.88, p = 0.002). The internal validity of OSATS was supported since total cognitive and checklist scores correlated with the global psychomotor skills score., Conclusions: In this multi-institutional study we successfully demonstrated the construct and internal validity of an objective assessment of cystoscopic and ureteroscopic cognitive and technical skills, including laser lithotripsy., (Copyright © 2015 American Urological Association Education and Research, Inc. Published by Elsevier Inc. All rights reserved.)

Objectives: Reducing preventable deaths because of uncontrolled hemorrhage, tension pneumothorax, and airway loss is a priority. As part of a research initiative comparing different training models, this study evaluated the reliability and validity of a test that assesses combat medic performance during a polytrauma scenario using live animal models., Methods: Nine procedural checklists and seven global rating scales were piloted with four cohorts of soldiers (n = 94) at two U.S. training sites. Cohorts represented "novice" to "proficient" trainees. Procedure scores and a mean global score were calculated per subject. The intraclass correlation was calculated per procedure, with 0.70 as the threshold for acceptability. An overall difference among cohorts was hypothesized: Cohort 4 (proficient) > Cohort 3 (competent) > Cohort 2 (beginners) > Cohort 1 (novice) trainees. Data were analyzed using Kruskal-Wallis and analysis of variance., Results: At Site A, intraclass correlation coefficients ranged from 74% to 93% for 6 of 9 procedures. Cohorts differed significantly on hemorrhage control, needle decompression, cricothyrotomy, amputation management, chest tube insertion, and mean global scores. Cohort 4 outperformed the others, and Cohorts 2 and 3 outperformed Cohort 1., Conclusion: The test differentiates novices from beginners, competent, and proficient trainees on difficult procedures and overall performance., (Reprint & Copyright © 2014 Association of Military Surgeons of the U.S.)

Objective: To determine the interrater reliability and construct validity of 3 separate assessment tools for assessing trainee skills in pediatric airway endoscopy simulation., Design: An Objective Structured Assessment of Technical Skills (OSATS) was developed in which examinees were asked to name and assemble the airway foreign body instruments and retrieve a foreign body from an infant airway mannequin. Each examinee's performance was assessed in a blinded fashion by 3 pediatric otolaryngology faculty at separate residency programs using 3 assessment tools: (1) objective quantifiable measures list (eg, assists needed, forceps openings, foreign body drops), (2) 15-point OSATS checklist, and (3) Global Rating Index for Technical Skills (GRITS). Setting Otolaryngology residency program. Subjects Examinees (medical students, n = 3; otolaryngology residents, n = 17; pediatric otolaryngology faculty, n = 3) and raters (n = 3). Main Outcome Measures Interrater reliability and construct validity. Results Anonymized split-screen videos simultaneously capturing each examinee's instrument handling and the endoscopic videos were created for all 23 examinees. Nineteen videos were chosen for review by 3 raters. The interrater reliability as measured by the intraclass correlation for objective quantifiable measures ranged from 0.46 to 0.98. The intraclass correlation coefficient was 0.95 for the 15-point OSATS checklist and 0.95 for the GRITS; both showed a high degree of construct validity with scores correlating with previous experience. Conclusion Assessment tools for skills assessments must have high interrater reliability and construct validity. When assessing trainee skills in pediatric airway foreign body scenarios, the 15-point OSATS checklist developed by this group or the GRITS meets these criteria.

Background and Purpose: The Fundamentals of Laparoscopic Surgery (FLS(™)) skills curriculum has validity evidence supporting use for assessing laparoscopic skills for general surgeons. As charged by the American Urological Association (AUA) Laparoscopy, Robotic, and New Surgical Technology Committee, we sought to develop and validate a urology-specific FLS, referred to as the Basic Laparoscopic Urologic Surgery (BLUS(©)) skills curriculum. The psychomotor component consists of three existing FLS tasks and one new clip-applying task., Materials and Methods: An animate renal artery model was designed for a clip-applying skills task. We assessed the acceptability and construct validity of using BLUS for basic laparoscopic skills assessment for urologists. A cohort of practicing urologists, fellows, residents, and medical students completed the tasks at the AUA Annual Meetings in 2010 and 2011., Results: All exercises were acceptable and demonstrated excellent face and content validity (>4.5/5 on a five-point Likert scale). Practicing clinical urologists (N=81) outperformed residents and medical students (N=35) in time to completion of circle cut (P<0.01) and in keeping scissor tips toward the center of the circle (P<0.01). Practicing urologists who reported >3 laparoscopic procedures per week were faster at the peg-transfer exercise (P<0.05) and the cutting exercise (P<0.01) than those reporting one to two procedures. More errors were committed for clip-applying among practicing urologists who perform one to two laparoscopic procedures (1.24) vs. those who perform >3 procedures (0.57) per week (P<0.01)., Conclusions: All exercises including the novel clip-applying model demonstrated good acceptability and evidence of construct validity (face, content, concurrent and convergent validity) for assessment of basic laparoscopic skill for urologic surgeons.

Objective: To develop a robust psychomotor skills curriculum to teach pediatric airway foreign body retrieval and to assess the effect of this curriculum on residents' confidence in and ability to perform the complete task in an infant airway mannequin., Study Design: Instructional course. Objective Structured Assessment of Technical Skills (OSATS)., Setting: Surgical simulation laboratory., Subjects and Methods: A half-day simulation-based course was developed to train otolaryngology residents in bronchoscopic foreign body retrieval. This complex psychomotor skill was deconstructed into subtasks. The following curricular learning objectives were presented and assessed: understanding of tracheobronchial anatomy, ability to adequately visualize the larynx with laryngoscopy, proficiency in rigid bronchoscopy, and familiarity with foreign body instrumentation. Residents were objectively evaluated on their ability to perform the complete task on a simulator before and after the course using an OSATS grading system. Confidence in successfully assembling the instruments and completing the task was assessed at these time periods., Results: Seventeen otolaryngology residents completed the study. Confidence in assembling the instruments and in performing the complete task increased on average by 81% and 43%, respectively (P < .001). Using a 15-point OSATS grading system, the average score for the precourse was 7 and for the postcourse was 11.3 (P < .001)., Conclusion: Simulation-based subtask training shows promise as an effective and reproducible method to teach the complex psychomotor task of airway foreign body retrieval. Completion of the curriculum led to a significant improvement in residents' confidence in and ability to perform bronchoscopic foreign body retrieval in an infant airway mannequin.

Objective: To present modern educational psychology theory and apply these concepts to validity and reliability of surgical skills training and assessment., Design: In a series of cross-disciplinary meetings, we applied a unified approach of behavioral science principles and theory to medical technical skills education given the recent advances in the theories in the field of behavioral psychology and statistics., Conclusions: While validation of the individual simulation tools is important, it is only one piece of a multimodal curriculum that in and of itself deserves examination and study. We propose concurrent validation throughout the design of simulation-based curriculum rather than once it is complete. We embrace the concept that validity and curriculum development are interdependent, ongoing processes that are never truly complete. Individual predictive, construct, content, and face validity aspects should not be considered separately but as interdependent and complementary toward an end application. Such an approach could help guide our acceptance and appropriate application of these exciting new training and assessment tools for technical skills training in medicine.

Purpose: To understand how urologists acquire resection skills we analyzed factors correlating with favorable resection metrics in groups defined as experts, residents and novices. We then evaluated discriminate validity by determining factors correlating with proficiency among individuals in the expert, resident and novice groups., Materials and Methods: A total of 136 subjects completed the protocol, including 72 urologists, 45 residents and 19 novices. After a pre-task questionnaire and training video subjects performed a standardized 5-minute resection task. Primary metrics were gm resected, blood loss, irrigant volume used, foot pedal use and differential time spent with orientation, cutting or coagulation., Results: Among experts larger resection correlated with more time spent cutting (p <0.001). In contrast, increased coagulation time correlated with gm resected in the novice group (p = 0.001). The number of transurethral prostate resections that residents reported having done in the real operating room correlated with gm resected (p = 0.043), use of more irrigating fluid (p = 0.024) and less time spent coagulating (p = 0.027) on the simulator. In residents and experts exclusively primary resection efficiency metrics, fluid use and blood loss correlated with cuts at tissue and correlated inversely with coagulation and orientation time (p <0.05)., Conclusions: Different factors determine transurethral prostate resection performance metrics among experts, residents and novices. These correlations reinforce discriminate validity and provide insight into specific factors that likely determine success at different training levels. Such data could be used to isolate and train skill subsets in the curriculum and they may elucidate the safest and most efficient approach to train resection skills.

Objectives: To assess the effect of the changing landscape of urologic residency education and training on resident operative exposure and inter-resident variability.Methods: The Accreditation Council for Graduate Medical Education (ACGME) case logs for graduating urology chief residents were reviewed from Academic Year (AY) 2009-2010 to 2016-2017. Cases were stratified into the 4 ACGME categories - general urology, endourology, oncology, and reconstruction. Linear regression models analyzed the association between training year, volume, and type of cases performed. Inter-resident variability in case exposure was calculated by the difference between the ACGME reported 10th and 90th percentiles.Results: During the study period, the mean number of cases performed per resident was 1092 (standard deviation 32.7). Although there was no significant change in total case volume, there were changes within case categories. Endoscopic, retroperitoneal oncology, and male reconstruction case volume all increased significantly (Δ20.1%, Δ 5.1%, Δ 8.2%, respectively, all P < .05). This was balanced with a concomitant decrease in pelvic oncology and female reconstruction cases (Δ 10.0% and Δ 14.5%, respectively, both P < .05). There was a 27.8% increase in laparoscopic/robotic cases (P < .001). The ratio difference between the 10th percentile and 90th percentile ranged from a low of 2.5 for retroperitoneal oncology cases to a high of 5.2 for extracorporeal shock wave lithotripsy and percutaneous nephrolithotomy.Conclusion: From AY2009-2010 to 2016-2017, residency case volume has remained constant, but there has been a change in types of cases performed and proliferation of minimally invasive techniques. Significant variability of inter-resident operative experience was noted. [ABSTRACT FROM AUTHOR]