Search

UNSTRUCTURED No abstract/not applicable

Natural language processing (NLP) may be a tool for automating trauma teamwork assessment in simulated scenarios.Using the Trauma Nontechnical Skills Assessment (T-NOTECHS), raters assessed video recordings of trauma teams in simulated pre-debrief (Sim1) and post-debrief (Sim2) trauma resuscitations. We developed codes through directed content analysis and created algorithms capturing teamwork-related discourse through NLP. Using a within subjects pre-post design (n = 150), we compared changes in teams' Sim1 versus Sim2 T-NOTECHS scores and automatically coded discourse to identify which NLP algorithms could identify skills assessed by the T-NOTECHS.Automatically coded behaviors revealed significant post-debrief increases in teams' simulation discourse: Verbalizing Findings, Acknowledging Communication, Directed Communication, Directing Assessment and Role Assignment, and Leader as Hub for Information.Our results suggest NLP can capture changes in trauma team discourse. These findings have implications for the expedition of team assessment and innovations in real-time feedback when paired with speech-to-text technology.

Effective self-assessment is a cornerstone of lifelong professional development; however, evidence suggests physicians have a limited ability to self-assess. Novel strategies to improve the accuracy of learner self-assessment are needed. Our institution's surgical entrustable professional activity (EPA) implementation strategy incorporates resident self-assessment to address this issue. This study evaluates the accuracy of resident self-assessment versus faculty assessment across 5 EPAs in general surgery.Within a single academic general surgery residency program, assessment data for 5 surgery EPAs was prospectively collected using a mobile application. Matched assessments (resident and faculty assessments for the same clinical encounter) were identified and the remainder excluded. Assessment scores were compared using Welch's t test. Agreement was analyzed using Cohen's kappa with squared weights.One thousand eight hundred and fifty-seven EPA assessments were collected in 17 months following implementation. One thousand one hundred and fifty-five (62.2%) were matched pairs. Residents under-rated their own performance relative to faculty assessments (2.36 vs 2.65, p0.01). This pattern held true for all subsets except for Postgraduate Year (PGY)2 residents and Inguinal Hernia EPAs. There was at least moderate agreement between matched resident and faculty EPA assessment scores (κ = 0.57). This was consistent for all EPAs except Trauma evaluations, which were completed by faculty from 2 different departments. Surgery resident self-assessments more strongly agreed with Surgery faculty assessments than Emergency Medicine faculty assessments (κ = 0.58 vs 0.36).Resident EPA self-assessments are equivalent or slightly lower than faculty assessments across a wide breadth of clinical scenarios. Resident and faculty matched assessments demonstrate moderate agreement.

OBJECTIVE: To determine differences in Entrustable Professional Activity (EPA) assessments between male and female general surgery residents. SUMMARY BACKGROUND DATA: Evaluations play a critical role in career advancement for physicians. However, female physicians in training receive lower evaluations and underrate their own performance. Competency-based assessment frameworks, such as EPAs, may help address gender bias in surgery by linking evaluations to specific, observable behaviors. METHODS: In this cohort study, EPA assessments were collected from July 2018 to May 2020. The effect of resident gender on EPA entrustment levels was analyzed using multiple linear and ordered logistic regressions. Narrative comments were analyzed using Latent Dirichlet Allocation (LDA) to identify topics correlated with resident gender. RESULTS: Of the 2,480 EPAs, 1,230 EPAs were submitted by faculty and 1,250 were submitted by residents. After controlling for confounding factors, faculty evaluations of residents were not impacted by resident gender (estimate = 0.09, p = 0.08). However, female residents rated themselves lower by 0.29 (on a 0–4 scale) compared to their male counterparts (p < 0.001). Within narrative assessments, topics associated with resident gender demonstrated that female residents focus on the ‘guidance’ and ‘supervision’ they received while performing an EPA, while male residents were more likely to report ‘independent’ action. CONCLUSIONS: Faculty assessments showed no difference in EPA levels between male and female residents. Female residents rate themselves lower by nearly an entire PGY level compared to male residents. LDA-identified topics suggest this difference in self-assessment is related to differences in perception of autonomy.

BACKGROUND: Entrustable professional activities (EPAs) are a new tool for assessing learners that represents a significant movement in graduate medical education (GME) toward competency‐based assessment and serves as a bridge between milestones and clinical practice. Whenever a major change is implemented to any system, resistance to change is expected. Many change management models have been proposed to overcome this resistance; a newer model is outlined in the book Switch. The objective was to describe the change management principles used to institute an EPA. METHODS: The model introduced in Switch was used as a framework for implementing a trauma resuscitation EPA in a joint effort between the departments of surgery and emergency medicine (EM) at the University of Wisconsin Hospitals and Clinics. The Department of Emergency Medicine used the principles of change management, completing 295 faculty evaluations of trauma resuscitations compared to the Department of Surgery, whose faculty completed 50 evaluations between the study period of July 2018 through October 2019. A survey completed winter 2019 of EM faculty was used to determine the most important principles toward successful implementation. RESULTS: Twenty‐seven of 35 (78% response rate) of EM faculty identified key concepts from Switch as being instrumental in the successful implementation. Internal discussion of the implementation approach used by the Department of Surgery identified several limitations that would have been overcome by using these same change management principles. CONCLUSION: We conclude that the principles of change management provide a useful framework for successfully implementing EPAs into GME.

Background This pilot study examined intraoperative instructional techniques during “takeovers,” defined as the act of an attending taking control of a case from a resident. This work describes what happens during takeovers and identifies possible reasons for takeovers. Methods Intraoperative audio-video recordings during 25 laparoscopic inguinal hernia repair procedures were collected. Participants included 2 postgraduate year–5 residents and 5 attendings. Postoperative evaluation forms were completed by attendings. Coding schemes for takeovers during hernia reduction and mesh placement steps were developed using conventional and directed content analysis in an iterative process by study team members, including individuals with expertise in education, surgery, and surgical education. Results Takeovers occurred in 72% of cases. Frequency of takeovers was not related to case difficulty or differences in resident technical skill levels, nor did they decrease over the duration of the 2-month rotation. Takeovers most commonly occurred when a resident struggled to progress the case. They also occurred when anatomy was unclear or when the attending wanted to teach a specific skill. Differences were identified among attendings regarding frequency of takeovers. The majority of takeover behaviors were directed at instructing residents; however, attendings’ teaching techniques did not vary by resident. Conclusion Attending teaching habits appear to be independent of resident skills and depend on the attending’s teaching style rather than residents’ learning needs. Findings highlight the need for faculty development to help surgical educators learn how to tailor instruction to individual trainees. Additionally, future research is needed to establish the effectiveness of instruction through takeovers in the operating room.

Background Entrustable Professional Activities (EPAs) contain narrative ‘entrustment roadmaps’ designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. Methods All text comments associated with EPA microassessments at a single institution were combined. EPA—entrustment level pairs (e.g. Gallbladder Disease—Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. Results Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). Conclusions LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Objective Concerns over resident ability to practice effectively after graduation have led to the competency-based medical education movement. Entrustable professional activities (EPAs) may facilitate competency-based medical education in surgery, but implementation is challenging. This manuscript describes 1 strategy used to implement EPAs into an academic general surgery residency. Design, Setting, Participants A mobile application was developed incorporating 5 EPAs developed by the American Board of Surgery; residents and faculty from the Departments of Surgery, Emergency Medicine, and Hospital Medicine at a single tertiary care center were trained in its use. Entrustment levels and free text feedback were collected. Self-assessment was paired with supervisor assessment, and faculty assessments were used to inform clinical competency committee entrustment decisions. Feedback was regularly solicited from app users and results distributed on a monthly basis. Results One thousand seven hundred and twenty microassessments were collected over the first 16 months of implementation; 898 (47.8%) were performed by faculty with 569 (66.0%) matched pairs. Engagement was skewed with small numbers of high performers in both resident and faculty groups. Continued development of resident and faculty was required to sustain engagement with the program. Nonsurgical specialties contributed significantly to resident assessments (496, 28.8%). Conclusions EPAs are being successfully integrated into the assessment framework at our institution. EPA implementation in surgery residency is a long-term process that requires investment, but may address limitations in the current assessment framework.

Background This paper explores a method for assessing intraoperative performance by modeling how surgeons integrate psychomotor, procedural, and cognitive skills to manage errors. Methods Audio-video data were collected from general surgery residents (N = 45) performing a simulated laparoscopic ventral hernia repair. Errors were identified using a standard checklist, and speech was coded for elements related to error recognition and management. Epistemic network analysis (ENA) was used to model the integration of error management skills. Results There was no correlation between number or type of errors committed and operative outcome. However, ENA models showed significant differences in the integration of error management skills between high-performing and low-performing residents. Conclusion These results suggest that error checklists and surgeons' speech can be used to model the integration of psychomotor, procedural, and cognitive aspects of intraoperative performance. Moreover, ENA can identify and quantify this integration, providing insight on performance gaps in both individuals and populations.

Background Interest is growing in simulation-based continuing medical education courses for practicing surgeons. However, little research has explored the instruction employed during these courses. This study examines instruction practices used during an annual simulation-based continuing medical education course. Methods Audio–video data were collected from surgeon instructors (n = 12) who taught a simulated laparoscopic hernia repair continuing medical education course across 2 years. Surgeon learners (n = 58) were grouped by their self-reported laparoscopic and hernia repair experience. Instructors’ transcribed dialogue was automatically coded for 5 types of responses to the following questions: anecdotes, confirming, correcting, guidance, and what not to do. Differences in these responses were measured against the progress of the simulations and across learners with different experience levels. Postcourse interviews with instructors were conducted for additional qualitative validation. Results Performing t tests of instructor responses revealed that they were significantly more likely to answer in forms coded as anecdotes when responding to relative experts and in forms coded as what not to do when responding to novices. Linear regressions of each code against normalized progressions of each simulation revealed a significant relationship between progression through a simulation and frequency of the what not to do code for less-experienced learners. Postcourse interviews revealed that instructors continuously assess participants throughout a session and modify their teaching strategies. Conclusion Instructors significantly modified the focus of their teaching as a function both of their learners’ self-reported experience levels, their assessment of learner needs, and learner progression through the training sessions.

Background Given their variegated backgrounds, surgeons taking continuing medical education (CME) courses possess different learning needs. This study examines the relationship between surgeons’ levels of experience and the questions they asked in a simulation-based CME course. Methods We analyzed transcribed audio-video data collected from surgeons participating in a simulated laparoscopic hernia repair CME course and identified four types of questions learners posed to their instructors. Linear regressions compared how often these questions were asked versus self-reported operative experience. Results Both Requesting Guidance and Requesting Confirmation were inversely proportional to experience, whereas Asking About a Specific Case was directly proportional to experience. Requesting Instructor Preference exhibited no significant correlation with experience. Conclusion Practicing surgeons with relatively less experience tend to ask for confirmation and guidance, whereas those with greater experience tend to focus on specific hypothetical scenarios. This data can be used to tailor instruction based on learners’ self-reported experience level.

We analyzed global patterns of expression in genes related to glutamatergic neurotransmission (glutamatergic genes) in healthy human adult brain before determining the effects of chronic alcohol and cocaine exposure on gene expression in the hippocampus. RNA-Seq data from ‘BrainSpan’ was obtained across 16 brain regions from nine control adults. We also generated RNA-Seq data from postmortem hippocampus from eight alcoholics, eight cocaine addicts and eight controls. Expression analyses were undertaken of 28 genes encoding glutamate ionotropic (AMPA, kainate, NMDA) and metabotropic receptor subunits, together with glutamate transporters. The expression of each gene was fairly consistent across the brain with the exception of the cerebellum, the thalamic mediodorsal nucleus and the striatum. GRIN1, encoding the essential NMDA subunit, had the highest expression across all brain regions. Six factors accounted for 84% of the variance in global gene expression. GRIN2B (encoding GluN2B), was up-regulated in both alcoholics and cocaine addicts (FDR corrected p = 0.008). Alcoholics showed up-regulation of three genes relative to controls and cocaine addicts: GRIA4 (encoding GluA4), GRIK3 (GluR7) and GRM4 (mGluR4). Expression of both GRM3 (mGluR3) and GRIN2D (GluN2D) was up-regulated in alcoholics and down-regulated in cocaine addicts relative to controls. Glutamatergic genes are moderately to highly expressed throughout the brain. Six factors explain nearly all the variance in global gene expression. At least in the hippocampus, chronic alcohol use largely up-regulates glutamatergic genes. The NMDA GluN2B receptor subunit might be implicated in a common pathway to addiction, possibly in conjunction with the GABAB1 receptor subunit.