Search

Background Surgeons provide patient care in complex health care systems and must be able to participate in improving both personal performance and the performance of the system. The Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery Milestones are utilized to assess vascular surgery fellows’ (VSF) achievement of graduation targets in the competencies of Systems Based Practice (SBP) and Practice Based Learning and Improvement (PBLI). We investigate the predictive value of semiannual milestones ratings for final achievement within these competencies at the time of graduation. Methods National ACGME milestones data were utilized for analysis. All trainees entering the 2-year vascular surgery fellowship programs in July 2016 were included in the analysis (n = 122). Predictive probability values (PPVs) were obtained for each SBP and PBLI sub-competencies by biannual review periods, to estimate the probability of VSFs not reaching the recommended graduation target based on their previous milestones ratings. Results The rate of nonachievement of the graduation target level 4.0 on the SBP and PBLI sub-competencies at the time of graduation for VSFs was 13.1–25.4%. At the first time point of assessment, 6 months into the fellowship program, the PPV of the SBP and PBLI milestones for nonachievement of level 4.0 upon graduation ranged from 16.3–60.2%. Six months prior to graduation, the PPVs across the 6 sub-competencies ranged from 14.6–82.9%. Conclusions A significant percentage of VSFs do not achieve the ACGME Vascular Surgery Milestone targets for graduation in the competencies of SBP and PBLI, suggesting a need to improve curricula and assessment strategies in these domains across vascular surgery fellowship programs. Reported milestones levels across all time point are predictive of ultimate achievement upon graduation and should be utilized to provide targeted feedback and individualized learning plans to ensure graduates are prepared to engage in personal and health care system improvement once in unsupervised practice.

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

Background With the implementation of competency-based assessment systems, education programs are collecting increasing amounts of data about medical learners. However, learning analytics are rarely employed to use this data to improve medical education. Objective We identified outstanding issues that are limiting the effective adoption of learning analytics in medical education. Methods Participants at an international summit on learning analytics in medical education generated key questions that need to be addressed to move the field forward. Small groups formulated questions related to data stewardship, learner perspectives, and program perspectives. Three investigators conducted an inductive qualitative content analysis on the participant questions, coding the data by consensus and organizing it into themes. One investigator used the themes to formulate representative questions that were refined by the other investigators. Results Sixty-seven participants from 6 countries submitted 195 questions. From them, we identified 3 major themes: implementation challenges (related to changing current practices to collect data and utilize learning analytics); data (related to data collection, security, governance, access, and analysis); and outcomes (related to the use of learning analytics for assessing learners and faculty as well as evaluating programs and systems). We present the representative questions and their implications. Conclusions Our analysis highlights themes regarding implementation, data management, and outcomes related to the use of learning analytics in medical education. These results can be used as a framework to guide stakeholder education, research, and policy development that delineates the benefits and challenges of using learning analytics in medical education.

PurposeInternationally, health professions education scholarship units (HPESUs) are often developed to promote engagement in educational scholarship, yet little is known about how HPESUs change over time or what factors support their longevity. In hopes of helping HPESUs thrive, this study explored factors that shaped the evolution of 8 HPESUs over the past 14 years.MethodThis study involved retrospective case-study analysis of the 8 American, Canadian, and Dutch HPESUs profiled in a 2004 publication. First, the research team summarized key elements of HPESUs from the 2004 articles, then conducted semistructured interviews with the current unit directors. In the first set of questions, directors were asked to reflect on how the unit had changed over time, what successes the unit enjoyed, what enabled these successes, what challenges the unit encountered, and how these challenges were managed. In the second set of questions, questions were tailored to each unit, following up on unique elements from the original article. The team used Braun and Clarke's 6-phase approach to thematic analysis to identify, analyze, and report themes.ResultsThe histories of the units varied widely-some had grown by following their original mandates, some had significant mission shifts, and others had nearly disappeared. Current HPESU directors identified 3 key factors that shaped their HPESU's longitudinal development: the people working within and overseeing the HPESU (the need for a critical mass of scholars, a pipeline for developing scholars, and effective leadership), institutional structures (issues of centralization, unit priorities, and clear messaging), and funding (the need for multiple funding sources).ConclusionsStudy findings offer insights that may help current HPESU directors to strategically plan for their unit's continued development. Tactically harnessing the factors identified could help directors ensure their HPESU's growth and contend with the challenges that threaten the unit's success.

ObjectivesThe national implementation of competency-based medical education (CBME) has prompted an increased interest in identifying and tracking clinical and educational outcomes for emergency medicine training programs. For the 2019 Canadian Association of Emergency Physicians (CAEP) Academic Symposium, we developed recommendations for measuring outcomes in emergency medicine training in the context of CBME to assist educational leaders and systems designers in program evaluation.MethodsWe conducted a three-phase study to generate educational and clinical outcomes for emergency medicine (EM) education in Canada. First, we elicited expert and community perspectives on the best educational and clinical outcomes through a structured consultation process using a targeted online survey. We then qualitatively analyzed these responses to generate a list of suggested outcomes. Last, we presented these outcomes to a diverse assembly of educators, trainees, and clinicians at the CAEP Academic Symposium for feedback and endorsement through a voting process.ConclusionAcademic Symposium attendees endorsed the measurement and linkage of CBME educational and clinical outcomes. Twenty-five outcomes (15 educational, 10 clinical) were derived from the qualitative analysis of the survey results and the most important short- and long-term outcomes (both educational and clinical) were identified. These outcomes can be used to help measure the impact of CBME on the practice of Emergency Medicine in Canada to ensure that it meets both trainee and patient needs.

Purpose A previous study found that milestone ratings at the end of training were higher for male than for female residents in emergency medicine (EM). However, that study was restricted to a sample of 8 EM residency programs and used individual faculty ratings from milestone reporting forms that were designed for use by the program's Clinical Competency Committee (CCC). The objective of this study was to investigate whether similar results would be found when examining the entire national cohort of EM milestone ratings reported by programs after CCC consensus review. Method This study examined longitudinal milestone ratings for all EM residents (n = 1,363; 125 programs) reported to the Accreditation Council for Graduate Medical Education every 6 months from 2014 to 2017. A multilevel linear regression model was used to estimate differences in slope for all subcompetencies, and predicted marginal means between genders were compared at time of graduation. Results There were small but statistically significant differences between males' and females' increase in ratings from initial rating to graduation on 6 of the 22 subcompetencies. Marginal mean comparisons at time of graduation demonstrated gender effects for 4 patient care subcompetencies. For these subcompetencies, males were rated as performing better than females; differences ranged from 0.048 to 0.074 milestone ratings. Conclusions In this national dataset of EM resident milestone assessments by CCCs, males and females were rated similarly at the end of their training for the majority of subcompetencies. Statistically significant but small absolute differences were noted in 4 patient care subcompetencies.

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets.ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation.National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 sub-competencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings.122 VSFs (95.3%) and 52 IVSRs (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6-25.4% across sub-competencies, which was not significantly different between the two groups for any of the sub-competencies (p=0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of ≤ 2.5, with one year remaining prior to graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% - 77.9% for VSFs and 33.3% - 75.0% for IVSRs.The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

Supplemental Digital Content is available in the text., Purpose To investigate the effectiveness of using national, longitudinal milestones data to provide formative assessments to identify residents at risk of not achieving recommended competency milestone goals by residency completion. The investigators hypothesized that specific, lower milestone ratings at earlier time points in residency would be predictive of not achieving recommended Level (L) 4 milestones by graduation. Method In 2018, the investigators conducted a longitudinal cohort study of emergency medicine (EM), family medicine (FM), and internal medicine (IM) residents who completed their residency programs from 2015 to 2018. They calculated predictive values and odds ratios, adjusting for nesting within programs, for specific milestone rating thresholds at 6-month intervals for all subcompetencies within each specialty. They used final milestones ratings (May–June 2018) as the outcome variables, setting L4 as the ideal educational outcome. Results The investigators included 1,386 (98.9%) EM residents, 3,276 (98.0%) FM residents, and 7,399 (98.0%) IM residents in their analysis. The percentage of residents not reaching L4 by graduation ranged from 11% to 31% in EM, 16% to 53% in FM, and 5% to 15% in IM. Using a milestone rating of L2.5 or lower at the end of post-graduate year 2, the predictive probability of not attaining the L4 milestone graduation goal ranged from 32% to 56% in EM, 32% to 67% in FM, and 15% to 36% in IM. Conclusions Longitudinal milestones ratings may provide educationally useful, predictive information to help individual residents address potential competency gaps, but the predictive power of the milestones ratings varies by specialty and subcompetency within these 3 adult care specialties.

Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.

Background Surgical observation is an integral part of surgical training. Junior residents, who have limited understanding of the procedures being performed, use observation to gain an initial exposure to surgical techniques. This study explores the challenges junior residents face from a cognitive standpoint when they are observing surgery. Methods Four focus groups were conducted with 18 general surgery junior residents. Transcripts from these focus groups were analyzed using a qualitative interpretative approach and the findings were explored through the lenses of discovery learning and cognitive load theory. Results Surgical observation is perceived by residents as a learning activity with rich potential. However, two main challenges were identified: directing their attention to the most pertinent element during observation and making sense of what is happening during procedures. Conclusions This study can inform strategies to help junior residents observe surgeries more efficiently to help make surgical observation a better learning experience.

Purpose To investigate whether milestone data obtained from clinical competency committee (CCC) ratings in a single specialty reflected the 6 general competency domains framework. Method The authors examined milestone ratings from all 275 U.S. Accreditation Council for Graduate Medical Education-accredited categorical obstetrics and gynecology (OBGYN) programs from July 1, 2018, to June 30, 2019. The sample size ranged from 1,371 to 1,438 residents from 275 programs across 4 postgraduate years (PGYs), each with 2 assessment periods. The OBGYN milestones reporting form consisted of 28 subcompetencies under the 6 general competency domains. Milestone ratings were determined by each program's CCC. Intraclass correlations (ICCs) and design effects were calculated for each subcompetency by PGY and assessment period. A multilevel confirmatory factor analysis (CFA) perspective was used, and the pooled within-program covariance matrix was obtained to compare the fit of the 6-domain factor model against 3 other plausible models. Results Milestone ratings from 5,618 OBGYN residents were examined. Moderate to high ICCs and design effects greater than 2.0 were prevalent among all subcompetencies for both assessment periods, warranting the use of the multilevel approach in applying CFA to the milestone data. The theory-aided split-patient care (PC) factor model, which used the 6 general competency domains but also included 3 factors within the PC domain (obstetric technical skills, gynecology technical skills, and ambulatory care), was consistently shown as the best-fitting model across all PGY by assessment period conditions, except for one. Conclusions The findings indicate that in addition to using the 6 general competency domains framework in their rating process, CCCs may have further distinguished the PC competency domain into 3 meaningful factors. This study provides internal structure validity evidence for the milestones within a single specialty and may shed light on CCCs' understanding of the distinctive content embedded within the milestones.

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Background The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation. Objective This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME). Methods ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models. Results At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups. Conclusions Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.

Soon after the identification of the novel coronavirus in Wuhan, the city experienced a massive outbreak of COVID-19 that severely strained the existing health system. All of the city's teaching hospitals repurposed buildings for COVID-19 patients, and recruited physicians and trainees of all backgrounds to care of these patients. The authors discuss some of the challenges faced by residency training programs during this period and how they were addressed.

Becoming a physician or other health care professional is a complex and intensely developmental process occurring over a prolonged period of time. The learning path for each medical student, resident, and fellow varies due to different individual learner abilities and curricular designs, clinical contexts, and assessments used by the training program. The slow and uneven evolution to outcomes-based medical education is partly the result of inadequate approaches to programmatic assessment that do not fully address all essential core competencies needed for practice or account for the developmental nature of training. Too many assessments in medical education still focus on single point-in-time performance or function as indirect proxies for actual performance in clinical care for patients and families.Milestones are a modest first step of providing predictive, longitudinal data on a national scale. Longitudinal Milestones data can facilitate the continuous improvement efforts of programs in assessment. However, Milestone judgments are only as good as the assessment data and group processes that inform them. Programmatic assessment should be longitudinally focused and provide all learners with comprehensive and actionable data to guide their professional development and support creation of meaningful individualized action plans. Efforts are urgently needed to rebalance programmatic assessment away from an overreliance on assessment proxies toward more effectively using developmentally focused work-based assessments, routinely incorporate clinical performance and patient experience data, and partner with learners through iterative coproduced assessment activities.

Background Since the launch of the Outcome Project in 2001, the graduate medical education community has been working to implement the 6 general competencies. In 2014, all Obstetrics and Gynecology residency programs implemented specialty-specific milestones to advance competency-based assessment. Each clinical competency committee of the Obstetrics and Gynecology program assesses all residents twice a year on the milestones. These data are reported to the Accreditation Council for Graduate Medical Education as part of a continuous quality improvement effort in graduate medical education. Objective This study aimed to evaluate the correlation between the Accreditation Council for Graduate Medical Education Obstetrics and Gynecology Milestones and residency program graduates’ performance on the American Board of Obstetrics and Gynecology qualifying (written) examination. Study Design We conducted a validity study of all graduating (postgraduate year 4) Obstetrics and Gynecology residents in 2017 within Accreditation Council for Graduate Medical Education–accredited United States training programs (1260 residents from 242 programs). This cohort of residents began receiving milestone assessments during their postgraduate year 2 in 2014; the first-year milestones were implemented for all Accreditation Council for Graduate Medical Education–accredited Obstetrics and Gynecology programs. This cohort completed their sixth and final milestone assessment at graduation in June 2017 for a total of 6 periods of milestone assessments. Data regarding each resident’s milestone ratings in each of the 28 Accreditation Council for Graduate Medical Education subcompetencies for Obstetrics and Gynecology were assessed for their association with candidates’ American Board of Obstetrics and Gynecology qualifying examination scores using a generalized estimating equation regression model. Results Data were available and analyzed from 1184 residents from 240 programs, representing 94% of the total academic year 2017 graduates of Obstetrics and Gynecology residency training programs. There was a substantial association between most milestone ratings at the 6 assessment points and candidates’ performance on the American Board of Obstetrics and Gynecology qualifying examination. The strongest associations with the American Board of Obstetrics and Gynecology were within all 7 of the subcompetencies of Medical Knowledge (range of slope correlation coefficients at final milestone ratings 3.84–5.17; slope coefficients can be interpreted as the gain in qualifying examination points per unit increase in milestone level). At the final milestone assessment, but more modest associations with the American Board of Obstetrics and Gynecology qualifying examination scores were also seen with 9 of the 11 Patient Care and Procedural Skills subcompetencies, the 2 of 2 Practice-Based Learning and Improvement subcompetencies, the 2 of 2 Systems-Based Practice subcompetencies, and 2 of the 3 Professionalism subcompetencies. Only 1 of the 3 Interpersonal and Communication Skills subcompetencies was associated with American Board of Obstetrics and Gynecology qualifying examination scores. Conclusion The pattern of associations between the qualifying examination scores and milestone ratings for the 2017 graduating cohort of Obstetrics and Gynecology residents followed a logical pattern, with the strongest associations seen in Medical Knowledge, and lower to no associations in subcompetencies not as effectively assessed on multiple-choice examinations. Although some positive associations were noted for non-Medical Knowledge milestones, these associations could be caused by correlational rating errors with further study needed to better understand these patterns.

Objectives The purpose of this study was to determine the effect of the Accreditation Council for Graduate Medical Education Milestones on the assessment of neurological surgery residents. The authors sought to determine the feasibility, acceptability, and utility of this new framework in making judgments of progressive competence, its implementation within programs, and the influence on curricula. Residents were also surveyed to elicit the effect of Milestones on their educational experience and professional development. Design, Setting, and Participants In 2015, program leadership and residents from 21 neurological surgery residency programs participated in an online survey and telephone interview in which they reflected on their experiences with the Milestones. Survey data were analyzed using descriptive statistics. Interview transcripts were analyzed using grounded theory. Results Response themes were categorized into 2 groups: outcomes of the Milestones implementation process, and facilitators and barriers. Because of Milestones implementation, participants reported changes to the quality of the assessment process, including the ability to identify struggling residents earlier and design individualized improvement plans. Some programs revised their curricula based on training gaps identified using the Milestones. Barriers to implementation included limitations to the adoption of a developmental progression model in the context of rotation block schedules and misalignment between progression targets and clinical experience. The shift from time-based to competency-based evaluation presented an ongoing adjustment for many programs. Organized preparation before clinical competency committee meetings and diverse clinical competency committee composition led to more productive meetings and perceived improvement in promotion decisions. Conclusions The results of this study can be used by program leadership to help guide further implementation of the Milestones and program improvement. These results also help to guide the evolution of Milestones language and their implementation across specialties.

Background In 2013, milestone ratings became a reporting requirement for emergency medicine (EM) residency programs. Programs rate each resident in the fall and spring on 23 milestone subcompetencies. Objective This study examined the incidence of straight line scoring (SLS) for EM Milestone ratings, defined as a resident being assessed the same score across the milestone subcompetencies. Methods This descriptive analysis measured the frequencies of SLS for all Accreditation Council for Graduate Medical Education (ACGME)–accredited EM programs during the 2015–2016 academic year. Outcomes were the frequency of SLS in the fall and spring milestone assessments, changes in the number of SLS reports, and reporting trends. Chi-square analysis compared nominal variables. Results There were 6257 residents in the fall and 6588 in the spring. Milestone scores were reported for 6173 EM residents in the fall (99% of 6257) and spring (94% of 6588). In the fall, 93% (5753 residents) did not receive SLS ratings and 420 (7%) did, with no significant difference compared with the spring (5776 [94%] versus 397 [6%]). Subgroup analysis showed higher SLS results for residents' first ratings (183 of 2136 versus 237 of 4220, P < .0001) and for their final ratings (200 of 2019 versus 197 of 4354, P < .0001). Twenty percent of programs submitted 10% or more SLS ratings, and a small percentage submitted more than 50% of ratings as SLS. Conclusions Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.

Importance Emergency resuscitation of critically ill patients can challenge team communication and situational awareness. Tools facilitating team performance may enhance patient safety. Objectives To determine resuscitation team members' perceptions of the Situational Awareness Display's utility. Design We conducted focus groups with healthcare providers during Situational Awareness Display development. After simulations assessing the display, we conducted debriefs with participants. Setting Dual site tertiary care level 1 trauma centre in Ottawa, Canada. Participants We recruited by email physicians, nurses and respiratory therapist. Intervention Situational Awareness Display, a visual cognitive aid that provides key clinical information to enhance resuscitation team communication and situational awareness. Main outcomes and measures Themes emerging from focus groups and simulation debriefs. Three reviewers independently coded and analysed transcripts using content qualitative analysis. Results We recruited a total of 33 participants in two focus groups (n = 20) and six simulation debriefs with three 4-5 member teams (n = 13). Majority of participants (10/13) strongly endorsed the Situational Awareness Display's utility in simulation (very or extremely useful). Focus groups and debrief themes included improved perception of patient data, comprehension of context and ability to project to future decisions. Participants described potentially positive and negative impacts on patient safety and positive impacts on provider performance and team communication. Participants expressed a need for easy data entry incorporated into clinical workflow and training on how to use the display. Conclusion Emergency resuscitation team participants felt the Situational Awareness Display has potential to improve provider performance, team communication and situational awareness, ultimately enhancing quality of care.

Background Parents can assess residents’ non-technical skills (NTS) in pediatric emergency departments (EDs). There are no assessment tools, with validity evidence, for parental use in pediatric EDs. The purpose of this study was to develop the Parents’ Assessment of Residents Enacting Non-Technical Skills (PARENTS) educational assessment tool and collect three sources of validity evidence (i.e., content, response process, internal structure) for it. Methods We established content evidence for the PARENTS through interviews with physician-educators and residents, focus groups with parents, a literature review, and a modified nominal group technique with experts. We collected response process evidence through cognitive interviews with parents. To examine the internal structure evidence, we administered the PARENTS and performed exploratory factor analysis. Results Initially, a 20-item PARENTS was developed. Cognitive interviews led to the removal of one closed-ended item, the addition of resident photographs, and wording/formatting changes. Thirty-seven residents and 434 parents participated in the administration of the resulting 19-item PARENTS. Following factor analysis, a one-factor model prevailed. Conclusions The study presents initial validity evidence for the PARENTS. It also highlights strategies for potentially: (a) involving parents in the assessment of residents, (b) improving the assessment of NTS in pediatric EDs, and (c) capturing parents’ perspectives to improve the preparation of future physicians. Electronic supplementary material The online version of this article (10.1186/s12909-017-1042-9) contains supplementary material, which is available to authorized users.

CONTEXT Although health professions education scholarship units (HPESUs) share a commitment to the production and dissemination of rigorous educational practices and research, they are situated in many different contexts and have a wide range of structures and functions.OBJECTIVES In this study, the authors explore the institutional logics common across HPESUs, and how these logics influence the organisation and activities of HPESUs.METHODS The authors analysed interviews with HPESU leaders in Canada (n = 12), Australia (n = 21), New Zealand (n = 3) and the USA (n = 11). Using an iterative process, they engaged in inductive and deductive analyses to identify institutional logics across all participating HPESUs. They explored the contextual factors that influence how these institutional logics impact each HPESU's structure and function.RESULTS Participants identified three institutional logics influencing the organisational structure and functions of an HPESU: (i) the logic of financial accountability; (ii) the logic of a cohesive education continuum, and (iii) the logic of academic research, service and teaching. Although most HPESUs embodied all three logics, the power of the logics varied among units. The relative power of each logic influenced leaders' decisions about how members of the unit allocate their time, and what kinds of scholarly contribution and product are valued by the HPESU.CONCLUSIONS Identifying the configuration of these three logics within and across HPESUs provides insights into the reasons why individual units are structured and function in particular ways. Having a common language in which to discuss these logics can enhance transparency, facilitate evaluation, and help leaders select appropriate indicators of HPESU success.

In response to the numerous challenges resident trainees currently face in their ability to competently acquire the requisite skills, knowledge and attitudes upon graduation, medical educators have looked to a competency-based medical education (CBME) approach as a possible solution. As CBME has already been implemented in many jurisdictions around the world, certain challenges in implementation have been experienced. One important challenge identified relates to how regulatory bodies can either assist or unintentionally hinder implementation. By examining the varied experiences from Canada, the USA and the Netherlands in implementing CBME, this paper identifies how regulatory bodies can support and advance worldwide efforts of furthering its implementation. If regulatory bodies restructure accreditation and regulatory criteria to align with CBME principles, work together in a coordinated fashion to ensure alignment of vital regulatory meaures throughout the training and practice continuum of a physician, and allow for (if not incentivize) individuals and programs to be innovative in adapting CBME to meet their local environments, it is likely that the worldwide implementation of CBME will occur successfully.

Purpose Health professions education scholarship units (HPESUs) in the United States are large in number and diverse in purpose, activities, and contributions. Although each of these units shares a commitment to scholarship, there is no synthetic framework to accurately represent and evaluate their activities and contributions. This study aimed to provide such a framework. Method The authors examined data collected from 11 U.S. HPESU directors. Interviews occurred between April 2015 and February 2016. The research team used a combination of deductive and inductive qualitative techniques to analyze the interview transcripts. The deductive portion drew on Boyer's four-part framework of scholarship; the inductive portion produced a new conceptualization of scholarship at the HPESU level. Results The scholarly activities of HPESUs generally align with Boyer's four types of scholarship-discovery, integration, application, and teaching. However, this categorization fails to capture the interconnectedness and variety of purposes served by these activities. Both are important when considering how best to represent the scholarly contributions made by HPESUs. From their analysis of interviews, the authors developed a three-part framework characterizing HPESU scholarly activities: supporting a scholarly approach to education, supporting educational scholarship within the institution, and supporting HPESU members' scholarship. Conclusions The authors contend that the three-part, unit-level framework for scholarship constructed in this study brings clarity and understanding to the purpose, activities, and contributions made by HPESUs in the United States. The proposed framework may allow unit directors to better justify and advocate for the resources needed to further promote the work of HPESUs.

Background The shift toward competency-based medical education has created a demand for feasible workplace-based assessment tools. Perhaps, more important than competence to assess an individual patient is the ability to successfully manage a surgical clinic. Trainee performance in clinic is a critical component of learning to manage a surgical practice, yet no assessment tool currently exists to assess daily performance in outpatient clinics for surgery residents. The development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT), is described here to address this gap. Study Design A consensus group of experts was gathered to generate dimensions of performance reflective of a competent “generalist” surgeon in clinic. A 6-month pilot study of the OCAT was conducted in orthopedics, general surgery, and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. In all, 2 subsequent feedback sessions and a survey for staff and residents evaluated the OCAT for clarity and utility. Results The OCAT is a 9-item tool, with a global assessment item and 2 short-answer questions. Among the 2 divisions, 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of validity. Analysis of feedback indicated that the entrustability rating scale was useful for surgeons and residents and that the items could be correlated with individual competencies. Conclusions Multiple sources of validity evidence collected in this study demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

There is limited information about how residents in surgical specialties view program strengths and opportunities for improvement (OFIs).Background This study aggregated surgical residents' perspectives on program strengths and OFIs to determine whether there was agreement in perspectives among residents in 5 surgical specialties.Objective Resident consensus lists of program strengths and areas for improvement were aggregated from site visits reports during 2012 and 2013 for obstetrics and gynecology, orthopaedic surgery, otolaryngology, plastic surgery, and surgery programs. Four trained individuals coded each strength or OFI in 1 of 3 categories: (1) factors common to all specialties; (2) program or institutional resources; and (3) factors unique to surgical specialties. Themes were classified as most frequent when listed by residents in more than 20% of the programs and less frequent when listed by residents in less than 20% of the programs.Methods This study included a total of 359 programs, representing 27% to 49% of the Accreditation Council for Graduate Medical Education accredited programs in the 5 specialties. The most frequent strengths were progressive autonomy, collegiality, program leadership, and operative volume. Improving research and didactics, increasing faculty teaching and attendance at educational sessions, and increasing the number of nurse practitioners and physician assistants were common OFIs.Results Factors identified as important by surgical residents related to their learning environment, their educational program, and program and institutional support. Across programs in the study, similar attributes were listed as both program strengths and OFIs.Conclusions

Display Omitted A large situation display can support teamwork.The display should integrate all high-priority information critical to all team members.The context of the situation can influence the extent that such a display is effective.A pilot study demonstrated that a team display supported ED resuscitation teamwork and situational awareness. Effective teamwork in ED resuscitations, including information sharing and situational awareness, could be degraded. Technological cognitive aids can facilitate effective teamwork. Objective: This study focused on the design of an ED situation display and pilot test its influence on teamwork and situational awareness during simulated resuscitation scenarios. Material and methods: The display design consisted of a central area showing the critical dynamic parameters of the interventions with an events time-line below it. Static information was placed at the sides of the display. We pilot tested whether the situation display could lead to higher scores on the Clinical Teamwork Scale (CTS), improved scores on a context-specific Situational Awareness Global Assessment Technique (SAGAT) tool, and team communication patterns that reflect teamwork and situational awareness. Results: Resuscitation teamwork, as measured by the CTS, was overall better with the presence of the situation display as compared with no situation display. Team members discussed interventions more with the situation display compared with not having the situation display. Situational awareness was better with the situation display only in the trauma scenario. Discussion: The situation display could be more effective for certain ED team members and in certain cases. Conclusions: Overall, this pilot study implies that a situation display could facilitate better teamwork and team communication in the resuscitation event.

Rational, aims and objectives Many quality problems exist in health care. We aim to investigate the feasibility and acceptability of using e-learning (defined as computer-based learning modules) to address gaps in quality of care. Methods We performed a qualitative evaluation of participants in a pilot e-learning program. Physician members of six medical teaching units (MTUs) at a multi-site tertiary care teaching hospital were asked to complete two e-learning modules addressing hand hygiene practices and management of community-acquired pneumonia (CAP). An e-learning design team created online modules that were made available to members of the six MTUs for 4 weeks using a password secured website. Use of the modules was voluntary. Participants' perceptions of module content, mode of delivery, and suggestions for improvement were determined through focus groups. We then performed content analysis on the transcripts. We used system data to define patterns of module access. Results Out of 55 eligible users, 30 (55%) logged onto the system at least once. Residents (14/30, 47%) were less likely to use the system than medical students (9/14, 64%) and attending staff (7/11, 64%). Learners at all levels thought the modules were easy to use. Participants liked the knowledge-based material in the CAP module because it directly applied to their work. There were less favourable opinions of the hand hygiene module Conclusions Generating e-learning modules targeted at gaps in quality of care is feasible and acceptable to learners. Future studies should assess whether these approaches lead to desired changes in behavior.

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2 %) and non-technical (83.2 vs. 75.1 %) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

To design and implement a needs assessment process that identifies gaps in caring for critically ill patients in a community hospital.This mixed-method study was conducted between June 2011 and February 2012. A conceptual framework, centered on the critically ill patient, guided the design and selection of the data collection instruments. Different perspectives sampled included regional leaders, healthcare professionals at the community hospital and its referral hospital, as well as family members of patients who had received care at the community ICU. Data sources included interviews (n = 22), walk-throughs (n = 5), focus groups (n = 31), database searches, context questionnaires (n = 8), family surveys (n = 16), and simulations (n = 13).None.Nine needs were identified. At the community hospital, needs identified included lack of access to human resources, gaps in expertise, poor patient flow and ICU bed use, communication, lack of educational opportunities, and gaps in end-of-life care and interprofessional teamwork. Needs were also identified in the interhospital interaction between the community and referral hospitals, which included an inadequate hospital network and gaps in transfer and repatriation of patients. The methodology uncovered the causes and widespread impact of each need and how they interacted with one another. Proposed solutions by the participants are presented including both organizational and educational/clinical solutions.This study captured needs in a complex, interprofessional, interhospital context, which can be targeted with tailored interventions to improve patient outcomes in a community hospital. Furthermore, this study provides a preliminary framework and rigorous methodology to performing a needs assessment in this setting.

OBJECTIVES Traditional training of electroconvulsive therapy (ECT) consists of a combination of didactic and hands-on demonstrations using ECT equipment. Our goal was to explore the potential of a high-fidelity patient simulator (HPS) to train these skills. To our knowledge, this is the first time an HPS has been used for skills training in psychiatry. METHODS Nineteen psychiatry residents participated in this randomized controlled trial to compare traditional training (n = 9) versus training using an HPS (n = 10). Two blinded raters assessed performance using a newly developed checklist and global rating scale for this task (ECT-OSATS) (Objective Structured Assessment of Technical Skills). Residents also completed a pretest-posttest knowledge test and confidence survey. RESULTS Residents in the HPS group performed significantly better in terms of ECT-OSATS when compared with the control group (P < 0.001). All 10 of the HPS group received a "pass" rating following training, whereas only 1 of the 9 control group received a "pass" rating. There were no significant group differences in posttest confidence (P = 0.21) or total knowledge gain scores from pretest to posttest (P = 0.36). CONCLUSIONS The level of clinical skill acquired by trainees in psychiatry for performing ECT is significantly superior using HPS- based training, in contrast to the domains of knowledge and confidence, which appear to be equally imparted using either training modality. The acquisition of skills in administering ECT seems to be an independent variable in relation to a clinician's level of knowledge and confidence in performing ECT.

Summarize the outcomes and best practices of simulation training for laparoscopic surgery.Simulation-based training for laparoscopic surgery has become a mainstay of surgical training. Much new evidence has accrued since previous reviews were published.We systematically searched the literature through May 2011 for studies evaluating simulation, in comparison with no intervention or an alternate training activity, for training health professionals in laparoscopic surgery. Outcomes were classified as satisfaction, skills (in a test setting) of time (to perform the task), process (eg, performance rating), product (eg, knot strength), and behaviors when caring for patients. We used random effects to pool effect sizes.From 10,903 articles screened, we identified 219 eligible studies enrolling 7138 trainees, including 91 (42%) randomized trials. For comparisons with no intervention (n = 151 studies), pooled effect size (ES) favored simulation for outcomes of knowledge (1.18; N = 9 studies), skills time (1.13; N = 89), skills process (1.23; N = 114), skills product (1.09; N = 7), behavior time (1.15; N = 7), behavior process (1.22; N = 15), and patient effects (1.28; N = 1), all P0.05. When compared with nonsimulation instruction (n = 3 studies), results significantly favored simulation for outcomes of skills time (ES, 0.75) and skills process (ES, 0.54). Comparisons between different simulation interventions (n = 79 studies) clarified best practices. For example, in comparison with virtual reality, box trainers have similar effects for process skills outcomes and seem to be superior for outcomes of satisfaction and skills time.Simulation-based laparoscopic surgery training of health professionals has large benefits when compared with no intervention and is moderately more effective than nonsimulation instruction.

Background The costs involved with technology-enhanced simulation remain unknown. Appraising the value of simulation-based medical education (SBME) requires complete accounting and reporting of cost. We sought to summarize the quantity and quality of studies that contain an economic analysis of SBME for the training of health professions learners. Methods We performed a systematic search of MEDLINE, EMBASE, CINAHL, ERIC, PsychINFO, Scopus, key journals, and previous review bibliographies through May 2011. Articles reporting original research in any language evaluating the cost of simulation, in comparison with nonsimulation instruction or another simulation intervention, for training practicing and student physicians, nurses, and other health professionals were selected. Reviewers working in duplicate evaluated study quality and abstracted information on learners, instructional design, cost elements, and outcomes. Results From a pool of 10,903 articles we identified 967 comparative studies. Of these, 59 studies (6.1%) reported any cost elements and 15 (1.6%) provided information on cost compared with another instructional approach. We identified 11 cost components reported, most often the cost of the simulator (n = 42 studies; 71%) and training materials (n = 21; 36%). Ten potential cost components were never reported. The median number of cost components reported per study was 2 (range, 1–9). Only 12 studies (20%) reported cost in the Results section; most reported it in the Discussion (n = 34; 58%). Conclusion Cost reporting in SBME research is infrequent and incomplete. We propose a comprehensive model for accounting and reporting costs in SBME.

Contains fulltext : 175815.pdf (Publisher’s version ) (Closed access) PROBLEM: Health professions education scholarship (HPES) is an important and growing field of inquiry. Problematically, consistent use of terminology regarding the individual roles and organizational structures that are active in this field are lacking. This inconsistency impedes the transferability of current and future findings related to the roles and organizational structures of HPES. APPROACH: Based on data collected during interviews with HPES leaders in Canada, Australia, New Zealand, the United States, and the Netherlands, the authors constructed working definitions for some of the professional roles and an organizational structure that support HPES. All authors reviewed the definitions to ensure relevance across multiple countries. OUTCOMES: The authors define and offer illustrative examples of three professional roles in HPES (clinician educator, HPES research scientist, and HPES administrative leader) and an organizational structure that can support HPES participation (HPES unit). These working definitions are foundational and not all-encompassing and, thus, are offered as stimulus for international dialogue and understanding. NEXT STEPS: With these working definitions, scholars and administrative leaders can examine HPES roles and organizational structures across and between national contexts to decide how lessons learned in other contexts can be applied to their local contexts. Although rigorously constructed, these definitions need to be vetted by the international HPES community. The authors argue that these definitions are sufficiently transferable to support such scholarly investigation and debate.

Residents must strive for excellence in their nontechnical skills (NTS). However, NTS have not traditionally been well-assessed in pediatric emergency departments (EDs). One underutilized assessment strategy is to have parents assess the residents caring for their children. Prior to involving parents in resident assessment, it is essential to identify which NTS parents in pediatric EDs can assess.To explore which resident NTS parents in pediatric EDs can assess.An exploratory qualitative study design was used. It included interviews with faculty members involved in the supervision and assessment of residents in a pediatric ED and residents who had experience working in a pediatric ED, as well as focus groups with parents who had visited a pediatric ED at least twice in the past year.Participants in this study suggested that parents, if provided with the opportunity, can assess residents' communication skills, comfort in a pediatric setting, adaptability, and collaboration.This study demystifies how parents can become involved in the assessment of residents' NTS. The findings will inform the development of assessment strategies and could be used to develop assessment instruments that enable parents to become actively involved in the assessment of residents in pediatric EDs.

High-quality assessment of resident performance is needed to guide individual residents' development and ensure their preparedness to provide patient care. To facilitate this aim, reporting milestones are now required across all internal medicine (IM) residency programs.To describe initial milestone ratings for the population of IM residents by IM residency programs.Cross-sectional study.IM residency programs.All IM residents whose residency program directors submitted milestone data at the end of the 2013-2014 academic year.Ratings addressed 6 competencies and 22 subcompetencies. A rating of "not assessable" indicated insufficient information to evaluate the given subcompetency. Descriptive statistics were calculated to describe ratings across competencies and training years.Data were available for all 21 774 U.S. IM residents from all 383 programs. Overall, 2889 residents (1621 in postgraduate year 1 [PGY-1], 902 in PGY-2, and 366 in PGY-3) had at least 1 subcompetency rated as not assessable. Summaries of average ratings by competency and training year showed higher ratings for PGY-3 residents in all competencies. Overall ratings for each of the 6 individual competencies showed that fewer than 1% of third-year residents were rated as "unsatisfactory" or "conditional on improvement." However, when subcompetency milestone ratings were used, 861 residents (12.8%) who successfully completed training had at least 1 competency with all corresponding subcompetencies graded below the threshold of "readiness for unsupervised practice."Data were derived from a point in time in the first reporting period in which milestones were used.The initial milestone-based evaluations of IM residents nationally suggest that documenting developmental progression of competency is possible over training years. Subcompetencies may identify areas in which residents might benefit from additional feedback and experience. Future work is needed to explore how milestones are used to support residents' development and enhance residency curricula.None.

Resident and faculty views of program strengths and opportunities for improvement (OFIs) offer insight into how stakeholders assess key elements of the learning environment.Background This study sought (1) to assess the degree to which residents and faculty in 359 programs in 5 surgical specialties (obstetrics and gynecology, orthopaedic surgery, otolaryngology, plastic surgery, and surgery) were aligned or divergent in their respective views of program strengths and OFIs; and (2) to evaluate whether responses to selected questions on the Accreditation Council for Graduate Medical Education (ACGME) Resident Survey correlated with strengths or OFIs identified by the residents during the site visit.Objective Faculty and resident lists of program strengths and OFIs in site visit reports for 2012 and 2013 were aggregated, analyzed, and compared to responses on the Resident Survey.Methods While there was considerable alignment in resident and faculty perceptions of program strengths and OFIs, some attributes were more important to one or the other group. Collegiality was valued highly by both stakeholder groups. Responses to 2 questions on the ACGME Resident Survey were associated with resident-identified OFIs in site visit reports pertaining to aspects of the didactic program and responsiveness to resident suggestions for improvement.Results The findings offer program leadership additional insight into how 2 key stakeholder groups view elements of the learning environment as program strengths or OFIs and may serve as useful focal areas for ongoing improvement activities.Conclusions

This article describes opportunities for scholarship in medical education, based on a brief overview of recent changes in medical education. The implications arising from these changes are discussed, with recommendations for focus, and suggestions and examples for making progress in this field. The author discusses 1) the historical context of the current shift toward competency-based medical education, 2) the potential contribution of social and behavioral sciences to medical education scholarship, 3) methods and approaches for supporting scholarship in medical education, and very briefly 4) trends in simulation. The author concludes with a call for quality in medical education scholarship and argues that the most promising and fruitful area of medical education scholarship for the future lies in the field of assessment of individual competence.

A plenary panel session at the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success" discussed barriers educators face in imagining, designing, and implementing studies to address educational challenges. This proceedings article presents a general approach to getting started in education research. Four examples of studies from the medical education literature that illustrate a distinct way to approach specific research questions are discussed. The study designs used are applicable to a variety of education research problems in emergency medicine (EM). Potential applications of studies are discussed, as well as effects and lessons learned.

Most assessment of surgical trainees is based on measures of knowledge, with limited evaluation of their competence to actually perform various surgical procedures. In this study, the authors evaluated a tool they designed to assess a trainee's competence to perform an entire surgical procedure independently, regardless of procedure type or postgraduate year (PGY).In phase 1, the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) was piloted in the University of Ottawa's Division of Orthopaedic Surgery. In phase 2, the refined 11-item tool (8 items rated on a 5-point competency scale, 1 item assessing procedural competence, 2 feedback items) was used in the Divisions of Orthopaedic Surgery and General Surgery to assess residents' performance on 11 common procedures. Quantitative and qualitative analyses were conducted.In phase 2, 34 orthopaedic and general surgeons assessed the performance of 37 residents in 163 procedures. ANOVA demonstrated an effect of PGY. Post hoc analysis found that total procedure scores for PGYs 1 and 2 were lower than those for PGY 3 (P.001), and PGY 3 scores were lower than those for PGYs 4 and 5 (P.02). Analysis of qualitative data indicated that the rating scale was practical and useful for surgeons and residents.This novel evaluation tool successfully discriminated between junior and senior residents and identified surgical competency across various PGY levels regardless of procedure type. Multiple sources of evidence support the O-SCORE as a valid tool for the assessment of trainee operative competency.

Background: Although technology-enhanced simulation is increasingly used in health professions education, features of effective simulation-based instructional design remain uncertain. Aims: Evaluate the effectiveness of instructional design features through a systematic review of studies comparing different simulation-based interventions. Methods: We systematically searched MEDLINE, EMBASE, CINAHL, ERIC, PsycINFO, Scopus, key journals, and previous review bibliographies through May 2011. We included original research studies that compared one simulation intervention with another and involved health professions learners. Working in duplicate, we evaluated study quality and abstracted information on learners, outcomes, and instructional design features. We pooled results using random effects meta-analysis. Results: From a pool of 10 903 articles we identified 289 eligible studies enrolling 18 971 trainees, including 208 randomized trials. Inconsistency was usually large (I 2 4 50%). For skills outcomes, pooled effect sizes ( positive numbers favoring the instructional design feature) were 0.68 for range of difficulty (20 studies; p5 0.001), 0.68 for repetitive practice (7 studies; p ¼ 0.06), 0.66 for distributed practice (6 studies; p ¼ 0.03), 0.65 for interactivity (89 studies; p5 0.001), 0.62 for multiple learning strategies (70 studies; p5 0.001), 0.52 for individualized learning (59 studies; p5 0.001), 0.45 for mastery learning (3 studies; p ¼ 0.57), 0.44 for feedback (80 studies; p5 0.001), 0.34 for longer time (23 studies; p ¼ 0.005), 0.20 for clinical variation (16 studies; p ¼ 0.24), and � 0.22 for group training (8 studies; p ¼ 0.09). Conclusions: These results confirm quantitatively the effectiveness of several instructional design features in simulation-based education.

Introduction Real-time assessment of operator performance during procedural simulation is a common practice that requires undivided attention by 1 or more reviewers, potentially over many repetitions of the same case. Objective To determine whether reviewers display better interrater agreement of procedural competency when observing recorded, rather than live, performance; and to develop an assessment tool for pediatric rapid sequence intubation (pRSI). Methods A framework of a previously established Objective Structured Assessment of Technical Skills (OSATS) tool was modified for pRSI. Emergency medicine residents (postgraduate year 1–4) were prospectively enrolled in a pRSI simulation scenario and evaluated by 2 live raters using the modified tool. Sessions were videotaped and reviewed by the same raters at least 4 months later. Raters were blinded to their initial rating. Interrater agreement was determined by using the Krippendorff generalized concordance method. Results Overall interrater agreement for live review was 0.75 (95% confidence interval [CI], 0.72–0.78) and for video was 0.79 (95% CI, 0.73–0.82). Live review was significantly superior to video review in only 1 of the OSATS domains (Preparation) and was equivalent in the other domains. Intrarater agreement between the live and video evaluation was very good, greater than 0.75 for all raters, with a mean of 0.81 (95% CI, 0.76–0.85). Conclusion The modified OSATS assessment tool demonstrated some evidence of validity in discriminating among levels of resident experience and high interreviewer reliability. With this tool, intrareviewer reliability was high between live and 4-months' delayed video review of the simulated procedure, which supports feasibility of delayed video review in resident assessment.

The purpose of this article is to consolidate some of the key concepts about scholarship in education related to the specialty of anesthesiology. We frame the discussion on two paradigm shifts in medical education, i.e., competency-based education and lifelong learning, and the scholarly approaches to lead these paradigm shifts in anesthesiology.Conventional medical education is being challenged by a shift from time-based education to competency-based education. This potential shift will also create a continuous need to foster a culture of lifelong learning in contrast with the traditional compartmentalized model of undergraduate, postgraduate, and continuing medical education. The specialty of anesthesia has the capacity to lead these changes by enhancing scholarship in education locally and nationally. The promotion of scholarship in education necessitates the creation of infrastructure and accountability frameworks to show return on investment. High-quality scholarship in medical education requires a solid rationale and, ultimately, a demonstrable benefit to patient care.Accountability of lifelong learning to established competency frameworks seems inevitable. Anesthesiology is one of only a few specialties that can truly protect faculty from clinical responsibilities in favour of scholarship pursuits. With appropriate support for scholarship in education, anesthesiologists have an opportunity to lead these paradigm shifts.

OBJECTIVE Resident work hour restrictions have been mandated in the USA largely out of concern that sleep deprivation compromises doctor performance and patient care. However, individuals ability to recognise the effects of sleep deprivation has not been studied in medical education. We examined the perceived impact of sleep deprivation among different groups of postgraduate medical trainees. METHODS A survey addressing work hours, sleepiness and daily functioning was mailed to all residents in the internal medicine, surgery and psychiatry programmes at the University of Toronto who were working at 6 different teaching hospitals. The mailing included the Epworth Sleepiness Scale (ESS), measuring acute sleepiness, and a new Sleep Deprivation Impact (SDI) scale, consisting of 12 items designed to measure the perceived impact of sleep deprivation on an individuals own performance. RESULTS Overall, 62.5% of surgery (95/152) and 59.5% of non-surgery residents (194/326) completed the survey. Surgery residents reported working longer hours per week (83.0 versus 62.5 hours; P < 0.01), and scored higher on the ESS (12.8 versus 9.2; P < 0.01) compared with other residents. Surgery residents scored significantly lower than others on the SDI scale (45.2 versus 51.5, P < 0.01), indicating less perceived impact of sleep deprivation on performance. CONCLUSIONS These results are consistent with the presence of an underlying culture within surgery in which individuals may be less willing to accept a natural limitation of individual performance. Whether these findings represent an actual resilience to sleep deprivation among surgery residents or a misperception within this group remains to be determined.

Background New Residency Review Committee requirements in general surgery require 50 colonoscopies. Simulators have been widely suggested to help prepare residents for live clinical experience. We assessed a computer-based colonoscopy simulator for effective transfer of skills to live patients. Methods A randomized controlled trial included general surgery and internal medicine residents with limited endoscopic experience. Following a pretest, the treatment group (n = 12) practiced on the simulator, while controls (n = 12) received no additional training. Both groups then performed a colonoscopy on a live patient. Technical ability was evaluated by expert endoscopists using previously validated assessment instruments. Results In the live patient setting, the treatment group scored significantly higher global ratings than controls (t (22) = 1.84, P = .04). Only 2 of the 8 computer-based performance metrics correlated significantly with previously validated global ratings of performance. Conclusions Residents trained on a colonoscopy simulator prior to their first patient-based colonoscopy performed significantly better in the clinical setting than controls, demonstrating skill transfer to live patients. The simulator's performance metrics showed limited concurrent validity, suggesting the need for further refinement.

Background We previously reported that a particular type of visual–spatial ability, mental rotation of visual forms, correlates with surgical performance in residents. In the current study, we used functional magnetic resonance imaging (fMRI) to identify patterns of cortical activation associated with mental rotation ability in those same residents. Methods Seventeen surgery residents underwent fMRI scan while performing a mental rotations test (MRT) and a perceptual matching task as a control (CON) for nonimagery components, such as visual attention. A contrast analysis (MRT greater than CON) revealed cortical regions engaged during mental rotation by all participants, and parametric statistical analysis identified regions having the strongest association with MRT performance. Results Significant bilateral (left greater than right) activation was seen in all participants for rotation-versus-perceptual CON contrast. Better MRT performance was associated with greater activation in several cortical regions related to visual imagery and motion processing. Comments Surgery residents represent a unique population in which to study individual differences in visual–spatial abilities and associated neural substrates because they may relate to technical skills. These findings suggest that variation in performance on spatially complex tasks involving imagery may reflect different spatial problem-solving strategies in surgery students.

Context Complete reporting of research is essential to enable consumers to accurately appraise, interpret and apply findings. Quality appraisal checklists are giving way to tools that judge the risk for bias. Objectives We sought to determine the prevalence of these complementary aspects of research reports (completeness of reporting and perceived risk for bias) of randomised studies in health professions education. Methods We searched bibliographic databases for randomised studies of health professions education. We appraised two cohorts representing different time periods (2008–2010 and 2014, respectively) and worked in duplicate to apply the CONSORT guidelines and Cochrane Risk of Bias tool. We explored differences between time periods using independent-samples t-tests or the chi-squared test, as appropriate. Results We systematically identified 180 randomised studies (2008–2010, n = 150; 2014, n = 30). Frequencies of reporting of CONSORT elements within full-text reports were highly variable and most elements were reported in fewer than 50% of studies. We found a statistically significant difference in the CONSORT reporting index (maximum score: 500) between the 2008–2010 (mean ± standard deviation [SD]: 242.7 ± 55.6) and 2014 (mean ± SD: 311.6 ± 53.2) cohorts (p

The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) collectively constitute the foundation of professional self-regulation in the United States. In February 1999, the 2 organizations approved 6 general competencies broadly relevant for all medical practice, followed by the official launch of the Outcomes Project in 2001. It was expected that the competencies would be an antidote to overspecification of accreditation standards, and that they would empower programs to create training programs grounded in meaningful outcomes in a developmental approach. As many programs can attest, the implementation of outcomes-based (eg, competency-based) medical education has been challenging. One reason has been the difficulty in implementing the competencies in both curriculum and assessment. Program leaders lacked shared mental models within their own training programs, accompanied by a lack of shared understanding nationally within disciplines. It is important to remember that 1 of the thorny problems the milestones were intended to address was the sources of unwanted and unwarranted variability in educational and, by extension, clinical outcomes. In addition, the community cannot improve at scale what cannot be measured, and prior frames and approaches to measurement were insufficient and ineffective. A key goal for milestones thus is to help improve the state and quality of measurement through better assessment in graduate medical education to facilitate the improved outcomes everyone desires. Approximately 10 years ago, conversations began on how to more effectively and meaningfully operationalize the competencies to help improve the design of residency and fellowship programs through the use of a developmental framework. In parallel, the ACGME began to explore mechanisms to move the accreditation system to a focus on outcomes using a continuous quality improvement philosophy. Developmental milestones, using narratives to describe in more descriptive terms the professional trajectories of residents, were seen as a way to move the outcomes project forward. Starting in 2007, the disciplines of internal medicine, pediatrics, and surgery began to create developmental milestones for the 6 competencies. Surgery would subsequently delay the development of their milestones focusing first on the SCORE curriculum. The ACGME began to restructure its accreditation processes in 2009, and soon after, milestone groups were constituted for all specialties. Milestone writing groups were cosponsored by the ACGME and the ABMS member certification boards. Early groups had significant latitude in developing their subcompetencies and milestones; specialties that started the process after 2010 used a standard template. Each milestone set was subjected to review by the educational community in the specialty. BOX 1 provides an overview of the purposes of the milestones across key stakeholders, and FIGURE 1 provides an example of a key driver diagram of milestones as an educational and clinical intervention. As FIGURE 1 highlights, milestones can potentially trigger a number of drivers, or mechanisms, to help enable changes in residency and fellowship education. In 2013, the milestones were officially launched in 7 core specialties (emergency medicine, internal medicine, neurological surgery, orthopaedic surgery, pediatrics, diagnostic radiology, and urology) as a formative, continuous quality improvement component of the new accreditation system. The remaining core disciplines and the majority of subspecialties implemented the milestones starting in July 2014. We have now reached an important ‘‘milestone’’ in the implementation process, and our commentary proDOI: http://dx.doi.org/10.4300/JGME-07-03-43

Objectives Increased use of point-of-care ultrasound (US) requires the development of assessment tools that measure the competency of learners. In this study, we developed and tested a tool to assess the quality of point-of-care cardiac US studies performed by novices. Methods In phase 1, the Rapid Assessment of Competency in Echocardiography (RACE) scale was developed on the basis of structured interviews with subject matter experts; the tool was then piloted on a small series of US studies in phase 2. In phase 3, the tool was applied to a sample of 154 point-of-care US studies performed by 12 learners; each study was independently rated by 2 experts, with quantitative analysis subsequently performed. Results Evidence of the content validity of the RACE scale was supported by a consensus exercise, wherein experts agreed on the assessment dimensions and specific items that made up the RACE scale. The tool showed good inter-rater reliability. An analysis of inter-item correlations provided support for the internal structure of the scale, and the tool was able to discriminate between learners early in their point-of-care US learning and those who were more advanced in their training. Conclusions The RACE scale provides a straightforward means to assess learner performance with minimal requirements for evaluator training. Our results support the conclusion that the tool is an effective means of making valid judgments regarding competency in point-of-care cardiac US.