Search

Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories.This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings.All ACGME-accredited vascular surgery training programs within the United States.All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%).As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

The quality and effectiveness of vascular surgery education should be evaluated based on patient care outcomes. To investigate predictive associations between trainee performance and subsequent patient outcomes, a critical first step is to determine the conceptual alignment of educational competencies with clinical outcomes in practice. We sought to generate expert consensus on the conceptual alignment of the Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery subcompetencies with patient care outcomes across different Vascular Quality Initiative (VQI) registries.A national panel of vascular surgeons with expertise in both clinical care and education were recruited to participate in a modified Delphi expert consensus building process to map ACGME Vascular Surgery subcompetencies (educational markers of resident performance) to VQI clinical modules (patient outcomes). A master list of items for rating was created, including the 31 ACGME Vascular Surgery subcompetencies and 8 VQI clinical registries (endovascular abdominal aortic aneurysm repair, open abdominal aortic aneurysm, thoracic endovascular aortic repair, carotid endarterectomy, carotid artery stent, infrainguinal, suprainguinal, and peripheral vascular intervention). These items were entered into an iterative Delphi process. Positive consensus was reached when 75% or more of the participants ranked an item as mandatory. Intraclass correlations (ICCs) were used to evaluate consistency between experts for each Delphi round.A total of 13 experts who contributed to the development of the Vascular Surgery Milestones participated; 12 experts (92%) participated in both rounds of the Delphi process. Two rounds of Delphi were conducted, as suggested by excellent expert agreement (round 1, ICC = 0.79 [95% confidence interval, 0.74-0.84]; round 2, ICC = 0.97 [95% confidence interval, 0.960-.98]). Using the predetermined consensus cutoff threshold, the Delphi process reduced the number of subcompetencies mapped to patient care outcomes from 31 to a range of 9 to 15 across the 8 VQI clinical registries. Practice-based learning and improvement, and professionalism subcompetencies were identified as less relevant to patient outcome variables captured by the VQI registries after the final round, and the only the systems-based practice subcompetency that was identified as relevant was radiation safety in two of the endovascular registries.A national panel of vascular surgeon experts reported a high degree of agreement on the relevance of ACGME subcompetencies to patient care outcomes as captured in the VQI clinical registry. Systems-based practice, practice-based learning and improvement, and professionalism competencies were identified as less relevant to patient outcomes after specific surgical procedures.

Background Surgeons provide patient care in complex health care systems and must be able to participate in improving both personal performance and the performance of the system. The Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery Milestones are utilized to assess vascular surgery fellows’ (VSF) achievement of graduation targets in the competencies of Systems Based Practice (SBP) and Practice Based Learning and Improvement (PBLI). We investigate the predictive value of semiannual milestones ratings for final achievement within these competencies at the time of graduation. Methods National ACGME milestones data were utilized for analysis. All trainees entering the 2-year vascular surgery fellowship programs in July 2016 were included in the analysis (n = 122). Predictive probability values (PPVs) were obtained for each SBP and PBLI sub-competencies by biannual review periods, to estimate the probability of VSFs not reaching the recommended graduation target based on their previous milestones ratings. Results The rate of nonachievement of the graduation target level 4.0 on the SBP and PBLI sub-competencies at the time of graduation for VSFs was 13.1–25.4%. At the first time point of assessment, 6 months into the fellowship program, the PPV of the SBP and PBLI milestones for nonachievement of level 4.0 upon graduation ranged from 16.3–60.2%. Six months prior to graduation, the PPVs across the 6 sub-competencies ranged from 14.6–82.9%. Conclusions A significant percentage of VSFs do not achieve the ACGME Vascular Surgery Milestone targets for graduation in the competencies of SBP and PBLI, suggesting a need to improve curricula and assessment strategies in these domains across vascular surgery fellowship programs. Reported milestones levels across all time point are predictive of ultimate achievement upon graduation and should be utilized to provide targeted feedback and individualized learning plans to ensure graduates are prepared to engage in personal and health care system improvement once in unsupervised practice.

Supplemental Digital Content is available in the text., Purpose The United States Medical Licensing Examination (USMLE) sequence and the Accreditation Council for Graduate Medical Education (ACGME) milestones represent 2 major components along the continuum of assessment from undergraduate through graduate medical education. This study examines associations between USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores and ACGME emergency medicine (EM) milestone ratings. Method In February 2019, subject matter experts (SMEs) provided judgments of expected associations for each combination of Step examination and EM subcompetency. The resulting sets of subcompetencies with expected strong and weak associations were selected for convergent and discriminant validity analysis, respectively. National-level data for 2013–2018 were provided; the final sample included 6,618 EM residents from 158 training programs. Empirical bivariate correlations between milestone ratings and Step scores were calculated, then those correlations were compared with the SMEs’ judgments. Multilevel regression analyses were conducted on the selected subcompetencies, in which milestone ratings were the dependent variable, and Step 1 score, Step 2 CK score, and cohort year were independent variables. Results Regression results showed small but statistically significant positive relationships between Step 2 CK score and the subcompetencies (regression coefficients ranged from 0.02 [95% confidence interval (CI), 0.01–0.03] to 0.12 [95% CI, 0.11–0.13]; all P < .05), with the degree of association matching the SMEs’ judgments for 7 of the 9 selected subcompetencies. For example, a 1 standard deviation increase in Step 2 CK score predicted a 0.12 increase in MK-01 milestone rating, when controlling for Step 1. Step 1 score showed a small statistically significant effect with only the MK-01 subcompetency (regression coefficient = 0.06 [95% CI, 0.05–0.07], P < .05). Conclusions These results provide incremental validity evidence in support of Step 1 and Step 2 CK score and EM milestone rating uses.

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

With the rapid uptake of entrustable professional activties and entrustment decision-making as an approach in undergraduate and graduate education in medicine and other health professions, there is a risk of confusion in the use of new terminologies. The authors seek to clarify the use of many words related to the concept of entrustment, based on existing literature, with the aim to establish logical consistency in their use. The list of proposed definitions includes independence, autonomy, supervision, unsupervised practice, oversight, general and task-specific trustworthiness, trust, entrust(ment), entrustable professional activity, entrustment decision, entrustability, entrustment-supervision scale, retrospective and prospective entrustment-supervision scales, and entrustment-based discussion. The authors conclude that a shared understanding of the language around entrustment is critical to strengthen bridges among stages of training and practice, such as undergraduate medical education, graduate medical education, and continuing professional development. Shared language and understanding provide the foundation for consistency in interpretation and implementation across the educational continuum.

Background With the implementation of competency-based assessment systems, education programs are collecting increasing amounts of data about medical learners. However, learning analytics are rarely employed to use this data to improve medical education. Objective We identified outstanding issues that are limiting the effective adoption of learning analytics in medical education. Methods Participants at an international summit on learning analytics in medical education generated key questions that need to be addressed to move the field forward. Small groups formulated questions related to data stewardship, learner perspectives, and program perspectives. Three investigators conducted an inductive qualitative content analysis on the participant questions, coding the data by consensus and organizing it into themes. One investigator used the themes to formulate representative questions that were refined by the other investigators. Results Sixty-seven participants from 6 countries submitted 195 questions. From them, we identified 3 major themes: implementation challenges (related to changing current practices to collect data and utilize learning analytics); data (related to data collection, security, governance, access, and analysis); and outcomes (related to the use of learning analytics for assessing learners and faculty as well as evaluating programs and systems). We present the representative questions and their implications. Conclusions Our analysis highlights themes regarding implementation, data management, and outcomes related to the use of learning analytics in medical education. These results can be used as a framework to guide stakeholder education, research, and policy development that delineates the benefits and challenges of using learning analytics in medical education.

Introduction To date, research studying health professions education scholarship units has overlooked the perspectives of research scientists in the field, despite their important role in these units. This research explores how health professions education scientists uphold and/or upend the institutional logics of the units they work within. Methods Recruited via snowball sampling, 29 Canadian health professions education scientists participated in semi-structured interviews that lasted between 32–55 min. Data analysis was informed by the theories of organizational institutionalism—specifically, the microfoundation element of sensemaking. Results Respondents’ narrations of career success were overtly linked to their research-oriented pursuits above other expectations (i.e., teaching, service). Discussion Respondents’ narrative revealed a mismatch between the value they associated with teaching- and service-related pursuits, and the value the institution associated with those pursuits. Participants indicated a need to reconceptualize the institutional value associated with these endeavors. Electronic supplementary material The online version of this article (10.1007/s40037-020-00577-1) contains supplementary material, which is available to authorized users.

PurposeInternationally, health professions education scholarship units (HPESUs) are often developed to promote engagement in educational scholarship, yet little is known about how HPESUs change over time or what factors support their longevity. In hopes of helping HPESUs thrive, this study explored factors that shaped the evolution of 8 HPESUs over the past 14 years.MethodThis study involved retrospective case-study analysis of the 8 American, Canadian, and Dutch HPESUs profiled in a 2004 publication. First, the research team summarized key elements of HPESUs from the 2004 articles, then conducted semistructured interviews with the current unit directors. In the first set of questions, directors were asked to reflect on how the unit had changed over time, what successes the unit enjoyed, what enabled these successes, what challenges the unit encountered, and how these challenges were managed. In the second set of questions, questions were tailored to each unit, following up on unique elements from the original article. The team used Braun and Clarke's 6-phase approach to thematic analysis to identify, analyze, and report themes.ResultsThe histories of the units varied widely-some had grown by following their original mandates, some had significant mission shifts, and others had nearly disappeared. Current HPESU directors identified 3 key factors that shaped their HPESU's longitudinal development: the people working within and overseeing the HPESU (the need for a critical mass of scholars, a pipeline for developing scholars, and effective leadership), institutional structures (issues of centralization, unit priorities, and clear messaging), and funding (the need for multiple funding sources).ConclusionsStudy findings offer insights that may help current HPESU directors to strategically plan for their unit's continued development. Tactically harnessing the factors identified could help directors ensure their HPESU's growth and contend with the challenges that threaten the unit's success.

ObjectivesThe national implementation of competency-based medical education (CBME) has prompted an increased interest in identifying and tracking clinical and educational outcomes for emergency medicine training programs. For the 2019 Canadian Association of Emergency Physicians (CAEP) Academic Symposium, we developed recommendations for measuring outcomes in emergency medicine training in the context of CBME to assist educational leaders and systems designers in program evaluation.MethodsWe conducted a three-phase study to generate educational and clinical outcomes for emergency medicine (EM) education in Canada. First, we elicited expert and community perspectives on the best educational and clinical outcomes through a structured consultation process using a targeted online survey. We then qualitatively analyzed these responses to generate a list of suggested outcomes. Last, we presented these outcomes to a diverse assembly of educators, trainees, and clinicians at the CAEP Academic Symposium for feedback and endorsement through a voting process.ConclusionAcademic Symposium attendees endorsed the measurement and linkage of CBME educational and clinical outcomes. Twenty-five outcomes (15 educational, 10 clinical) were derived from the qualitative analysis of the survey results and the most important short- and long-term outcomes (both educational and clinical) were identified. These outcomes can be used to help measure the impact of CBME on the practice of Emergency Medicine in Canada to ensure that it meets both trainee and patient needs.

Purpose A previous study found that milestone ratings at the end of training were higher for male than for female residents in emergency medicine (EM). However, that study was restricted to a sample of 8 EM residency programs and used individual faculty ratings from milestone reporting forms that were designed for use by the program's Clinical Competency Committee (CCC). The objective of this study was to investigate whether similar results would be found when examining the entire national cohort of EM milestone ratings reported by programs after CCC consensus review. Method This study examined longitudinal milestone ratings for all EM residents (n = 1,363; 125 programs) reported to the Accreditation Council for Graduate Medical Education every 6 months from 2014 to 2017. A multilevel linear regression model was used to estimate differences in slope for all subcompetencies, and predicted marginal means between genders were compared at time of graduation. Results There were small but statistically significant differences between males' and females' increase in ratings from initial rating to graduation on 6 of the 22 subcompetencies. Marginal mean comparisons at time of graduation demonstrated gender effects for 4 patient care subcompetencies. For these subcompetencies, males were rated as performing better than females; differences ranged from 0.048 to 0.074 milestone ratings. Conclusions In this national dataset of EM resident milestone assessments by CCCs, males and females were rated similarly at the end of their training for the majority of subcompetencies. Statistically significant but small absolute differences were noted in 4 patient care subcompetencies.

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets.ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation.National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 sub-competencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings.122 VSFs (95.3%) and 52 IVSRs (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6-25.4% across sub-competencies, which was not significantly different between the two groups for any of the sub-competencies (p=0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of ≤ 2.5, with one year remaining prior to graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% - 77.9% for VSFs and 33.3% - 75.0% for IVSRs.The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

ImportanceEvaluation of trainees in graduate medical education training programs using Milestones has been in place since 2013. It is not known whether trainees who have lower ratings during the last year of training go on to have concerns related to interactions with patients in posttraining practice.ObjectiveTo investigate the association between resident Milestone ratings and posttraining patient complaints.Design, Setting, and ParticipantsThis retrospective cohort study included physicians who completed Accreditation Council for Graduate Medical Education (ACGME)–accredited programs between July 1, 2015, and June 30, 2019, and worked at a site that participated in the national Patient Advocacy Reporting System (PARS) program for at least 1 year. Milestone ratings from ACGME training programs and patient complaint data from PARS were collected. Data analysis was conducted from March 2022 to February 2023.ExposuresLowest professionalism (P) and interpersonal and communication skills (ICS) Milestones ratings 6 months prior to the end of training.Main Outcomes and MeasuresPARS year 1 index scores, based on recency and severity of complaints.ResultsThe cohort included 9340 physicians with median (IQR) age of 33 (31-35) years; 4516 (48.4%) were women physicians. Overall, 7001 (75.0%) had a PARS year 1 index score of 0, 2023 (21.7%) had a score of 1 to 20 (moderate), and 316 (3.4%) had a score of 21 or greater (high). Among physicians in the lowest Milestones group, 34 of 716 (4.7%) had high PARS year 1 index scores, while 105 of 3617 (2.9%) with Milestone ratings of 4.0 (proficient), had high PARS year 1 index scores. In a multivariable ordinal regression model, physicians in the 2 lowest Milestones rating groups (0-2.5 and 3.0-3.5) were statistically significantly more likely to have higher PARS year 1 index scores than the reference group with Milestones ratings of 4.0 (0-2.5 group: odds ratio, 1.2 [95% CI, 1.0-1.5]; 3.0-3.5 group: odds ratio, 1.2 [95% CI, 1.1-1.3]).Conclusions and RelevanceIn this study, trainees with low Milestone ratings in P and ICS near the end of residency were at increased risk for patient complaints in their early posttraining independent physician practice. Trainees with lower Milestone ratings in P and ICS may need more support during graduate medical education training or in the early part of their posttraining practice career.

Organizational readiness is critical for successful implementation of an innovation. We evaluated program readiness to implement Competence by Design (CBD), a model of Competency-Based Medical Education (CBME), among Canadian postgraduate training programs. A survey of program directors was distributed 1 month prior to CBD implementation in 2019. Questions were informed by the R = MC2 framework of organizational readiness and addressed: program motivation, general capacity for change, and innovation-specific capacity. An overall readiness score was calculated. An ANOVA was conducted to compare overall readiness between disciplines. Survey response rate was 42% (n = 79). The mean overall readiness score was 74% (30���98%). There was no difference in scores between disciplines. The majority of respondents agreed that successful implementation of CBD was a priority (74%), and that their leadership (94%) and faculty and residents (87%) were supportive of change. Fewer perceived that CBD was a move in the right direction (58%) and that implementation was a manageable change (53%). Curriculum mapping, competence committees and programmatic assessment activities were completed by >90% of programs, while

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice.To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents.This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019.Longitudinal formative assessment using the Milestones assessment system reported to the ACGME.To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories.This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth.These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Supplemental Digital Content is available in the text., Purpose To investigate the effectiveness of using national, longitudinal milestones data to provide formative assessments to identify residents at risk of not achieving recommended competency milestone goals by residency completion. The investigators hypothesized that specific, lower milestone ratings at earlier time points in residency would be predictive of not achieving recommended Level (L) 4 milestones by graduation. Method In 2018, the investigators conducted a longitudinal cohort study of emergency medicine (EM), family medicine (FM), and internal medicine (IM) residents who completed their residency programs from 2015 to 2018. They calculated predictive values and odds ratios, adjusting for nesting within programs, for specific milestone rating thresholds at 6-month intervals for all subcompetencies within each specialty. They used final milestones ratings (May–June 2018) as the outcome variables, setting L4 as the ideal educational outcome. Results The investigators included 1,386 (98.9%) EM residents, 3,276 (98.0%) FM residents, and 7,399 (98.0%) IM residents in their analysis. The percentage of residents not reaching L4 by graduation ranged from 11% to 31% in EM, 16% to 53% in FM, and 5% to 15% in IM. Using a milestone rating of L2.5 or lower at the end of post-graduate year 2, the predictive probability of not attaining the L4 milestone graduation goal ranged from 32% to 56% in EM, 32% to 67% in FM, and 15% to 36% in IM. Conclusions Longitudinal milestones ratings may provide educationally useful, predictive information to help individual residents address potential competency gaps, but the predictive power of the milestones ratings varies by specialty and subcompetency within these 3 adult care specialties.

Objective The Surgical Exploration and Discovery (SEAD) program was established to facilitate career decision-making by providing preclerkship students with comprehensive exposure to surgical specialties. Our short-term findings demonstrated that, compared to a control group, SEAD participants showed significantly greater career-related learning. The purpose of this study was to understand the long-term impact of the SEAD program. Design This was a prospective cohort study. One group of students participated in a 2-week surgical curriculum (SEAD group) while another group read only the program manual (Manual group). Students were surveyed following their residency selection 3 years later. The outcome measures were final specialty preference (medical or surgical), program utility, and program satisfaction. Setting Undergraduate Medical Education, Faculty of Medicine, at the University of Ottawa in Ottawa, Ontario, Canada. Participants A total of 18 medical students in the SEAD group, and 18 in the Manual group. Results Survey response rate was 100%. There was no significant difference in the number of students who pursued surgical careers in the SEAD and Manual groups. All students who pursued a surgical residency were ‘very interested’ in surgery prior to SEAD—initial interest in surgery had a significant influence on final residency preference. Ninety-four percent (n = 17) of SEAD participants described the SEAD program as valuable to facilitating their career decision-making. Conclusions Although SEAD does not generate sustained new interest in surgical disciplines, graduating students believe the program is valuable in facilitating career decision-making and perceive the program as a worthwhile time investment. These findings were true for students who selected both surgical and medical specialties, suggesting that early, multifaceted, exposure to surgery is a valuable addition to career exploration even for students who ultimately don't pursue surgical specialties. Going forward, integrating a longitudinal mentorship program may further improve the value of SEAD.

Milestones specific to orthopaedic surgical training document individual resident progress through skill development in multiple dimensions. Residents increasingly interact with and are assessed by surgeons in both academic and private practice environments. Milestones describe the skills that support competence. One of the primary goals of milestones is to provide continuous data for educational quality improvement of residency programs. They provide a dialogue between surgeons who supervise residents or fellows and the program's Clinical Competency Committee throughout a resident's education. The orthopaedic milestones were developed jointly by the Accreditation Council for Graduate Medical Education and the American Board of Orthopaedic Surgery. The working team was designed with broad representation within the specialty. The milestones were introduced to orthopaedic residencies in 2013. Orthopaedics is a 5-year training program; the first comprehensive longitudinal data set is now available for study. This summary provides historical perspective on the development of the milestones, state of current milestone implementation, attempts to establish validity, challenges with the milestones, and the development of next-generation assessment tools.

Supplemental Digital Content is available in the text., Purpose To investigate whether clinical competency committees (CCCs) were consistent in applying milestone ratings for first-year residents over time or whether ratings increased or decreased. Method Beginning in December 2013, the Accreditation Council for Graduate Medical Education (ACGME) initiated a phased-in requirement for reporting milestones; emergency medicine (EM), diagnostic radiology (DR), and urology (UR) were among the earliest reporting specialties. The authors analyzed CCC milestone ratings of first-year residents from 2013 to 2016 from all ACGME-accredited EM, DR, and UR programs for which they had data. The number of first-year residents in these programs ranged from 2,838 to 2,928 over this time period. The program-level average milestone rating for each subcompetency was regressed onto the time of observation using a random coefficient multilevel regression model. Results National average program-level milestone ratings of first-year residents decreased significantly over the observed time period for 32 of the 56 subcompetencies examined. None of the other subcompetencies showed a significant change. National average in-training examination scores for each of the specialties remained essentially unchanged over the time period, suggesting that differences between the cohorts were not likely an explanatory factor. Conclusions The findings indicate that CCCs tend to become more stringent or maintain consistency in their ratings of beginning residents over time. One explanation for these results is that CCCs may become increasingly comfortable in assigning lower ratings when appropriate. This finding is consistent with an increase in confidence with the milestone rating process and the quality of feedback it provides.

Background Surgical observation is an integral part of surgical training. Junior residents, who have limited understanding of the procedures being performed, use observation to gain an initial exposure to surgical techniques. This study explores the challenges junior residents face from a cognitive standpoint when they are observing surgery. Methods Four focus groups were conducted with 18 general surgery junior residents. Transcripts from these focus groups were analyzed using a qualitative interpretative approach and the findings were explored through the lenses of discovery learning and cognitive load theory. Results Surgical observation is perceived by residents as a learning activity with rich potential. However, two main challenges were identified: directing their attention to the most pertinent element during observation and making sense of what is happening during procedures. Conclusions This study can inform strategies to help junior residents observe surgeries more efficiently to help make surgical observation a better learning experience.

Purpose To investigate whether milestone data obtained from clinical competency committee (CCC) ratings in a single specialty reflected the 6 general competency domains framework. Method The authors examined milestone ratings from all 275 U.S. Accreditation Council for Graduate Medical Education-accredited categorical obstetrics and gynecology (OBGYN) programs from July 1, 2018, to June 30, 2019. The sample size ranged from 1,371 to 1,438 residents from 275 programs across 4 postgraduate years (PGYs), each with 2 assessment periods. The OBGYN milestones reporting form consisted of 28 subcompetencies under the 6 general competency domains. Milestone ratings were determined by each program's CCC. Intraclass correlations (ICCs) and design effects were calculated for each subcompetency by PGY and assessment period. A multilevel confirmatory factor analysis (CFA) perspective was used, and the pooled within-program covariance matrix was obtained to compare the fit of the 6-domain factor model against 3 other plausible models. Results Milestone ratings from 5,618 OBGYN residents were examined. Moderate to high ICCs and design effects greater than 2.0 were prevalent among all subcompetencies for both assessment periods, warranting the use of the multilevel approach in applying CFA to the milestone data. The theory-aided split-patient care (PC) factor model, which used the 6 general competency domains but also included 3 factors within the PC domain (obstetric technical skills, gynecology technical skills, and ambulatory care), was consistently shown as the best-fitting model across all PGY by assessment period conditions, except for one. Conclusions The findings indicate that in addition to using the 6 general competency domains framework in their rating process, CCCs may have further distinguished the PC competency domain into 3 meaningful factors. This study provides internal structure validity evidence for the milestones within a single specialty and may shed light on CCCs' understanding of the distinctive content embedded within the milestones.

Background Quality of training is determined through programs’ compliance with accreditation standards, often set for a number of years. However, perspectives on quality of training within these standards may differ from the clinicians’ perspectives on quality of training. Knowledge on how standards relate to clinicians’ perspectives on quality of training is currently lacking yet is expected to lead to improved accreditation design. Methods This qualitative study design was based on a case-study research approach. We analyzed accreditation standards and conducted 29 interviews with accreditors, clinical supervisors and trainees across Australia and the Netherlands about the quality and accreditation of specialist medical training programs. The perspectives were coded and either if applicable compared to national accreditation standards of both jurisdictions, or thematized to the way stakeholders encounter accreditation standards in practice. Results There were two evident matches and four mismatches between the perspectives of clinicians and the accreditation standards. The matches are: (1) accreditation is necessary (2) trainees are the best source for quality measures. The mismatches are: (3) fundamental training aspects that accreditation standards do not capture: the balance between training and service provision, and trainee empowerment (4) using standards lack dynamism and (5) quality improvement; driven by standards or intrinsic motivation of healthcare professionals. Conclusion In our Australian and Dutch health education cases accreditation is an accepted phenomenon which may be improved by trainee empowerment, a dynamic updating process of standards and by flexibility in its use.

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Background The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation. Objective This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME). Methods ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models. Results At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups. Conclusions Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

The iconic Miller's pyramid, proposed in 1989, characterizes 4 levels of assessment in medical education ("knows," "knows how," "shows how," "does"). The frame work has created a worldwide awareness of the need to have different assessment approaches for different expected outcomes of education and training. At the time, Miller stressed the innovative use of simulation techniques, geared at the third level ("shows how"); however, the "does" level, assessment in the workplace, remained a largely uncharted area. In the 30 years since Miller's conference address and seminal paper, much attention has been devoted to procedures and instrument development for workplace-based assessment. With the rise of competency-based medical education (CBME), the need for approaches to determine the competence of learners in the clinical workplace has intensified. The proposal to use entrustable professional activities as a framework of assessment and the related entrustment decision making for clinical responsibilities at designated levels of supervision of learners (e.g., direct, indirect, and no supervision) has become a recent critical innovation of CBME at the "does" level. Analysis of the entrustment concept reveals that trust in a learner to work without assistance or supervision encompasses more than the observation of "doing" in practice (the "does" level). It implies the readiness of educators to accept the inherent risks involved in health care tasks and the judgment that the learner has enough experience to act appropriately when facing unexpected challenges. Earning this qualification requires qualities beyond observed proficiency, which led the authors to propose adding the level "trusted" to the apex of Miller's pyramid.

Soon after the identification of the novel coronavirus in Wuhan, the city experienced a massive outbreak of COVID-19 that severely strained the existing health system. All of the city's teaching hospitals repurposed buildings for COVID-19 patients, and recruited physicians and trainees of all backgrounds to care of these patients. The authors discuss some of the challenges faced by residency training programs during this period and how they were addressed.

Becoming a physician or other health care professional is a complex and intensely developmental process occurring over a prolonged period of time. The learning path for each medical student, resident, and fellow varies due to different individual learner abilities and curricular designs, clinical contexts, and assessments used by the training program. The slow and uneven evolution to outcomes-based medical education is partly the result of inadequate approaches to programmatic assessment that do not fully address all essential core competencies needed for practice or account for the developmental nature of training. Too many assessments in medical education still focus on single point-in-time performance or function as indirect proxies for actual performance in clinical care for patients and families.Milestones are a modest first step of providing predictive, longitudinal data on a national scale. Longitudinal Milestones data can facilitate the continuous improvement efforts of programs in assessment. However, Milestone judgments are only as good as the assessment data and group processes that inform them. Programmatic assessment should be longitudinally focused and provide all learners with comprehensive and actionable data to guide their professional development and support creation of meaningful individualized action plans. Efforts are urgently needed to rebalance programmatic assessment away from an overreliance on assessment proxies toward more effectively using developmentally focused work-based assessments, routinely incorporate clinical performance and patient experience data, and partner with learners through iterative coproduced assessment activities.

Background Since 2013, US residency programs have used the competency-based framework of the Milestones to report resident progress and to provide feedback to residents. The implementation of Milestones-based assessments, clinical competency committee (CCC) meetings, and processes for providing feedback varies among programs and warrants systematic examination across specialties. Objective We sought to determine how varying assessment, CCC, and feedback implementation strategies result in different outcomes in resource expenditure and stakeholder engagement, and to explore the contextual forces that moderate these outcomes. Methods From 2017 to 2018, interviews were conducted of program directors, CCC chairs, and residents in emergency medicine (EM), internal medicine (IM), pediatrics, and family medicine (FM), querying their experiences with Milestone processes in their respective programs. Interview transcripts were coded using template analysis, with the initial template derived from previous research. The research team conducted iterative consensus meetings to ensure that the evolving template accurately represented phenomena described by interviewees. Results Forty-four individuals were interviewed across 16 programs (5 EM, 4 IM, 5 pediatrics, 3 FM). We identified 3 stages of Milestone-process implementation, including a resource-intensive early stage, an increasingly efficient transition stage, and a final stage for fine-tuning. Conclusions Residency program leaders can use these findings to place their programs along an implementation continuum and gain an understanding of the strategies that have enabled their peers to progress to improved efficiency and increased resident and faculty engagement.

Background Since the launch of the Outcome Project in 2001, the graduate medical education community has been working to implement the 6 general competencies. In 2014, all Obstetrics and Gynecology residency programs implemented specialty-specific milestones to advance competency-based assessment. Each clinical competency committee of the Obstetrics and Gynecology program assesses all residents twice a year on the milestones. These data are reported to the Accreditation Council for Graduate Medical Education as part of a continuous quality improvement effort in graduate medical education. Objective This study aimed to evaluate the correlation between the Accreditation Council for Graduate Medical Education Obstetrics and Gynecology Milestones and residency program graduates’ performance on the American Board of Obstetrics and Gynecology qualifying (written) examination. Study Design We conducted a validity study of all graduating (postgraduate year 4) Obstetrics and Gynecology residents in 2017 within Accreditation Council for Graduate Medical Education–accredited United States training programs (1260 residents from 242 programs). This cohort of residents began receiving milestone assessments during their postgraduate year 2 in 2014; the first-year milestones were implemented for all Accreditation Council for Graduate Medical Education–accredited Obstetrics and Gynecology programs. This cohort completed their sixth and final milestone assessment at graduation in June 2017 for a total of 6 periods of milestone assessments. Data regarding each resident’s milestone ratings in each of the 28 Accreditation Council for Graduate Medical Education subcompetencies for Obstetrics and Gynecology were assessed for their association with candidates’ American Board of Obstetrics and Gynecology qualifying examination scores using a generalized estimating equation regression model. Results Data were available and analyzed from 1184 residents from 240 programs, representing 94% of the total academic year 2017 graduates of Obstetrics and Gynecology residency training programs. There was a substantial association between most milestone ratings at the 6 assessment points and candidates’ performance on the American Board of Obstetrics and Gynecology qualifying examination. The strongest associations with the American Board of Obstetrics and Gynecology were within all 7 of the subcompetencies of Medical Knowledge (range of slope correlation coefficients at final milestone ratings 3.84–5.17; slope coefficients can be interpreted as the gain in qualifying examination points per unit increase in milestone level). At the final milestone assessment, but more modest associations with the American Board of Obstetrics and Gynecology qualifying examination scores were also seen with 9 of the 11 Patient Care and Procedural Skills subcompetencies, the 2 of 2 Practice-Based Learning and Improvement subcompetencies, the 2 of 2 Systems-Based Practice subcompetencies, and 2 of the 3 Professionalism subcompetencies. Only 1 of the 3 Interpersonal and Communication Skills subcompetencies was associated with American Board of Obstetrics and Gynecology qualifying examination scores. Conclusion The pattern of associations between the qualifying examination scores and milestone ratings for the 2017 graduating cohort of Obstetrics and Gynecology residents followed a logical pattern, with the strongest associations seen in Medical Knowledge, and lower to no associations in subcompetencies not as effectively assessed on multiple-choice examinations. Although some positive associations were noted for non-Medical Knowledge milestones, these associations could be caused by correlational rating errors with further study needed to better understand these patterns.

Objectives The purpose of this study was to determine the effect of the Accreditation Council for Graduate Medical Education Milestones on the assessment of neurological surgery residents. The authors sought to determine the feasibility, acceptability, and utility of this new framework in making judgments of progressive competence, its implementation within programs, and the influence on curricula. Residents were also surveyed to elicit the effect of Milestones on their educational experience and professional development. Design, Setting, and Participants In 2015, program leadership and residents from 21 neurological surgery residency programs participated in an online survey and telephone interview in which they reflected on their experiences with the Milestones. Survey data were analyzed using descriptive statistics. Interview transcripts were analyzed using grounded theory. Results Response themes were categorized into 2 groups: outcomes of the Milestones implementation process, and facilitators and barriers. Because of Milestones implementation, participants reported changes to the quality of the assessment process, including the ability to identify struggling residents earlier and design individualized improvement plans. Some programs revised their curricula based on training gaps identified using the Milestones. Barriers to implementation included limitations to the adoption of a developmental progression model in the context of rotation block schedules and misalignment between progression targets and clinical experience. The shift from time-based to competency-based evaluation presented an ongoing adjustment for many programs. Organized preparation before clinical competency committee meetings and diverse clinical competency committee composition led to more productive meetings and perceived improvement in promotion decisions. Conclusions The results of this study can be used by program leadership to help guide further implementation of the Milestones and program improvement. These results also help to guide the evolution of Milestones language and their implementation across specialties.

Background In 2013, milestone ratings became a reporting requirement for emergency medicine (EM) residency programs. Programs rate each resident in the fall and spring on 23 milestone subcompetencies. Objective This study examined the incidence of straight line scoring (SLS) for EM Milestone ratings, defined as a resident being assessed the same score across the milestone subcompetencies. Methods This descriptive analysis measured the frequencies of SLS for all Accreditation Council for Graduate Medical Education (ACGME)–accredited EM programs during the 2015–2016 academic year. Outcomes were the frequency of SLS in the fall and spring milestone assessments, changes in the number of SLS reports, and reporting trends. Chi-square analysis compared nominal variables. Results There were 6257 residents in the fall and 6588 in the spring. Milestone scores were reported for 6173 EM residents in the fall (99% of 6257) and spring (94% of 6588). In the fall, 93% (5753 residents) did not receive SLS ratings and 420 (7%) did, with no significant difference compared with the spring (5776 [94%] versus 397 [6%]). Subgroup analysis showed higher SLS results for residents' first ratings (183 of 2136 versus 237 of 4220, P < .0001) and for their final ratings (200 of 2019 versus 197 of 4354, P < .0001). Twenty percent of programs submitted 10% or more SLS ratings, and a small percentage submitted more than 50% of ratings as SLS. Conclusions Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.

Importance Emergency resuscitation of critically ill patients can challenge team communication and situational awareness. Tools facilitating team performance may enhance patient safety. Objectives To determine resuscitation team members' perceptions of the Situational Awareness Display's utility. Design We conducted focus groups with healthcare providers during Situational Awareness Display development. After simulations assessing the display, we conducted debriefs with participants. Setting Dual site tertiary care level 1 trauma centre in Ottawa, Canada. Participants We recruited by email physicians, nurses and respiratory therapist. Intervention Situational Awareness Display, a visual cognitive aid that provides key clinical information to enhance resuscitation team communication and situational awareness. Main outcomes and measures Themes emerging from focus groups and simulation debriefs. Three reviewers independently coded and analysed transcripts using content qualitative analysis. Results We recruited a total of 33 participants in two focus groups (n = 20) and six simulation debriefs with three 4-5 member teams (n = 13). Majority of participants (10/13) strongly endorsed the Situational Awareness Display's utility in simulation (very or extremely useful). Focus groups and debrief themes included improved perception of patient data, comprehension of context and ability to project to future decisions. Participants described potentially positive and negative impacts on patient safety and positive impacts on provider performance and team communication. Participants expressed a need for easy data entry incorporated into clinical workflow and training on how to use the display. Conclusion Emergency resuscitation team participants felt the Situational Awareness Display has potential to improve provider performance, team communication and situational awareness, ultimately enhancing quality of care.

Background Parents can assess residents’ non-technical skills (NTS) in pediatric emergency departments (EDs). There are no assessment tools, with validity evidence, for parental use in pediatric EDs. The purpose of this study was to develop the Parents’ Assessment of Residents Enacting Non-Technical Skills (PARENTS) educational assessment tool and collect three sources of validity evidence (i.e., content, response process, internal structure) for it. Methods We established content evidence for the PARENTS through interviews with physician-educators and residents, focus groups with parents, a literature review, and a modified nominal group technique with experts. We collected response process evidence through cognitive interviews with parents. To examine the internal structure evidence, we administered the PARENTS and performed exploratory factor analysis. Results Initially, a 20-item PARENTS was developed. Cognitive interviews led to the removal of one closed-ended item, the addition of resident photographs, and wording/formatting changes. Thirty-seven residents and 434 parents participated in the administration of the resulting 19-item PARENTS. Following factor analysis, a one-factor model prevailed. Conclusions The study presents initial validity evidence for the PARENTS. It also highlights strategies for potentially: (a) involving parents in the assessment of residents, (b) improving the assessment of NTS in pediatric EDs, and (c) capturing parents’ perspectives to improve the preparation of future physicians. Electronic supplementary material The online version of this article (10.1186/s12909-017-1042-9) contains supplementary material, which is available to authorized users.

CONTEXT Although health professions education scholarship units (HPESUs) share a commitment to the production and dissemination of rigorous educational practices and research, they are situated in many different contexts and have a wide range of structures and functions.OBJECTIVES In this study, the authors explore the institutional logics common across HPESUs, and how these logics influence the organisation and activities of HPESUs.METHODS The authors analysed interviews with HPESU leaders in Canada (n = 12), Australia (n = 21), New Zealand (n = 3) and the USA (n = 11). Using an iterative process, they engaged in inductive and deductive analyses to identify institutional logics across all participating HPESUs. They explored the contextual factors that influence how these institutional logics impact each HPESU's structure and function.RESULTS Participants identified three institutional logics influencing the organisational structure and functions of an HPESU: (i) the logic of financial accountability; (ii) the logic of a cohesive education continuum, and (iii) the logic of academic research, service and teaching. Although most HPESUs embodied all three logics, the power of the logics varied among units. The relative power of each logic influenced leaders' decisions about how members of the unit allocate their time, and what kinds of scholarly contribution and product are valued by the HPESU.CONCLUSIONS Identifying the configuration of these three logics within and across HPESUs provides insights into the reasons why individual units are structured and function in particular ways. Having a common language in which to discuss these logics can enhance transparency, facilitate evaluation, and help leaders select appropriate indicators of HPESU success.

In response to the numerous challenges resident trainees currently face in their ability to competently acquire the requisite skills, knowledge and attitudes upon graduation, medical educators have looked to a competency-based medical education (CBME) approach as a possible solution. As CBME has already been implemented in many jurisdictions around the world, certain challenges in implementation have been experienced. One important challenge identified relates to how regulatory bodies can either assist or unintentionally hinder implementation. By examining the varied experiences from Canada, the USA and the Netherlands in implementing CBME, this paper identifies how regulatory bodies can support and advance worldwide efforts of furthering its implementation. If regulatory bodies restructure accreditation and regulatory criteria to align with CBME principles, work together in a coordinated fashion to ensure alignment of vital regulatory meaures throughout the training and practice continuum of a physician, and allow for (if not incentivize) individuals and programs to be innovative in adapting CBME to meet their local environments, it is likely that the worldwide implementation of CBME will occur successfully.

Purpose Health professions education scholarship units (HPESUs) in the United States are large in number and diverse in purpose, activities, and contributions. Although each of these units shares a commitment to scholarship, there is no synthetic framework to accurately represent and evaluate their activities and contributions. This study aimed to provide such a framework. Method The authors examined data collected from 11 U.S. HPESU directors. Interviews occurred between April 2015 and February 2016. The research team used a combination of deductive and inductive qualitative techniques to analyze the interview transcripts. The deductive portion drew on Boyer's four-part framework of scholarship; the inductive portion produced a new conceptualization of scholarship at the HPESU level. Results The scholarly activities of HPESUs generally align with Boyer's four types of scholarship-discovery, integration, application, and teaching. However, this categorization fails to capture the interconnectedness and variety of purposes served by these activities. Both are important when considering how best to represent the scholarly contributions made by HPESUs. From their analysis of interviews, the authors developed a three-part framework characterizing HPESU scholarly activities: supporting a scholarly approach to education, supporting educational scholarship within the institution, and supporting HPESU members' scholarship. Conclusions The authors contend that the three-part, unit-level framework for scholarship constructed in this study brings clarity and understanding to the purpose, activities, and contributions made by HPESUs in the United States. The proposed framework may allow unit directors to better justify and advocate for the resources needed to further promote the work of HPESUs.

Current controversies in medical education associated with professionalism, including disagreements about curriculum, pedagogy, and assessment, are rooted in part in the differing frameworks that are used to address professionalism. Three dominant frameworks, which have evolved in the medical education community, are described. The oldest framework is virtue based and focuses on the inner habits of the heart, the development of moral character and reasoning, plus humanistic qualities of caring and compassion: The good physician is a person of character. The second framework is behavior based, which emphasizes milestones, competencies, and measurement of observable behaviors: The good physician is a person who consistently demonstrates competence in performing patient care tasks. The third framework is identity formation, with a focus on identity development and socialization into a community of practice: The good physician integrates into his or her identity a set of values and dispositions consonant with the physician community and aspires to a professional identity reflected in the very best physicians. Although each professionalism framework is useful and valid, the field of medical education is currently engaged in several different discourses resulting in misunderstanding and differing recommendations for strategies to facilitate professionalism. In this article, the assumptions and contributions of each framework are described to provide greater insight into the nature of professionalism. By examining each discourse in detail, underlying commonalities and differences can be highlighted to assist educators in more effectively creating professionalism curricula, pedagogy, and assessment.

Background The shift toward competency-based medical education has created a demand for feasible workplace-based assessment tools. Perhaps, more important than competence to assess an individual patient is the ability to successfully manage a surgical clinic. Trainee performance in clinic is a critical component of learning to manage a surgical practice, yet no assessment tool currently exists to assess daily performance in outpatient clinics for surgery residents. The development of a competency-based assessment tool, the Ottawa Clinic Assessment Tool (OCAT), is described here to address this gap. Study Design A consensus group of experts was gathered to generate dimensions of performance reflective of a competent “generalist” surgeon in clinic. A 6-month pilot study of the OCAT was conducted in orthopedics, general surgery, and obstetrics and gynecology with quantitative and qualitative evidence of validity collected. In all, 2 subsequent feedback sessions and a survey for staff and residents evaluated the OCAT for clarity and utility. Results The OCAT is a 9-item tool, with a global assessment item and 2 short-answer questions. Among the 2 divisions, 44 staff surgeons completed 132 OCAT assessments of 79 residents. Psychometric data was collected as evidence of validity. Analysis of feedback indicated that the entrustability rating scale was useful for surgeons and residents and that the items could be correlated with individual competencies. Conclusions Multiple sources of validity evidence collected in this study demonstrate that the OCAT can measure resident clinic competency in a valid and feasible manner.

There is limited information about how residents in surgical specialties view program strengths and opportunities for improvement (OFIs).Background This study aggregated surgical residents' perspectives on program strengths and OFIs to determine whether there was agreement in perspectives among residents in 5 surgical specialties.Objective Resident consensus lists of program strengths and areas for improvement were aggregated from site visits reports during 2012 and 2013 for obstetrics and gynecology, orthopaedic surgery, otolaryngology, plastic surgery, and surgery programs. Four trained individuals coded each strength or OFI in 1 of 3 categories: (1) factors common to all specialties; (2) program or institutional resources; and (3) factors unique to surgical specialties. Themes were classified as most frequent when listed by residents in more than 20% of the programs and less frequent when listed by residents in less than 20% of the programs.Methods This study included a total of 359 programs, representing 27% to 49% of the Accreditation Council for Graduate Medical Education accredited programs in the 5 specialties. The most frequent strengths were progressive autonomy, collegiality, program leadership, and operative volume. Improving research and didactics, increasing faculty teaching and attendance at educational sessions, and increasing the number of nurse practitioners and physician assistants were common OFIs.Results Factors identified as important by surgical residents related to their learning environment, their educational program, and program and institutional support. Across programs in the study, similar attributes were listed as both program strengths and OFIs.Conclusions

Background The declining popularity of surgical specialties among North American medical students has been attributed partially to limited early exposure and minimal involvement of surgeons in preclerkship education; in response, the Surgical Exploration and Discovery (SEAD) program was developed at the University of Toronto in 2012. SEAD is a 2-week curriculum that provides first-year medical students comprehensive exposure to surgical specialties through operating room observerships, simulation workshops, and career discussions. This study is the first to examine implementation of the SEAD program at another site. Methods This prospective cohort study evaluated the effectiveness of the SEAD program in improving surgical knowledge and facilitating career decision making when compared with a control group. In all, 18 students participated in the SEAD program, and 18 students from the same class read only the program's instructional manual; both the groups completed multiple-choice tests and questionnaires at baseline and at completion to assess knowledge acquisition and career-related learning. Results Both the groups significantly improved their surgical knowledge, but there was no difference between groups. SEAD participants made significantly greater progress in refining their career decisions when compared with the control group; all but one SEAD participant either gained interest in or ruled out a surgical specialty as a potential career choice compared with only 10 of the participants reading the manual only. Conclusions The SEAD program provides a meaningful opportunity for medical students to explore surgical careers during preclerkship and to make better-informed career decisions. This expansion demonstrates that the program can be successfully reproduced at another institution.

To identify common and overlapping themes among the interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP) milestones of the transitional year and 26 specialties.In May 2017, milestones were accessed from the Accreditation Council for Graduate Medical Education specialties website. A thematic analysis of the ICS, PBLI, PROF, and SBP milestones was performed to determine unique and common themes across these competencies and across specialties. Keywords from the common program requirements were initially applied as codes to the milestones. Codes were then grouped into common themes.Twenty-two themes were identified: 15 (68%) were unique to a given competency (3 related to ICS, 4 related to PBLI, 5 related to PROF, and 3 related to SBP), and 7 (32%) appeared in the milestones of more than one core competency. Eleven themes (50%) were used by 20 or more specialties, and 6 themes (27%) by 10 or fewer specialties. No theme was present across all specialties.The ICS, PBLI, PROF, and SBP milestones contain multiple themes with areas of overlap among these four competencies and substantial variability across specialties. This variability may create differential expectations of residents across specialties, complicate faculty development, and make sharing assessment tools difficult. The thematic analysis provides important insights into how individual specialties interpret and operationalize the ICS, PBLI, PROF, and SBP competency domains and can inform future revisions of milestones to enable harmonization and shared understanding of these competencies across specialties where appropriate.

STATEMENT Fidelity has become a ubiquitous feature of discourse in simulation studies. Recent studies have highlighted the often ambiguous and contradictory manner in which fidelity has been defined, with each definition emphasizing different physical and functional features of simulation. We suggest that regarding fidelity as an objective property of a simulation obscures the interactive nature of the educator-learner relationship and should be abandoned. Rather than conceiving training as tasks performed by an individual in isolation, we suggest that it is more accurately understood as the social learning of affordances. Affordances represent the functional features of a simulator, which are taken as relevant in a specific learning context by means of analogy. Training is successful to the extent that educators and learners share an understanding of those affordances. Even when explicitly formulated, the concept of fidelity has greater difficulty accounting for the complex, interactional features of the training situation in comparison with accounts based on social learning. We conclude that continued attempts to redefine and use fidelity in the context of training will likely yield little benefit to the field compared with an interactive social learning framework.