Search

Program directors in surgical disciplines need more tools from the ACGME to help them use Milestone ratings to improve trainees' performance. This is especially true in competencies that are notoriously difficult to measure, such as professionalism (PROF) and interpersonal and communication skills (ICS). It is now widely understood that skills in these two areas have direct impact on patient care outcomes. This study investigated the potential for generating early predictors of final Milestone ratings within the PROF and ICS competency categories.This retrospective cohort study utilized Milestone ratings from all ACGME-accredited vascular surgery training programs, covering residents and fellows who completed training in June 2019. The outcome measure studied was the rate of achieving the recommended graduation target of Milestone Level 4 (possible range: 1-5), while the predictors were the Milestone ratings attained at earlier stages of training. Predictive probability values (PPVs) were calculated for each of the 3 PROF and two ICS sub-competencies to estimate the probability of trainees not reaching the recommended graduation target based on their previous Milestone ratings.All ACGME-accredited vascular surgery training programs within the United States.All trainees completing a 2 year vascular surgery fellowship (VSF) in June 2019 (n = 119) or a 5 year integrated vascular surgery residency (IVSR) in June 2019 (n = 52) were included in the analyses.The overall rate of failing to achieve the recommended graduation target across all PROF and ICS sub-competencies ranged from 7.7% to 21.8% of all trainees. For trainees with a Milestone rating at ≤ 2.5 with 1 year remaining in their training program, the predictive probability of not achieving the recommended graduation target ranged from 37.0% to 71.5% across sub-competencies, with the highest risks observed under PROF for "Administrative Tasks" (71.5%) and under ICS for "Communication with the Healthcare Team" (56.7%).As many as 1 in 4 vascular surgery trainees did not achieve the ACGME vascular surgery Milestones targets for graduation in at least one of the PROF and ICS sub-competencies. Biannual ACGME Milestone assessment ratings of PROF and ICS during early training can be used to predict achievement of competency targets at time of graduation. Early clues to problems in PROF and ICS enable programs to address potential deficits early in training to ensure competency in these essential non-technical skills prior to entering unsupervised practice.

The quality and effectiveness of vascular surgery education should be evaluated based on patient care outcomes. To investigate predictive associations between trainee performance and subsequent patient outcomes, a critical first step is to determine the conceptual alignment of educational competencies with clinical outcomes in practice. We sought to generate expert consensus on the conceptual alignment of the Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery subcompetencies with patient care outcomes across different Vascular Quality Initiative (VQI) registries.A national panel of vascular surgeons with expertise in both clinical care and education were recruited to participate in a modified Delphi expert consensus building process to map ACGME Vascular Surgery subcompetencies (educational markers of resident performance) to VQI clinical modules (patient outcomes). A master list of items for rating was created, including the 31 ACGME Vascular Surgery subcompetencies and 8 VQI clinical registries (endovascular abdominal aortic aneurysm repair, open abdominal aortic aneurysm, thoracic endovascular aortic repair, carotid endarterectomy, carotid artery stent, infrainguinal, suprainguinal, and peripheral vascular intervention). These items were entered into an iterative Delphi process. Positive consensus was reached when 75% or more of the participants ranked an item as mandatory. Intraclass correlations (ICCs) were used to evaluate consistency between experts for each Delphi round.A total of 13 experts who contributed to the development of the Vascular Surgery Milestones participated; 12 experts (92%) participated in both rounds of the Delphi process. Two rounds of Delphi were conducted, as suggested by excellent expert agreement (round 1, ICC = 0.79 [95% confidence interval, 0.74-0.84]; round 2, ICC = 0.97 [95% confidence interval, 0.960-.98]). Using the predetermined consensus cutoff threshold, the Delphi process reduced the number of subcompetencies mapped to patient care outcomes from 31 to a range of 9 to 15 across the 8 VQI clinical registries. Practice-based learning and improvement, and professionalism subcompetencies were identified as less relevant to patient outcome variables captured by the VQI registries after the final round, and the only the systems-based practice subcompetency that was identified as relevant was radiation safety in two of the endovascular registries.A national panel of vascular surgeon experts reported a high degree of agreement on the relevance of ACGME subcompetencies to patient care outcomes as captured in the VQI clinical registry. Systems-based practice, practice-based learning and improvement, and professionalism competencies were identified as less relevant to patient outcomes after specific surgical procedures.

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets.ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation.National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 sub-competencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings.122 VSFs (95.3%) and 52 IVSRs (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6-25.4% across sub-competencies, which was not significantly different between the two groups for any of the sub-competencies (p=0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of ≤ 2.5, with one year remaining prior to graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% - 77.9% for VSFs and 33.3% - 75.0% for IVSRs.The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

ImportanceEvaluation of trainees in graduate medical education training programs using Milestones has been in place since 2013. It is not known whether trainees who have lower ratings during the last year of training go on to have concerns related to interactions with patients in posttraining practice.ObjectiveTo investigate the association between resident Milestone ratings and posttraining patient complaints.Design, Setting, and ParticipantsThis retrospective cohort study included physicians who completed Accreditation Council for Graduate Medical Education (ACGME)–accredited programs between July 1, 2015, and June 30, 2019, and worked at a site that participated in the national Patient Advocacy Reporting System (PARS) program for at least 1 year. Milestone ratings from ACGME training programs and patient complaint data from PARS were collected. Data analysis was conducted from March 2022 to February 2023.ExposuresLowest professionalism (P) and interpersonal and communication skills (ICS) Milestones ratings 6 months prior to the end of training.Main Outcomes and MeasuresPARS year 1 index scores, based on recency and severity of complaints.ResultsThe cohort included 9340 physicians with median (IQR) age of 33 (31-35) years; 4516 (48.4%) were women physicians. Overall, 7001 (75.0%) had a PARS year 1 index score of 0, 2023 (21.7%) had a score of 1 to 20 (moderate), and 316 (3.4%) had a score of 21 or greater (high). Among physicians in the lowest Milestones group, 34 of 716 (4.7%) had high PARS year 1 index scores, while 105 of 3617 (2.9%) with Milestone ratings of 4.0 (proficient), had high PARS year 1 index scores. In a multivariable ordinal regression model, physicians in the 2 lowest Milestones rating groups (0-2.5 and 3.0-3.5) were statistically significantly more likely to have higher PARS year 1 index scores than the reference group with Milestones ratings of 4.0 (0-2.5 group: odds ratio, 1.2 [95% CI, 1.0-1.5]; 3.0-3.5 group: odds ratio, 1.2 [95% CI, 1.1-1.3]).Conclusions and RelevanceIn this study, trainees with low Milestone ratings in P and ICS near the end of residency were at increased risk for patient complaints in their early posttraining independent physician practice. Trainees with lower Milestone ratings in P and ICS may need more support during graduate medical education training or in the early part of their posttraining practice career.

Organizational readiness is critical for successful implementation of an innovation. We evaluated program readiness to implement Competence by Design (CBD), a model of Competency-Based Medical Education (CBME), among Canadian postgraduate training programs. A survey of program directors was distributed 1 month prior to CBD implementation in 2019. Questions were informed by the R = MC2 framework of organizational readiness and addressed: program motivation, general capacity for change, and innovation-specific capacity. An overall readiness score was calculated. An ANOVA was conducted to compare overall readiness between disciplines. Survey response rate was 42% (n = 79). The mean overall readiness score was 74% (30���98%). There was no difference in scores between disciplines. The majority of respondents agreed that successful implementation of CBD was a priority (74%), and that their leadership (94%) and faculty and residents (87%) were supportive of change. Fewer perceived that CBD was a move in the right direction (58%) and that implementation was a manageable change (53%). Curriculum mapping, competence committees and programmatic assessment activities were completed by >90% of programs, while