Search

Canada relies heavily on foreign-trained physicians. As a Federation, with health care being a Provincial jurisdiction, this often translates into varied processes that international medical graduates (IMGs) must undertake to obtain a Canadian medical license. Two decades ago, several government officials and representatives of many physician organizations, including regulatory bodies, met and proposed 6 recommendations to make the processes standardized, simpler, and more transparent to aid internationally trained physicians in their pursuit of Canadian medical licenses. The Medical Council of Canada (MCC) was one of the organizations at the 2002 meeting in Calgary, Alberta. As an organization responsible for the assessment of physicians’ knowledge and skills and the issuant of the Licentiate of the MCC (LMCC), a prerequisite for Canadian medical license, the MCC was one of the institutions tasked with implementation of the recommendations. The purpose of this manuscript is to evaluate how well the recommendations were met. To do this, we explored whether the IMGs’ journey to obtain Canadian medical licenses in 2022 is more challenging or less challenging than in 2002. The MCC’s role in helping to effect changes in the licensing process was highlighted.

Capillary refill time (CRT) to assess peripheral perfusion in children with suspected shock may be subject to poor reproducibility. Our objectives were to compare video-based and bedside CRT assessment using a standardized protocol and evaluate interrater and intrarater consistency of video-based CRT (VB-CRT) assessment. We hypothesized that measurement errors associated with raters would be low for both standardized bedside CRT and VB-CRT as well as VB-CRT across raters.Ninety-nine children (aged 1-12 y) had 5 consecutive bedside CRT assessments by an experienced critical care clinician following a standardized protocol. Each CRT assessment was video recorded on a black background. Thirty video clips (10 with bedside CRT1 s, 10 with CRT 1-2 s, and 10 with CRT2 s) were randomly selected and presented to 10 clinicians twice in randomized order. They were instructed to push a button when they visualized release of compression and completion of a capillary refill. The correlation and absolute difference between bedside and VB-CRT were assessed. Consistency across raters and within each rater was analyzed using the intraclass correlation coefficient (ICC). A Generalizability study was performed to evaluate sources of variation.We found moderate agreement between bedside and VB-CRT observations (r = 0.65; P0.001). The VB-CRT values were shorter by 0.17 s (95% confidence interval, 0.09-0.25; P0.001) on average compared with bedside CRT. There was moderate agreement in VB-CRT across raters (ICC = 0.61). Consistency of repeated VB-CRT within each rater was moderate (ICC = 0.71). Generalizability study revealed the source of largest variance was from individual patient video clips (57%), followed by interaction of the VB-CRT reviewer and patient video clip (10.7%).Bedside and VB-CRT observations showed moderate consistency. Using video-based assessment, moderate consistency was also observed across raters and within each rater. Further investigation to standardize and automate CRT measurement is warranted.

BackgroundAlthough little is known about why opioid prescribing practices differ between physicians, clinical competence, specialty training and country of origin may play a role. We hypothesised that physicians with stronger clinical competence and communication skills are less likely to prescribe opioids and prescribe lower doses, as do medical specialists and physicians from Asia.MethodsOpioid prescribing practices were examined among international medical graduates (IMGs) licensed to practise in the USA who evaluated Medicare patients for chronic pain problems in 2014–2015. Clinical competence was assessed by the Educational Commission for Foreign Medical Graduates (ECFMG) Clinical Skills Assessment. Physicians in the ECFMG database were linked to the American Medical Association Masterfile. Patients evaluated for chronic pain were obtained by linkage to Medicare outpatient and prescription files. Opioid prescribing was measured within 90 days of evaluation visits. Prescribed dose was measured using morphine milligram equivalents (MMEs). Generalised estimating equation logistic and linear regression estimated the association of clinical competence, specialty, and country of origin with opioid prescribing and dose.Results7373 IMGs evaluated 65 012 patients for chronic pain; 15.2% received an opioid prescription. Increased clinical competence was associated with reduced opioid prescribing, but only among female physicians. For every 10% increase in the clinical competence score, the odds of prescribing an opioid decreased by 16% for female physicians (OR 0.84, 95% CI 0.75 to 0.94) but not male physicians (OR 0.99, 95% CI 0.92 to 1.07). Country of origin was associated with prescribed opioid dose; US and Canadian citizens prescribed higher doses (adjusted MME difference +3.56). Primary care physicians were more likely to prescribe opioids, but surgical and hospital-based specialists prescribed higher doses.ConclusionsClinical competence at entry into US graduate training, physician gender, specialty and country of origin play a role in opioid prescribing practices.

Objective: This study describes the supply, distribution, and characteristics of international medical graduate (IMG) psychiatrists who provide services in the USA. Methods: Cross-sectional study design, using descriptive statistics based on combined data from the American Medical Association (2020 Physician Masterfile) and the Educational Commission for Foreign Medical Graduates. Results: International medical graduates continue to make significant contributions to the US physician workforce. As a group, they represent 29% of active psychiatrists in the USA, compared to 23% in all other medical specialties. Many IMG psychiatrists were US citizens who obtained their medical degrees outside the USA or Canada, often in the Caribbean. In some states (i.e., Florida, New Jersey), over 40% of active psychiatrists are IMGs. Over 30% of IMG psychiatrists graduated from medical schools in India and Pakistan. Conclusions: This study provides an overview of the psychiatric workforce in the USA, quantifying the specific contribution of IMGs. Several factors, including immigration policies, continued expansion of US medical schools, and the number of available residency positions, could impact the flow of IMGs to the US. Longitudinal studies are needed to better understand the implications for workforce composition and distribution, and their potential impact on the care of psychiatric patients.

Background Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees. Methods Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards. Results For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario. Conclusions Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains.

From national and international workforce perspectives, Canadians studying medicine abroad (CSAs) are a growing provider group. Some were born in Canada whereas others immigrated as children. They study medicine in various countries, often attempting both American and Canadian medical licensure pathways.Using data from the Educational Commission for Foreign Medical Graduates (ECFMG) and the Medical Council of Canada (MCC), we looked at CSAs who attempted to secure residency positions in both Canada and the United States. We detailed the CSAs' countries of birth and medical education. We tracked these individuals through their postgraduate education programs to enumerate their success rate and categorize the geographic locations of their training.The majority of CSAs study medicine in one of 10 countries. The remainder are disbursed across 88 other countries. Most CSAs were born in Canada (62%). Approximately 1/3 of CSA from the 2004-2016 cohort had no record of entering a residency program in Canada or the United States (U.S.). Recently graduated CSAs were most likely to secure residency training in Ontario and New York.Many CSAs attempt to secure residency training in both Canada and the U.S. Quantifying success rates may be helpful for Canadians thinking about studying medicine abroad. Understanding the educational pathways of CSAs will be useful for physician labour workforce planning.Selon une perspective nationale et internationale des effectifs, les Canadiens qui étudient la médecine à l’étranger (CEE) représentent un groupe en croissance. Certains sont nés au Canada, alors que d’autres ont immigré durant leur enfance. Ils étudient la médecine dans divers pays, essayant souvent parallèlement d’obtenir un permis américain et canadien pour exercer la médecine.À l’aide de données de l’Educational Commission for Foreign Medical Graduates (ECFMG) et du Conseil médical du Canada (CMC), nous avons examiné les CEE qui avaient tenté d’obtenir des postes de résidence à la fois au Canada et aux États-Unis. Nous avons extrait des données quant au pays de naissance et à la formation médicale de ces CEE. Nous avons suivi ces personnes dans leurs processus de demande d’admission à des programmes de formation postdoctorale pour rapporter leur taux de succès et catégoriser les emplacements géographiques de leur formation.Nous avons identifié 10 pays d’où provenaient la plupart de ces CEE. Les autres CEE provenaient de 88 autres pays. La plupart de ces CEE sont nées au Canada (62 %). Environ 1/3 des CEE de la cohorte de 2004 à 2016 ne possède pas de dossier d’inscription à un programme de résidence au Canada ou aux États-Unis. Les CEE récemment diplômés étaient les plus susceptibles de suivre une formation en résidence en Ontario et dans l’État de New York.De nombreux CEE ont tenté d’obtenir un poste de résidence au Canada et aux États-Unis. Quantifier les taux de succès pourrait se révéler utile pour les Canadiens qui pensent à étudier la médecine à l’étranger. Comprendre les parcours éducatifs des CEE sera utile à la planification des effectifs médicaux.

Certification by the Educational Commission for Foreign Medical Graduates (ECFMG®) is required for international medical graduates (IMGs) to enter U.S. graduate medical education (GME). As a gatekeeper to the U.S. health care system, ECFMG has a duty to verify that these individuals have met minimum standards for undergraduate medical education. Historically, ECFMG has focused on evaluating individual graduates, not medical schools. However, in response to the rapid growth of medical schools around the world and increasing physician migration, ECFMG decided in 2010 to institute medical school accreditation as a future requirement for ECFMG certification. More specifically, beginning in 2023, individuals applying for ECFMG certification will be required to be a student or graduate of a medical school that is accredited by an agency recognized by the World Federation for Medical Education (WFME). By requiring accreditation by an agency that has met WFME's standards, ECFMG seeks to improve the quality, consistency and transparency of undergraduate medical education worldwide. The 2023 Medical School Accreditation Requirement is intended to stimulate global accreditation efforts, increase the information publicly available about medical schools, and provide greater assurance to medical students, regulatory authorities, and the public that these future physicians will be appropriately educated.

PURPOSE International medical graduates (IMGs), approximately 25% of the U.S. physician workforce, have unique needs as they enter residency programs. This study identified wellness barriers and challenges that IMGs encounter as they transition to the United States. METHOD The authors analyzed results from 3 open-ended questions in a 21-item survey. This survey was administered in December 2019 to 11,504 IMG resident physicians sponsored by the Educational Commission for Foreign Medical Graduate's J-1 visa program. These questions asked respondents to describe challenges to their wellness, how they maintain wellness, and resources that would have aided their transition. Data were analyzed using a mixed-methods approach, including both qualitative descriptions and category frequencies. RESULTS Of the surveys administered, 7,817 responses (68% response rate) were received. Respondents identified challenges navigating cultural differences (1,314, 17%), health care system (1,108, 14%), distance from family and friends (890, 11%), bureaucratic barriers (724, 9%), and language and communication and finances (575, 7%; 565, 7%, respectively). They also specified that friendships/relationships (2,800, 36%) followed by exercise (2,318, 30%), family (1,822, 23%), socialization (1,001, 13%), and healthy eating (775 10%,) were factors important to their wellness. Respondents requested more information about socialization (741, 9%), bureaucratic support (456, 6%), IMG support networks (427, 5%), financial support (404, 5%), and greater online resources (240, 3%). CONCLUSIONS IMGs have needs and concerns specific to their demographic group. Participants' responses suggested they wanted additional support in the workplace and their personal lives. Answers also indicated that IMGs experienced a unique set of stressors such as fluctuating immigration laws that U.S. medical graduates do not face. Finally, this study supports a body of research that connects social and physical wellness. By identifying and describing these challenges, the authors seek to inform the development of specific programs and resources to improve IMG resident wellness.

After extensive stakeholder discussion, the Federation of State Medical Boards and the National Board of Medical Examiners announced in February 2020 that United States Medical Licensing Examination Step 1 will transition to a pass/fail exam. Program directors have historically used Step 1 scores in deciding which residency applicants to interview. The lack of numerical scores will force changes to the residency selection process, which could have both positive and negative consequences for international medical graduates (IMGs). In this Invited Commentary, the authors discuss how some of the issues associated with the transition to Step 1 pass/fail are likely to impact IMGs. The authors also provide insights into how this and other policy changes could help spur the medical education community to improve the process by which medical school graduates transition to graduate medical education.

Background To describe the supply, distribution, and characteristics of international medical graduates (IMGs) in family medicine who provide patient care in the U.S. Methods A cross-sectional study design, using descriptive statistics on combined data from the Educational Commission for Foreign Medical Graduates and the American Medical Association, including medical school attended, country of medical school, and citizenship when entering medical school. Results In total, 118,817 physicians in family medicine were identified, with IMGs representing 23.8% (n = 28,227) of the U.S. patient care workforce. Of all 9579 residents in family medicine, 36.0% (n = 3452) are IMGS. In total, 35.9% of IMGs attended medical school in the Caribbean (n = 10,136); 19.9% in South-Central Asia (n = 5607) and 9.1% in South-Eastern Asia (n = 2565). The most common countries of medical school training were Dominica, Mexico, and Sint Maarten. Of all IMGs in family medicine who attended medical school in the Caribbean, 74.5% were U.S. citizens. In total, 40.5% of all IMGs in family medicine held U.S. citizenship at entry to medical school. IMGs comprise almost 40% of the family medicine workforce in Florida, New Jersey and New York. Conclusions IMGs play an important role in the U.S. family medicine workforce. Many IMGs are U.S. citizens who studied abroad and then returned to the U.S. for graduate training. Given the shortage of family physicians, and the large number of IMGs in graduate training programs, IMGs will continue to play a role in the U.S. physician workforce for some time to come. Many factors, including the supply of residency training positions, could eventually restrict the number of IMGs entering the U.S., including those contributing to family practice.

PURPOSE In 2024, international medical graduates seeking Educational Commission for Foreign Medical Graduates (ECFMG) certification will be required to graduate from an accredited medical school. This study's goal was to examine relationships between medical school accreditation variables and ECFMG certification for a global sample. METHOD Using ECFMG databases, the authors created a 10-year cohort (January 1, 2007-December 31, 2016) of certification applicants, defined as individuals who had attempted at least 2 examinations required for certification. The authors aggregated applicant data at the school level, excluding schools with < 80 applicants. School accreditation statuses were based on agency websites. School region, age, and time of first accreditation were included. Analyses included descriptive and bivariate statistics and multiple linear regressions adjusting for school start year and year of first accreditation. RESULTS The cohort included 128,046 applicants from 1,973 medical schools across 162 countries. After excluding low-volume schools, 318 schools across 81 countries remained. These provided 99,598 applicants and 77,919 certificate holders, three-quarters of whom came from the Caribbean, South-Central Asia, and West Asia regions. Two hundred and fifty (78.6%) schools were accredited; 68 (21.4%) were not. Most ECFMG applicants (n = 84,776, 85.1%) and certificate holders (n = 68,444, 87.8%) attended accredited medical schools. Accredited schools had higher rates of ECFMG certification among graduates than nonaccredited schools in comparisons that included all schools (75.0% [standard deviation (SD) = 10.6%] vs 68.3% (SD = 15.9%), P < .001), and for countries that had both accredited and nonaccredited schools (73.9% [SD = 11.4%] vs 67.3% [SD = 16.8%], P = .023). After adjusting for age of school, longer duration of accreditation was associated with higher certification rates (P < .001). CONCLUSIONS Accreditation had a positive association with ECFMG certification rates. Future studies should investigate how accreditation processes might account for higher certification rates.

BACKGROUND AND OBJECTIVES: To describe the supply, distribution, and characteristics of international medical graduates (IMGs) in pediatrics who provide patient care in the United States. METHODS: Cross-sectional study, combining data from the 2019 Physician Masterfile of the American Medical Association and the Educational Commission for Foreign Medical Graduates database. RESULTS: In total, 92 806 pediatric physicians were identified, comprising 9.4% of the entire US physician workforce. Over half are general pediatricians. IMGs account for 23.2% of all general pediatricians and pediatric subspecialists. Of all IMGs in pediatrics, 22.1% or 4775 are US citizens who obtained their medical degree outside the United States or Canada, and 15.4% (3246) attended medical school in the Caribbean. Fifteen non-US medical schools account for 29.9% of IMGs currently in active practice in pediatrics in the United States. IMGs are less likely to work in group practice or hospital-based practice and are more likely to be employed in solo practice (compared with US medical school graduates). CONCLUSIONS: With this study, we provide an overview of the pediatric workforce, quantifying the contribution of IMGs. Many IMGs are US citizens who attend medical school abroad and return to the United States for postgraduate training. Several factors, including the number of residency training positions, could affect future numbers of IMGs entering the United States. Longitudinal studies are needed to better understand the implications that workforce composition and distribution may have for the care of pediatric patients.

Introduction Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established. Methods Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters. Results The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability. Conclusions The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Purpose Family medicine residency programs can be cited for low pass or take rates on the American Board of Family Medicine (ABFM) certification examination, and the relationships among standardized medical education assessments and performance on board certification examinations and eventual board certification have not been comprehensively studied. The objective of this study was to evaluate the associations of all required standardized examinations in medical education with ABFM certification examination scores and eventual ABFM certification. Method All graduates of U.S. MD-granting family medicine residency programs from 2008 to 2012 were included. Data on ABFM certification examination score, ABFM certification status (as of December 31, 2014), Medical College Admission Test (MCAT) section scores, undergraduate grade point average, all United States Medical Licensing Examination (USMLE) Step scores, and all ABFM in-training examination scores were linked. Nested logistic and linear regression models, controlling for clustering by residency program, determined associations between assessments and both certification examination scores and board certification status. As many international medical graduates (IMGs) do not take the MCAT, separate models for U.S. medical graduates (USMG) and IMGs were run. Results The study sample was 15,902 family medicine graduates, of whom 92.1% (14,648/15,902) obtained board certification. In models for both IMGs and USMGs, the addition of more recent assessments weakened the associations of earlier assessments. USMLE Step 2 Clinical Knowledge was predictive of certification examination scores and certification status in all models in which it was included. Conclusions For family medicine residents, more recent assessments generally have stronger associations with board certification score and status than earlier assessments. Solely using medical school admissions (grade point average and MCAT) and licensure (USMLE) scores for resident selection may not adequately predict ultimate board certification.

INTRODUCTION The purpose of this study was to examine Uniformed Services University (USU) women graduates in terms of percent of graduates', specialty choices and practice choices as compared to civilian women who graduate and practice medicine in the USA. This is a perspective that is currently not well understood. MATERIALS AND METHODS We conducted a retrospective cohort study of all USU women graduates (1980-2015) using the 2016 American Medical Association (AMA) Physician Masterfile that included data from the American Board of Medical Specialties (ABMS). To describe USU women graduates' current practice status we queried for: (1) medical school; (2) year of graduation; (3) practice state; (4) primary specialty board; and (5) major professional activity (office-based practice vs. full-time hospital staff). Data were analyzed using descriptive statistics. RESULTS Our findings indicate that the percentage of USU women graduates has increased over time and stands at 29% for the 2010-2015 cohort as compared to 48% for women graduating from all U.S. medical schools. USU women graduates have a slightly higher board certification rate (89%) than the national cohort (88%). USU women graduates also have a higher percentage in family medicine (19%) than the national cohort (14%). USU women graduates practice in 48 states and were equally split between full-time hospital staff and office-based practice which differs from the national cohort that has a much higher proportion in office-based practice (85%). CONCLUSIONS Women are making significant gains in enrollment at USU, obtaining board certification at similar, and in some cases, higher rates than their civilian peers, and practicing in diverse specialties. This study provides a descriptive picture of women's enrollment and practice characteristics from a military-based medical school. Future work could examine underlying factors that may influence their school choice, career experiences, and trajectories. Future research could also focus on women's experiences of mentoring and support to better understand these factors.

The acquisition and maintenance of individual competency is a critical component of effective emergency care systems. This article summarizes consensus working group deliberations and recommendations focusing on the topic "Simulation-based education to ensure provider competency within the healthcare system." The authors presented this work for discussion and feedback at the 2017 Academic Emergency Medicine Consensus Conference on "Catalyzing System Change Through Healthcare Simulation: Systems, Competency, and Outcomes," held on May 16, 2017, in Orlando, Florida. Although simulation-based training is a quality and safety imperative in other high-reliability professions such as aviation, nuclear power, and the military, health care professions still lag behind in applying simulation more broadly. This is likely a result of a number of factors, including cost, assessment challenges, and resistance to change. This consensus subgroup focused on identifying current gaps in knowledge and process related to the use of simulation for developing, enhancing, and maintaining individual provider competency. The resulting product is a research agenda informed by expert consensus and literature review.

Background We sought to determine whether mannequin-based simulation can reliably characterize how board-certified anesthesiologists manage simulated medical emergencies. Our primary focus was to identify gaps in performance and to establish psychometric properties of the assessment methods. Methods A total of 263 consenting board-certified anesthesiologists participating in existing simulation-based maintenance of certification courses at one of eight simulation centers were video recorded performing simulated emergency scenarios. Each participated in two 20-min, standardized, high-fidelity simulated medical crisis scenarios, once each as primary anesthesiologist and first responder. Via a Delphi technique, an independent panel of expert anesthesiologists identified critical performance elements for each scenario. Trained, blinded anesthesiologists rated video recordings using standardized rating tools. Measures included the percentage of critical performance elements observed and holistic (one to nine ordinal scale) ratings of participant’s technical and nontechnical performance. Raters also judged whether the performance was at a level expected of a board-certified anesthesiologist. Results Rater reliability for most measures was good. In 284 simulated emergencies, participants were rated as successfully completing 81% (interquartile range, 75 to 90%) of the critical performance elements. The median rating of both technical and nontechnical holistic performance was five, distributed across the nine-point scale. Approximately one-quarter of participants received low holistic ratings (i.e., three or less). Higher-rated performances were associated with younger age but not with previous simulation experience or other individual characteristics. Calling for help was associated with better individual and team performance. Conclusions Standardized simulation-based assessment identified performance gaps informing opportunities for improvement. If a substantial proportion of experienced anesthesiologists struggle with managing medical emergencies, continuing medical education activities should be reevaluated.

One challenge when implementing case-based learning, and other approaches to contextualized learning, is determining which clinical problems to include. This article illustrates how health care utilization data, readily available from the National Center for Health Statistics (NCHS), can be incorporated into an educational needs assessment to identify medical problems physicians are likely to encounter in clinical practice. The NCHS survey data summarize patient demographics, diagnoses, and interventions for tens of thousands of patients seen in various settings, including emergency departments (EDs), clinics, and hospitals.Selected data from the National Hospital Ambulatory Medical Care Survey: Emergency Department illustrate how instructional materials can be derived from the results of such public-use health care data. Using fever as the reason for visit to the ED, the patient management path is depicted in the form of a case drill-down by exploring the most common diagnoses, blood tests, diagnostic studies, procedures, and medications associated with fever.Although these types of data are quite useful, they should not serve as the sole basis for determining which instructional cases to include. Additional sources of information should be considered to ensure the inclusion of cases that represent infrequent but high-impact problems and those that illustrate fundamental principles that generalize to other cases.

When the safety of the public is at stake, it is particularly relevant for licensing and credentialing exam agencies to use defensible standard setting methods to categorize candidates into competence categories (e.g., pass/fail). The aim of this study was to gather evidence to support change to the Comprehensive Osteopathic Medical Licensing-USA Level 2-Performance Evaluation standard setting design and administrative process. Twenty-two video recordings of candidates assessed for clinical competence were randomly selected from the 2014-2015 Humanistic domain test score distribution ranging from the highest to lowest quintile of performance. Nineteen panelists convened at the same site to receive training and practice prior to generating judgments of qualified or not qualified performance to each of the twenty videos. At the end of training, one panel remained onsite to complete their judgments and the second panel was released and given 1 week to observe the same twenty videos and complete their judgments offsite. The two one-sided test procedure established equivalence between panel group means at the 0.05 confidence level, controlling for rater errors within each panel group. From a practical cost-effective and administrative resource perspective, results from this study suggest it is possible to diverge from typical panel groups, who are sequestered the entire time onsite, to larger numbers of panelists who can make their judgments offsite with little impact on judged samples of qualified performance. Standard setting designs having panelists train together and then allowing those to provide judgments yields equivalent ratings and, ultimately, similar cut scores.

Introduction We developed a taxonomy of simulation delivery and documentation deviations noted during a multicenter, high-fidelity simulation trial that was conducted to assess practicing physicians' performance. Eight simulation centers sought to implement standardized scenarios over 2 years. Rules, guidelines, and detailed scenario scripts were established to facilitate reproducible scenario delivery; however, pilot trials revealed deviations from those rubrics. A taxonomy with hierarchically arranged terms that define a lack of standardization of simulation scenario delivery was then created to aid educators and researchers in assessing and describing their ability to reproducibly conduct simulations. Methods Thirty-six types of delivery or documentation deviations were identified from the scenario scripts and study rules. Using a Delphi technique and open card sorting, simulation experts formulated a taxonomy of high-fidelity simulation execution and documentation deviations. The taxonomy was iteratively refined and then tested by 2 investigators not involved with its development. Results The taxonomy has 2 main classes, simulation center deviation and participant deviation, which are further subdivided into as many as 6 subclasses. Inter-rater classification agreement using the taxonomy was 74% or greater for each of the 7 levels of its hierarchy. Cohen kappa calculations confirmed substantial agreement beyond that expected by chance. All deviations were classified within the taxonomy. Conclusions This is a useful taxonomy that standardizes terms for simulation delivery and documentation deviations, facilitates quality assurance in scenario delivery, and enables quantification of the impact of deviations upon simulation-based performance assessment.

Although the authors do not agree with medical students' bid to end the United States Medical Licensing Examination Step 2 Clinical Skills or Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation tests, they concur with Ecker and colleagues that conducting further research to support the validity argument, providing greater feedback on performance, and exploring options to reduce costs are important for addressing students' concerns. Evidence to support the validity of clinical skills exam scores and associated inferences already exists. What is lacking, and would help further justify the use of these examinations, is more evidence to support the "extrapolation" argument-that is, is performance on these examinations related to actual patient care? Enhanced feedback on exam performance should also be considered. While performance data from licensing examinations should be used judiciously given the primary purpose of these tests, additional data would be helpful to learners and their institutions. Centralized testing remains the least costly design, but efficiencies of standardized patient training, case development, and scoring can be reviewed. Scoring modifications made in the past several years at substantial cost should be evaluated to determine whether they have achieved desired goals. Testing organizations can and should embrace these essential elements of transparency and accountability to address concerns about the value of clinical skills examinations.

BACKGROUND: The Comprehensive Osteopathic Medical Licensure Examination (COMLEX-USA) Level 2–Cognitive Examination (CE) and the Comprehensive Osteopathic Medical Achievement Test (COMAT) are administered to similar populations (third- and fourth-year osteopathic students) at similar points in time. Examining the relationship between scores on the 2 assessments that measure similar constructs ultimately supports the validity of both. OBJECTIVE: The purpose of this study is to provide empirical evidence of the concurrent and predictive validity of COMAT and COMLEX-USA Level 2-CE. METHODS: In 2018, first-attempt scores on Level 2-CE were aggregated from June 2015 to May 2018 and matched with first-attempt scores on each COMAT clinical subject. We conducted correlational analyses between performance on COMAT and Level 2-CE, and COMAT scores and Level 2-CE discipline subscores. Additionally, we used multivariate regression to analyze the predictive relationship between performance on all COMAT clinical subjects and Level 2-CE. RESULTS: The results from correlational analyses indicated statistically significant, positive associations between COMAT and Level 2-CE scores (r = 0.49–0.68, P < .0001), and statistically significant, but slightly weaker relationships between COMAT scores and Level 2-CE discipline subscores (r = 0.31–0.60, P < .0001). Furthermore, results from the multiple regression indicated that scores on COMAT explained 68% of the variance in Level 2-CE scores, and that COMAT internal medicine and emergency medicine were weighted more heavily than other specialties. CONCLUSIONS: The findings from this study can inform assessment practices by supporting the use of COMAT for osteopathic medical schools that do not administer COMAT.

Background Migration of physicians has been a cause for global concern. In China, reforms of the higher education and healthcare systems have led to a shortage of postgraduate training positions relative to the number of medical graduates. Medical graduates opt for non-clinical roles or move abroad to pursue further training and practice opportunities. The impact of this physician migration is not known. This study quantifies where Chinese migrant physicians to the U.S. were educated, where they went to practice, and how these trends have changed over time. Methods We combined data on physician characteristics from the 2008 and 2017 American Medical Association Physician Masterfiles with demographic information from the Educational Commission for Foreign Medical Graduates. Using a repeated cross-sectional approach, we reviewed the available data, including citizenship at entry to medical school, medical school attended, practice specialty, and practice location. Results The number of Chinese-educated physicians (CEPs) to the United States (US) has increased over the past 10 years, from 3,878 in 2008 to 5,355 in 2017 (+38.1%). The majority held Chinese citizenship at entry to medical school (98.4% vs 97.1%) with the remainder being citizens of other East Asian nations. Of the Chinese citizens identified in 2008, 913 (19.3%) attended medical school outside of China; in 2017, 376 (6.7%) attended medical school outside of China, representing a decrease of 58.8%. Overall, in 2017, four Chinese medical schools provided 32.1% of all Chinese-educated physicians in the US. Over 50% of the CEPs were practicing in Internal Medicine, Anatomic/ Clinical Pathology, Anesthesiology, Family Medicine or Neurology. Compared with all IMGs, CEPs are more likely to be Anatomic/ Clinical Pathologists and Anesthesiologists. CEPs were concentrated in several states, including New York, California and Massachusetts. In 2017, a lower proportion of CEPs in the US healthcare workforce were in residency training, compared to 2008 (13.2% vs 22.8%). Conclusions Unlike trends from some other South Asian countries, the number of CEPs in the US has increased over the past 10 years. Migration trends may vary depending on citizenship and country of medical school training. The majority of Chinese-educated graduates come to the US from relatively few medical schools. Fewer CEPs currently in residency training might indicate lower success rates in securing GME training in the US.

Studies in healthcare simulation research are often based on performance scores. These scores can be used to compare provider groups, establish the efficacy of competing educational programs, and identify clinical skills deficiencies. The following chapter provides an overview of the development and use of assessment tools. Researchers need to select tools that align with the purpose of the assessment. Where human evaluators are employed, they should have sufficient expertise in the domains being assessed. Training is also necessary to ensure that evaluators are using the rubrics as intended. In the future, technology may be helpful for gathering accurate data from, and providing standardized scoring for, various simulation-based assessments. Healthcare simulation researchers who employ assessment tools need to evaluate whether the scores that are produced represent reliable and valid estimates of ability. Without some assurance of the psychometric rigor of the scores, their use in any research study could be questioned.

Background Peer-assisted learning may augment health care provider training. We studied four questions surrounding the extension of faculty resources by using trained peer mentors in the low-stakes formative assessment of IV skills in sixty-two second semester associate degree nursing students. Method Peer mentors rated performances and provided immediate feedback. Faculty rated recorded performances. Faculty rated performances at follow-up. Results Peer mentor checklist scores were moderately associated with faculty ratings. Both identified low performers. No change in scores occurred over time. Conclusions This study found that peer mentors can extend faculty resources and provide feedback. This may not impact student long-term skills retention.

Contains fulltext : 172025.pdf (Publisher’s version ) (Open Access) INTRODUCTION: Clinical encounters are often assessed using a checklist. However, without direct faculty observation, the timing and sequence of questions are not captured. We theorized that the sequence of questions can be captured and measured using coherence scores that may distinguish between low and high performing candidates. METHODS: A logical sequence of key features was determined using the standard case checklist for an observed structured clinical exam (OSCE). An independent clinician educator reviewed each encounter to provide a global rating. Coherence scores were calculated based on question sequence. These scores were compared with global ratings and checklist scores. RESULTS: Coherence scores were positively correlated to checklist scores and to global ratings, and these correlations increased as global ratings improved. Coherence scores explained more of the variance in student performance as global ratings improved. DISCUSSION: Logically structured question sequences may indicate a higher performing student, and this information is often lost when using only overall checklist scores. CONCLUSIONS: The sequence test takers ask questions can be accurately recorded, and is correlated to checklist scores and to global ratings. The sequence of questions during a clinical encounter is not captured by traditional checklist scoring, and may represent an important dimension of performance.

Purpose To compare the relative utility of openbook examinations (OBEs) and closedbook examinations (CBEs) given the rapid expansion and accessibility of knowledge. Method A systematic review of peer-reviewed articles retrieved from MEDLINE, ERIC, Embase, and PsycINFO (through June 2013). In 2013-2014, articles that met inclusion criteria were reviewed by at least two investigators and coded for six outcome categories: (1) examination preparation, (2) test anxiety, (3) exam performance, (4) psychometrics and logistics, (5) testing effects, and (6) public perception. Results From 4,192 identified studies, 37 were included. The level of learner and subject studied varied. The frequency of each outcome category was as follows: (1) exam preparation (n = 20; 54%); (2) test anxiety (n = 14; 38%); (3) exam performance (n = 30; 81%); (4) psychometrics and logistics (n = 5; 14%); (5) testing effects (n = 24; 65%); and (6) public perception (n = 5; 14%). Preexamination outcome findings were equivocal, but students may prepare more extensively for CBEs. For during-examination outcomes, examinees appear to take longer to complete OBEs. Studies addressing examination performance favored CBE, particularly when preparation for CBE was greater than for OBE. Postexamination outcomes suggest little difference in testing effects or public perception. Conclusions Given the data available, there does not appear to be sufficient evidence for exclusively using CBE or OBE. As such, a combined approach could become a more significant part of testing protocols as licensing bodies seek ways to assess competencies other than the maintenance of medical knowledge.

Key Points Question What is the prevalence of international medical graduates from Muslim-majority nations in the US physician workforce, and what are the trends in this group’s contribution to the labor force? Findings In this cross-sectional study, citizens from Muslim-majority nations made up 4.5% of the US physician workforce in 2019. Applications to the Educational Commission for Foreign Medical Graduates increased from 2009 to 2015 and decreased from 2016 to 2018. Meaning The findings of this study suggest that the contribution of international medical graduates from Muslim-majority nations is likely to decrease, which may exacerbate gaps in the US physician workforce., This cross-sectional study provides an overview of the characteristics of international medical graduates from Muslim-majority nations, including their contributions to the US physician workforce, and describes trends in the number of applications for certification to the Educational Commission for Foreign Medical Graduates between 2019 and 2018, both overall and for citizens of Muslim-majority nations., Importance Historically, the US physician workforce has included a large number of international medical graduates (IMGs). Recent US immigration policies may affect the inflow of IMGs, particularly those who are citizens of Muslim-majority nations. Objectives To provide an overview of the characteristics of IMGs from Muslim-majority nations, including their contributions to the US physician workforce, and to describe trends in the number of applications for certification to the Educational Commission for Foreign Medical Graduates between 2019 and 2018, both overall and for citizens of Muslim-majority nations. Design, Setting, and Participants This cross-sectional study, which included 1 065 606 US physicians listed in the 2019 American Medical Association Physician Masterfile and 156 017 applicants to the Educational Commission for Foreign Medical Graduates certification process between 2009 and 2018, used a repeated cross-sectional study design to review the available data, including country of medical school attended, citizenship when entering medical school, and career information, such as present employment, specialty, and type of practice. Exposures Country of citizenship when entering medical school. Main Outcomes and Measures Physician counts and demographic information from the 2019 American Medical Association Physician Masterfile and applicant data from the Educational Commission for Foreign Medical Graduates from 2009 to 2018. Results Of 1 065 606 physicians in the American Medical Association Physician Masterfile, 263 029 (24.7%) were IMGs, of whom 48 354 were citizens of Muslim-majority countries at time of entry to medical school, representing 18.4% of all IMGs. Overall, 1 in 22 physicians in the US was an IMG from a Muslim-majority nation, representing 4.5% of the total US physician workforce. More than half of IMGs from Muslim-majority nations (24 491 [50.6%]) come from 3 countries: Pakistan (14 352 [29.7%]), Iran (5288 [10.9%]), and Egypt (4851 [10.0%]). The most prevalent specialties include internal medicine (10 934 [23.6%]), family medicine (3430 [7.5%]), pediatrics (2767 [5.9%]), and psychiatry (2251 [4.8%]), with 18 229 (38.1%) practicing in primary care specialties. The number of applicants for Educational Commission for Foreign Medical Graduates certification from Muslim-majority countries increased from 2009 (3227 applicants) to 2015 (4244 applicants), then decreased by 2.1% in 2016 to 4254 applicants, 4.3% in 2017 to 4073 applicants, and 11.5% in 2018 to 3604 applicants. Much of this decrease could be attributed to fewer citizens from Pakistan (1042 applicants in 2015 to 919 applicants in 2018), Egypt (493 applicants in 2015 to 309 applicants in 2018), Iran (281 applicants in 2015 to 182 applicants in 2018), and Saudi Arabia (337 applicants in 2015 to 163 applicants in 2018) applying for certification. Conclusions and Relevance Based on the findings of this study, the number of ECFMG applicants from Muslim-majority countries decreased from 2015 to 2018. The US physician workforce will continue to rely on IMGs for some time to come. To the extent that citizens from some countries no longer seek residency positions in the US, gaps in the physician workforce could widen.

Educational outcome measures, known to be associated with the quality of care, are needed to support improvements in graduate medical education (GME). This retrospective observational study sought to determine whether there was a relationship between the specialty board certification rates of GME training institutions and the quality of care delivered by their graduates. It is based on 7 years of hospitalizations in Pennsylvania ( N = 354,767) with diagnoses of acute myocardial infarction, congestive heart failure, gastrointestinal hemorrhage, or pneumonia. The 2,265 attending physicians were self-identified internists, and they completed their training in 59 institutions. The percentage of board-certified physicians from each training institution, excluding the physician herself or himself, was calculated and an indicator of whether it exceeded 80% was created. This was analyzed against inhospital mortality and length of stay, adjusted for patient/physician/hospital characteristics. There were significantly lower odds of mortality (adjusted Odd’s ratio [ OR] = .92, 95% CI [0.86, 0.98]) and log length of stay (adjusted OR = .98, 95% CI [.94, .99]) when the attending physician trained in a residency program with an 80% or greater certification rate. The results suggest that specialty certification rates may be a useful educational outcome for residency training programs.

Purpose Health care professionals are expected to acquire decision-making skills during their training, but few methods are available to assess progress in acquiring these essential skills. The purpose of this study was to determine whether a simulation methodology could be used to assess whether decision-making skills improve during critical care training. Materials and methods Sixteen simulated scenarios were designed to assess a critical care provider's ability to make decisions in the care of a critical ill patient. Seventeen (17) critical care providers managed 8 of the scenarios early during their training and then managed a second set of 8 scenarios (T2) at the conclusion of their training. Results Provider's mean global scenario scores (0–9) increased significantly fromT1 and T2 (5.64 ± 0.74) and (6.54 ± 0.64) with a large effect size (1.3). Acute care nurse practitioners and fellows achieved similar overall scores at the conclusion of their training (ACNP 6.43 ± 0.57; Fellows 6.64 ± 0.72). Conclusions These findings provide evidence to support the validity of a simulation-based method to assess progress in decision-making skills. A simulation methodology could be used to establish a performance standard that determined a provider's ability to make independent decisions.

Introduction The education of health professionals and the assessment of their long-term goals are essential. Additionally, in the military health system (MHS), such programs also have the goals of improving readiness, providing high-quality care and meeting the needs and challenges of the MHS. The Uniformed Services University F. Edward Hebert School of Medicine (USUHS) has existed for over 40 years and the assessment of the longer term outcomes of its medical school graduates is an important endeavor. The purpose of this study is to describe the relationship of USUHS on the care in the MHS by reporting specialty choices, practice characteristics and locations of two consecutive cohorts of USUHS graduates and to compare these cohort findings with national data. Two cohorts were chosen as the first cohort has reached retirement and we sought to describe the impact of our graduates following their military service commitment. Materials and Methods We performed a retrospective analysis of our graduates (1980-2009) using data obtained from the American Medical Association Physician Masterfile in two 15-year cohorts. USU graduate data are described and compared with national allopathic medical school data in order to explore USUHS graduates' practice characteristics and how these contrast with national data. Results Family medicine was the most commonly identified primary care specialty across both USUHS cohorts. Other primary care specialties such as Internal Medicine and Pediatrics became more popular among USUHS graduates over time. There were a lower percentage of inactive USUHS graduates from the 1980-1994 cohorts than the U.S. national sample (3.23% vs. 3.98%). The proportion of USUHS graduates working in an office-based practice increased from 34.8% (1980-1994) to 43.6% (1995-2009) yet still was lower than U.S. national data (1980-1994: 73.5%; 1995-2009: 73.8%). The type of practice in which the majority of USUHS and national graduates were involved was direct patient care (over 70% of the population). Practice locations for medical school graduates included all 50 states. Furthermore, though several states such as Mississippi, Nevada, South Carolina, and Utah were among the top 20 practice locations among USUHS graduates, they did not appear among the top 20 practicing states for U.S. graduates. Conclusions USUHS can play a major role by training health care professionals needed to meet the large health care strategic goals of a transforming military health system. A high percentage of USUHS graduates had significant changes in practice characteristics over time, are staying in practice for longer time compared with U.S. graduates, and practice in all 50 states including those that are underserved.

Develop a standardized simulation method to assess clinical skills of ICU providers.Simulation assessment.Simulation laboratory.Residents, Critical Care Medicine fellows, acute care nurse practitioner students.Performance scoring in scenarios from multiple Critical Care Medicine competency domains.Three-hundred eighty-four performances by 48 participants were scored using checklists (% correct) and holistic "global" ratings (1 [unprepared] to 9 [expert]). One-hundred eighty were scored by two raters. Mean checklist and global scores (± SD) ranged from 65.0% (± 16.3%) to 84.5% (± 17.3%) and 4.7 (± 1.4) to 7.2 (± 1.2). Checklist and global scores for Critical Care Medicine fellows and senior acute care nurse practitioner students (Experienced group, n = 26) were significantly higher than those for the Novice acute care nurse practitioner students (Novice group, n = 14) (75.6% ± 15.6% vs 68.8% ± 21.0% and 6.1 ± 1.6 vs 5.4 ± 1.5, respectively; p0.05). Residents (Intermediate group, n = 8) scored between the two (75.4% ± 18.3% and 5.7 ± 1.7). 38.5% of the Experienced group scored in the top quartile for mean global score, compared with 12.5% of the Intermediate and 7.1% of the Novice groups. Conversely, 50% of the Novice group scored in the lower quartile (5.3), compared with 37.5% of the Intermediate and 11.5% of the Experienced groups. Psychometric analyses yielded discrimination values greater than 0.3 for most scenarios and reliability for the eight-scenario assessments of 0.51 and 0.60, with interrater reliability of 0.71 and 0.75, for checklist and global scoring, respectively.The simulation assessments yielded reasonably reliable measures of Critical Care Medicine decision-making skills. Despite a wide range of performance, those with more ICU training and experience performed better, providing evidence to support the validity of the scores. Simulation-based assessments may ultimately prove useful to determine readiness to assume decision-making roles in the ICU.

Breaking bad news (BBN) is particularly difficult in perinatology. Previous research has shown that BBN skills can be learned and improved when taught and practiced. This project evaluated whether a structured training session would enhance perinatology residents' skills in BBN. This was a randomized controlled intervention study with year 1 to 4 Perinatology residents from a medical school in Brazil, during the 2014/15 school year. A total of 61 out of 100 (61%) eligible residents volunteered to a structured training program involving communicating a perinatal loss to a simulated patient (SP) portraying the mother followed by the SP's immediate feedback, both video recorded. Later, residents were randomly assigned to BBN training based on a setting, perception, invitation, knowledge, emotion and summary (SPIKES) strategy with video reviews (intervention) or no training (control group). All residents returned for a second simulation with the same SP blinded to the intervention and portraying a similar case. Residents' performances were then evaluated by the SP with a checklist. The statistical analysis included a repeated measures analysis of covariance (RM-ANCOVA). Complementarily, the residents provided their perceptions about the simulation with feedback activities. Fifty-eight residents completed the program. The simulations lasted on average 12 minutes, feedback 5 minutes and SPIKES training between 1h and 2h30m. There was no significant difference in the residents' performances according to the SPs' evaluations ( The SPIKES training did not significantly impact the residents' performance. The residents endorsed the simulation with feedback as a useful training modality. Further research is needed to determine which modality is more effective. É uma tarefa particularmente difícil transmitir más notícias em perinatologia. Habilidades de comunicação podem ser aprendidas, ensinadas e praticadas. O presente estudo avalia se uma sessão de treinamento estruturado para comunicar más notícias ampliaria as habilidades dos residentes de perinatologia. MéTODOS: Estudo de intervenção controlado e aleatorizado com residentes do 1° ao 4° ano do curso de perinatologia de uma faculdade de ciências médicas no ano letivo de 2014/15. Um total de 61 dos 100 residentes elegíveis (61%) voluntariaram-se para um programa de treinamento envolvendo comunicar uma perda perinatal para uma paciente simulada no papel da mãe, seguido do feedback imediato da atriz, ambos filmados. Posteriormente, os residentes foram aleatoriamente designados para um grupo de treinamento em más notícias baseado na estratégia SPIKES e revisão dos vídeos (intervenção) ou para um grupo-controle, sem treinamento. Todos os residentes retornaram numa segunda simulação análoga à primeira, com a mesma paciente simulada cega à intervenção. Avaliou-se as habilidades dos residentes segundo um checklist preenchido pela atriz. A análise estatística incluiu análise de covariância para medidas repetidas (ANCOVA-MR). Os residentes avaliaram a atividade de simulação com feedback. O programa foi completado por 58 residentes. As simulações duraram em média 12 minutos, o feedback 5 minutos, e o treinamento SPIKES entre 1h e 2,5h. Não houve diferença significativa nas atuações dos residentes segundo a paciente simulada (

Context As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society. Objectives We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role. Methods We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies. Discussion As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians.

BackgroundObtaining reliable and valid information on resident performance is critical to patient safety and training program improvement. The goals were to characterize important anesthesia resident performance gaps that are not typically evaluated, and to further validate scores from a multiscenario simulation-based assessment.MethodsSeven high-fidelity scenarios reflecting core anesthesiology skills were administered to 51 first-year residents (CA-1s) and 16 third-year residents (CA-3s) from three residency programs. Twenty trained attending anesthesiologists rated resident performances using a seven-point behaviorally anchored rating scale for five domains: (1) formulate a clear plan, (2) modify the plan under changing conditions, (3) communicate effectively, (4) identify performance improvement opportunities, and (5) recognize limits. A second rater assessed 10% of encounters. Scores and variances for each domain, each scenario, and the total were compared. Low domain ratings (1, 2) were examined in detail.ResultsInterrater agreement was 0.76; reliability of the seven-scenario assessment was r = 0.70. CA-3s had a significantly higher average total score (4.9 ± 1.1 vs. 4.6 ± 1.1, P = 0.01, effect size = 0.33). CA-3s significantly outscored CA-1s for five of seven scenarios and domains 1, 2, and 3. CA-1s had a significantly higher proportion of worrisome ratings than CA-3s (chi-square = 24.1, P < 0.01, effect size = 1.50). Ninety-eight percent of residents rated the simulations more educational than an average day in the operating room.ConclusionsSensitivity of the assessment to CA-1 versus CA-3 performance differences for most scenarios and domains supports validity. No differences, by experience level, were detected for two domains associated with reflective practice. Smaller score variances for CA-3s likely reflect a training effect; however, worrisome performance scores for both CA-1s and CA-3s suggest room for improvement.