Search

Background: Assessment of pediatric anesthesia trainees is complicated by the random nature of adverse patient events and the vagaries of clinical exposure. However, assessment is critical to improve patient safety. In previous studies, a multiple scenario assessment provided reliable and valid measures of the abilities of anesthesia residents. The purpose of this study was to develop a set of relevant simulated pediatric perioperative scenarios and to determine their effectiveness in the assessment of anesthesia residents and pediatric anesthesia fellows.Methods: Ten simulation scenarios were designed to reflect situations encountered in perioperative pediatric anesthesia care. Anesthesiology residents and fellows consented to participate and were debriefed after each scenario. Two pediatric anesthesiologists scored each scenario by key action checklist. The psychometric properties (reliability, validity) of the scores were studied.Results: Thirty-five anesthesiology residents and pediatric anesthesia fellows participated. The participants with greater experience administering pediatric anesthetics generally outperformed those with less experience. Score variance attributable to raters was low, yielding a high interrater reliability.Conclusions: A multiple-scenario, simulation-based assessment of pediatric perioperative care was designed and administered to residents and fellows. The scores obtained from the assessment indicated the content was relevant and that raters could reliably score the scenarios. Participants with more training achieved higher scores, but there was a wide range of ability among subjects. This method has the potential to contribute to pediatric anesthesia performance assessment, but additional measures of validity including correlations with more direct measures of clinical performance are needed to establish the utility of this approach. [ABSTRACT FROM AUTHOR]

Simulations have taken a central role in the education and assessment of medical students, residents, and practicing physicians. The introduction of simulation-based assessments in anesthesiology, especially those used to establish various competencies, has demanded fairly rigorous studies concerning the psychometric properties of the scores. Most important, major efforts have been directed at identifying, and addressing, potential threats to the validity of simulation-based assessment scores. As a result, organizations that wish to incorporate simulation-based assessments into their evaluation practices can access information regarding effective test development practices, the selection of appropriate metrics, the minimization of measurement errors, and test score validation processes. The purpose of this article is to provide a broad overview of the use of simulation for measuring physician skills and competencies. For simulations used in anesthesiology, studies that describe advances in scenario development, the development of scoring rubrics, and the validation of assessment results are synthesized. Based on the summary of relevant research, psychometric requirements for practical implementation of simulation-based assessments in anesthesiology are forwarded. As technology expands, and simulation-based education and evaluation takes on a larger role in patient safety initiatives, the groundbreaking work conducted to date can serve as a model for those individuals and organizations that are responsible for developing, scoring, or validating simulation-based education and assessment programs in anesthesiology. [ABSTRACT FROM AUTHOR]

Background: Standardized patients can be trained to assess the communication and interpersonal skills of medical students and graduates. Purpose: The purpose of this study is to present data to support the psychometric adequacy of the communication ratings provided by standardized patients. Methods: Using the data from testing of 3, 450 examinees over a 1-year period, a number of psychometric analyses were undertaken. These included a variance component analysis, the calculation of various validity coefficients, the comparison of communication ratings for select examinee cohorts and case characteristics, and the investigation of some potential sources of score invalidity. Results: Communication skills scores are moderately correlated to other competencies (knowledge, skills) and may be influenced by candidate characteristics such as gender and English language proficiency. They are not dependant on the age of the examinees, the clinical case content, or the gender of the standardized patients. Conclusions: For a multistation assessment, a reasonably precise and valid estimate of a candidate's communication ability can be obtained from trained standardized patients. [ABSTRACT FROM AUTHOR]

BACKGROUND AND PURPOSE: Questions remain about the congruence between students' written notes and checklists as summaries of encounters. METHODS: Students examined standardized patients and summarized findings in postencounter notes. The patients completed checklists. A physician read the students' notes and completed parallel checklists to document the history and physical items performed. Rates of under- and overdocumentation were calculated. RESULTS: Students documented findings for 71% of items performed - an underdocumentation rate of 29%. Approximately 94% of their documented findings were consistent with what they had done. Their rate of overdocumentation was 6%, in which they documented findings inconsistent with the checklists. About half the students had no instances of overdocumentation. CONCLUSION: Students' rate of underdocumentation was comparable to experienced clinicians. Although their overdocumentation rate was low overall, it was high for a few students. Evaluation of the congruence between checklists and postencounter notes provides useful information and informs checklist development. [ABSTRACT FROM AUTHOR]

Background: Anesthesiologists and anesthesia residents are expected to acquire and maintain skills to manage a wide range of acute intraoperative anesthetic events. The purpose of this study was to determine whether an inventory of simulated intraoperative scenarios provided a reliable and valid measure of anesthesia residents' and anesthesiologists' skill.Methods: Twelve simulated acute intraoperative scenarios were designed to assess the performance of 64 residents and 35 anesthesiologists. The participants were divided into four groups based on their training and experience. There were 31 new CA-1, 12 advanced CA-1, and 22 CA-2/CA-3 residents as well as a group of 35 experienced anesthesiologists who participated in the assessment. Each participant managed a set of simulated events. The advanced CA-1 residents, CA-2/CA-3 residents, and 35 anesthesiologists managed 8 of 12 intraoperative simulation exercises. The 31 CA-1 residents each managed 3 intraoperative scenarios.Results: The new CA-1 residents received lower scores on the simulated intraoperative events than the other groups of participants. The advanced CA-1 residents, CA-2/CA-3 residents, and anesthesiologists performed similarly on the overall assessment. There was a wide range of scores obtained by individuals in each group. A number of the exercises were difficult for the majority of participants to recognize and treat, but most events effectively discriminated among participants who achieved higher and lower overall scores.Conclusion: This simulation-based assessment provided a valid method to distinguish the skills of more experienced anesthesia residents and anesthesiologists from residents in early training. The overall score provided a reliable measure of a participant's ability to recognize and manage simulated acute intraoperative events. Additional studies are needed to determine whether these simulation-based assessments are valid measures of clinical performance. [ABSTRACT FROM AUTHOR]

Objective: Develop a scoring system that can assess the management of septic shock by individuals and teams.Design: Retrospective review of videotapes of critical care house staff managing a standardized simulation of septic shock.Setting: Academic medical center; videotapes were made in a recreated intensive care unit environment using a high-fidelity patient simulator.Subjects: Residents in medicine, surgery, and anesthesiology who had participated in the intensive care unit rotation.Interventions: The septic patient was managed by the intensive care unit team in a graded manner with interns present for the first 10 mins and more senior-level help arriving after 10 mins. The intern was graded separately for the first 10 mins, and the team was graded for the entire 35-min performance.Measurements and Main Results: Both technical and nontechnical scoring systems were developed to rate the management of septic shock. Technical scores are based on guidelines and principles of managing septic shock. Team leadership, communication, contingency planning, and resource utilization were addressed by the nontechnical rating. Technical scores were calculated for both interns and teams; nontechnical scores applied only to the team. Of 16 technical checklist items, interns completed a mean of 7 with a range of 1.5-11. Team technical ratings had a mean of 9.3 with a range of 3.3-13. Nontechnical scores showed similar intergroup variability with a mean of 26 and a range of 10-35. Technical and nontechnical scores showed a modest correlation (r = .40, p = .05). Interrater reliabilities for intern and team technical scores were both r = .96 and for nontechnical scores r = .88.Conclusions: Objective measures of both knowledge-based and behavioral skills pertinent to the management of septic shock were made. Scores identified both adequate and poor levels of performance. Such assessments can be used to benchmark clinical skills of individuals and groups over time and may allow the identification of interventions that improve clinical effectiveness in sepsis management. [ABSTRACT FROM AUTHOR]

International medical graduates (IMGs) represent a large proportion of the population entering graduate medical education (GME) programs. Many of these internationally trained physicians go on to practice medicine in the United States. To be eligible for GME, IMGs must be certified by the Educational Commission for Foreign Medical Graduates (ECFMG). The number of certificates issued by the ECFMG has varied over time and historically has exceeded the number of available training positions. More detailed longitudinal analyses are required to better understand the interwoven issues of physician supply, consumers' needs, and the role of IMGs in the U.S. health care system. [ABSTRACT FROM AUTHOR]

BACKGROUND: Failing candidates often make multiple attempts on licensure and certification examinations. For performance-based assessments, where available test material is often limited, overlap in examination content is frequently inevitable. PURPOSE: The purpose of this study was to investigate the performance of repeat candidates, both on new and exposed material, on a standardized patient clinical skills assessment. METHODS: Analysis of variance techniques were used to investigate the longitudinal performance of repeat candidates. Score differences for the second test session were analyzed as function of prior exposure to assessment materials.RESULTS: Although the performance of candidates improved between the first and second assessment, score increases could not be attributed to the random exposure of examination material.CONCLUSIONS: The exposure of case content on clinical skills examinations does not appear to provide any advantage, or disadvantage, to repeat test takers. This finding, although based on average scores for repeat examinees, suggests that valid assessment scores can be obtained even if there is some overlap in assessment content. [ABSTRACT FROM AUTHOR]

Background The pivotal role of doctor-patient communication in effective health care delivery led the Educational Commission for Foreign Medical Graduates (ECFMG) to incorporate the assessment of interpersonal skills and spoken English proficiency into its Clinical Skills Assessment (CSA). Furthermore, it was decided that to pass the CSA, a candidate would need to meet or surpass defined performance standards for doctor-patient communication as a discrete component. This requirement, among others, is designed to ensure the readiness of graduates of foreign medical schools (FMGs) to enter postgraduate medical education programmes in the United States. Objective The primary focus of this study was to determine the extent to which performance in a simulated testing environment is related to performance in the clinical setting. Method Nurses were trained to rate the communication skills of residents from the patient's perspective. A total of 43 first-year residents were evaluated. The survey ratings (n=225) were compared with the residents' CSA communication scores. Results Corrected correlations between CSA ratings and those obtained from nurses ranged from 0·61 to 0·73. Conclusion This study provides evidence for the validity of the communication ratings provided by standardized patients. The reasonably strong associations between ratings obtained during testing and those obtained through observation of 'real' patient interactions suggest that external observers can provide accurate evaluations of doctor-patient communication. [ABSTRACT FROM AUTHOR]

The Uniformed Services University's (USU) F. Edward Hébert School of Medicine was chartered in 1972, with the goal of providing high-quality physicians for the Uniformed Services. In exchange for their education, USU graduates incur an active duty service obligation, after which they may choose to stay on active duty or transition to civilian practice. The purpose of this study is to describe the practice characteristics of USU graduates after this obligation has been completed in order to determine the societal benefits during this phase of their careers. To accomplish this purpose, we performed a retrospective cohort study of the first 20 years of USU graduates (1980-1999). We used the American Medical Association Physician Masterfile to determine the graduates' current practice location and characteristics, as well as their board certification status. Of these 2,760 graduates, nearly all (91%) were involved in active clinical practice in over 100 self-declared specialties, the vast majority (89%) practiced in locations other than the immediate vicinity of the medical school (i.e., Maryland and the District of Columbia), and most still worked for the federal government (71%). Finally, USU graduates in full-time clinical practice had a board certification rate of 93%, which was better than the average of all other graduates of U.S. Medical Schools (88%) in the same time period. Thus, it seems USU is attaining its goal of producing high-quality physicians who continue to benefit the nation after their service obligation has been completed, with many still in federal service. [ABSTRACT FROM AUTHOR]

Context: The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores., Objectives: The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework., Methods: Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE)., Results: All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure., Conclusions: This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series., (© 2024 the author(s), published by De Gruyter, Berlin/Boston.)

Objectives: Capillary refill time (CRT) to assess peripheral perfusion in children with suspected shock may be subject to poor reproducibility. Our objectives were to compare video-based and bedside CRT assessment using a standardized protocol and evaluate interrater and intrarater consistency of video-based CRT (VB-CRT) assessment. We hypothesized that measurement errors associated with raters would be low for both standardized bedside CRT and VB-CRT as well as VB-CRT across raters., Methods: Ninety-nine children (aged 1-12 y) had 5 consecutive bedside CRT assessments by an experienced critical care clinician following a standardized protocol. Each CRT assessment was video recorded on a black background. Thirty video clips (10 with bedside CRT < 1 s, 10 with CRT 1-2 s, and 10 with CRT > 2 s) were randomly selected and presented to 10 clinicians twice in randomized order. They were instructed to push a button when they visualized release of compression and completion of a capillary refill. The correlation and absolute difference between bedside and VB-CRT were assessed. Consistency across raters and within each rater was analyzed using the intraclass correlation coefficient (ICC). A Generalizability study was performed to evaluate sources of variation., Results: We found moderate agreement between bedside and VB-CRT observations (r = 0.65; P < 0.001). The VB-CRT values were shorter by 0.17 s (95% confidence interval, 0.09-0.25; P < 0.001) on average compared with bedside CRT. There was moderate agreement in VB-CRT across raters (ICC = 0.61). Consistency of repeated VB-CRT within each rater was moderate (ICC = 0.71). Generalizability study revealed the source of largest variance was from individual patient video clips (57%), followed by interaction of the VB-CRT reviewer and patient video clip (10.7%)., Conclusions: Bedside and VB-CRT observations showed moderate consistency. Using video-based assessment, moderate consistency was also observed across raters and within each rater. Further investigation to standardize and automate CRT measurement is warranted., Competing Interests: The other authors declare no conflict of interest., (Copyright © 2022 Wolters Kluwer Health, Inc. All rights reserved.)

Objective: This study describes the supply, distribution, and characteristics of international medical graduate (IMG) psychiatrists who provide services in the USA., Methods: Cross-sectional study design, using descriptive statistics based on combined data from the American Medical Association (2020 Physician Masterfile) and the Educational Commission for Foreign Medical Graduates., Results: International medical graduates continue to make significant contributions to the US physician workforce. As a group, they represent 29% of active psychiatrists in the USA, compared to 23% in all other medical specialties. Many IMG psychiatrists were US citizens who obtained their medical degrees outside the USA or Canada, often in the Caribbean. In some states (i.e., Florida, New Jersey), over 40% of active psychiatrists are IMGs. Over 30% of IMG psychiatrists graduated from medical schools in India and Pakistan., Conclusions: This study provides an overview of the psychiatric workforce in the USA, quantifying the specific contribution of IMGs. Several factors, including immigration policies, continued expansion of US medical schools, and the number of available residency positions, could impact the flow of IMGs to the US. Longitudinal studies are needed to better understand the implications for workforce composition and distribution, and their potential impact on the care of psychiatric patients., (© 2022. The Author(s), under exclusive licence to American Association of Chairs of Departments of Psychiatry, American Association of Directors of Psychiatric Residency Training, Association for Academic Psychiatry and Association of Directors of Medical Student Education in Psychiatry.)

Purpose: The reproducibility and consistency of assessments of entrustable professional activities (EPAs) in undergraduate medical education (UME) have been identified as potential areas of concern. EPAs were designed to facilitate workplace-based assessments by faculty with a shared mental model of a task who could observe a trainee complete the task multiple times. In UME, trainees are frequently assessed outside the workplace by faculty who only observe a task once., Method: In November 2019, the authors conducted a generalizability study (G-study) to examine the impact of student, faculty, case, and faculty familiarity with the student on the reliability of 162 entrustment assessments completed in a preclerkship environment. Three faculty were recruited to evaluate 18 students completing 3 standardized patient (SP) cases. Faculty familiarity with each student was determined. Decision studies were also completed. Secondary analysis of the relationship between student performance and entrustment (scoring inference) compared average SP checklist scores and entrustment scores., Results: G-study analysis revealed that entrustment assessments struggled to achieve moderate reliability. The student accounted for 30.1% of the variance in entrustment scores with minimal influence from faculty and case, while the relationship between student and faculty accounted for 26.1% of the variance. G-study analysis also revealed a difference in generalizability between assessments by unfamiliar (φ = 0.75) and familiar (φ = 0.27) faculty. Subanalyses showed that entrustment assessments by familiar faculty were moderately correlated to average SP checklist scores (r = 0.44, P < .001), while those by unfamiliar faculty were weakly correlated (r = 0.16, P = .13)., Conclusions: While faculty and case had a limited impact on the generalizability of entrustment assessments made outside the workplace in UME, faculty who were familiar with a student's ability had a notable impact on generalizability and potentially on the scoring validity of entrustment assessments, which warrants further study., (Copyright © 2021 by the Association of American Medical Colleges.)

Background: Accreditation systems strive to ensure the quality of undergraduate (basic) medical education and encourage ongoing improvements. Despite increasing global emphasis on quality assurance activities, there is limited research linking accreditation of medical education to improved student and graduate outcomes. The purpose of this study is to compare the United States Medical Licensing Examination® (USMLE®) performance of students and graduates who attended international medical schools accredited by an agency recognized by the World Federation of Medical Education (WFME) to individuals who attended schools that did not meet this criterion., Methods: During the 2018-2020 study period, 39,650 individuals seeking Educational Commission for Foreign Medical Graduates® (ECFMG®) certification took one or more USMLE examinations. We cross-tabulated USMLE performance (first-attempt pass/fail result) and medical school accreditation status., Results: Individuals seeking ECFMG certification who attended international medical schools accredited by an agency recognized by WFME had higher or comparable USMLE first-attempt pass rates compared to individuals who attended medical schools that did not meet this criterion., Conclusions: Implementing and maintaining meaningful accreditation systems requires substantial resources. These results provide important positive evidence that external evaluation of educational programs is associated, on average, with better educational outcomes, including in the domains of basic science, clinical knowledge, and clinical skills performance., (© 2022. The Author(s).)

Purpose: In 2024, international medical graduates seeking Educational Commission for Foreign Medical Graduates (ECFMG) certification will be required to graduate from an accredited medical school. This study's goal was to examine relationships between medical school accreditation variables and ECFMG certification for a global sample., Method: Using ECFMG databases, the authors created a 10-year cohort (January 1, 2007-December 31, 2016) of certification applicants, defined as individuals who had attempted at least 2 examinations required for certification. The authors aggregated applicant data at the school level, excluding schools with < 80 applicants. School accreditation statuses were based on agency websites. School region, age, and time of first accreditation were included. Analyses included descriptive and bivariate statistics and multiple linear regressions adjusting for school start year and year of first accreditation., Results: The cohort included 128,046 applicants from 1,973 medical schools across 162 countries. After excluding low-volume schools, 318 schools across 81 countries remained. These provided 99,598 applicants and 77,919 certificate holders, three-quarters of whom came from the Caribbean, South-Central Asia, and West Asia regions. Two hundred and fifty (78.6%) schools were accredited; 68 (21.4%) were not. Most ECFMG applicants (n = 84,776, 85.1%) and certificate holders (n = 68,444, 87.8%) attended accredited medical schools. Accredited schools had higher rates of ECFMG certification among graduates than nonaccredited schools in comparisons that included all schools (75.0% [standard deviation (SD) = 10.6%] vs 68.3% (SD = 15.9%), P < .001), and for countries that had both accredited and nonaccredited schools (73.9% [SD = 11.4%] vs 67.3% [SD = 16.8%], P = .023). After adjusting for age of school, longer duration of accreditation was associated with higher certification rates (P < .001)., Conclusions: Accreditation had a positive association with ECFMG certification rates. Future studies should investigate how accreditation processes might account for higher certification rates., (Copyright © 2021 by the Association of American Medical Colleges.)

Background: Health care professionals must be able to make frequent and timely decisions that can alter the illness trajectory of intensive care patients. A competence standard for this ability is difficult to establish yet assuring practitioners can make appropriate judgments is an important step in advancing patient safety. We hypothesized that simulation can be used effectively to assess decision-making competence. To test our hypothesis, we used a "standard-setting" method to derive cut scores (standards) for 16 simulated ICU scenarios targeted at decision-making skills and applied them to a cohort of critical care trainees., Methods: Panelists (critical care experts) reviewed digital audio-video performances of critical care trainees managing simulated critical care scenarios. Based on their collectively agreed-upon definition of "readiness" to make decisions in an ICU setting, each panelist made an independent judgment (ready, not ready) for a large number of recorded performances. The association between the panelists' judgments and the assessment scores was used to derive scenario-specific performance standards., Results: For all 16 scenarios, the aggregate panelists' ratings (ready/not ready for independent decision making) were positively associated with the performance scores, permitting derivation of performance standards for each scenario., Conclusions: Minimum competence standards for high-stakes decision making can be established through standard-setting techniques. We effectively identified "front-line" providers who are, or are not, ready to make independent decisions in an ICU setting. Our approach may be used to assure stakeholders that clinicians are competent to make appropriate judgments. Further work is needed to determine whether our approach is effective in simulation-based assessments in other domains., Competing Interests: The authors declare no conflicts of interest., (Copyright © 2020 International Anesthesia Research Society.)

Introduction: Even physicians who routinely work in complex, dynamic practices may be unprepared to optimally manage challenging critical events. High-fidelity simulation can realistically mimic critical clinically relevant events, however the reliability and validity of simulation-based assessment scores for practicing physicians has not been established., Methods: Standardised complex simulation scenarios were developed and administered to board-certified, practicing anesthesiologists who volunteered to participate in an assessment study during formative maintenance of certification activities. A subset of the study population agreed to participate as the primary responder in a second scenario for this study. The physicians were assessed independently by trained raters on both teamwork/behavioural and technical performance measures. Analysis using Generalisability and Decision studies were completed for the two scenarios with two raters., Results: The behavioural score was not more reliable than the technical score. With two raters > 20 scenarios would be required to achieve a reliability estimate of 0.7. Increasing the number of raters for a given scenario would have little effect on reliability., Conclusions: The performance of practicing physicians on simulated critical events may be highly context-specific. Realistic simulation-based assessment for practicing physicians is resource-intensive and may be best-suited for individualized formative feedback. More importantly, aggregate data from a population of participants may have an even higher impact if used to identify skill or knowledge gaps to be addressed by training programs and inform continuing education improvements across the profession.

Background and Objectives: To describe the supply, distribution, and characteristics of international medical graduates (IMGs) in pediatrics who provide patient care in the United States., Methods: Cross-sectional study, combining data from the 2019 Physician Masterfile of the American Medical Association and the Educational Commission for Foreign Medical Graduates database., Results: In total, 92 806 pediatric physicians were identified, comprising 9.4% of the entire US physician workforce. Over half are general pediatricians. IMGs account for 23.2% of all general pediatricians and pediatric subspecialists. Of all IMGs in pediatrics, 22.1% or 4775 are US citizens who obtained their medical degree outside the United States or Canada, and 15.4% (3246) attended medical school in the Caribbean. Fifteen non-US medical schools account for 29.9% of IMGs currently in active practice in pediatrics in the United States. IMGs are less likely to work in group practice or hospital-based practice and are more likely to be employed in solo practice (compared with US medical school graduates)., Conclusions: With this study, we provide an overview of the pediatric workforce, quantifying the contribution of IMGs. Many IMGs are US citizens who attend medical school abroad and return to the United States for postgraduate training. Several factors, including the number of residency training positions, could affect future numbers of IMGs entering the United States. Longitudinal studies are needed to better understand the implications that workforce composition and distribution may have for the care of pediatric patients., Competing Interests: POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose., (Copyright © 2020 by the American Academy of Pediatrics.)

Educational outcome measures, known to be associated with the quality of care, are needed to support improvements in graduate medical education (GME). This retrospective observational study sought to determine whether there was a relationship between the specialty board certification rates of GME training institutions and the quality of care delivered by their graduates. It is based on 7 years of hospitalizations in Pennsylvania ( N = 354,767) with diagnoses of acute myocardial infarction, congestive heart failure, gastrointestinal hemorrhage, or pneumonia. The 2,265 attending physicians were self-identified internists, and they completed their training in 59 institutions. The percentage of board-certified physicians from each training institution, excluding the physician herself or himself, was calculated and an indicator of whether it exceeded 80% was created. This was analyzed against inhospital mortality and length of stay, adjusted for patient/physician/hospital characteristics. There were significantly lower odds of mortality (adjusted Odd's ratio [ OR ] = .92, 95% CI [0.86, 0.98]) and log length of stay (adjusted OR = .98, 95% CI [.94, .99]) when the attending physician trained in a residency program with an 80% or greater certification rate. The results suggest that specialty certification rates may be a useful educational outcome for residency training programs.

Purpose: Family medicine residency programs can be cited for low pass or take rates on the American Board of Family Medicine (ABFM) certification examination, and the relationships among standardized medical education assessments and performance on board certification examinations and eventual board certification have not been comprehensively studied. The objective of this study was to evaluate the associations of all required standardized examinations in medical education with ABFM certification examination scores and eventual ABFM certification., Method: All graduates of U.S. MD-granting family medicine residency programs from 2008 to 2012 were included. Data on ABFM certification examination score, ABFM certification status (as of December 31, 2014), Medical College Admission Test (MCAT) section scores, undergraduate grade point average, all United States Medical Licensing Examination (USMLE) Step scores, and all ABFM in-training examination scores were linked. Nested logistic and linear regression models, controlling for clustering by residency program, determined associations between assessments and both certification examination scores and board certification status. As many international medical graduates (IMGs) do not take the MCAT, separate models for U.S. medical graduates (USMG) and IMGs were run., Results: The study sample was 15,902 family medicine graduates, of whom 92.1% (14,648/15,902) obtained board certification. In models for both IMGs and USMGs, the addition of more recent assessments weakened the associations of earlier assessments. USMLE Step 2 Clinical Knowledge was predictive of certification examination scores and certification status in all models in which it was included., Conclusions: For family medicine residents, more recent assessments generally have stronger associations with board certification score and status than earlier assessments. Solely using medical school admissions (grade point average and MCAT) and licensure (USMLE) scores for resident selection may not adequately predict ultimate board certification.

After extensive stakeholder discussion, the Federation of State Medical Boards and the National Board of Medical Examiners announced in February 2020 that United States Medical Licensing Examination Step 1 will transition to a pass/fail exam. Program directors have historically used Step 1 scores in deciding which residency applicants to interview. The lack of numerical scores will force changes to the residency selection process, which could have both positive and negative consequences for international medical graduates (IMGs). In this Invited Commentary, the authors discuss how some of the issues associated with the transition to Step 1 pass/fail are likely to impact IMGs. The authors also provide insights into how this and other policy changes could help spur the medical education community to improve the process by which medical school graduates transition to graduate medical education.

Background: From national and international workforce perspectives, Canadians studying medicine abroad (CSAs) are a growing provider group. Some were born in Canada whereas others immigrated as children. They study medicine in various countries, often attempting both American and Canadian medical licensure pathways., Methods: Using data from the Educational Commission for Foreign Medical Graduates (ECFMG) and the Medical Council of Canada (MCC), we looked at CSAs who attempted to secure residency positions in both Canada and the United States. We detailed the CSAs' countries of birth and medical education. We tracked these individuals through their postgraduate education programs to enumerate their success rate and categorize the geographic locations of their training., Results: The majority of CSAs study medicine in one of 10 countries. The remainder are disbursed across 88 other countries. Most CSAs were born in Canada (62%). Approximately 1/3 of CSA from the 2004-2016 cohort had no record of entering a residency program in Canada or the United States (U.S.). Recently graduated CSAs were most likely to secure residency training in Ontario and New York., Conclusion: Many CSAs attempt to secure residency training in both Canada and the U.S. Quantifying success rates may be helpful for Canadians thinking about studying medicine abroad. Understanding the educational pathways of CSAs will be useful for physician labour workforce planning., Competing Interests: Conflicts of interest: The authors note there are no conflicts of interest due to financial nor personal relationships that could potentially bias their work., (© 2020 Bartman, Boulet, Qin, Bowmer; licensee Synergies Partners.)

Importance: Historically, the US physician workforce has included a large number of international medical graduates (IMGs). Recent US immigration policies may affect the inflow of IMGs, particularly those who are citizens of Muslim-majority nations., Objectives: To provide an overview of the characteristics of IMGs from Muslim-majority nations, including their contributions to the US physician workforce, and to describe trends in the number of applications for certification to the Educational Commission for Foreign Medical Graduates between 2019 and 2018, both overall and for citizens of Muslim-majority nations., Design, Setting, and Participants: This cross-sectional study, which included 1 065 606 US physicians listed in the 2019 American Medical Association Physician Masterfile and 156 017 applicants to the Educational Commission for Foreign Medical Graduates certification process between 2009 and 2018, used a repeated cross-sectional study design to review the available data, including country of medical school attended, citizenship when entering medical school, and career information, such as present employment, specialty, and type of practice., Exposures: Country of citizenship when entering medical school., Main Outcomes and Measures: Physician counts and demographic information from the 2019 American Medical Association Physician Masterfile and applicant data from the Educational Commission for Foreign Medical Graduates from 2009 to 2018., Results: Of 1 065 606 physicians in the American Medical Association Physician Masterfile, 263 029 (24.7%) were IMGs, of whom 48 354 were citizens of Muslim-majority countries at time of entry to medical school, representing 18.4% of all IMGs. Overall, 1 in 22 physicians in the US was an IMG from a Muslim-majority nation, representing 4.5% of the total US physician workforce. More than half of IMGs from Muslim-majority nations (24 491 [50.6%]) come from 3 countries: Pakistan (14 352 [29.7%]), Iran (5288 [10.9%]), and Egypt (4851 [10.0%]). The most prevalent specialties include internal medicine (10 934 [23.6%]), family medicine (3430 [7.5%]), pediatrics (2767 [5.9%]), and psychiatry (2251 [4.8%]), with 18 229 (38.1%) practicing in primary care specialties. The number of applicants for Educational Commission for Foreign Medical Graduates certification from Muslim-majority countries increased from 2009 (3227 applicants) to 2015 (4244 applicants), then decreased by 2.1% in 2016 to 4254 applicants, 4.3% in 2017 to 4073 applicants, and 11.5% in 2018 to 3604 applicants. Much of this decrease could be attributed to fewer citizens from Pakistan (1042 applicants in 2015 to 919 applicants in 2018), Egypt (493 applicants in 2015 to 309 applicants in 2018), Iran (281 applicants in 2015 to 182 applicants in 2018), and Saudi Arabia (337 applicants in 2015 to 163 applicants in 2018) applying for certification., Conclusions and Relevance: Based on the findings of this study, the number of ECFMG applicants from Muslim-majority countries decreased from 2015 to 2018. The US physician workforce will continue to rely on IMGs for some time to come. To the extent that citizens from some countries no longer seek residency positions in the US, gaps in the physician workforce could widen.

Introduction: The purpose of this study was to examine Uniformed Services University (USU) women graduates in terms of percent of graduates', specialty choices and practice choices as compared to civilian women who graduate and practice medicine in the USA. This is a perspective that is currently not well understood., Materials and Methods: We conducted a retrospective cohort study of all USU women graduates (1980-2015) using the 2016 American Medical Association (AMA) Physician Masterfile that included data from the American Board of Medical Specialties (ABMS). To describe USU women graduates' current practice status we queried for: (1) medical school; (2) year of graduation; (3) practice state; (4) primary specialty board; and (5) major professional activity (office-based practice vs. full-time hospital staff). Data were analyzed using descriptive statistics., Results: Our findings indicate that the percentage of USU women graduates has increased over time and stands at 29% for the 2010-2015 cohort as compared to 48% for women graduating from all U.S. medical schools. USU women graduates have a slightly higher board certification rate (89%) than the national cohort (88%). USU women graduates also have a higher percentage in family medicine (19%) than the national cohort (14%). USU women graduates practice in 48 states and were equally split between full-time hospital staff and office-based practice which differs from the national cohort that has a much higher proportion in office-based practice (85%)., Conclusions: Women are making significant gains in enrollment at USU, obtaining board certification at similar, and in some cases, higher rates than their civilian peers, and practicing in diverse specialties. This study provides a descriptive picture of women's enrollment and practice characteristics from a military-based medical school. Future work could examine underlying factors that may influence their school choice, career experiences, and trajectories. Future research could also focus on women's experiences of mentoring and support to better understand these factors., (Published by Oxford University Press on behalf of the Association of Military Surgeons of the United States 2018.)

Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA's further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public's health.

Background: To describe the supply, distribution, and characteristics of international medical graduates (IMGs) in family medicine who provide patient care in the U.S., Methods: A cross-sectional study design, using descriptive statistics on combined data from the Educational Commission for Foreign Medical Graduates and the American Medical Association, including medical school attended, country of medical school, and citizenship when entering medical school., Results: In total, 118,817 physicians in family medicine were identified, with IMGs representing 23.8% (n = 28,227) of the U.S. patient care workforce. Of all 9579 residents in family medicine, 36.0% (n = 3452) are IMGS. In total, 35.9% of IMGs attended medical school in the Caribbean (n = 10,136); 19.9% in South-Central Asia (n = 5607) and 9.1% in South-Eastern Asia (n = 2565). The most common countries of medical school training were Dominica, Mexico, and Sint Maarten. Of all IMGs in family medicine who attended medical school in the Caribbean, 74.5% were U.S. citizens. In total, 40.5% of all IMGs in family medicine held U.S. citizenship at entry to medical school. IMGs comprise almost 40% of the family medicine workforce in Florida, New Jersey and New York., Conclusions: IMGs play an important role in the U.S. family medicine workforce. Many IMGs are U.S. citizens who studied abroad and then returned to the U.S. for graduate training. Given the shortage of family physicians, and the large number of IMGs in graduate training programs, IMGs will continue to play a role in the U.S. physician workforce for some time to come. Many factors, including the supply of residency training positions, could eventually restrict the number of IMGs entering the U.S., including those contributing to family practice.

Introduction: The education of health professionals and the assessment of their long-term goals are essential. Additionally, in the military health system (MHS), such programs also have the goals of improving readiness, providing high-quality care and meeting the needs and challenges of the MHS. The Uniformed Services University F. Edward Hebert School of Medicine (USUHS) has existed for over 40 years and the assessment of the longer term outcomes of its medical school graduates is an important endeavor. The purpose of this study is to describe the relationship of USUHS on the care in the MHS by reporting specialty choices, practice characteristics and locations of two consecutive cohorts of USUHS graduates and to compare these cohort findings with national data. Two cohorts were chosen as the first cohort has reached retirement and we sought to describe the impact of our graduates following their military service commitment., Materials and Methods: We performed a retrospective analysis of our graduates (1980-2009) using data obtained from the American Medical Association Physician Masterfile in two 15-year cohorts. USU graduate data are described and compared with national allopathic medical school data in order to explore USUHS graduates' practice characteristics and how these contrast with national data., Results: Family medicine was the most commonly identified primary care specialty across both USUHS cohorts. Other primary care specialties such as Internal Medicine and Pediatrics became more popular among USUHS graduates over time. There were a lower percentage of inactive USUHS graduates from the 1980-1994 cohorts than the U.S. national sample (3.23% vs. 3.98%). The proportion of USUHS graduates working in an office-based practice increased from 34.8% (1980-1994) to 43.6% (1995-2009) yet still was lower than U.S. national data (1980-1994: 73.5%; 1995-2009: 73.8%). The type of practice in which the majority of USUHS and national graduates were involved was direct patient care (over 70% of the population). Practice locations for medical school graduates included all 50 states. Furthermore, though several states such as Mississippi, Nevada, South Carolina, and Utah were among the top 20 practice locations among USUHS graduates, they did not appear among the top 20 practicing states for U.S. graduates., Conclusions: USUHS can play a major role by training health care professionals needed to meet the large health care strategic goals of a transforming military health system. A high percentage of USUHS graduates had significant changes in practice characteristics over time, are staying in practice for longer time compared with U.S. graduates, and practice in all 50 states including those that are underserved.

Context: As the practice of medicine evolves, the knowledge, skills and attitudes required to provide patient care will continue to change. These competency-based changes will necessitate the restructuring of assessment systems. High-quality assessment programmes are needed to fulfil health professions education's contract with society., Objectives: We discuss several issues that are important to consider when developing assessments in health professions education. We organise the discussion along the continuum of medical education, outlining the tension between what has been deemed important to measure and what should be measured. We also attempt to alleviate some of the apprehension associated with measuring evolving competencies by discussing how emerging technologies, including simulation and artificial intelligence, can play a role., Methods: We focus our thoughts on the assessment of competencies that, at least historically, have been difficult to measure. We highlight several assessment challenges, discuss some of the important issues concerning the validity of assessment scores, and argue that medical educators must do a better job of justifying their use of specific assessment strategies., Discussion: As in most professions, there are clear tensions in medicine in relation to what should be assessed, who should be responsible for administering assessment content, and how much evidence should be gathered to support the evaluation process. Although there have been advances in assessment practices, there is still room for improvement. From the student's, resident's and practising physician's perspectives, assessments need to be relevant. Knowledge is certainly required, but there are other qualities and attributes that are important, and perhaps far more important. Research efforts spent now on delineating what makes a good physician, and on aligning new and upcoming assessment tools with the relevant competencies, will ensure that assessment practices, whether aimed at establishing competence or at fostering learning, are effective with respect to their primary goal: to produce qualified physicians., (© 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)