Search

BACKGROUND: A contextual error occurs when a physician overlooks elements of a patient's environment or behavior that are essential to planning appropriate care. In contrast to biomedical errors, which are not patient-specific, contextual errors represent a failure to individualize care. OBJECTIVE: To explore the frequency and circumstances under which physicians probe contextual and biomedical red flags and avoid treatment error by incorporating what they learn from these probes. DESIGN: An incomplete randomized block design in which unannounced, standardized patients visited 111 internal medicine attending physicians between April 2007 and April 2009 and presented variants of 4 scenarios. In all scenarios, patients presented both a contextual and a biomedical red flag. Responses to probing about flags varied in whether they revealed an underlying complicating biomedical or contextual factor (or both) that would lead to errors in management if overlooked. SETTING: 14 practices, including 2 academic clinics, 2 community-based primary care networks with multiple sites, a core safety net provider, and 3 U.S. Department of Veterans Affairs facilities. MEASUREMENTS: Primary outcomes were the proportion of visits in which physicians probed for contextual and biomedical factors in response to hints or red flags and the proportion of visits that resulted in error-free treatment plans. RESULTS: Physicians probed fewer contextual red flags (51%) than biomedical red flags (63%). Probing for contextual or biomedical information in response to red flags was usually necessary but not sufficient for an error-free plan of care. Physicians provided error-free care in 73% of the uncomplicated encounters, 38% of the biomedically complicated encounters, 22% of the contextually complicated encounters, and 9% of the combined biomedically and contextually complicated encounters. Limitations: Only 4 case scenarios were used. The study assessed physicians' propensity to make errors when every encounter provided an opportunity to do so and did not measure actual error rates that occur in primary care settings because of inattention to context. CONCLUSION: Inattention to contextual information, such as a patient's transportation needs, economic situation, or caretaker responsibilities, can lead to contextual error, which is not currently measured in assessments of physician performance. PRIMARY FUNDING SOURCE: U.S. Department of Veterans Affairs Health Services Research and Development Service [ABSTRACT FROM AUTHOR]

PROBLEM. Nurse-physician communication affects patient safety. Such communication has been well studied using a variety of survey and observational methods; however, missing from the literature is an investigation of what constitutes effective and ineffective interprofessional communication from the perspective of the professionals involved. The purpose of this study was to explore nurse and physician perceptions of effective and ineffective communication between the two professions. METHODS. Using focus group methodology, we asked nurses and physicians with at least 5 years' acute care hospital experience to reflect on effective and ineffective interprofessional communication and to provide examples. Three focus groups were held with 6 participants each (total sample 18). Sessions were audio recorded and transcribed verbatim. Transcripts were coded into categories of effective and ineffective communication. FINDINGS. The following themes were found. For effective communication: clarity and precision of message that relies on verification, collaborative problem solving, calm and supportive demeanor under stress, maintenance of mutual respect, and authentic understanding of the unique role. For ineffective communication: making someone less than, dependence on electronic systems, and linguistic and cultural barriers. CONCLUSION. These themes may be useful in designing learning activities to promote effective interprofessional communication. [ABSTRACT FROM AUTHOR]

BACKGROUND: Despite the highly acclaimed psychometric features of a 360-degree assessment in the fields of economics, military, and education, there has been increased interest in developing 360-degree instruments to assess competencies in graduate medical education only in the past recent years. Most of the effort to date, however, has focused on developing instruments and testing their reliability and feasibility. Insufficient attention has gone into issues of construct validity and particularly understanding the underlying constructs on which the instruments are based as well as the phenomena that affect ratings. PURPOSE: In preparation for developing a 360-degree assessment instrument, we explored variations in evaluators' opinion type of a competent resident and offer observation about evaluator's professional background and opinions. METHOD: Evaluators from two residency programs ranked 36 opinion statements, using a relative-ranking model, based on their opinion of a competent resident. By-person factor analysis was used to structure opinion types. RESULTS: Factor analysis of 156 responses identified four factors interpreted as four different opinion types of a competent resident: (a) altruistic, compassionate healer (n = 42 evaluators), (b) scientifically grounded clinician (n = 30), (c) holistic, humanistic clinician (n = 62), and (d) patient-focused, health manager (n = 31). Although 72% of nurses/respiratory therapist evaluators expressed type C, 28% expressed other types just as often. Only 14% of evaluator physicians expressed type D, and the remainders were evenly split among the other types. CONCLUSIONS: Our evaluators in 360-degree system expressed four opinion types of a competent resident. The individual opinion and not professional background influences the characteristics an evaluator values in a competent resident. We propose that these values will have an impact on competency assessment and should be taken into account in a 360-degree assessment. [ABSTRACT FROM AUTHOR]

Background: The whole-test Ebel and Yes/No Angoff methods offer the possibility of faster and easier standard-setting exercises for local, medium-stakes performance exams. Purpose: We sought to establish if these less demanding methods might be substituted for the traditional but complex case-level Ebel. Methods: Six faculty judges each used all three methods to set standards for six standardized-patient checklists used for 178 fourth-year medical students. We compared the cut scores, passing rates, reliability, ease of use, and credibility of the three methods. Results: The three methods produced roughly equivalent cut scores. Generalizability of judgments was .94 and .96 for the case-level and whole-test Ebel, and .76 for the Yes/No Angoff. Judges found the simplified methods more credible than the case-level Ebel. The Yes/No Angoff method was preferred by five of six judges. Conclusions: The whole-test Ebel and the Yes/No Angoff may be simple and realistic options for setting standards for local performance exams. [ABSTRACT FROM AUTHOR]

OBJECTIVES: Clinical decision making requires 2 distinct cognitive skills: the ability to classify patients' conditions into diagnostic and management categories that permit the application of research evidence and the ability to individualize or-more specifically-to contextualize care for patients whose circumstances and needs require variation from the standard approach to care. The purpose of this study was to develop and test a methodology for measuring physicians' performance at contextualizing care and compare it to their performance at planning biomedically appropriate care. METHODS: First, the authors drafted 3 cases, each with 4 variations, 3 of which are embedded with biomedical and/or contextual information that is essential to planning care. Once the cases were validated as instruments for assessing physician performance, 54 internal medicine residents were then presented with opportunities to make these preidentified biomedical or contextual errors, and data were collected on information elicitation and error making. RESULTS: The case validation process was successful in that, in the final iteration, the physicians who received the contextual variant of cases proposed an alternate plan of care to those who received the baseline variant 100% of the time. The subsequent piloting of these validated cases unmasked previously unmeasured differences in physician performance at contextualizing care. The findings, which reflect the performance characteristics of the study population, are presented. CONCLUSIONS: This pilot study demonstrates a methodology for measuring physician performance at contextualizing care and illustrates the contribution of such information to an overall assessment of physician practice. [ABSTRACT FROM AUTHOR]

OBJECTIVE: This exploratory study investigated whether prior task experience and comfort correlate with scores on an assessment of patient-centered communication. METHODS: A six-station standardized patient exam assessed patient-centered communication of 79 PGY2-3 residents in Internal Medicine and Family Medicine. A survey provided information on prior experiences. t-tests, correlations, and multi-factorial ANOVA explored relationship between scores and experiences. RESULTS: Experience with a task predicted comfort but did not predict communication scores. Comfort was moderately correlated with communication scores for some tasks; residents who were less comfortable were indeed less skilled, but greater comfort did not predict higher scores. Female gender and medical school experiences with standardized patients along with training in patient-centered interviewing were associated with higher scores. Residents without standardized patient experiences in medical school were almost five times more likely to be rejected by patients. CONCLUSIONS: Task experience alone does not guarantee better communication, and may instill a false sense of confidence. Experiences with standardized patients during medical school, especially in combination with interviewing courses, may provide an element of 'deliberate practice' and have a long-term impact on communication skills. PRACTICE IMPLICATIONS: The combination of didactic courses and practice with standardized patients may promote a patient-centered approach. [ABSTRACT FROM AUTHOR]

Describes a workshop in curriculum development for international medical fellows designed and implemented at the Department of Medical Education of the University of Illinois at Chicago. Definition of curriculum; Basic principles involved in the course; Educational strategies.

Introduction: Simulation-based education is a recognized way of developing medical competencies, and there is overwhelming scientific evidence to support its efficacy. However, it is still underused, which can often be related to poor implementation process. In addition, best practices for implementation of simulation-based courses based on implementation science are not widely known nor applied. The purpose of this study was to develop a rubric, the Implementation Quality Rubric for Simulation (IQR-SIM), to evaluate the implementation quality of simulation-based courses., Methods: A 3-round, modified Delphi process involving international simulation and implementation experts was initiated to gather and converge opinions regarding criteria for evaluating the implementation quality of simulation-based courses. Candidate items for Round 1 were developed based on the Adapted Implementation Model for Simulation. Items were revised and expanded to include descriptive anchors for evaluation in Round 2. Criterion for inclusion was 70% of respondents selecting an importance rating of 4 or 5/5. Round 3 provided refinement and final approval of items and anchors., Results: Thirty-three experts from 9 countries participated. The initial rubric of 32 items was reduced to 18 items after 3 Delphi rounds, resulting in the IQR-SIM: a 3-point rating scale, with nonscored options "Don't know/can't assess" and "Not applicable," and a comments section., Conclusions: The IQR-SIM is an operational tool that can be used to evaluate the implementation quality of simulation-based courses and aid in the implementation process to identify gaps, monitor the process, and promote the achievement of desired implementation and learning outcomes., Competing Interests: The authors declare no conflict of interest., (Copyright © 2022 Society for Simulation in Healthcare.)

Purpose: Scoring postencounter patient notes (PNs) yields significant insights into student performance, but the resource intensity of scoring limits its use. Recent advances in natural language processing (NLP) and machine learning allow application of automated short answer grading (ASAG) for this task. This retrospective study evaluated psychometric characteristics and reliability of an ASAG system for PNs and factors contributing to implementation, including feasibility and case-specific phrase annotation required to tune the system for a new case., Method: PNs from standardized patient (SP) cases within a graduation competency exam were used to train the ASAG system, applying a feed-forward neural networks algorithm for scoring. Using faculty phrase-level annotation, 10 PNs per case were required to tune the ASAG system. After tuning, ASAG item-level ratings for 20 notes were compared across ASAG-faculty (4 cases, 80 pairings) and ASAG-nonfaculty (2 cases, 40 pairings). Psychometric characteristics were examined using item analysis and Cronbach's alpha. Inter-rater reliability (IRR) was examined using kappa., Results: ASAG scores demonstrated sufficient variability in differentiating learner PN performance and high IRR between machine and human ratings. Across all items the ASAG-faculty scoring mean kappa was .83 (SE ± .02). The ASAG-nonfaculty pairings kappa was .83 (SE ± .02). The ASAG scoring demonstrated high item discrimination. Internal consistency reliability values at the case level ranged from a Cronbach's alpha of .65 to .77. Faculty time cost to train and supervise nonfaculty raters for 4 cases was approximately $1,856. Faculty cost to tune the ASAG system was approximately $928., Conclusions: NLP-based automated scoring of PNs demonstrated a high degree of reliability and psychometric confidence for use as learner feedback. The small number of phrase-level annotations required to tune the system to a new case enhances feasibility. ASAG-enabled PN scoring has broad implications for improving feedback in case-based learning contexts in medical education., (Copyright © 2023 by the Association of American Medical Colleges.)

Objectives: To conduct a pilot investigation about the alignment between didactic multimedia materials utilized by pharmacy faculty, with Mayer's Principles for Multimedia Learning and faculty characteristics associated with greater alignment., Methods: An investigatory systematic process was used which included a modified Learning Object Review Instrument (LORI) to evaluate the faculty video-recorded lectures for alignment with Mayer's Principles of Multimedia Learning, hence capturing the number and type of misalignments. Correlations were performed to evaluate the association between faculty characteristics; and ratings and proportions of misalignments., Results: Five hundred fifty-five PowerPoint slides of 13 lectures from 13 faculty members were reviewed. The average (SD) LORI score per slide was 4.44 (0.84) out of 5 with an average score per lecture ranging from 3.83 (0.96) to 4.95 (0.53). Across all lecture slides, misalignments with multimedia principles were captured in 20.2% of slides. For each lecture, the average percentage of misalignments was 27.6% ranging from 0% to 49%. Principal misalignments included violation of the principles of coherence (66.1%), signaling (15.2%), and segmenting (8%). No faculty characteristics were significantly associated with LORI ratings or proportion of misalignments within lectures., Conclusions: Faculty had high LORI ratings for their multimedia material but these varied significantly between lectures. Misalignments with multimedia principles were identified and were related primarily to extraneous processing. These misalignments, when addressed, have the potential to improve learning, thus suggesting an opportunity for the faculty to develop ways to optimize multimedia educational delivery. Future investigation is needed to clarify how clinical pharmacy faculty can develop multimedia material and the impact of faculty development on the application of multimedia principles and learning outcomes., (Copyright © 2023 American Association of Colleges of Pharmacy. Published by Elsevier Inc. All rights reserved.)

Introduction: Engaging learners in continuing medical education (CME) is challenging. Recently, CME courses have transitioned to livestreamed CME, with learners viewing live, in-person courses online. The authors aimed to (1) compare learner engagement and teaching effectiveness in livestreamed with in-person CME and (2) determine how livestream engagement and teaching effectiveness is associated with (A) interactivity metrics, (B) presentation characteristics and (C) medical knowledge., Methods: A 3-year, non-randomised study of in-person and livestream CME was performed. The course was in-person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and Teaching Effectiveness Instrument after each presentation. Both instruments were supported by content, internal structure and relations to other variables' validity evidence. Interactivity metrics included learner use of audience response, questions asked by learners and presentation views. Presentation characteristics included presentations using audience response, using pre/post-test format, time of day and words per slide. Medical knowledge was assessed by audience response. A repeated measures analysis of variance (anova) was used for comparisons and a mixed model approach for correlations., Results: A total of 159 learners (response rate 27%) completed questionnaires. Engagement did not significantly differ between in-person or livestream CME. (4.56 versus 4.53, p = 0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in-person compared with livestream (4.77 versus 4.71 p = 0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45, p < .0001), use of a pre/post-test (yes = 4.62, no = 4.54, p < .0001) and time of presentation (morning = 4.58, afternoon = 4.53, p = .0002). Significant associations were not seen for interactivity metrics or medical knowledge., Discussion: Livestreaming may be as engaging as in-person CME. Although teaching effectiveness in livestreaming was lower, this difference was small. CME course planners should consider offering livestream CME while exploring strategies to enhance teaching effectiveness in livestreamed settings., (© 2022 Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Background: The move toward telemedicine has markedly accelerated with the COVID-19 pandemic. Anesthesia residents must learn to provide preoperative assessments on a virtual platform. We created a pilot telemedicine curriculum for postgraduate year-2 (PGY2) anesthesiology., Methods: The curriculum included a virtual didactic session and a simulated virtual preoperative assessment with a standardized patient (SP). A faculty member and the SP provided feedback using a checklist based on the American Medical Association Telehealth Visit Etiquette Checklist and the American Board of Anesthesiology Applied Examination Objective Structured Clinical Examination content outline. Residents completed surveys assessing their perceptions of the effectiveness and helpfulness of the didactic session and simulated encounter, as well as the cognitive workload of the encounter., Results: A total of 12 PGY2 anesthesiology residents in their first month of clinical anesthesia residency training participated in this study. Whereas most (11/12) residents felt confident, very confident, or extremely confident in being able to conduct a telemedicine preoperative assessment after the didactic session, only 42% ensured adequate lighting and only 33% ensured patient privacy before conducting the visit. Postencounter survey comments indicated that the SP encounter was of greater value (more effective and helpful) than the didactic session. Residents perceived the encounter as demanding, but they felt successful in accomplishing it and did not feel rushed. Faculty and SP indicated that the checklist guided them in providing clear and useful formative feedback., Conclusions: A virtual SP encounter can augment didactics to help residents learn and practice essential telemedicine skills for virtual preoperative assessments., Competing Interests: Conflicts of interest: None, (© 2023 Society for Education in Anesthesia.)

Purpose: Post-standardized patient (SP) encounter patient notes used to assess students' clinical reasoning represent a significant time burden for faculty who traditionally score them. To reduce this burden, the authors previously reported a complex faculty-developed scoring method to assess patient notes rated by nonclinicians. The current study explored whether a simplified scoring procedure for nonclinician raters could further optimize patient note assessments by reducing time, cost, and creating additional opportunities for formative feedback., Method: Ten nonclinician raters scored patient notes of 141 students across 5 SP cases by identifying case-specific patient note checklist items. The authors identified the bottom quintile of students using the proportion of correct items identified in the note (percent-scores) and case-specific faculty-generated scoring formulas (formula-scores). Five faculty raters scored a subset of notes from low, borderline, and high-performing students (n = 30 students) using a global rating scale. The authors performed analyses to gather validity evidence for percent-scores (i.e., relationship to other variables), investigate its reliability (i.e., generalizability study), and evaluate its costs (i.e., faculty time)., Results: Nonclinician percent- and formula-scores were highly correlated ( r = .88) and identified similar lists of low-performing students. Both methods demonstrated good agreement for pass-fail determinations with each other (Kappa = .68) and with faculty global ratings (Kappa percent =.61; Kappa formula =.66). The G-coefficient of percent-scores was .52, with 38% of variability attributed to checklist items nested in cases. Using percent-scores saved an estimated $746 per SP case (including 6 hours of faculty time) in development costs over formula-scores., Conclusions: Nonclinician percent-scores reliably identified low-performing students without the need for complex faculty-developed scoring formulas. Combining nonclinician analytic and faculty holistic ratings can reduce the time and cost of patient note scoring and afford faculty more time to coach at-risk students and provide targeted assessment input for high-stakes summative exams., (Copyright © 2022 by the Association of American Medical Colleges.)

Objectives: Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC., Design: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions., Setting: Academic surgical residency program., Participants: Forty-eight surgical residents (PGY-1 to PGY-5)., Results: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions., Conclusions: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments., Competing Interests: Other Disclosures No conflicts of interest., (Copyright © 2022. Published by Elsevier Inc.)

Background: A formative hepato-pancreato-biliary (HPB) ultrasound (US) skills practicum is offered annually to graduating HPB fellows, using entrustment assessments for open (IOUS) and laparoscopic (LAPUS) US. It is hypothesized that validity evidence will support the use of these assessments to determine if graduating fellows are well prepared to perform HPB US independently., Methods: Expert faculty were surveyed to set Mastery Entrustment standards for fellow performance. Standards were applied to fellow performances during two annual US skills practicums., Results: 11 faculty questionnaires were included. Mean Entrustment cut scores across all items were 4.9/5.0 and 4.8/5.0 and Global Entrustment cut scores were 5.0/5.0 and 4.8/5.0 for IOUS and LAPUS, respectively. 78.5% (29/37) fellows agreed to have their de-identified data evaluated. Mean fellow Entrustments (across all skills) were 4.1 (SD 0.6; 2.6-4.9) and 3.9 (SD 0.7; 2.7-5), while the Global Entrustments were 3.6 (SD 0.8; 2-5) and 3.5 (SD 1.0; 2-5) for IOUS and LAPUS, respectively., Conclusions: Two cohorts of graduating HPB fellows are not meeting Mastery Standards for HPB US performance determined by a panel of expert faculty., (Copyright © 2021 Elsevier Inc. All rights reserved.)

Purpose: With the growing importance of professionalism in medical education, it is imperative to develop professionalism assessments that demonstrate robust validity evidence. The Professionalism Mini-Evaluation Exercise (P-MEX) is an assessment that has demonstrated validity evidence in the authentic clinical setting. Identifying the factorial structure of professionalism assessments determines professionalism constructs that can be used to provide diagnostic and actionable feedback. This study examines validity evidence for the P-MEX, a focused and standardized assessment of professionalism, in a simulated patient setting., Method: The P-MEX was administered to 275 pediatric residency applicants as part of a 3-station standardized patient encounter, pooling data over an 8-year period (2012 to 2019 residency admission years). Reliability and construct validity for the P-MEX were evaluated using Cronbach's alpha, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA)., Results: Cronbach's alpha for the P-MEX was 0.91. The EFA yielded 4 factors: doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. The CFA demonstrated good model fit with a root-mean-square error of approximation of .058 and a comparative fit index of .92, confirming the reproducibility of the 4-factor structure of professionalism., Conclusions: The P-MEX demonstrates construct validity as an assessment of professionalism, with 4 underlying subdomains in doctor-patient relationship skills, interprofessional skills, professional demeanor, and reflective skills. These results yield new confidence in providing diagnostic and actionable subscores within the P-MEX assessment. Educators may wish to integrate the P-MEX assessment into their professionalism curricula., (Copyright © 2021 by the Association of American Medical Colleges.)

Background: Professionalism is a core competency that is difficult to assess. We examined the incidence of publication inaccuracies in Electronic Residency Application Service applications to our training program as potential indicators of unprofessional behavior., Study Design: We reviewed all 2019-2020 National Resident Matching Program applicants being considered for interview. Applicant demographic characteristics recorded included standardized examination scores, gender, medical school, and medical school ranking (2019 US News & World Report). Publication verification by a medical librarian was performed for peer-reviewed journal articles/abstracts, peer-reviewed book chapters, and peer-reviewed online publications. Inaccuracies were classified as "nonserious" (eg incorrect author order without author rank promotion) or "serious" (eg miscategorization, non-peer-reviewed journal, incorrect author order with author rank promotion, nonauthorship of cited existing publication, and unverifiable publication). Multivariate logistic regression analysis was performed for demographic characteristics to identify predictors of overall inaccuracy and serious inaccuracy., Results: Of 319 applicants, 48 (15%) had a total of 98 inaccuracies; after removing nonserious inaccuracies, 37 (12%) with serious inaccuracies remained. Seven publications were reported in predatory open access journals. In the regression model, none of the variables (US vs non-US medical school, gender, or medical school ranking) were significantly associated with overall inaccuracy or serious inaccuracy., Conclusions: One in 8 applicants (12%) interviewing at a general surgery residency program were found to have a serious inaccuracy in publication reporting on their Electronic Residency Application Service application. These inaccuracies might represent inattention to detail or professionalism transgressions., (Copyright © 2021 American College of Surgeons. Published by Elsevier Inc. All rights reserved.)

Competency-based medical education (CBME) is being implemented worldwide. In CMBE, residency training is designed around competencies required for unsupervised practice and use entrustable professional activities (EPAs) as workplace "units of assessment". Well-designed workplace-based assessment (WBA) tools are required to document competence of trainees in authentic clinical environments. In this study, we developed a WBA instrument to assess residents' performance of intra-operative pathology consultations and conducted a validity investigation. The entrustment-aligned pathology assessment instrument for intra-operative consultations (EPA-IC) was developed through a national iterative consultation and used clinical supervisors to assess residents' performance at an anatomical pathology program. Psychometric analyses and focus groups were conducted to explore the sources of evidence using modern validity theory: content, response process, internal structure, relations to other variables, and consequences of assessment. The content was considered appropriate, the assessment was feasible and acceptable by residents and supervisors, and it had a positive educational impact by improving performance of intra-operative consultations and feedback to learners. The results had low reliability, which seemed to be related to assessment biases, and supervisors were reluctant to fully entrust trainees due to cultural issues. With CBME implementation, new workplace-based assessment tools are needed in pathology. In this study, we showcased the development of the first instrument for assessing resident's performance of a prototypical entrustable professional activity in pathology using modern education principles and validity theory., (© 2021. The Author(s).)

Importance: The sociopolitical and cultural context of graduate surgical education has changed considerably over the past 2 decades. Although new structures of graduate surgical training programs have been developed in response and the comparative value of formats are continually debated, it remains unclear how different time-based structural paradigms are preparing trainees for independent practice after program completion., Objective: To investigate the factors associated with trainees' and program directors' perception of trainee preparedness for independent surgical practice., Design, Setting, and Participants: This qualitative study used an instrumental case study approach and obtained information through semistructured interviews, which were analyzed using open-and-focused coding. Participants were recent graduates and program directors of vascular surgery training programs in the United States. The 2 training paradigms analyzed were the integrated vascular surgery residency program (0 + 5, with 0 indicating that the general surgery training experiences are fully integrated into the 5 years of overall training and 5 indicating the total number of years of training) and the traditional vascular surgery fellowship program (5 + 2, with 5 indicating the number of years of general surgery training and 2 indicating the number of years of vascular surgery training). All graduates completed their training in 2018. All interviews were conducted between July 1, 2018, and September 30, 2018., Main Outcomes and Measures: A conceptual framework to inform current and ongoing efforts to optimize graduate surgical training programs across specialties., Results: A total of 22 semistructured interviews were completed, involving 7 graduates of 5 + 2 programs, 9 graduates of 0 + 5 programs, and 6 vascular surgery program directors. Of the 22 participants, 15 were men (68%). Participants described 4 interconnected domains that were associated with trainees' perceived preparedness for practice: structural, individual, relational, and organizational. Structural factors included the overall and vascular surgery-specific time spent in training, whereas individual factors included innate technical skills, confidence, maturity, and motivation. Faculty-trainee relationships (or relational factors) were deemed important for building trust and granting of autonomy. Organizational factors included features of the local organization, including patient population, case volume, and case mix., Conclusions and Relevance: Findings suggest that restructuring training paradigms alone is insufficient to address the issue of trainees' perceived preparedness for practice. A framework was created from the results for evaluating and improving residency and fellowship programs as well as for developing graduate surgical training paradigms that incorporate all 4 domains associated with preparedness.

Health care errors are a national concern. Although considerable attention has been placed on reducing errors since a 2000 Institute of Medicine report, adverse events persist. The purpose of this pilot study was to evaluate the effect of mindfulness training, employing the standardized approach of an eight-week mindfulness-based, stress reduction program on reduction of nurse errors in simulated clinical scenarios. An experimental, pre- and post-test control group design was employed with 20 staff nurses and senior nursing students. Although not statistically significant, there were numerical differences in clinical performance scores from baseline when comparing mindfulness and control groups immediately following mindfulness training and after three months. A number of benefits of mindfulness training, such as improved listening skills, were identified. This pilot study supports the benefits of mindfulness training in improving nurse clinical performance and illustrates a novel approach to employ in future research.

Purpose: The residency admissions process is a high-stakes assessment system with the purpose of identifying applicants who best meet standards of the residency program and the medical specialty. Prior studies have found that professionalism issues contribute significantly to residents in difficulty during training. This study examines the reliability (internal structure) and predictive (relations to other variables) validity evidence for a standardized patient (SP)-based Professionalism Mini-Evaluation Exercise (P-MEX) using longitudinal data from pediatrics candidates from admission to the end of the first year of postgraduate training., Method: Data from 5 cohorts from 2012 to 2016 (195 invited applicants) were analyzed from the University of Geneva (Switzerland) Pediatrics Residency Program. Generalizability theory was used to examine the reliability and variance components of the P-MEX scores, gathered across 3 cases. Correlations and mixed-effects regression analyses were used to examine the predictive utility of SP-based P-MEX scores (gathered as part of the admissions process) with rotation evaluation scores (obtained during the first year of residency)., Results: Generalizability was moderate (G coefficient = 0.52). Regression analyses predicting P-MEX scores to first-year rotation evaluations indicated significant standardized effect sizes for attitude and personality (β = 0.36, P = .02), global evaluation (β = 0.27, P = .048), and total evaluation scores (β = 0.34, P = .04)., Conclusions: Validity evidence supports the use of P-MEX scores as part of the admissions process to assess professionalism. P-MEX scores provide a snapshot of an applicant's level of professionalism and may predict performance during the first year of residency.

Purpose: Clinical reasoning is often assessed through patient notes (PNs) following standardized patient (SP) encounters. While nonclinicians can score PNs using analytic tools such as checklists, these do not sufficiently encompass the holistic judgments of clinician faculty. To better model faculty judgments, the authors developed checklists with faculty-specified scoring formulas embedded in spreadsheets and studied the resulting interrater reliability (IRR) of nonclinician raters (SPs and medics) and student pass/fail status., Method: In Study 1, nonclinician and faculty raters rescored PNs of 55 third-year medical students across 5 cases of the 2017 Graduation Competency Examination (GCE) to determine IRR. In Study 2, nonclinician raters scored all notes of the 5-case 2018 GCE (178 students). Faculty rescored all notes of failing students and could modify formula-derived scores if faculty felt appropriate. Faculty also rescored and corrected scores of additional notes for a total of 90 notes (3 cases, including failing notes)., Results: Mean overall percent exact agreement between nonclinician and faculty ratings was 87% (weighted kappa, 0.86) and 83% (weighted kappa, 0.88) for Study 1 and Study 2, respectively. SP and medic IRRs did not differ significantly. Four students failed the note section in 2018; 3 passed after faculty corrections. Few corrections were made to nonfailing students' notes., Conclusions: Nonclinician PN raters using checklists and scoring rules may provide a feasible alternative to faculty raters for low-stakes assessments and for the bulk of well-performing students. Faculty effort can be targeted strategically at rescoring notes of low-performing students and providing more detailed feedback.

Context: Medical school admissions committees use a variety of criteria to determine which candidates to admit to their programmes. Effective communication is increasingly considered a key requisite to the practice of effective medicine. Medical students with pre-medical backgrounds in the humanities and social sciences may be more likely to acquire skills relevant to patient-centred communication, either prior to or during medical school., Objectives: The purpose of this study was to investigate the relationship between pre-medical backgrounds in the humanities and social sciences and outcomes in medical school, including in communication and interpersonal skills (CIS), licensure examination results and postgraduate specialty choice (primary care versus non-primary care specialties)., Methods: The American Medical College Application Service database was used to identify pre-medical college majors, demographic characteristics, Medical College Admission Test scores and college grade point averages for medical students at a large, midwestern medical school. Data were obtained for 465 medical students across three cohorts (classes of 2014-2016). Correlation and regression analyses were used to examine relationships between pre-medical background, performance on graduation competency examination standardised patient encounter CIS scores and on United States Medical Licensing Examination (USMLE) Step 1 and Step 2 Clinical Knowledge scores, and postgraduate specialty choice., Results: Graduating medical students with pre-medical humanities or social sciences majors performed significantly better in terms of CIS than those with natural science majors (Cohen's d = 0.28, p = 0.011). There were no significant associations between pre-medical majors and USMLE Step 1 and Step 2 Clinical Knowledge scores or postgraduate specialty choice., Conclusions: These results suggest that considering humanistic factors as part of admissions criteria may promote the selection and training of physicians with good communication skills., (© 2018 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Objective: Communication and interpersonal skills (CIS) are essential elements of competency-based education. We examined defensible CIS passing levels for medical students completing basic sciences (second-year students) and clinical training (fourth-year students), using five standard setting methods., Methods: A 14-item CIS scale was used. Data from second-year (n = 190) and fourth-year (n = 170) students were analyzed using descriptive statistics and generalizability studies. Fifteen judges defined borderline CIS performance. Cut scores and fail rates from five standard setting methods (Angoff, Borderline-Group, Borderline-Regression, Contrasting-Groups, and Normative methods) were examined., Results: CIS performance was similar during second-year (Mean = 74%, SD = 6%) and fourth-year (Mean = 72%, SD = 5%) students. Judges using the Angoff method expected greater competence at the fourth-year level, as reflected in the Angoff cut scores (second-year = 53% with 0% fail, fourth-year = 66% with 10% fail). Cut scores from the remaining methods did not differentiate between training levels. We found evidence of case specificity., Conclusion: Performance on CIS may be case specific. Passing standards for communication skills may require employing approaches such as the Angoff method that are sensitive to expectations of learner performance for different levels of training, competencies, and milestone levels., Practice Implications: Institutions that want to encourage continued growth in CIS should apply appropriate standard setting methods., (Copyright © 2018 Elsevier B.V. All rights reserved.)

Purpose: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools. The authors developed an online KFE to assess third-year medical students' decision-making abilities during internal medicine (IM) clerkships in the United States. They used Messick's unified validity framework to gather validity evidence regarding response process, internal structure, and relationship to other variables., Method: From February 2012 through January 2013, 759 students (at eight U.S. medical schools) had 75 minutes to complete one of four KFE forms during their IM clerkship. They also completed a survey regarding their experiences. The authors performed item analyses and generalizability studies, comparing KFE scores with prior clinical experience and National Board of Medical Examiners Subject Examination (NBME-SE) scores., Results: Five hundred fifteen (67.9%) students consented to participate. Across KFE forms, mean scores ranged from 54.6% to 60.3% (standard deviation 8.4-9.6%), and Phi-coefficients ranged from 0.36 to 0.52. Adding five cases to the most reliable form would increase the Phi-coefficient to 0.59. Removing the least discriminating case from the two most reliable forms would increase the alpha coefficient to, respectively, 0.58 and 0.57. The main source of variance came from the interaction of students (nested in schools) and cases. Correlation between KFE and NBME-SE scores ranged from 0.24 to 0.47 (P < .01)., Conclusions: These results provide strong evidence for response-process and relationship-to-other-variables validity and moderate internal structure validity for using a KFE to complement other assessments in U.S. IM clerkships.

Background: The Emergency Medicine (EM) Milestone Project provides guidance for assessment of resident trainee airway management proficiency (PC10). Although milestones provide a general structure for assessment, they do not define performance standards. The objective of this project was to establish comprehensive airway management performance standards for EM trainees at both novice and mastery levels of proficiency., Methods: Comprehensive airway management standards were derived using standard-setting procedures. A panel of residency education and airway management experts was convened to determine how trainees would be expected to perform on 51 individual tasks in a standardized airway management simulation encompassing preparation, endotracheal intubation, backup airway use, and ventilation. Experts participated in facilitated exercises in which they were asked to 1) define which items were critical for patient safety, 2) predict the performance of a "novice" learner, and 3) predict the performance of a "mastery" learner nearing independent practice. Experts were given a worksheet to complete and descriptive statistics were calculated using STATA 14., Results: Experts identified 39 of 51 (76%) airway management items as critical for patient safety. Experts also noted that novice trainees do not need to complete all the items deemed to be critical prior to starting practice since they will be supervised by a board-certified EM physician. In contrast, mastery-level trainees would be expected to successfully complete not only the critical tasks, but also nearly all the items in the assessment (49/51, 96%) since they are nearing independent practice., Conclusion: In this study, we established EM resident performance standards for comprehensive airway management during a simulation scenario. Future work will focus on validating these performance standards in current resident trainees as they move from simulation to actual patient care.

Purpose: Defensible minimum passing standards (MPSs) must be used to evaluate learner performance outcomes in health professions education. In this study, the authors compared the results of traditional Angoff and Hofstee standard-setting exercises with the Mastery Angoff and Patient-Safety approaches for central venous catheter (CVC) insertion skills examinations. The authors also evaluated how these standards affected the historical performance of residents who participated in a simulation-based mastery learning (SBML) curriculum for CVC insertion skills., Method: In April and May 2015, 12 physicians with expertise in CVC insertion set MPSs for previously published internal jugular (IJ) and subclavian (SC) CVC insertion checklists using Angoff, Hofstee, Mastery Angoff, and Patient-Safety approaches. The resulting MPSs were compared using historical performance of internal medicine and emergency medicine residents who participated in CVC insertion SBML., Results: The MPSs were set as follows: Angoff: IJ 91% checklist items correct, SC 90%. Hofstee: IJ 88%, SC 90%. Mastery Angoff: IJ 98%, SC 98%. Patient-Safety: IJ 98%, SC 98%. Based on the historical performance of 143 residents assessed on IJ and SC insertion, applying the 98% MPS would result in additional practice and retesting of 55/123 residents (45%) who had previously passed the IJ examination and 36/130 residents (28%) who had passed the SC examination using the Angoff and Hofstee MPSs., Conclusions: The Mastery Angoff and Patient-Safety standard-setting approaches resulted in higher CVC insertion SBML MPSs compared with traditional standard-setting methods. Further study should assess the impact of these more rigorous standards on patient outcomes.

Construct: This study aims to provide validity evidence for the standardized Assessment Letter for Pediatrics as a measure of competencies expected of a 1st-year pediatrics resident as part of a pediatric residency admissions process., Background: The Narrative Letter of Recommendation is a frequently used tool in the residency admissions process even though it has poor interrater reliability, lacks pertinent content, and does not correlate with residency performance. A newer tool, the Standardized Letter, has shown validity evidence for content and interrater reliability in other specialties. We sought to develop and provide validity evidence for the standardized Assessment Letter for Pediatrics., Approach: All 2012 and 2013 applicants invited to interview at the University of Geneva Pediatrics Residency Program provided 2 standardized Assessment Letters. Content for the letter was based on CanMEDS roles and ratings of 6 desired competencies and an overall assessment. Validity evidence was gathered for internal structure (Cronbach's alpha and generalizability), response process (interrater reliability with intraclass correlation), relations to other variables (Pearson's correlation coefficient), and consequences (logistic regression to predict admission)., Results: One hundred fourteen faculty completed 142 standardized Assessment Letters for 71 applicants. Average overall assessment was 3.0 of 4 (SD = 0.59). Cronbach's alpha was 0.93. The G-coefficient was 0.59. The decision study projected that four Assessment Letters are needed to attain a G-coefficient of 0.73. Applicant variance (28.5%) indicated high applicant differentiation. The Assessment Letter intraclass coefficient was 0.51, 95% confidence interval (CI) [0.43, 0.59]. Assessment Letter scores were correlated with the structured interview (r = .28), 95% CI [0.05, 0.51]; global rating (r = .36), 95% CI [0.13, 0.58]; and admissions decision (r = .25), 95% CI [0.02, 0.46]. Assessment Letter scores did not predict the admissions decision (odds ratio = 1.67, p = .37) after controlling for the unique contribution of the structured interview and global rating scores., Conclusion: Validity evidence supports use of the Assessment Letter for Pediatrics; future studies should refine items to improve predictive validity and explore how to best integrate the Assessment Letter into the residency admissions process.

Objective: Develop new performance evaluation standards for the clinical breast examination (CBE)., Summary Background Data: There are several, technical aspects of a proper CBE. Our recent work discovered a significant, linear relationship between palpation force and CBE accuracy. This article investigates the relationship between other technical aspects of the CBE and accuracy., Methods: This performance assessment study involved data collection from physicians (n = 553) attending 3 different clinical meetings between 2013 and 2014: American Society of Breast Surgeons, American Academy of Family Physicians, and American College of Obstetricians and Gynecologists. Four, previously validated, sensor-enabled breast models were used for clinical skills assessment. Models A and B had solitary, superficial, 2 cm and 1 cm soft masses, respectively. Models C and D had solitary, deep, 2 cm hard and moderately firm masses, respectively. Finger movements (search technique) from 1137 CBE video recordings were independently classified by 2 observers. Final classifications were compared with CBE accuracy., Results: Accuracy rates were model A = 99.6%, model B = 89.7%, model C = 75%, and model D = 60%. Final classification categories for search technique included rubbing movement, vertical movement, piano fingers, and other. Interrater reliability was (k = 0.79). Rubbing movement was 4 times more likely to yield an accurate assessment (odds ratio 3.81, P < 0.001) compared with vertical movement and piano fingers. Piano fingers had the highest failure rate (36.5%). Regression analysis of search pattern, search technique, palpation force, examination time, and 6 demographic variables, revealed that search technique independently and significantly affected CBE accuracy (P < 0.001)., Conclusions: Our results support measurement and classification of CBE techniques and provide the foundation for a new paradigm in teaching and assessing hands-on clinical skills. The newly described piano fingers palpation technique was noted to have unusually high failure rates. Medical educators should be aware of the potential differences in effectiveness for various CBE techniques.

Purpose: To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs)., Method: Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring. Data included scores from standardized patient encounters for history taking, physical examination, and PNs. Descriptive statistics were used to examine scores from the different assessment components. Generalizability studies (G-studies) using variance components were conducted to estimate reliability for composite scores., Results: Validity evidence was collected for response process (rater perception), internal structure (variance components, reliability), relations to other variables (interassessment correlations), and consequences (composite score). Student performance varied by case and task. In the PNs, justification of differential diagnosis was the most discriminating task. G-studies showed that schools accounted for less than 1% of total variance; however, for the PNs, there were differences in scores for varying cases and tasks across schools, indicating a school effect. Composite score reliability was maximized when the PN was weighted between 30% and 40%. Raters preferred using case-specific scoring guidelines with clear point-scoring systems., Conclusions: This multisite study presents validity evidence for PN scores based on scoring rubric and case-specific scoring guidelines that offer rigor and feedback for learners. Variability in PN scores across participating sites may signal different approaches to teaching clinical reasoning among medical schools.

The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane's formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.

Purpose: Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence., Method: Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used. The reliability of SP encounter (history taking and physical examination), PN, and communication and interpersonal skills scores were estimated with generalizability studies. Composite score reliability was estimated for varying weight combinations. Faculty were surveyed for preferred weights on the SP encounter and PN scores. Composite scores based on Kane's method were compared with weighted mean scores., Results: Faculty suggested weighting PNs higher (60%-70%) than the SP encounter scores (30%-40%). Statistically, composite score reliability was maximized when PN scores were weighted at 40% to 50%. Composite score reliability of ICE scores increased by up to 0.20 points when SP-history taking (SP-Hx) scores were included; excluding SP-Hx only increased composite score reliability by 0.09 points. Classification accuracy for pass-fail decisions between composite and weighted mean scores was 0.77; misclassification was < 5%., Conclusions: Medical schools and certification agencies should consider implications of assigning weights with respect to composite score reliability and consequences on pass-fail decisions.

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was .79 (Documentation = .63, Differential Diagnosis = .90, Justification = .48, and Workup = .54). The majority of score variance was due to case specificity (13 %) and case-task specificity (31 %), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5 %). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

Background In the past few years, there has been rapid growth in the number of simulation fellowships for physicians in the United States and Canada, with the objective of producing faculty with expertise and leadership training in medical simulation. Relatively little is known about the collective content and structure of these new fellowship opportunities. Objective We sought to identify a common set of core curricular elements among existing simulation fellowships and to obtain demographic background information on participants and leadership. Methods We designed a web-based survey and circulated it to simulation fellowship directors in the United States and Canada. The questions explored aspects of the fellowship curriculum. A grounded theory approach was used to qualitatively analyze fellowship goals and objectives. Results Of the 29 program directors surveyed, 23 responded (79%). The most commonly listed goals and objectives were to increase skills in simulation curriculum development, simulation operations and training environment setup, research, educational theory, administration, and debriefing. The majority of the responding fellowship directors (17 of 22, 77%) indicated that a set of consensus national guidelines would benefit their fellowship program. Conclusions Simulation fellowships are experiencing a period of rapid growth. Development of a common set of program guidelines is a widely shared objective among fellowship directors.

Theoretical and empirical support is increasing for mastery learning, in which learners must demonstrate a minimum level of proficiency before completing a given educational unit. Mastery learning approaches aim for uniform achievement of key objectives by allowing learning time to vary and as such are a course-level analogue to broader competency-based curricular strategies. Sound assessment is the cornerstone of mastery learning systems, yet the nature of assessment validity and justification for mastery learning differs in important ways from standard assessment models. Specific validity issues include (1) the need for careful definition of what is meant by "mastery" in terms of learners' achievement or readiness to proceed, the expected retention of mastery over time, and the completeness of content mastery required in a particular unit; (2) validity threats associated with increased retesting; (3) the need for reliability estimates that account for the specific measurement error at the mastery versus nonmastery cut score; and (4) changes in item- and test-level score variance over retesting, which complicate the analysis of evidence related to reliability, internal structure, and relationships to other variables. The positive and negative consequences for learners, educational systems, and patients resulting from the use of mastery learning assessments must be explored to determine whether a given mastery assessment and pass/fail cut score are valid and justified. In this article, the authors outline key considerations for the validation and justification of mastery learning assessments, with the goal of supporting insightful research and sound practice as the mastery model becomes more widespread.

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential.Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past examinees. The marginally competent "borderline student" or "borderline group" referenced in traditional item-based and examinee-based procedures will generally need to be redefined in mastery settings. Patient safety considerations support conjunctive standards for key knowledge and skill subdomains and for items that have an impact on clinical outcomes. Finally, traditional psychometric indices used to evaluate the quality of standards do not necessarily reflect critical measurement properties of mastery assessments. Mastery learning and testing are essential to the achievement and assessment of entrustable professional activities and residency milestones. With careful attention, sound mastery standard-setting procedures can provide an essential step toward improving the effectiveness of health professions education, patient safety, and patient care.

Guidelines for reporting several types of medical studies have been described in the literature. However, there are no current guidelines to report studies on mastery learning curriculum development and trainee evaluation in medical education. Such guidelines will be important because medical education is moving toward a competency-based model. The authors sought to define standards for the evaluation of mastery learning curricula using previously published guidelines in related fields and expert review.The authors reviewed previously published guidelines from clinical medicine, medical education, and the social sciences. Six authors with expertise in mastery learning curricula, performance assessment, and medical education compiled and reached agreement about a list of guidelines. The authors later circulated the list to 12 other experts and made revisions. A final list of guidelines was established and received group consensus. The Reporting Mastery Education Research in Medicine (ReMERM) guidelines have 22 categories with 38 items considered to be imperative for reporting a mastery learning research study. Details about each item, with a specific focus on those unique to mastery learning, are discussed.The ReMERM guidelines highlight the importance of developing rigorous curricula that embody reliable measures which yield valid decisions about achievement among medical learners. These guidelines should improve the quality of reporting and help educators, authors, peer reviewers, journal editors, and readers to better understand and evaluate mastery learning research. With this shift to competency-based medical education, the ReMERM guidelines should help meet medical educators' needs to achieve these new goals.