Search

The medical education continuum is interrupted by several transition periods that can adversely affect performance. Most of what has been learned about such periods focuses upon movement from one stage of training to another and movement from training to practice. Established physicians, however, experience transitions throughout their careers at idiosyncratic times and with little assistance. Better understanding how physicians experience transition, where they struggle and how they adapt, would enable better support to be provided. We investigated the COVID-19-forced transition in clinical practice to virtual care, particularly its effect on physician roles and the ways that established physicians faced challenges they encountered when transitioning to virtual care.Ten semistructured interviews were conducted between November 2020 and February 2021 with physicians across different specialties and practice contexts who transitioned their practice to virtual care during the COVID-19 pandemic. Interview data were analyzed iteratively using "generic qualitative methodology" with constant comparison to identify themes in relation to observations.The transition to telehealth had implications that extended beyond the patient encounter, appearing to affect all aspects of the physician's practice. To reflect that, CanMEDS was chosen as a useful organizing framework. The effects, captured in the theme "changes to the physician's roles," were nuanced, illustrated a consistent need to adapt to context, and could be framed positively or negatively or both. Additionally identified themes were labeled "physicians' mental health" and "strategies to mitigate challenges." These themes highlighted that, despite the effort involved and novelty of the situation, all participants found remarkably similar ways of grappling with the challenges faced.While the basic roles of the physician do not appear to have changed through the transition to telehealth, our findings indicated that these roles were redefined in fundamental ways in response to changing societal needs.

Introduction\ud Scholarship is a key activity in health professions education (HPE). When disseminating scholarly work, how one selects the journal to which they submit is often argued to be a key determinant of subsequent success. To draw more evidence-based recommendations in this regard, we surveyed successful scholars working in HPE regarding their perspectives and experiences with journal selection.\ud \ud Methods\ud We conducted an international survey of HPE scholars, investigating their decisions regarding journal choice. Corresponding authors were identified from a sample of 4000 papers published in 2019 and 2020. They were invited via email with up to four reminders. We describe their experience and use principle component and regression analyses to identify factors associated with successful acceptance.\ud \ud Results\ud In total, 863 responses were received (24.7% response rate), 691 of which were included in our analyses. Two thirds of respondents had their manuscripts accepted at their first-choice journal with revisions required in 98% of cases. We identified six priority factors when choosing journals. In descending order of importance, they were: fit, impact, editorial reputation, speed of dissemination, breadth of dissemination, and guidance from others. Authors who prioritised fit higher and who selected a journal earlier were more likely to have their manuscripts accepted at their first-choice journal.\ud \ud Discussion\ud Based on our results we make three recommendations for authors when writing manuscripts: do not be disheartened by a revise decision, consider journal choice early in the research process, and use the fit between your manuscript and the journal as the main factor driving journal choice.

PURPOSE To optimize learning, health professional training programs need to achieve the right balance between depth of practice (gaining more experience with particular skills) and breadth of practice (spreading experience across an array of activities). Better understanding how training for a particular skill set is impacted by periods of focus on a different skill set would allow improved curriculum and assessment design, thereby enhancing the efficiency of training and effectiveness of care. To this end, learning curves were used to compare performance in surgery after prolonged periods of practice to performance after gaps in surgical training. METHOD Daily operative assessments from the Dalhousie obstetrics and gynecology program were analyzed retrospectively and learning curves were generated. In addition to examining the variability in learning trajectories, the impact of gaps was systematically assessed by comparing resident scores after 2 successive months in which they were not assessed operatively to those collected after 2 successive months in which they were assessed at least once. RESULTS Four thousand four hundred sixteen scores for 33 residents over a 10-year period were analyzed. Trajectories and peak performances were identified. Residents performed better during their third sequential month of being assessed (mean = 4.40, 95% CI = 4.33-4.46) relative to during months following a period of being away from the operating room for at least 2 months (mean = 4.21, 95% CI = 4.13-4.29; P < .01; d = 0.7). However, maximum performance achieved was more strongly related to the number of times residents experienced a gap in training (r = 0.50) than to the number of times residents experienced 3 consecutive months of training (r = 0.25). CONCLUSIONS Distinct patterns of development exist for individual residents. Time away from surgical practice and assessment negatively impacted short-term performance, but may improve long-term learning trajectories. This speaks to the value of spaced education and is important for the design of longitudinal skills-based training programs.

IntroductionElectronic health records (EHRs) are increasingly common platforms used in medical settings to capture and store patient information, but very little has been published about how EHR implementation affects educational practice from the point of view of clinician-learner interactions. This research sought to examine how EHR implementation is anticipated to affect clinician-learner interactions and, in turn, impact upon educational priorities and outcomes. MethodsSemi-structured interviews were conducted with a group of practicing oncologists who work in outpatient clinics while also providing education to medical student and resident trainees. Data regarding perceived impact on the teaching dynamic between clinicians and learners were collected prior to implementation of an EHR. ResultsPhysician educators expected EHR implementation to influence the learning they themselves normally gain through teaching interactions as well as their engagement in teaching. Additionally, EHR implementation was expected to influence learners by changing what is taught, what is modelled, and their role in both clinical care and the educational dynamic. ConclusionUnderstanding the concerns clinicians have about EHR implementation both offers potential to enable changes to be made that could minimize disruptions caused by implementation and provides a foundation from which to assess actual educational impacts.

Introduction When medical education programs have difficulties recruiting or retaining clinical teachers, they often introduce incentives to help improve motivation. Previous research, however, has shown incentives can unfortunately have unintended consequences. When and why that is the case in the context of incentivizing clinical teachers remains unclear. The purposes of this study, therefore, were to understand what values and motivations influence teaching decisions; and to delve deeper into how teaching incentives have been perceived.Methods An interpretive description methodology was used to improve understanding of the development and delivery of teaching incentives. A purposeful sampling strategy identified a heterogenous sample of clinical faculty teaching in undergraduate and postgraduate contexts. Sixteen semi-structured interviews were conducted and transcripts were analyzed using an iterative process to develop a thematic structure that accounts for general trends and individual variations.Results Clinicians articulated interrelated and dynamic personal and environmental factors that had linear, dual-edged and inverted U-shaped impacts on their motivations towards teaching. Barriers were frequently rationalized away, but cumulative barriers often led to teaching attrition. Clinical teachers were motivated when they felt valued and connected to their learners, peers, leadership, and/or the medical education community. While incentives aimed at producing these connections could be perceived as supportive, they could also negatively impact motivation if they were impersonal, inequitable, inefficient, or poorly framed.Discussion/Conclusion These findings reinforce the literature suggesting that it is necessary to proceed with caution when labeling any particular factor as a motivator or barrier to teaching. They take us deeper, however, towards understanding how and why clinical teachers' perceptions are unique, dynamic and fluid. Incentive schemes can be beneficial for teacher recruitment and retention, but must be designed with nuance that takes into account what makes clinicians feel valued if the strategy is to do more good than harm.

Given the widespread use of Multiple Mini Interviews (MMIs), their impact on the selection of candidates and the considerable resources invested in preparing and administering them, it is essential to ensure their quality. Given the variety of station formats used and the degree to which that factor resides in the control of training programmes that we know so little about, format's effect on MMI quality is a considerable oversight. This study assessed the effect of two popular station formats (interview vs. role-play) on the psychometric properties of MMIs.We analysed candidate data from the first 8 years of the Integrated French MMIs (IF-MMI) (2010-2017, n = 11 761 applicants), an MMI organised yearly by three francophone universities and administered at four testing sites located in two Canadian provinces. There were 84 role-play and 96 interview stations administered, totalling 180 stations. Mixed design analyses of variance (ANOVAs) were used to test the effect of station format on candidates' scores and stations' discrimination. Cronbach's alpha coefficients for interview and role-play stations were also compared. Predictive validity of both station formats was estimated with a mixed multiple linear regression model testing the relation between interview and role-play scores with average clerkship performance for those who gained entry to medical school (n = 462).Role-play stations (M = 20.67, standard deviation [SD] = 3.38) had a slightly lower mean score than interview stations (M = 21.36, SD = 3.08), p 0.01, Cohen's d = 0.2. The correlation between role-play and interview stations scores was r = 0.5 (p 0.01). Discrimination coefficients, Cronbach's alpha and predictive validity statistics did not vary by station format.Interview and role-play stations have comparable psychometric properties, suggesting format to be interchangeable. Programmes should select station format based on match to the personal qualities for which they are trying to select.

PURPOSE Written comments are gaining traction as robust sources of assessment data. Compared with the structure of numeric scales, what faculty choose to write is ad hoc, leading to idiosyncratic differences in what is recorded. This study offers exploration of what aspects of writing styles are determined by the faculty offering comment and what aspects are determined by the trainee being commented upon. METHOD The authors compiled in-training evaluation report comment data, generated from 2012 to 2015 by 4 large North American Internal Medicine training programs. The Linguistic Index and Word Count (LIWC) was used to categorize and quantify the language contained. Generalizability theory was used to determine whether faculty could be reliably discriminated from one another based on writing style. Correlations and ANOVAs were used to determine what styles were related to faculty or trainee demographics. RESULTS Datasets contained 23-142 faculty who provided 549-2,666 assessments on 161-989 trainees. Faculty could easily be discriminated from one another using a variety of LIWC metrics including word count, words per sentence, and the use of "clout" words. These patterns appeared person specific and did not reflect demographic factors such as gender or rank. These metrics were similarly not consistently associated with trainee factors such as postgraduate year or gender. CONCLUSIONS Faculty seem to have detectable writing styles that are relatively stable across the trainees they assess, which may represent an under-recognized source of construct irrelevance. If written comments are to meaningfully contribute to decision making, we need to understand and account for idiosyncratic writing styles.

PURPOSE Physician shortages and maldistribution, particularly within family medicine, have led many medical schools worldwide to create regional medical campuses (RMCs) for clerkship training. However, Canadian medical schools have developed a number of RMCs in which all years of training (i.e., a combined model that includes both preclerkship and clinical training) are provided geographically separate from the main campus. This study addresses the question: Are combined model RMC graduates more likely to enter postgraduate training in family medicine and rural-focused programs relative to main campus graduates? METHOD The authors used a quasi-experimental research design and analyzed 2006-2016 data from the Canadian Resident Matching Service (CaRMS). Graduating students (N = 26,525) from 16 Canadian medical schools who applied for the CaRMS match in their year of medical school graduation were eligible for inclusion. The proportions of graduates who matched to postgraduate training in (1) family medicine and (2) rural-focused programs were compared for combined model RMCs and main campuses. RESULTS Of RMC graduates, 48.4% matched to family medicine (95% confidence interval [CI] = 46.1-50.7) compared with 37.1% of main campus graduates (95% CI = 36.5-37.7; P < .001). Of RMC graduates, 23.9% matched to rural-focused training programs (95% CI = 21.8-25.9) compared with 10.4% of main campus graduates (95% CI = 10.0-10.8; P < .001). Subanalyses ruled out a variety of potentially confounding variables. CONCLUSIONS Combined model RMCs, in which all years of training take place away from the medical school's main campus, are associated with greater proportions of medical students entering family medicine postgraduate training and rural-focused training programs. These findings should encourage policymakers, health services agencies, and medical schools to continue seeking complements to academic medical center-based medical education.

Near-peer interactions (NPIs) provide formal and informal mentorship that can allow medical students to share strategies for successful training. Such capacity to convey valuable advice, however, m...

Background: Exploring how motivations for conference attendance can change over the course of one’s career might help organisers better cater to the diverse range of conference attendees Aims and Objectives: To explore if and how motivations for conference attendance have varied across the timeline of professional careers. Methods: Semi-structured interviews were conducted with 13 European Respiratory Society stakeholders who were experts in their field and frequent conference attenders. Transcripts were coded using inductive thematic analysis. Results: Preliminary analyses have identified three themes of relevance to this study’s aim: (1) Early career ERS members seek knowledge gain and career inspiration from experts while requiring guidance about how to integrate into the community; (2) Mid career members attend to advance their reputation and career progression through active participation in the conference program; (3) Late career members shift their conference focus to, and draw pleasure from, helping others (e.g., PhD students) Conclusion: Motivations for conference attendance change over the course of one’s career. It may, therefore, be helpful to tailor sessions to different stages of individuals’ careers. Future research is needed to investigate what other variables contribute to variability in motivations to optimise conference utility for all.

INTRODUCTION Effective continuing professional development (CPD) is critical for safe and effective health care. Recent shifts have called for a move away from didactic CPD, which often fails to affect practice, toward workplace learning such as clinical coaching. Unfortunately, coaching programs are complex, and adoption does not guarantee effectiveness. To resolve this problem, thus ensuring resources are well spent, there is a critical need to understand what physicians try to achieve and how they engage. Therefore, we examined the types of change physicians pursue through clinical coaching and the impact of context on their desired changes. METHODS In the context of two clinical coaching programs for rural physicians, we applied a generic qualitative approach. Coachees (N = 15) participated in semistructured interviews. Analysis involved iterative cycles of initial, focused, and theoretical coding. RESULTS Coachees articulated desired practice changes along a spectrum, ranging from honing their current practice to making larger changes that involved new skills outside their current practice; changes also ranged from those focused on individual physicians to those focused on the practice system. Desired changes were affected by factors in the learning/practice environment, including those related to the individual coachee, coach, and learning/practice context. DISCUSSION These results suggest that the current focus on acquiring new knowledge through CPD may miss important learning that involves subtle shifts in practice as well as learning that focusses on systems change. Moreover, an appreciation of the contextual nature of CPD can ensure that contextual affordances are leveraged and barriers are acknowledged.

Background Video review processes for evaluation and coaching are often incorporated into medical education as a means to accurately capture physician-patient interactions. Compared to direct observation they offer the advantage of overcoming many logistical challenges. However, the suitability and viability of using video-based peer consultations for professional development requires further investigation. This study aims to explore the acceptability and feasibility of video-based peer feedback to support professional development and quality improvement in patient care. Methods Five rheumatologists each provided four videos of patient consultations. Peers evaluated the videos using five-point scales, providing annotations in the video recordings, and offering recommendations. The rheumatologists reviewed the videos of their own four patient interactions along with the feedback. They were asked to document if they would make practice changes based on the feedback. Focus groups were conducted and analysed to explore the effectiveness of video-based peer feedback in assisting physicians to improve clinical practice. Results Participants felt the video-based feedback provided accurate and detailed information in a more convenient, less intrusive manner than direct observation. Observations made through video review enabled participants to evaluate more detailed information than a chart review alone. Participants believed that reviewing recorded consultations allowed them to reflect on their practice and gain insight into alternative communication methods. Conclusions Video-based peer feedback and self-review of clinical performance is an acceptable and pragmatic approach to support professional development and improve clinical care among peer clinicians. Further investigation into the effectiveness of this approach is needed.

The authors describe influences associated with the incorporation of modern technologies into medical school admissions processes. Their purpose is not to critique or support specific technologies but, rather, to prompt reflection on the evolution that is afoot. Technology is now integral to the administration of multiple admissions tools, including the Medical College Admission Test, situational judgment tests, and standardized video interviews. Consequently, today's admissions landscape is transforming into an online, globally interconnected marketplace for health professions admissions tools. Academic capitalism and distance-based technologies combine to enable global marketing and dissemination of admissions tests beyond the national jurisdictions in which they are designed. As predicted by disruptive business theory, they are becoming key drivers of transformative change. The seeds of technological disruption are present now rather than something to be wary of in the future. The authors reflect on this transformation and the need for tailoring test modifications to address issues of medical student diversity and social responsibility. They comment on the online assessment of applicants' personal competencies and the potential detriments if this method were to replace admissions methods involving human contact, thanks to the ease with which institutions can implement them without cost to themselves and without adequate consideration of measurement utility or contextual appropriateness. The authors advocate for socially responsible academic capitalism within this interconnected admissions marketplace: Attending to today's transformative challenges may inform how health professions education responds to tomorrow's admissions technologies and, in turn, how tomorrow's health professionals respond to their patients' needs.

Background Audit and feedback (A&F) interventions are one of the most common approaches for implementing evidence-based practices. A key barrier to more effective A&F interventions is the lack of a theory-guided approach to the accumulation of evidence. Recent interviews with theory experts identified 313 theory-informed hypotheses, spread across 30 themes, about how to create more effective A&F interventions. In the current survey, we sought to elicit from stakeholders which hypotheses were most likely to advance the field if studied further. Methods From the list of 313, three members of the research team identified 216 that were clear and distinguishable enough for prioritization. A web-based survey was then sent to 211 A&F intervention stakeholders asking them to choose up to 50 ‘priority’ hypotheses following the header “A&F interventions will be more effective if…”. Analyses included frequencies of endorsement of the individual hypotheses and themes into which they were grouped. Results 68 of the 211 invited participants responded to the survey. Seven hypotheses were chosen by > 50% of respondents, including A&F interventions will be more effective… “if feedback is provided by a trusted source”; “if recipients are involved in the design/development of the feedback intervention”; “if recommendations related to the feedback are based on good quality evidence”; “if the behaviour is under the control of the recipient”; “if it addresses barriers and facilitators (drivers) to behaviour change”; “if it suggests clear action plans”; and “if target/goal/optimal rates are clear and explicit”. The most endorsed theme was Recipient Priorities (four hypotheses were chosen 92 times as a ‘priority’ hypotheses). Conclusions This work determined a set of hypotheses thought by respondents to be to be most likely to advance the field through future A&F intervention research. This work can inform a coordinated research agenda that may more efficiently lead to more effective A&F interventions.

Want to know how COVID‐19 changes health professional education research priorities? Our Editor‐in‐chief argues that in the most important ways it shouldn't.

Introduction Using assessment to facilitate learning is a well-established priority in education but has been associated with variable effectiveness for continuing professional development. What factors modulate the impact of testing in practitioners are unclear. We aimed to improve capacity to support maintenance of competence by exploring variables that influence the value of web-based pretesting. Methods Family physicians belonging to a practice-based learning program studied two educational modules independently or in small groups. Before learning sessions they completed a needs assessment and were assigned to either sit a pretest intervention or read a relevant review article. After the learning session, they completed an outcome test, indicated plans to change practice, and subsequently documented changes made. Results One hundred twelve physicians completed the study, 92 in small groups. The average lag between tests was 6.3 weeks. Relative to those given a review article, physicians given a pretest intervention: (1) reported spending less time completing the assigned task (16.7 versus 25.7 minutes); (2) performed better on outcome test questions that were repeated from the pretest (65.9% versus 58.7%); and (3) when the learning module was completed independently, reported making a greater proportion of practice changes to which they committed (80.0% versus 45.0%). Knowledge gain was unrelated to physicians' stated needs. Discussion Low-stakes formative quizzes, delivered with feedback, can influence the amount of material practicing physicians remember from an educational intervention independent of perceptions regarding the need to engage in continuing professional development on the particular topic.

Purpose Accreditation aims to ensure all training programs meet agreed-upon standards of quality. The process is complex, resource intensive, and costly. Its benefits are difficult to assess because contextual confounds obscure comparisons between systems that do and do not include accreditation. This study explores accreditation's influence "within system" by investigating the relationship between accreditation cycle and performance on a national licensing examination. Method Scores on the computer-based portion of the Medical Council of Canada Qualifying Examination Part I, from 1993 to 2017, were examined for all 17 Canadian medical schools. Typically completed upon graduation from medical school, results within each year were transformed for comparability across administrations and linked to timing within each school's accreditation cycle. ANOVAs were used to assess the relationship between accreditation timing and examination scores. Secondary analyses isolated 4-year from 3-year training programs and separated data generated before versus after implementation of a national midcycle informal review program. Results Performance on the licensing exam was highest during and shortly after an accreditation site visit, falling significantly until the midpoint in the accreditation cycle (d = 0.47) before rising again. This pattern disappeared after introduction of informal interim review, but too little data have accumulated post implementation to determine if interim review is sufficient to break the influence of accreditation cycle. Conclusions Formal, externally driven, accreditation cycles appear associated with educational processes in ways that translated into student outcomes on a national licensing examination. Whether informal, internal, interim reviews can mediate this effect remains to be seen.

Medical education is dependent on clinicians and other faculty who volunteer time and expertise to teaching. Unfortunately, the literature reports increasing levels of dissatisfaction, burnout, and attrition. Incentivization provides an obvious intervention, but rewards must be implemented judiciously or risk unintended consequences. With little known about the effects of incentives in medical education, the authors investigate key insights across three disciplines to explain how, why, and when incentives can be used effectively.In this critical synthesis, a purposeful and iterative literature search was conducted by exploring a variety of databases to identify seminal articles, key concepts, and generative search terms. Particularly fruitful disciplines were then explored more deliberately.Psychologists argue that the impact of an incentive depends on an individual's motivational drives. Organizational behaviorists draw attention to environmental incentives and disincentives that build or detract from motivation. Behavioral economists posit that size, type, and way in which an incentive is provided affect motivation differently.The influence of an incentive depends on how it interacts with underlying mechanisms deemed important for motivation. These mechanisms change across tasks, individuals, and contexts. Recommendations derived from the effort include being deliberate about (1) determining what is driving the individual to act, (2) considering the unique interactions between incentives and motivation types, and (3) considering barriers that may interfere with incentive effectiveness. In examining each of these, the authors argue that the field needs greater clarity regarding how, when, and why incentives operate within the many contexts in which medical educators work.

Selection and recruitment into healthcare education and practice is a key area of interest for educators with significant developments in research, policy, and practice in recent years. This updated consensus statement, developed through a multi-stage process, examines future opportunities and challenges in selection and recruitment. There is both a gap in the literature around and a compelling case for further theoretical and empirical literature to underpin the development of overall selection philosophes and policies and their enactment. More consistent evidence has emerged regarding the quality of different selection methods. Approaches to selection are context-dependent, requiring the consideration of an institution's philosophy regarding what they are trying to achieve, the communities it purports to serve, along with the system within which they are used. Diversity and globalization issues continue to be critically important topics. Further research is required to explore differential attainment and explain why there are substantial differences in culturally acceptable ways of approaching diversity and widening access. More sophisticated evaluation approaches using multi-disciplinary theoretical frameworks are required to address the issues. Following a discussion of these areas, 10 recommendations are presented to guide future research and practice and to encourage debate between colleagues across the globe.

The assessment of healthcare professionals is a critical determinant of patient safety and healthcare quality, playing both a gatekeeping function and a path toward performance improvement. Given the complexity of physician–patient interactions and the inadequacy of self-assessment for judging one's own strengths and weaknesses, medicine exemplifies a domain in which adequate assessment is dependent on the perceptions and perspectives of observers. Such perspectives are susceptible to influences that range well beyond the performance itself, offering an opportunity to consider how cognitive psychology can guide improved practices and how examination of psychological processes in real-world environments can inform thinking about cognition. In this target article I will provide an overview of the challenges facing health professional educators, the insights that have been gained from the application of cognitive psychology toward deriving solutions, and some reflections on the current state of the science and ongoing needs. These efforts lead to the conclusion that performance assessment protocols stand a better chance of success when they are designed to accommodate limitations of attentional capacity, working memory, and the idiosyncratic influence of prior experience rather than striving to change these fundamental aspects of human cognition through rater training or response form alterations.

In workplace-based assessment, research has suggested that aligning rating scales with how clinical supervisors naturally conceptualize trainee performance improves reliability and makes assessment more efficient. This study examined the generalizability of those findings by determining if construct alignment improves the reliability of competency-focused program evaluation ratings provided by 135 Year 3 medical students and 185 faculty supervisors. Participants were randomly assigned to complete the Readiness for Clerkship survey using one of two construct-aligned rating scales or a competence-focused rating scale. Mean differences, Spearman correlations, and Generalizability analyses compared the scales’ utility. Relative to the competence-oriented scale, the construct-aligned scales showed greater consistency in the rank order of items between students and faculty. Eight student raters were needed to achieve G=0.8 using the independence scale, 12 were needed using the behavioral/independence scale, and 22 were needed using the competence-oriented scale. Faculty were less consistent in their ratings and required more raters (k=25-32), regardless of the scale used. These results extend previous research into the benefits of construct-aligned scales by suggesting that students offer better ratings when the scales used to collect their judgments are aligned with the constructs of independence and sophistication rather than being phrased in terms of competence.

Health professionals are expected to routinely assess their weaknesses, set learning goals, and monitor their achievement. Unfortunately, it is well known that these professionals often struggle with effectively integrating external data and self-perceptions. To know how best to intervene, it is critical that the health professionals community understand the cues students and practitioners use to assess their abilities. Here the authors aimed to gain insights into how and why medical students set learning goals, monitor their progress, and demonstrate their learning.In 2012, the authors conducted semistructured interviews with Year 2 students (n = 20), applying an inductive approach to data analysis by iteratively developing, refining, and testing coding structures.Themes were constructed through discussion and consensus: (1) Students were diverse in how they set learning goals, (2) they used a range of approaches to monitor their progress, and (3) they struggled to balance studying for exams with preparation for clinical training. Tensions observed highlight assumptions embedded in medical curricula that can be problematic.Educators often treat medical students as a cohesive whole, thereby creating a mismatch between assessments that are intended to be formative and information students use to monitor their progress. Despite limited exposure to clinical contexts, goal generation and monitoring often stem from a desire to prepare for clinical practice. In grappling with these tensions, it is important to be mindful that students are individualistic in how they balance their commitment to prepare for clinical work and the need to concentrate on exams.

Item does not contain fulltext Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.

Background Audit and feedback (A&F) is a common strategy for helping health providers to implement evidence into practice. Despite being extensively studied, health care A&F interventions remain variably effective, with overall effect sizes that have not improved since 2003. Contributing to this stagnation is the fact that most health care A&F interventions have largely been designed without being informed by theoretical understanding from the behavioral and social sciences. To determine if the trend can be improved, the objective of this study was to develop a list of testable, theory-informed hypotheses about how to design more effective A&F interventions. Methods Using purposive sampling, semi-structured 60–90-min telephone interviews were conducted with experts in theories related to A&F from a range of fields (e.g., cognitive, health and organizational psychology, medical decision-making, economics). Guided by detailed descriptions of A&F interventions from the health care literature, interviewees described how they would approach the problem of designing improved A&F interventions. Specific, theory-informed hypotheses about the conditions for effective design and delivery of A&F interventions were elicited from the interviews. The resulting hypotheses were assigned by three coders working independently into themes, and categories of themes, in an iterative process. Results We conducted 28 interviews and identified 313 theory-informed hypotheses, which were placed into 30 themes. The 30 themes included hypotheses related to the following five categories: A&F recipient (seven themes), content of the A&F (ten themes), process of delivery of the A&F (six themes), behavior that was the focus of the A&F (three themes), and other (four themes). Conclusions We have identified a set of testable, theory-informed hypotheses from a broad range of behavioral and social science that suggest conditions for more effective A&F interventions. This work demonstrates the breadth of perspectives about A&F from non-healthcare-specific disciplines in a way that yields testable hypotheses for healthcare A&F interventions. These results will serve as the foundation for further work seeking to set research priorities among the A&F research community. Electronic supplementary material The online version of this article (10.1186/s13012-017-0646-0) contains supplementary material, which is available to authorized users.

Item does not contain fulltext CONTEXT: Interest is growing in the use of qualitative data for assessment. Written comments on residents' in-training evaluation reports (ITERs) can be reliably rank-ordered by faculty attendings, who are adept at interpreting these narratives. However, if residents do not interpret assessment comments in the same way, a valuable educational opportunity may be lost. OBJECTIVES: Our purpose was to explore residents' interpretations of written assessment comments using mixed methods. METHODS: Twelve internal medicine (IM) postgraduate year 2 (PGY2) residents were asked to rank-order a set of anonymised PGY1 residents (n = 48) from a previous year in IM based solely on their ITER comments. Each PGY1 was ranked by four PGY2s; generalisability theory was used to assess inter-rater reliability. The PGY2s were then interviewed separately about their rank-ordering process, how they made sense of the comments and how they viewed ITERs in general. Interviews were analysed using constructivist grounded theory. RESULTS: Across four PGY2 residents, the G coefficient was 0.84; for a single resident it was 0.56. Resident rankings correlated extremely well with faculty member rankings (r = 0.90). Residents were equally adept at reading between the lines to construct meaning from the comments and used language cues in ways similarly reported in faculty attendings. Participants discussed the difficulties of interpreting vague language and provided perspectives on why they thought it occurs (time, discomfort, memorability and the permanency of written records). They emphasised the importance of face-to-face discussions, the relative value of comments over scores, staff-dependent variability of assessment and the perceived purpose and value of ITERs. They saw particular value in opportunities to review an aggregated set of comments. CONCLUSIONS: Residents understood the 'hidden code' in assessment language and their ability to rank-order residents based on comments matched that of faculty. Residents seemed to accept staff-dependent variability as a reality. These findings add to the growing evidence that supports the use of narrative comments and subjectivity in assessment.