Search

Clinical supervisors are known to assess trainee performance idiosyncratically, causing concern about the validity of their ratings. The literature on this issue relies heavily on retrospective collection of decisions, resulting in the risk of inaccurate information regarding what actually drives raters' perceptions. Capturing in-the-moment information about supervisors' impressions could yield better insight into how to intervene. The purpose of this study, therefore, was to gather "real-time" judgments to explore what drives preceptors' judgments of student performance. We performed a prospective study in which physicians were asked to adjust a rating scale in real-time while watching two video-recordings of trainee clinical performances. Scores were captured in 1-s increments, examined for frequency, direction, and magnitude of adjustments, and compared to assessors' final entrustability judgment as measured by the modified Ottawa Clinic Assessment Tool. The standard deviation in raters' judgment was examined as a function of time to determine how long it takes impressions to begin to vary. 20 participants viewed 2 clinical vignettes. Considerable variability in ratings was observed with different behaviours triggering scale adjustments for different raters. That idiosyncrasy occurred very quickly, with the standard deviation in raters' judgments rapidly increasing within 30 s of case onset. Particular moments appeared to generally be influential, but their degree of influence still varied. Correlations between the final assessment and (a) score assigned upon first adjustment of the scale, (b) upon last adjustment, and (c) the mean score, were r = 0.13, 0.32, and 0.57 for one video and r = 0.30, 0.50, and 0.52 for the other, indicating the degree to which overall impressions reflected accumulation of raters' idiosyncratic moment-by-moment observations. Our results demonstrated that variability in raters' impressions begins very early in a case presentation and is associated with different behaviours having different influence on different raters. More generally, this study outlines a novel methodology that offers a new path for gaining insight into factors influencing assessor judgments.

Safe and effective healthcare requires that new knowledge or skills, once learned, are incorporated into professional practice. However, this process is not always straightforward. Learning takes place in complex contexts, requiring practitioners to overcome various motivational, systemic, emotional, and social barriers to the application of knowledge. This paper explores the mechanisms through which individuals translate knowledge into action to provide insight into why disconnects between knowledge and action can arise. As a critical review, the aim was to draw on key literature from multiple fields to analyse and synthesize existing schools of thought and lay a strong conceptual foundation on which knowledge to action gaps might be considered. We iteratively consulted clinicians and experts in various fields to guide literature searches focused on theoretical perspectives that could inform educational and research efforts around knowledge-to-action gaps. Key theoretical perspectives on motivation address when and how individuals decide to take action. Literatures from cognitive science address how clinicians and learners self-regulate to (sometimes) overcome barriers to action. Sociocultural theories examine the ways in which action might be prevented by social norms that conflict with what the individual knows and believes, potentially also giving rise to counter-normative action. No single perspective will entirely explain how health professionals and learners implement knowledge in practice. As a result, the authors offer multiple lenses through which to view the problem, and then propose how each of these lenses might better guide educational and research efforts to untangle this challenging but important issue.

Learning to take safe and effective action in complex settings rife with uncertainty is essential for patient safety and quality care. Doing so is not easy for trainees, as they often consider certainty to be a necessary precursor for action and subsequently struggle in these settings. Understanding how skillful clinicians work comfortably when uncertain, therefore, offers an important opportunity to facilitate trainees' clinical reasoning development. This critical review aims to define and elaborate the concept of 'comfort with uncertainty' in clinical settings by juxtaposing a variety of frameworks and theories in ways that generate more deliberate ways of thinking about, and researching, this phenomenon. We used Google Scholar to identify theoretical concepts and findings relevant to the topics of 'uncertainty,' 'ambiguity,' 'comfort,' and 'confidence,' and then used preliminary findings to pursue parallel searches within the social cognition, cognition, sociology, sociocultural, philosophy of medicine, and medical education literatures. We treat uncertainty as representing the lived experience of individuals, reflecting the lack of confidence one feels that he/she has an incomplete mental representation of a particular problem. Comfort, in contrast, references confidence in one's capabilities to act (or not act) in a safe and effective manner given the situation. Clinicians' 'comfort with uncertainty' is informed by a variety of perceptual, emotional, and situational cues, and is enabled through a combination of self-monitoring and forward planning. Potential implications of using 'comfort with uncertainty' as a framework for educational and research programs are explored.

Many models of safe and effective clinical decision making in medical practice emphasize the importance of recognizing moments of uncertainty and seeking help accordingly. This is not always done effectively, but we know little about what cues prompt health professionals to call on resources beyond their own knowledge or skill set. Such information would offer guidance regarding how systems might be designed to offer better individual support. In this study, the authors explored the situational factors that are present during moments of uncertainty that lead primary care physicians to access external resources. To do so, a generic qualitative exploratory analysis was conducted on 72 narratives collected through audio recorder-based, self-observational, journaling completed by 12 purposively selected family physicians. Participants were asked to provide a detailed descriptive account of the circumstances surrounding their consultation of external resources immediately after 6 sequential patient encounters in which they felt compelled to seek such support. Thematic analysis of the transcripts was performed to better understand participants' experiences of the social, contextual, and personal features surrounding decisions to seek support. When doing so we observed that specific features of patient encounters were routinely present when physicians decided to access external sources for help. These included medical aspects of the case (e.g., complex presentations), social aspects (e.g., the presence of another individual), and personal factors (e.g., feeling a need for reassurance). External resources were seen as an opportunity for verification, a mechanism to increase patient satisfaction, and a means through which to defend decision-making. Accessing such resources appeared to influence the physician-patient relationship for various reasons. Recognition and further study of the cues that prompt use of external information will further our understanding of physicians' behavioural responses to challenging/uncertain situations, highlight mechanisms through which a culture of self-directed assessment seeking might be encouraged, and offer guidance regarding ways in which physicians can be encouraged to practice mindfully. Our results make it clear that reasons for which primary care physicians seek the support of external resources may be multifactorial and, therefore, one should be cautious when inferring reasons for the pursuit of such support.

ObjectiveReceptiveness to interprofessional feedback, which is important for optimal collaboration, may be influenced by 'in-group or out-group' categorisation, as suggested by social identity theory. We used an experimental design to explore how nurses and resident physicians perceive feedback from people within and outside their own professional group.MethodsPaediatric residents and nurses participated in a simulation-based team exercise. Two nurses and two physicians wrote anonymous performance feedback for each participant. Participants each received a survey containing these feedback comments with prompts to rate (i) the usefulness (ii) the positivity and (iii) their agreement with each comment. Half of the participants received feedback labelled with the feedback provider's profession (two comments correctly labelled and two incorrectly labelled). Half received unlabelled feedback and were asked to guess the provider's profession. For each group, we performed separate three-way anovas on usefulness, positivity and agreement ratings to examine interactions between the recipient's profession, actual provider profession and perceived provider profession.ResultsForty-five out of 50 participants completed the survey. There were no significant interactions between profession of the recipient and the actual profession of the feedback provider for any of the 3 variables. Among participants who guessed the source of the feedback, we found significant interactions between the profession of the feedback recipient and the guessed source of the feedback for both usefulness (F1,48 = 25.6; p < 0.001; η(2) = 0.35) and agreement ratings (F1,48 = 8.49; p < 0.01; η(2) = 0.15). Nurses' ratings of feedback they guessed to be from nurses were higher than ratings of feedback they guessed to be from physicians, and vice versa. Among participants who received labelled feedback, we noted a similar interaction between the profession of the feedback recipient and labelled source of feedback for usefulness ratings (F1,92 = 4.72; p < 0.05; η(2) = 0.05).ConclusionOur data suggest that physicians and nurses are more likely to attribute favourably perceived feedback to the in-group than to the out-group. This finding has potential implications for interprofessional feedback practices.

Information is generally more memorable after it is studied and tested than when it is only studied. One must be cautious to use this phenomenon strategically, however, due to uncertainty about whether testing improves memorability for only tested material, facilitates learning of related non-tested content, or inhibits memory of non-tested material. 52 second-year Pharmacy students were asked to study therapeutic aspects of gastroesophageal reflux disease and peptic ulcer disease. One group was given 30 min to study. Another was given 20 min to study and 10 min to complete a 10-item test. Two weeks later a 40-item test was delivered to both groups that contained (a) the 10 learning phase questions, (b) 10 new questions drawn from the studied material, (c) 10 new questions about therapeutics in different disease states, and (d) 10 new questions drawn from more general pharmaceutical knowledge (e.g., basic physiology and drug characteristics). Moderate to large retrieval-enhanced learning effects were observed for both questions about material that was tested (22.9% difference in scores, p < 0.05, d = 0.60) and questions about material that was studied without being tested (18.9% difference, p < 0.05, d = 0.75). Such effects were not observed for questions that were not part of the study material: therapeutic questions that addressed different disease states (1.8% difference, p > 0.7, d = 0.08) or generic pharmaceutical questions (7.4% difference, p > 0.2, d = 0.32). Being tested made it more likely that students would report reviewing the material after the initial learning session, but such reports were not associated with better test performance. The benefit of mentally retrieving information from studied material appears to facilitate the retrieval of information that was studied without being tested. Such generalization of the benefit of testing can increase the flexibility of test-based pedagogic interventions.

Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting "idiosyncratic rater variance" is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.

Validity is one of the most debated constructs in our field; debates abound about what is legitimate and what is not, and the word continues to be used in ways that are explicitly disavowed by current practice guidelines. The resultant tensions have not been well characterized, yet their existence suggests that different uses may maintain some value for the user that needs to be better understood. We conducted an empirical form of Discourse Analysis to document the multiple ways in which validity is described, understood, and used in the health professions education field. We created and analyzed an archive of texts identified from multiple sources, including formal databases such as PubMED, ERIC and PsycINFO as well as the authors' personal assessment libraries. An iterative analytic process was used to identify, discuss, and characterize emerging discourses about validity. Three discourses of validity were identified. Validity as a "test characteristic" is underpinned by the notion that validity is an intrinsic property of a tool and could, therefore, be seen as content and context independent. Validity as an argument-based "evidentiary-chain" emphasizes the importance of supporting the interpretation of assessment results with ongoing analysis such that validity does not belong to the tool/instrument itself. The emphasis is on process-based validation (emphasizing the journey instead of the goal). Validity as a "social imperative" foregrounds the consequences of assessment at the individual and societal levels, be they positive or negative. The existence of different discourses may explain--in part--results observed in recent systematic reviews that highlighted discrepancies and tensions between recommendations for practice and the validation practices that are actually adopted and reported. Some of these practices, despite contravening accepted validation "guidelines", may nevertheless respond to different and somewhat unarticulated needs within health professional education.

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

Educators often seek to demonstrate the equivalence of groups, such as whether or not students achieve comparable success regardless of the site at which they trained. A methodological consideration that is often underappreciated is how to operationalize equivalence. This study examined whether a distribution-based approach, based on effect size, can identify an appropriate equivalence threshold for medical education data. Thirty-nine individuals rated program site equivalence on a series of simulated pairwise bar graphs representing one of four measures with which they had prior experience: (1) undergraduate academic achievement, (2) a student experience survey, (3) an Objective Structured Clinical Exam global rating scale, or (4) a licensing exam. Descriptive statistics and repeated measures ANOVA examined the effects on equivalence ratings of (a) the difference between means, (b) variability in scores, and (c) which program site (the larger or smaller) scored higher. The equivalence threshold was defined as the point at which 50% of participants rated the sites as non-equivalent. Across the four measures, the equivalence thresholds converged to average effect size of Cohen's d = 0.57 (range of 0.50-0.63). This corresponded to an average mean difference of 10% (range of 3-13%). These results are discussed in reference to findings from the health-related quality of life field that has demonstrated that d = 0.50 represents a consistent threshold for perceived change. This study provides preliminary empirically-based guidance for defining an equivalence threshold for researchers and evaluators conducting equivalence tests.

Written comments on residents' evaluations can be useful, yet the literature suggests that the language used by assessors is often vague and indirect. The branch of linguistics called pragmatics argues that much of our day to day language is not meant to be interpreted literally. Within pragmatics, the theory of "politeness" suggests that non-literal language and other strategies are employed in order to "save face". We conducted a rigorous, in-depth analysis of a set of written in-training evaluation report (ITER) comments using Brown and Levinson's influential theory of 'politeness' to shed light on the phenomenon of vague language use in assessment. We coded text from 637 comment boxes from first year residents in internal medicine at one institution according to politeness theory. Non-literal language use was common and "hedging", a key politeness strategy, was pervasive in comments about both high and low rated residents, suggesting that faculty may be working to "save face" for themselves and their residents. Hedging and other politeness strategies are considered essential to smooth social functioning; their prevalence in our ITERs may reflect the difficult social context in which written assessments occur. This research raises questions regarding the "optimal" construction of written comments by faculty.

When assessing the likelihood of an event, human judgment is often inconsistent with the rules inherent in standard probability theory. For example, the judged probability of an event can be heavily influenced by the alternatives that are explicitly presented. Tversky and Koehier (1994) attempted to account for this phenomenon by arguing that probability judgments are made by comparing the amount of cognitive support one holds in favour of the event in question relative to all other possibilities. They suggested that different descriptions of the same event elicit different amounts of support resulting in different probability ratings. In addition to the role played by explicitly considered alternatives, the present paper suggests that people are also sensitive to the influence of alternatives that are not considered explicitly. We present the term "implied numerosity' in an attempt to indicate that probability ratings are influenced by a general impression of the number of potential alternatives that exist. Systematic differences in probability estimations may result from systematic changes in the perceived size of the category being evaluated.

Using heuristics offers several cognitive advantages, such as increased speed and reduced effort when making decisions, in addition to allowing us to make decision in situations where missing data do not allow for formal reasoning. But the traditional view of heuristics is that they trade accuracy for efficiency. Here the authors discuss sources of bias in the literature implicating the use of heuristics in diagnostic error and highlight the fact that there are also data suggesting that under certain circumstances using heuristics may lead to better decisions that formal analysis. They suggest that diagnostic error is frequently misattributed to the use of heuristics and propose an alternative view whereby content knowledge is the root cause of diagnostic performance and heuristics lie on the causal pathway between knowledge and diagnostic error or success.