Search

Background: The widespread implementation of longitudinal assessment (LA) to document trainees' progression to independent practice rests more on speculative rather than evidence-based benefits. We aimed to document stakeholders' knowledge of- and attitudes towards LA, and identify how the supports and barriers can help or hinder the uptake and sustainable use of LA. Methods: We interviewed representatives from four stakeholder groups involved in LA. The interview protocols were based on the Theoretical Domains Framework (TDF), which contains a total of 14 behaviour change determinants. Two team members coded the interviews deductively to the TDF, with a third resolving differences in coding. The qualitative data analysis was completed with iterative consultations and discussions with team members until consensus was achieved. Saliency analysis was used to identify dominant domains. Results: Forty-one individuals participated in the study. Three dominant domains were identified. Participants perceive that LA has more positive than negative consequences and requires substantial resources. All the elements and characteristics of LA are present in our data, with differences between stakeholders. Conclusion: Going forward, we could develop and implement tailored and theory driven interventions to promote a shared understanding of LA, and maintain potential positive outcomes while reducing negative ones. Furthermore, resources to support LA implementation need to be addressed to facilitate its uptake.

The COVID-19 pandemic provoked an urgency for many educators to integrate digital information and communication technologies in their educational practices. We explored how faculty members tackled the task of adapting their assessment practices during the pandemic to identify what is required to sustain and favour future quality development and implementation of e-assessment in higher education. Employing a qualitative descriptive approach, we conducted semi-structured interviews with thirty-one individuals six months into the COVID-19 pandemic. We identified four major themes in participants' discourse about the integration of e-assessment during the COVID-19 pandemic: (a) the considerations they had for the potential consequences on students and how they considered this while deciding how to move forward, (b) the preoccupations for the potential for cheating, (c) the importance of pedagogical alignment, and (d) the affordances available to them. While the COVID-19 pandemic highlighted the fact that higher education institutions were not prepared for a pivot to- or greater integration of- e-assessment, it also provided the tipping-point to do so. In other words, it offered an unprecedented opportunity to critically appraise and change assessment practices, this opportunity was also a very challenging balancing act of considering the social consequences of assessment, and the alignment within set affordances.

Assessment is more educationally effective when learners engage with assessment processes and perceive the feedback received as credible. With the goal of optimizing the educational value of assessment in medical education, we mapped the primary literature to identify factors that may affect a learner's perceptions of the credibility of assessment and assessment-generated feedback (i.e., scores or narrative comments). For this scoping review, search strategies were developed and executed in five databases. Eligible articles were primary research studies with medical learners (i.e., medical students to post-graduate fellows) as the focal population, discussed assessment of individual learners, and reported on perceived credibility in the context of assessment or assessment-generated feedback. We identified 4705 articles published between 2000 and November 16, 2020. Abstracts were screened by two reviewers; disagreements were adjudicated by a third reviewer. Full-text review resulted in 80 articles included in this synthesis. We identified three sets of intertwined factors that affect learners' perceived credibility of assessment and assessment-generated feedback: (i) elements of an assessment process, (ii) learners' level of training, and (iii) context of medical education. Medical learners make judgments regarding the credibility of assessments and assessment-generated feedback, which are influenced by a variety of individual, process, and contextual factors. Judgments of credibility appear to influence what information will or will not be used to improve later performance. For assessment to be educationally valuable, design and use of assessment-generated feedback should consider how learners interpret, use, or discount assessment-generated feedback.

Validity is one of the most debated constructs in our field; debates abound about what is legitimate and what is not, and the word continues to be used in ways that are explicitly disavowed by current practice guidelines. The resultant tensions have not been well characterized, yet their existence suggests that different uses may maintain some value for the user that needs to be better understood. We conducted an empirical form of Discourse Analysis to document the multiple ways in which validity is described, understood, and used in the health professions education field. We created and analyzed an archive of texts identified from multiple sources, including formal databases such as PubMED, ERIC and PsycINFO as well as the authors' personal assessment libraries. An iterative analytic process was used to identify, discuss, and characterize emerging discourses about validity. Three discourses of validity were identified. Validity as a "test characteristic" is underpinned by the notion that validity is an intrinsic property of a tool and could, therefore, be seen as content and context independent. Validity as an argument-based "evidentiary-chain" emphasizes the importance of supporting the interpretation of assessment results with ongoing analysis such that validity does not belong to the tool/instrument itself. The emphasis is on process-based validation (emphasizing the journey instead of the goal). Validity as a "social imperative" foregrounds the consequences of assessment at the individual and societal levels, be they positive or negative. The existence of different discourses may explain--in part--results observed in recent systematic reviews that highlighted discrepancies and tensions between recommendations for practice and the validation practices that are actually adopted and reported. Some of these practices, despite contravening accepted validation "guidelines", may nevertheless respond to different and somewhat unarticulated needs within health professional education.

Supplemental Digital Content is Available in the Text. Introduction: Contextual factors can influence healthcare professionals' (HCPs) competencies, yet there is a scarcity of research on how to optimally measure these factors. The aim of this study was to develop and validate a comprehensive tool for HCPs to document the contextual factors likely to influence the maintenance, development, and deployment of professional competencies. Methods: We used DeVellis' 8-step process for scale development and Messick's unified theory of validity to inform the development and validation of the context tool. Building on results from a scoping review, we generated an item pool of contextual factors articulated around five themes: Leadership and Agency, Values, Policies, Supports, and Demands. A first version of the tool was pilot tested with 127 HCPs and analyzed using the classical test theory. A second version was tested on a larger sample (n = 581) and analyzed using the Rasch rating scale model. Results: First version of the tool: we piloted 117 items that were grouped as per the themes related to contextual factors and rated on a 5-point Likert scale. Cronbach alpha for the set of 12 retained items per scale ranged from 0.75 to 0.94. Second version of the tool included 60 items: Rasch analysis showed that four of the five scales (ie, Leadership and Agency, Values, Policies, Supports) can be used as unidimensional scales, whereas the fifth scale (Demands) had to be split into two unidimensional scales (Demands and Overdemands). Discussion: Validity evidence documented for content and internal structure is encouraging and supports the use of the McGill context tool. Future research will provide additional validity evidence and cross-cultural translation. [ABSTRACT FROM AUTHOR]

Since cognitive abilities have been shown to decrease with age, it is expected that older physicians would not perform as well as their younger counterparts on clinical cases unless their expertise can counteract the cognitive effects of aging. However, studies on the topic have shown contradictory results. This study aimed to further investigate the effect of aging on physicians' diagnostic accuracy when diagnosing prevalent and less prevalent cases based on clinical vignettes. A mixed design was used to assess the influence of case prevalence (high vs. low) as a within-subjects factor, and age group as a between subjects factor (<30; n = 23, 30-39; n = 19, 40-49; n = 27, >50 years old; n = 19) on the diagnostic accuracy of 65 family physicians and 25 residents. Repeated Measure ANOVA revealed a significant effect of case prevalence (p < 0.001) and age group (p < 0.001). Post-hoc analyses revealed that younger physicians showed the best performance. This study did not demonstrate the positive effect of experience in older physicians. In line with previous studies on expertise development, findings of the present study suggest that skills should be actively maintained to assure a high performance level throughout one's lifespan. If not, performance level could gradually decline with age.

Performance-based assessment (PBA) is a valued assessment approach in medical education, be it in a clerkship, residency, or practice context. Raters are intrinsic to PBA and the increased use of PBA has lead to an increased interest in rater cognition. Although several researchers have tackled factors that may influence the variability in rater judgment, the critical examination of rater observation of performance and the translation of that data into judgements are being investigated. The purpose of this study was to qualitatively investigate the cognitive processes of raters, and to create a framework that conceptualizes those processes when raters assess a complex performance. We conducted semi-structured interviews with 11 faculty members (nominated as excellent assessors) from a Department of Medicine to investigate how raters observe, interpret, and translate performance into judgments. The transcribed verbal protocols were analyzed using Constructivist Grounded Theory in order to develop a theoretical model of raters' assessment processes. Several themes emerged from the data and were grouped according to three macro-level themes describing how the raters balance two sources of data [(1) external sources of information and (2) internal/personal sources of information] by relying on specific cognitive processes to assess an examinee performance. The results from our study demonstrate that assessment is a difficult cognitive task that involves nuance using specific cognitive processes to weigh external and internal data against each other. Our data clearly draws attention to the constant struggle between objectivity and subjectivity that is observed in assessment as illustrated by the importance given to nuancing the examinee's observed performance.

Educational strategies that promote the development of clinical reasoning in students remain scarce. Generating self-explanations (SE) engages students in active learning and has shown to be an effective technique to improve clinical reasoning in clerks. Example-based learning has been shown to support the development of accurate knowledge representations. The purpose of this study was to investigate the effect of combining student's SE and observation of peer's or expert's SE examples on diagnostic performance. Fifty-three third-year medical students were assigned to a peer SE example, an expert SE example or control (no example) group. All participants solved a set of the same four clinical cases (training cases), 1--after SE, 2--after listening to a peer or expert SE example or after a control task, and 3--1 week later. They solved a new set of four different cases (transfer cases) also 1 week later. For training cases, students improved significantly their diagnostic performance overtime but the main effect of group was not significant suggesting that students' SE mainly drives the observed effect. On transfer cases, there was no difference between the three groups (p > 0.05). Educational implications are discussed and further studies on different types of examples and additional strategies to help students actively process examples are proposed.

Testing has been shown to enhance retention of learned information beyond simple studying, a phenomena known as test-enhanced learning (TEL). Research has shown that TEL effects are greater for tests that require the production of responses [e.g., short-answer questions (SAQs)] relative to tests that require the recognition of correct answers [e.g., multiple-choice questions (MCQs)]. High stakes licensure examinations have recently differentiated MCQs that require the application of clinical knowledge (context-rich MCQs) from MCQs that rely on the recognition of "facts" (context-free MCQs). The present study investigated the influence of different types of educational activities (including studying, SAQs, context-rich MCQs and context-free MCQs) on later performance on a mock licensure examination. Fourth-year medical students (n = 224) from four Quebec universities completed four educational activities: one reading-based activity and three quiz-based activities (SAQs, context-rich MCQs, and context-free MCQs). We assessed the influence of the type of educational activity on students' subsequent performance in a mock licensure examination, which consisted of two types of context-rich MCQs: (1) verbatim replications of previous items and (2) items that tested the same learning objective but were new. Mean accuracy scores on the mock licensure exam were higher when intervening educational activities contained either context-rich MCQs (Mean z-score = 0.40) or SAQs (M = 0.39) compared to context-free MCQs (M = -0.38) or study only items (M = -0.42; all p < 0.001). Higher mean scores were only present for verbatim items (p < 0.001). The benefit of testing was observed when intervening educational activities required either the generation of a response (SAQs) or the application of knowledge (context-rich MCQs); however, this effect was only observed for verbatim test items. These data provide evidence that context-rich MCQs and SAQs enhance learning through testing compared to context-free MCQs or studying alone. The extent to which these findings generalize beyond verbatim questions remains to be seen.

Supplemental Digital Content is Available in the Text. Introduction: Health care professionals work in different contexts, which can influence professional competencies. Despite existing literature on the impact of context on practice, the nature and influence of contextual characteristics, and how context is defined and measured, remain poorly understood. The aim of this study was to map the breadth and depth of the literature on how context is defined and measured and the contextual characteristics that may influence professional competencies. Methods: A scoping review using Arksey and O'Malley's framework. We searched MEDLINE (Ovid) and CINAHL (EBSCO). Our inclusion criteria were studies that reported on context or relationships between contextual characteristics and professional competencies or that measured context. We extracted data on context definitions, context measures and their psychometric properties, and contextual characteristics influencing professional competencies. We performed numerical and qualitative analyses. Results: After duplicate removal, 9106 citations were screened and 283 were retained. We compiled a list of 67 context definitions and 112 available measures, with or without psychometric properties. We identified 60 contextual factors and organized them into five themes: Leadership and Agency, Values, Policies, Supports, and Demands. Discussion: Context is a complex construct that covers a wide array of dimensions. Measures are available, but none include the five dimensions in one single measure or focus on items targeting the likelihood of context influencing several competencies. Given that the practice context plays a critical role in health care professionals' competencies, stakeholders from all sectors (education, practice, and policy) should work together to address those contextual characteristics that can adversely influence practice. [ABSTRACT FROM AUTHOR]