Search

Background: Globalization results in a rapidly diversifying population, increased inequities, and more complex health problems affecting populations. This forces medical schools to integrate global health (GH) into the training of health-care professionals from curriculum development to practical learning activities, here and abroad. Approach: The approach aims at enriching existing programs in GH competencies in an interdisciplinary context. The goal is to ensure that all health-science students develop a certain level of GH competency. The main actions are the mobilization of key stakeholders, the development of a competency framework (CF) to perform gap analysis, tool formalization, and monitoring and evaluation activities. Subsequent to scoping review and stakeholder consultations, ten principles are identified and used to guide the enrichment process. Results: Actual outputs cover a broad scope, from key decision-makers’ support and endorsement to the formalization of tools and the consolidation and creation of activities such as service-learning activities, rotations among underserved populations, and training for international rotations. Conclusion: While this unique approach is proving to be a major challenge, the preliminary results are well worth the effort. The project’s tangible impacts on health-sciences teaching, the GH competence of graduates, and care delivery are topics of interest for future investigation.

Medical school admissions is a contentious and high stakes selection activity. Many assessment approaches are available to support selection; but how are decisions about building, monitoring, and adapting admissions systems made? What shapes the processes and practices that underpin selection decisions? We explore how these decisions are made across several Canadian medical schools, and how values shape the creation, monitoring, and adaptation of admissions systems. Using phenomenography (a qualitative method suited to examining variability), the authors analyzed interviews with 10 current or previous heads of admissions from 10 different undergraduate medical education programs in Canada. Interviews were conducted in English and French, and data was collected between 2016 and 2017 (therefore participants no longer hold these roles). Data was coded and analyzed iteratively, focusing on identifying underlying values, and exploring how these values shape admissions practices and considerations for validity. Eight different intersecting values were identified. Of these, four were shared across all participants: critically questioning the process and tools, aiming for equity, striving for better, and embracing the challenges of change. The expression of these values depended on different contextual variables (e.g., geographic location, access to expertise, resource availability), and values shaped how admissions systems were built, enacted, and monitored for quality. Ultimately, values shaped: (1) admissions practices resulting in different candidates being offered admission, and (2) how arguments supporting score interpretation are built (i.e., validity). This study documents various values that influence admissions processes, practices, and quality monitoring. The values that shape what is assessed, how it is assessed, and how fair and defensible practices are conceptualized have significant impact, ultimately determining who is selected. These values-whether implicit or explicit-result in intended and unintended consequences in selection processes. However, these values are rarely explicitly examined and questioned, leaving it uncertain as to which consequences are the intended outcomes of deliberately chosen values, and which are unintended consequences of implicitly held values of admissions systems and their actors., (© 2023. The Author(s), under exclusive licence to Springer Nature B.V.)

Introduction: Contextual factors can influence healthcare professionals' (HCPs) competencies, yet there is a scarcity of research on how to optimally measure these factors. The aim of this study was to develop and validate a comprehensive tool for HCPs to document the contextual factors likely to influence the maintenance, development, and deployment of professional competencies., Methods: We used DeVellis' 8-step process for scale development and Messick's unified theory of validity to inform the development and validation of the context tool. Building on results from a scoping review, we generated an item pool of contextual factors articulated around five themes: Leadership and Agency, Values, Policies, Supports, and Demands. A first version of the tool was pilot tested with 127 HCPs and analyzed using the classical test theory. A second version was tested on a larger sample (n = 581) and analyzed using the Rasch rating scale model., Results: First version of the tool: we piloted 117 items that were grouped as per the themes related to contextual factors and rated on a 5-point Likert scale. Cronbach alpha for the set of 12 retained items per scale ranged from 0.75 to 0.94. Second version of the tool included 60 items: Rasch analysis showed that four of the five scales (ie, Leadership and Agency, Values, Policies, Supports) can be used as unidimensional scales, whereas the fifth scale (Demands) had to be split into two unidimensional scales (Demands and Overdemands)., Discussion: Validity evidence documented for content and internal structure is encouraging and supports the use of the McGill context tool. Future research will provide additional validity evidence and cross-cultural translation., Competing Interests: The authors declare no conflict of interest., (Copyright © 2024 The Alliance for Continuing Education in the Health Professions, the Association for Hospital Medical Education, and the Society for Academic Continuing Medical Education.)

Background: While developing reflection skills is considered important by educators, the assessment of these skills is often associated with unintended negative consequences. In the context of a mandatory longitudinal course that aims to promote the development of reflection on professional identity, we assessed students' commitment to reflection. This study explores students' perception of this assessment by their mentor., Methods: We conducted a qualitative descriptive study using semi-structured interviews with twenty-one 1 st and six 2 nd year medical students. Thematic analysis was informed by Braun and Clarke's six-step approach., Results: We identified four main themes: 1- assessment as a motivator, 2- consequences on authenticity, 3- perception of inherent subjectivity, and 4 - relationship with the mentor., Conclusions: In the context of assessing reflection skills in future physicians, we observed that students -when assessed on the process of reflection- experienced high motivation but were ambivalent on the question of authenticity. The subjectivity of the assessment as well as the relationship with their mentor also raises questions. Nevertheless, this assessment approach for reflective skills appears to be promising in terms of limiting the negative consequences of assessment., Competing Interests: None, (© 2023 Poirier, Ouellet, Désilets, Graillon, Xhignesse, St-Onge; licensee Synergies Partners.)

Introduction: Health care professionals work in different contexts, which can influence professional competencies. Despite existing literature on the impact of context on practice, the nature and influence of contextual characteristics, and how context is defined and measured, remain poorly understood. The aim of this study was to map the breadth and depth of the literature on how context is defined and measured and the contextual characteristics that may influence professional competencies., Methods: A scoping review using Arksey and O'Malley's framework. We searched MEDLINE (Ovid) and CINAHL (EBSCO). Our inclusion criteria were studies that reported on context or relationships between contextual characteristics and professional competencies or that measured context. We extracted data on context definitions, context measures and their psychometric properties, and contextual characteristics influencing professional competencies. We performed numerical and qualitative analyses., Results: After duplicate removal, 9106 citations were screened and 283 were retained. We compiled a list of 67 context definitions and 112 available measures, with or without psychometric properties. We identified 60 contextual factors and organized them into five themes: Leadership and Agency, Values, Policies, Supports, and Demands., Discussion: Context is a complex construct that covers a wide array of dimensions. Measures are available, but none include the five dimensions in one single measure or focus on items targeting the likelihood of context influencing several competencies. Given that the practice context plays a critical role in health care professionals' competencies, stakeholders from all sectors (education, practice, and policy) should work together to address those contextual characteristics that can adversely influence practice., Competing Interests: The authors declare no conflict of interest., (Copyright © 2023 The Alliance for Continuing Education in the Health Professions, the Association for Hospital Medical Education, and the Society for Academic Continuing Medical Education.)

Background & Need for Innovation: Appraising the quality of narratives used in assessment is challenging for educators and administrators. Although some quality indicators for writing narratives exist in the literature, they remain context specific and not always sufficiently operational to be easily used. Creating a tool that gathers applicable quality indicators and ensuring its standardized use would equip assessors to appraise the quality of narratives., Steps Taken for Development and Implementation of Innovation: We used DeVellis' framework to develop a checklist of evidence-informed indicators for quality narratives. Two team members independently piloted the checklist using four series of narratives coming from three different sources. After each series, team members documented their agreement and achieved a consensus. We calculated frequencies of occurrence for each quality indicator as well as the interrater agreement to assess the standardized application of the checklist., Outcomes of Innovation: We identified seven quality indicators and applied them on narratives. Frequencies of quality indicators ranged from 0% to 100%. Interrater agreement ranged from 88.7% to 100% for the four series., Critical Reflection: Although we were able to achieve a standardized application of a list of quality indicators for narratives used in health sciences education, it does not exclude the fact that users would need training to be able to write good quality narratives. We also noted that some quality indicators were less frequent than others and we suggested a few reflections on this., Competing Interests: The authors have no competing interests to declare., (Copyright: © 2023 The Author(s).)

Decisions to set aside Structured Oral Examinations (SOE) are, almost invariably, based on their poor psychometric properties. However, considering the perspectives of the stakeholders might help us to understand its potential contribution. To explore this, we conducted focus groups and individual interviews with stakeholders: students, assessors, and administrators. Students and assessors perceived the SOE as a window on students' clinical reasoning, as an authentic assessment, but as a subjective and stressful method. Administrators emphasized the organizational consequences such as logistical challenges. Consequences must be considered when making decisions about SOE and our results support important positive consequences., Competing Interests: Competing InterestsThe authors declare no competing interests., (© The Author(s) under exclusive licence to International Association of Medical Science Educators 2023.)

Purpose: Narrative comments are increasingly used in assessment to document trainees' performance and to make important decisions about academic progress. However, little is known about how to document the quality of narrative comments, since traditional psychometric analysis cannot be applied. The authors aimed to generate a list of quality indicators for narrative comments, to identify recommendations for writing high-quality narrative comments, and to document factors that influence the quality of narrative comments used in assessments in higher education., Method: The authors conducted a scoping review according to Arksey & O'Malley's framework. The search strategy yielded 690 articles from 6 databases. Team members screened abstracts for inclusion and exclusion, then extracted numerical and qualitative data based on predetermined categories. Numerical data were used for descriptive analysis. The authors completed the thematic analysis of qualitative data with iterative discussions until they achieved consensus for the interpretation of the results., Results: After the full-text review of 213 selected articles, 47 were included. Through the thematic analysis, the authors identified 7 quality indicators, 12 recommendations for writing quality narratives, and 3 factors that influence the quality of narrative comments used in assessment. The 7 quality indicators are (1) describes performance with a focus on particular elements (attitudes, knowledge, skills); (2) provides a balanced message between positive elements and elements needing improvement; (3) provides recommendations to learners on how to improve their performance; (4) compares the observed performance with an expected standard of performance; (5) provides justification for the mark/score given; (6) uses language that is clear and easily understood; and (7) uses a nonjudgmental style., Conclusions: Assessors can use these quality indicators and recommendations to write high-quality narrative comments, thus reinforcing the appropriate documentation of trainees' performance, facilitating solid decision making about trainees' progression, and enhancing the impact of narrative feedback for both learners and programs., (Copyright © 2022 by the Association of American Medical Colleges.)

Purpose: Progress testing is an increasingly popular form of assessment in which a comprehensive test is administered to learners repeatedly over time. To inform potential users, this scoping review aimed to document barriers, facilitators, and potential outcomes of the use of written progress tests in higher education., Method: The authors followed Arksey and O'Malley's scoping review methodology to identify and summarize the literature on progress testing. They searched 6 databases (Academic Search Complete, CINAHL, ERIC, Education Source, MEDLINE, and PsycINFO) on 2 occasions (May 22, 2018, and April 21, 2020) and included articles written in English or French and pertaining to written progress tests in higher education. Two authors screened articles for the inclusion criteria (90% agreement), then data extraction was performed by pairs of authors. Using a snowball approach, the authors also screened additional articles identified from the included reference lists. They completed a thematic analysis through an iterative process., Results: A total of 104 articles were included. The majority of progress tests used a multiple-choice and/or true-or-false question format (95, 91.3%) and were administered 4 times a year (38, 36.5%). The most documented source of validity evidence was internal consistency (38, 36.5%). Four major themes were identified: (1) barriers and challenges to the implementation of progress testing (e.g., need for additional resources); (2) established collaboration as a facilitator of progress testing implementation; (3) factors that increase the acceptance of progress testing (e.g., formative use); and (4) outcomes and consequences of progress test use (e.g., progress testing contributes to an increase in knowledge)., Conclusions: Progress testing appears to have a positive impact on learning, and there is significant validity evidence to support its use. Although progress testing is resource- and time-intensive, strategies such as collaboration with other institutions may facilitate its use., (Copyright © 2021 by the Association of American Medical Colleges.)

Purpose: Written examinations such as multiple-choice question (MCQ) exams are a key assessment strategy in health professions education (HPE), frequently used to provide feedback, to determine competency, or for licensure decisions. However, traditional psychometric approaches for monitoring the quality of written exams, defined as items that are discriminant and contribute to increase the overall reliability and validity of the exam scores, usually warrant larger samples than are typically available in HPE contexts. The authors conducted a descriptive exploratory study to document how undergraduate medical education (UME) programs ensure the quality of their written exams, particularly MCQs., Method: Using a qualitative descriptive methodology, the authors conducted semistructured interviews with 16 key informants from 10 Canadian UME programs in 2018. Interviews were transcribed, anonymized, coded by the primary investigator, and co-coded by a second team member. Data collection and analysis were conducted iteratively. Research team members engaged in analysis across phases, and consensus was reached on the interpretation of findings via group discussion., Results: Participants focused their answers around MCQ-related practices, reporting using several indicators of quality such as alignment between items and course objectives and psychometric properties (difficulty and discrimination). The authors clustered findings around 5 main themes: processes for creating MCQ exams, processes for building quality MCQ exams, processes for monitoring the quality of MCQ exams, motivation to build quality MCQ exams, and suggestions for improving processes., Conclusions: Participants reported engaging multiple strategies to ensure the quality of MCQ exams. Assessment quality considerations were integrated throughout the development and validation phases, reflecting recent work regarding validity as a social imperative.

Purpose: Using item analyses is an important quality-monitoring strategy for written exams. Authors urge caution as statistics may be unstable with small cohorts, making application of guidelines potentially detrimental. Given the small cohorts common in health professions education, this study's aim was to determine the impact of cohort size on outcomes arising from the application of item analysis guidelines., Method: The authors performed a Monte Carlo simulation study in fall 2015 to examine the impact of applying 2 commonly used item analysis guidelines on the proportion of items removed and overall exam reliability as a function of cohort size. Three variables were manipulated: Cohort size (6 levels), exam length (6 levels), and exam difficulty (3 levels). Study parameters were decided based on data provided by several Canadian medical schools., Results: The analyses showed an increase in proportion of items removed with decreases in exam difficulty and decreases in cohort size. There was no effect of exam length on this outcome. Exam length had a greater impact on exam reliability than did cohort size after applying item analysis guidelines. That is, exam reliability decreased more with shorter exams than with smaller cohorts., Conclusions: Although program directors and assessment creators have little control over their cohort sizes, they can control the length of their exams. Creating longer exams makes it possible to remove items without as much negative impact on the exam's reliability relative to shorter exams, thereby reducing the negative impact of small cohorts when applying item removal guidelines.

Background: Although clinical teachers can often identify struggling learners readily and reliably, they can be reluctant to act upon their impressions, resulting in failure to fail. In the absence of a clear process for identifying and remediating struggling learners, clinical teachers can be put off by the prospect of navigating the politically and personally charged waters of remediation and potential failing of students., Methods: To address this gap, we developed a problem-solving algorithm to support clinical teachers from the identification through the remediation of learners with clinical reasoning difficulties, which have significant implications for patient care. Based on this algorithm, a mobile application (Pdx) was developed and assessed in two emergency departments at a Canadian university, from 2015 to 2016, using interpretive description as our research design. Semi-structured interviews were conducted before and after a three-month trial with the application. Interviews were analysed both deductively, using pre-determined categories, and inductively, using emerging categories., Results: Twelve clinical teachers were interviewed. Their experience with the application revealed their need to first validate their impressions of difficulties in learners and to find the right words to describe them before difficulties could be addressed. The application was unanimously considered helpful regarding both these aspects, while the mobile format appeared instrumental in allowing clinical teachers to quickly access targeted information during clinical supervision., Conclusions: The value placed on verifying impressions and finding the right words to pinpoint difficulties should be further explored in endeavours that aim to address the failure to fail phenomenon. Moreover, just-in-time mobile solutions, which mirror habitual clinical practices, may be used profitably for knowledge transfer in medical education, as an alternative form of faculty development.

Background: Collaboration is an important competence to be acquired by residents. Although improving residents' collaboration via interprofessional education has been investigated in many studies, little is known about the residents' spontaneous collaborative behavior. The purpose of this exploratory study was to describe how residents spontaneously collaborate., Methods: Seven first-year residents (postgraduate year 1; three from family medicine and one each from ear, nose, and throat, obstetrics/gynecology, general surgery, and orthopedic surgery) participated in two collaborative meetings with actors performing the part of other health professionals (ie, occupational therapist, physiotherapist, nurse, or social worker). Both meetings were built around an issue or conflict with the patients' families reported by one professional. The residents were required to lead the meeting to collect proper information to reach a joint decision. Two team members analyzed the video recordings of the meetings using an emerging-theme qualitative methodology., Results: Although the residents spontaneously knew how to successfully communicate with other professionals, they seemed to struggle with the patient-centered approach and the shared decision-making process., Discussion: Even if the residents performed communication-wise in their collaborative role, they seemed to have perceived themselves as decision makers instead of collaborators in the joint decision process. The results of this study can inform future studies on learning strategies to improve behaviors that would more likely need attention in interprofessional education.