Search

Background: Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively., Methods: The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score., Results: A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument., Conclusion: There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research., (© 2023 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

The use of Artificial Intelligence (AI) in medical education has the potential to facilitate complicated tasks and improve efficiency. For example, AI could help automate assessment of written responses, or provide feedback on medical image interpretations with excellent reliability. While applications of AI in learning, instruction, and assessment are growing, further exploration is still required. There exist few conceptual or methodological guides for medical educators wishing to evaluate or engage in AI research. In this guide, we aim to: 1) describe practical considerations involved in reading and conducting studies in medical education using AI, 2) define basic terminology and 3) identify which medical education problems and data are ideally-suited for using AI.

The purpose of this scoping review was to explore how errors are conceptualized in medical education contexts by examining different error perspectives and practices. This review used a scoping methodology with a systematic search strategy to identify relevant studies, written in English, and published before January 2021. Four medical education journals (Medical Education, Advances in Health Science Education, Medical Teacher, and Academic Medicine) and four clinical journals (Journal of the American Medical Association, Journal of General Internal Medicine, Annals of Surgery, and British Medical Journal) were purposively selected. Data extraction was charted according to a data collection form. Of 1505 screened studies, 79 studies were included. Three overarching perspectives were identified: 'understanding errors') (n = 31), 'avoiding errors' (n = 25), 'learning from errors' (n = 23). Studies that aimed at'understanding errors' used qualitative methods (19/31, 61.3%) and took place in the clinical setting (19/31, 61.3%), whereas studies that aimed at 'avoiding errors' and 'learning from errors' used quantitative methods ('avoiding errors': 20/25, 80%, and 'learning from errors': 16/23, 69.6%, p = 0.007) and took place in pre-clinical (14/25, 56%) and simulated settings (10/23, 43.5%), respectively (p < 0.001). The three perspectives differed significantly in terms of inclusion of educational theory: 'Understanding errors' studies 16.1% (5/31),'avoiding errors' studies 48% (12/25), and 'learning from errors' studies 73.9% (17/23), p < 0.001. Errors in medical education and clinical practice are defined differently, which makes comparisons difficult. A uniform understanding is not necessarily a goal but improving transparency and clarity of how errors are currently conceptualized may improve our understanding of when, why, and how to use and learn from errors in the future., (© 2022. The Author(s), under exclusive licence to Springer Nature B.V.)

Context: A long-standing myth in medical education research is a divide between two different poles: research aiming to advance theory with little focus on practical applications ('ivory tower' research) and practically oriented research aiming to serve educators and decision-makers with little focus on advancing theory ('in-the-trenches' practice). We explored this myth in a sample of randomised medical education studies using Stokes' four-quadrant framework for the classification of research perspective., Methods: We searched MEDLINE, Embase, CINAHL, PsychINFO, ERIC, Web of Science and Scopus for studies in medical education using a randomised design that were published between 1 January 2018 and 31 December 2018. We used Stokes' four-quadrant framework to categorise the studies according to their use of theory, concepts and their justification for practical use. We compared medical education research published in medical education journals and clinical journals., Results: A total of 150 randomised studies were included in the analysis. The largest segment of studies (46.7%) was categorised as use-inspired basic research (Pasteur's Quadrant), closely followed by pure applied research (40.7%, Edison's Quadrant). Only a few studies were categorised as aiming to advance knowledge with no thought for practical educational application (2.0%, Bohr's Quadrant). The proportion of studies that included educational concepts and theory differed according to publication in clinical journals or medical education journals: 40.5% vs 71.8%, respectively, P < .001. There were no differences between journals with regard to the proportion of studies that included a practical educational or clinical rationale (P = .99)., Conclusion: In a large sample of studies using randomised designs, we found no evidence to support the myth that medical education research divides between two singular poles represented by 'ivory tower research' and 'in-the-trenches practice'. We did confirm prevailing assumptions regarding an emphasis on non-theoretical medical education research in clinical journals., (© 2020 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

This AMEE Guide summarizes fundamentals of a major experimental design option for medical education researchers: the randomised study. Medical education researchers face an overwhelming taxonomy of study design options; given the breadth of information on experimental design, the purpose of this Guide is to offer a resource for medical education researchers wishing to equip themselves with helpful information for when to match a study's objective and the use of randomised designs. Once a research question has been formulated study design is the cornerstone of the intricate, nested activities of any research project. Researchers negotiate many decisions in the pursuit of choosing an appropriate design approach; failure to do so can undermine a project's capacity to, for example, sufficiently test a hypothesis or theory. Written as an introduction, this Guide is intended for medical education researchers seeking to build on and synthesise the existing corpus of literature on experimental and quasi-experimental design approaches. While not comprehensive, presented are key concepts alongside relevant examples from the field of health professions education.

Objective: From 2012 to 2015, two Departments of Obstetrics and Gynecology and two Departments of Pediatrics at the University of Copenhagen implemented an English medium international project. The project allowed international students to work in pairs with local Danish speaking students in a clinical setting. The student cohort was supported by Danish doctors who were responsible for student-pair supervision in English and, ultimately, patient care. Drawing on survey responses of 113 Danish doctors, this study considers the doctors' overall evaluation of the program and their perception of the international students' knowledge, skills and attitudes compared with local students., Results: The Danish doctors rated the international and local students comparable in respect to professional commitment (p = 0.347), academic level (p = 0.134), and English proficiency (p = 0.080). The Danish doctors rated the international students significantly lower than the local students regarding communication with Danish doctors, other hospital staff, and patients (p < 0.001 in all cases). Ninety percent of the doctors involved in the project supported continuing working with internationalization if it included mixed pairs of students and a Danish doctor assigned each day to be exclusively responsible for student supervision. Language barriers for international medical students could be overcome but required substantial faculty support.

Objective: The aim of this systematic review was to identify studies on hysteroscopic training and assessment., Design: PubMed, Excerpta Medica, the Cochrane Library, and Web of Science were searched in January 2015. Manual screening of references and citation tracking were also performed. Studies on hysteroscopic educational interventions were selected without restrictions on study design, populations, language, or publication year. A qualitative data synthesis including the setting, study participants, training model, training characteristics, hysteroscopic skills, assessment parameters, and study outcomes was performed by 2 authors working independently. Effect sizes were calculated when possible. Overall, 2 raters independently evaluated sources of validity evidence supporting the outcomes of the hysteroscopy assessment tools., Results: A total of 25 studies on hysteroscopy training were identified, of which 23 were performed in simulated settings. Overall, 10 studies used virtual-reality simulators and reported effect sizes for technical skills ranging from 0.31 to 2.65; 12 used inanimate models and reported effect sizes for technical skills ranging from 0.35 to 3.19. One study involved live animal models; 2 studies were performed in clinical settings. The validity evidence supporting the assessment tools used was low. Consensus between the 2 raters on the reported validity evidence was high (94%)., Conclusions: This systematic review demonstrated large variations in the effect of different tools for hysteroscopy training. The validity evidence supporting the assessment of hysteroscopic skills was limited., (Copyright © 2016 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Objectives: This study is designed to provide an overview of why, how, when and for whom collaborative learning of clinical skills may work in health professions education., Why: Collaborative learning of clinical skills may influence learning positively according to the non-medical literature. Training efficiency may therefore be improved if the outcomes of collaborative learning of clinical skills are superior or equivalent to those attained through individual learning., How: According to a social interaction perspective, collaborative learning of clinical skills mediates its effects through social interaction, motivation, accountability and positive interdependence between learners. Motor skills learning theory suggests that positive effects rely on observational learning and action imitation, and negative effects may include decreased hands-on experience. Finally, a cognitive perspective suggests that learning is dependent on cognitive co-construction, shared knowledge and reduced cognitive load., When and for Whom: The literature on the collaborative learning of clinical skills in health science education is reviewed to support or contradict the hypotheses provided by the theories outlined above. Collaborative learning of clinical skills leads to improvements in self-efficacy, confidence and performance when task processing is observable or communicable. However, the effects of collaborative learning of clinical skills may decrease over time as benefits in terms of shared cognition, scaffolding and cognitive co-construction are outweighed by reductions in hands-on experience and time on task., Conclusions: Collaborative learning of clinical skills has demonstrated promising results in the simulated setting. However, further research into how collaborative learning of clinical skills may work in clinical settings, as well as into the role of social dynamics between learners, is required., (© 2015 John Wiley & Sons Ltd.)

Introduction: Research in medical education has increased in volume over the past decades but concerns have been raised regarding the quality of trials conducted within this field. Randomised controlled trials (RCTs) involving educational interventions that are reported in biomedical journals have been criticised for their insufficient conceptual, theoretical framework. RCTs published in journals dedicated to medical education, on the other hand, have been questioned regarding their methodological rigour. The aim of this study is therefore to assess the quality of RCTs of educational interventions reported in 2012 and 2013 in journals dedicated to medical education compared to biomedical journals with respect to objective quality criteria., Methods and Analysis: RCTs published between 1 January 2012 and 31 December 2013 in English are included. The search strategy is developed with the help of experienced librarians to search online databases for key terms. All of the identified RCTs are screened based on their titles and abstracts individually by the authors and then compared in pairs to assess agreement. Data are extracted from the included RCTs by independently scoring each RCT using a data collection form. The data collection form consists of four steps. Step 1 includes confirmation of RCT eligibility; step 2 consists of the CONSORT checklist; step 3 consists of the Medical Education Research Study Quality Instrument framework; step 4 consists of a Medical Education Extension (MEdEx) to the CONSORT checklist. The MEdEx includes the following elements: Description of scientific background, explanation of rationale, quality of research questions and hypotheses, clarity in the description of the use of the intervention and control as well as interpretation of results., Ethics and Dissemination: This review is the first to systematically examine the quality of RCTs conducted in medical education. We plan to disseminate the results through publications and presentation at relevant conferences. Ethical approval is not sought for this review., (Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.)