Search

Objectives: This study compares the performance of the artificial intelligence (AI) platform Chat Generative Pre-Trained Transformer (ChatGPT) to Otolaryngology trainees on board-style exam questions., Methods: We administered a set of 30 Otolaryngology board-style questions to medical students (MS) and Otolaryngology residents (OR). 31 MSs and 17 ORs completed the questionnaire. The same test was administered to ChatGPT version 3.5, five times. Comparisons of performance were achieved using a one-way ANOVA with Tukey Post Hoc test, along with a regression analysis to explore the relationship between education level and performance., Results: The average scores increased each year from MS1 to PGY5. A one-way ANOVA revealed that ChatGPT outperformed trainee years MS1, MS2, and MS3 (p = <0.001, 0.003, and 0.019, respectively). PGY4 and PGY5 otolaryngology residents outperformed ChatGPT (p = 0.033 and 0.002, respectively). For years MS4, PGY1, PGY2, and PGY3 there was no statistical difference between trainee scores and ChatGPT (p = .104, .996, and 1.000, respectively)., Conclusion: ChatGPT can outperform lower-level medical trainees on Otolaryngology board-style exam but still lacks the ability to outperform higher-level trainees. These questions primarily test rote memorization of medical facts; in contrast, the art of practicing medicine is predicated on the synthesis of complex presentations of disease and multilayered application of knowledge of the healing process. Given that upper-level trainees outperform ChatGPT, it is unlikely that ChatGPT, in its current form will provide significant clinical utility over an Otolaryngologist., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Patel et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Introduction: Clinical reasoning (CR) is required for physicians. Pediatric residents often gain CR skills through experiential learning. Currently, deliberate education on CR targeted toward pediatric residents is inconsistent. Our objective was to implement a pilot CR curriculum, including five hour-long sessions, and evaluate its impact on self-identified CR Milestones and comfort with CR skills., Methods: We used Kern's six steps for curriculum development to develop our curriculum. Five morning report sessions included didactics and small-group activities. Pre/post surveys assessed resident self-identified level on ACGME Milestones related to CR skills (Patient Care 4 [PC4] and Medical Knowledge 2 [MK2]) and comfort with CR skills. The postsurvey assessed resident attitudes toward the sessions. Paired samples for Milestone and comfort-based questions were analyzed using Wilcoxon signed rank tests. Attitude questions were reported with descriptive statistics., Results: Each of the five curricular sessions was attended by 40-50 pediatric residents. Seventy-one trainees (58% of residency) and 51 trainees (42% of residency) completed the pre- and postsurveys, respectively, with 20 paired samples. Self-assessment of PC4 ( p = .006) and resident comfort with all measured CR skills increased significantly. Of trainees who attended at least one session ( n = 44), most reported finding the sessions helpful (97%), relevant to their clinical work (97%), and impactful on their clinical practice (73%)., Discussion: Following exposure to this CR curriculum, pediatric residents reported increased self-identified competency levels on the evaluated Milestones and improved comfort with CR skills. Dedicated CR education may advance pediatric resident understanding of and comfort with CR., (© 2024 Bowen et al.)

Objectives: This study assessed the relationship between academic performance and National Licensing Examination (NLE) scores of Ethiopian anaesthetists and proposed academic performance thresholds that predict failing the NLE., Design: A retrospective cross-sectional study., Setting: Academic performance measures were collected from eight universities and matched to total and subdomain NLE scores for anaesthetists. Four universities were first generation (oldest), three were second generation and one was third generation (newest)., Participants: 524 anaesthetists., Measures: Selected course grades, yearly (Y1-Y4) and cumulative grade point averages (GPAs) and in-school exit exam status were correlated with NLE scores. Linear regression was used to identify predictors of NLE score. Receiver operating characteristic curves were plotted to estimate GPA thresholds that predict failing the NLE., Results: Total and general surgery anaesthesia subdomain NLE scores showed moderate to strong linear relationships with most academic performance indicators. However, correlations between the rest of the subdomain NLE scores and composite course scores were either small or negligible (r<0.5). For every 1-point increase in a 4-point cumulative GPA and composite senior year practicum course score, the first-attempt NLE score increased by 12.61 (95% CI 10.03 to 15.19; p<0.0001) and 3.98 (95% CI 1.77 to 6.19; p=0.001) percentage points, respectively. Those entering university directly from high school and those attending a third-generation university would, on average, score 2.79 (95% CI 0.23 to 5.35; p=0.033) and 4.34 (95% CI 1.42 to 7.26; p=0.004) higher NLE scores compared with nurse entrants and first-generation university graduates, respectively. On a 4-point grading system, the thresholds for failing the NLE were below 2.68 for year-1 GPA, 2.62 for year-2 GPA, 2.72 for year-3 GPA, 3.04 for year-4 GPA and 2.87 for cumulative GPA., Conclusions: The Ethiopian anaesthetist's NLE score modestly correlated with most academic performance indicators. Based on pass/fail thresholds, the NLE could assist training programmes in improving NLE pass rates., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Medical interviewing is a critical skill in clinical practice, yet opportunities for practical training are limited in Japanese medical schools, necessitating urgent measures. Given advancements in artificial intelligence (AI) technology, its application in the medical field is expanding. However, reports on its application in medical interviews in medical education are scarce., Objective: This study aimed to investigate whether medical students' interview skills could be improved by engaging with AI-simulated patients using large language models, including the provision of feedback., Methods: This nonrandomized controlled trial was conducted with fourth-year medical students in Japan. A simulation program using large language models was provided to 35 students in the intervention group in 2023, while 110 students from 2022 who did not participate in the intervention were selected as the control group. The primary outcome was the score on the Pre-Clinical Clerkship Objective Structured Clinical Examination (pre-CC OSCE), a national standardized clinical skills examination, in medical interviewing. Secondary outcomes included surveys such as the Simulation-Based Training Quality Assurance Tool (SBT-QA10), administered at the start and end of the study., Results: The AI intervention group showed significantly higher scores on medical interviews than the control group (AI group vs control group: mean 28.1, SD 1.6 vs 27.1, SD 2.2; P=.01). There was a trend of inverse correlation between the SBT-QA10 and pre-CC OSCE scores (regression coefficient -2.0 to -2.1). No significant safety concerns were observed., Conclusions: Education through medical interviews using AI-simulated patients has demonstrated safety and a certain level of educational effectiveness. However, at present, the educational effects of this platform on nonverbal communication skills are limited, suggesting that it should be used as a supplementary tool to traditional simulation education., (©Akira Yamamoto, Masahide Koda, Hiroko Ogawa, Tomoko Miyoshi, Yoshinobu Maeda, Fumio Otsuka, Hideo Ino. Originally published in JMIR Medical Education (https://mededu.jmir.org), 23.09.2024.)

Background: Large language models (LLMs), such as ChatGPT, have considerable implications for various medical applications. However, ChatGPT's training primarily draws from English-centric internet data and is not tailored explicitly to the medical domain. Thus, an ophthalmic LLM in Chinese is clinically essential for both healthcare providers and patients in mainland China., Methods: We developed an LLM of ophthalmology (MOPH) using Chinese corpora and evaluated its performance in three clinical scenarios: ophthalmic board exams in Chinese, answering evidence-based medicine-oriented ophthalmic questions and diagnostic accuracy for clinical vignettes. Additionally, we compared MOPH's performance to that of human doctors., Results: In the ophthalmic exam, MOPH's average score closely aligned with the mean score of trainees (64.7 (range 62-68) vs 66.2 (range 50-92), p=0.817), but achieving a score above 60 in all seven mock exams. In answering ophthalmic questions, MOPH demonstrated an adherence of 83.3% (25/30) of responses following Chinese guidelines (Likert scale 4-5). Only 6.7% (2/30, Likert scale 1-2) and 10% (3/30, Likert scale 3) of responses were rated as 'poor or very poor' or 'potentially misinterpretable inaccuracies' by reviewers. In diagnostic accuracy, although the rate of correct diagnosis by ophthalmologists was superior to that by MOPH (96.1% vs 81.1%, p>0.05), the difference was not statistically significant., Conclusion: This study demonstrated the promising performance of MOPH, a Chinese-specific ophthalmic LLM, in diverse clinical scenarios. MOPH has potential real-world applications in Chinese-language ophthalmology settings., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background/aims: This study assesses the proficiency of Generative Pre-trained Transformer (GPT)-4 in answering questions about complex clinical ophthalmology cases., Methods: We tested GPT-4 on 422 Journal of the American Medical Association Ophthalmology Clinical Challenges, and prompted the model to determine the diagnosis (open-ended question) and identify the next-step (multiple-choice question). We generated responses using two zero-shot prompting strategies, including zero-shot plan-and-solve+ (PS+), to improve the reasoning of the model. We compared the best-performing model to human graders in a benchmarking effort., Results: Using PS+ prompting, GPT-4 achieved mean accuracies of 48.0% (95% CI (43.1% to 52.9%)) and 63.0% (95% CI (58.2% to 67.6%)) in diagnosis and next step, respectively. Next-step accuracy did not significantly differ by subspecialty (p=0.44). However, diagnostic accuracy in pathology and tumours was significantly higher than in uveitis (p=0.027). When the diagnosis was accurate, 75.2% (95% CI (68.6% to 80.9%)) of the next steps were correct. Conversely, when the diagnosis was incorrect, 50.2% (95% CI (43.8% to 56.6%)) of the next steps were accurate. The next step was three times more likely to be accurate when the initial diagnosis was correct (p<0.001). No significant differences were observed in diagnostic accuracy and decision-making between board-certified ophthalmologists and GPT-4. Among trainees, senior residents outperformed GPT-4 in diagnostic accuracy (p≤0.001 and 0.049) and in accuracy of next step (p=0.002 and 0.020)., Conclusion: Improved prompting enhances GPT-4's performance in complex clinical situations, although it does not surpass ophthalmology trainees in our context. Specialised large language models hold promise for future assistance in medical decision-making and diagnosis., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: Chat Generative Pre-trained Transformer (ChatGPT), a large language model by OpenAI, and Bard, Google's artificial intelligence (AI) chatbot, have been evaluated in various contexts. This study aims to assess these models' proficiency in the part 1 Fellowship of the Royal College of Ophthalmologists (FRCOphth) Multiple Choice Question (MCQ) examination, highlighting their potential in medical education., Methods: Both models were tested on a sample question bank for the part 1 FRCOphth MCQ exam. Their performances were compared with historical human performance on the exam, focusing on the ability to comprehend, retain and apply information related to ophthalmology. We also tested it on the book 'MCQs for FRCOpth part 1', and assessed its performance across subjects., Results: ChatGPT demonstrated a strong performance, surpassing historical human pass marks and examination performance, while Bard underperformed. The comparison indicates the potential of certain AI models to match, and even exceed, human standards in such tasks., Conclusion: The results demonstrate the potential of AI models, such as ChatGPT, in processing and applying medical knowledge at a postgraduate level. However, performance varied among different models, highlighting the importance of appropriate AI selection. The study underlines the potential for AI applications in medical education and the necessity for further investigation into their strengths and limitations., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. No commercial re-use. See rights and permissions. Published by BMJ.)

In the UK, student nurses, nursing associates and midwives are supervised by practice supervisors and practice assessors while completing the placement-based component of their training. Because of increased clinical commitments, staff shortages and rising student numbers, demands for the supervision and assessment of student nurses have increased in recent years and this was exacerbated by COVID-19. The Nursing and Midwifery Council's Standards for Student Supervision and Assessment separated the roles of practice supervisor and assessor, giving rise to opportunities to assess students in a different way. Some integrated care systems are exploring the new role of roaming assessor. Roaming assessors can fulfil the role of practice assessors while providing additional support for student nurses. This role could be expanded to other health professions. This article explores the use of practice learning facilitators as roaming assessors for student nurses in the UK and evaluates the potential benefits and limitations of this approach. Roaming assessors may provide a useful service to deliver practice assessments, increasing the reliability and availability of assessments.

Background: ChatGPT has been tested in health care, including the US Medical Licensing Examination and specialty exams, showing near-passing results. Its performance in the field of anesthesiology has been assessed using English board examination questions; however, its effectiveness in Korea remains unexplored., Objective: This study investigated the problem-solving performance of ChatGPT in the fields of anesthesiology and pain medicine in the Korean language context, highlighted advancements in artificial intelligence (AI), and explored its potential applications in medical education., Methods: We investigated the performance (number of correct answers/number of questions) of GPT-4, GPT-3.5, and CLOVA X in the fields of anesthesiology and pain medicine, using in-training examinations that have been administered to Korean anesthesiology residents over the past 5 years, with an annual composition of 100 questions. Questions containing images, diagrams, or photographs were excluded from the analysis. Furthermore, to assess the performance differences of the GPT across different languages, we conducted a comparative analysis of the GPT-4's problem-solving proficiency using both the original Korean texts and their English translations., Results: A total of 398 questions were analyzed. GPT-4 (67.8%) demonstrated a significantly better overall performance than GPT-3.5 (37.2%) and CLOVA-X (36.7%). However, GPT-3.5 and CLOVA X did not show significant differences in their overall performance. Additionally, the GPT-4 showed superior performance on questions translated into English, indicating a language processing discrepancy (English: 75.4% vs Korean: 67.8%; difference 7.5%; 95% CI 3.1%-11.9%; P=.001)., Conclusions: This study underscores the potential of AI tools, such as ChatGPT, in medical education and practice but emphasizes the need for cautious application and further refinement, especially in non-English medical contexts. The findings suggest that although AI advancements are promising, they require careful evaluation and development to ensure acceptable performance across diverse linguistic and professional settings., (©Soo-Hyuk Yoon, Seok Kyeong Oh, Byung Gun Lim, Ho-Jin Lee. Originally published in JMIR Medical Education (https://mededu.jmir.org), 16.09.2024.)

Background: The ability of an expert's item difficulty ratings to predict test-taker actual performance is an important aspect of licensure examinations. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of raters involved in predicting item difficulty is central to set credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual Multiple-Choice Questions' difficulty of the undergraduate medicine licensure examination (UGMLE) in Ethiopia., Method: 815 examinees' responses to 200 Multiple-Choice Questions (MCQs) were used in this study. The study also included experts' item difficulty ratings of seven physicians who participated in the standard settings of UGMLE. Then, analysis was conducted to understand experts' rating variation in predicting the actual difficulty levels of examinees. Descriptive statistics was used to profile the mean rater's and actual difficulty value for MCQs, and ANOVA was used to compare the mean differences between raters' prediction of item difficulty. Additionally, regression analysis was used to understand the interrater variations in item difficulty predictions compared to the actual difficulty. The proportion of variance of actual difficulty explained from rater prediction was computed using regression analysis., Results: In this study, the mean difference between raters' prediction and examinees' actual performance was inconsistent across the exam domains. The study revealed a statistically significant strong positive correlation between the actual and predicted item difficulty in exam domains eight and eleven. However, a non-statistically significant very weak positive correlation was reported in exam domains seven and twelve. The multiple comparison analysis showed significant differences in mean item difficulty ratings between raters. In the regression analysis, experts' item difficulty ratings of the UGMLE had 33% power in predicting the actual difficulty level. The regression model also showed a moderate positive correlation (R = 0.57) that was statistically significant at F (6, 193) = 15.58, P = 0.001., Conclusion: This study demonstrated the complex process for assessing the difficulty level of MCQs in the UGMLE and emphasized the benefits of using experts' ratings in advance. To ensure the exams maintain the necessary reliable and valid scores, raters' accuracy on the UGMLE must be improved. To achieve this, techniques that align with the evolving assessment methodologies must be developed., (© 2024. The Author(s).)

Validity has long held a venerated place in education, leading some authors to refer to it as the "sine qua non" or "cardinal virtue" of assessment. And yet, validity has not held a fixed meaning; rather it has shifted in its definition and scope over time. In this Eye Opener, the authors explore if and how current conceptualizations of validity fit a next era of assessment that prioritizes patient care and learner equity. They posit that health profession education's conceptualization of validity will change in three related but distinct ways. First, consequences of assessment decisions will play a central role in validity arguments. Second, validity evidence regarding impacts of assessment on patients and society will be prioritized. Third, equity will be seen as part of validity rather than an unrelated concept. The authors argue that health professions education has the agency to change its ideology around validity, and to align with values that predominate the next era of assessment such as high-quality care and equity for learners and patients., Competing Interests: The authors have no competing interests to declare., (Copyright: © 2024 The Author(s).)

Chat Generative Pretrained Transformer (ChatGPT; OpenAI) is a state-of-the-art large language model that can simulate human-like conversations based on user input. We evaluated the performance of GPT-4 V in the Japanese National Clinical Engineer Licensing Examination using 2,155 questions from 2012 to 2023. The average correct answer rate for all questions was 86.0%. In particular, clinical medicine, basic medicine, medical materials, biological properties, and mechanical engineering achieved a correct response rate of ≥ 90%. Conversely, medical device safety management, electrical and electronic engineering, and extracorporeal circulation obtained low correct answer rates ranging from 64.8% to 76.5%. The correct answer rates for questions that included figures/tables, required numerical calculation, figure/table ∩ calculation, and knowledge of Japanese Industrial Standards were 55.2%, 85.8%, 64.2% and 31.0%, respectively. The reason for the low correct answer rates is that ChatGPT lacked recognition of the images and knowledge of standards and laws. This study concludes that careful attention is required when using ChatGPT because several of its explanations lack the correct description., (© 2024. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Background: History-taking is an essential clinical competency for qualified doctors. The limitations of the standardized patient (SP) in taking history can be addressed by the virtual standardized patient (VSP). This paper investigates the accuracy of virtual standardized patient simulators and evaluates the applicability of the improved system's accuracy for diagnostic teaching support and performance assessment., Methods: Data from the application of VSP to medical residents and students were gathered for this prospective study. In a human-machine collaboration mode, students completed exams involving taking SP histories while VSP provided real-time scoring. Every participant had VSP and SP scores. Lastly, using the voice and text records as a guide, the technicians will adjust the system's intention recognition accuracy and speech recognition accuracy., Results: The research revealed significant differences in scoring across several iterations of VSP and SP (p < 0.001). Across various clinical cases, there were differences in application accuracy for different versions of VSP (p < 0.001). Among training groups, the diarrhea case showed significant differences in speech recognition accuracy (Z = -2.719, p = 0.007) and intent recognition accuracy (Z = -2.406, p = 0.016). Scoring and intent recognition accuracy improved significantly after system upgrades., Conclusion: VSP has a comprehensive and detailed scoring system and demonstrates good scoring accuracy, which can be a valuable tool for history-taking training., (© 2024. The Author(s).)

Standardized tests intend to reduce information asymmetry by providing a common and objective measure of students' academic performance. The basic assumption underlying standardized testing is that differences in student performance on standardized tests should be attributed primarily to differences in the quality of education received by students. However, there is evidence that environmental factors can affect standardized test scores, which may result in anomalous observations or outliers that show a distortion of student performance. In this regard, the exclusion of Spain from PISA 2018 is particularly interesting as Spanish data met PISA 2018 Technical Standards but showed implausible student-response behavior. The aim of this paper is to complement the OECD's analysis of Spain's exclusion from PISA 2018 by exploring the potential reasons behind the outlier results, focusing on the Madrid region., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 van Grieken et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Objective: to reflect on the perspectives of adopting the Progress Test in undergraduate nursing education., Methods: this is a reflective study, based on authors' critical thinking and supported by national and international literature on the Progress Test application in undergraduate health courses., Results: the Progress Test is as a valuable teaching-learning strategy, with potential applications and benefits for students, professors, and academic management., Final Considerations: systematic nursing education assessment indicates valuable information for different stakeholders. Understanding its potential benefits, the Progress Test is presented as a strategy that can be replicated in undergraduate nursing education, either individually, by institutions, or collaboratively, by the establishment of partnerships or consortiums of institutions.

Objective structured clinical examinations (OSCEs) are a widely used performance assessment for medical and dental students. A common limitation of OSCEs is that the evaluation results depend on the characteristics of raters and a scoring rubric. To overcome this limitation, item response theory (IRT) models such as the many-facet Rasch model have been proposed to estimate examinee abilities while taking into account the characteristics of raters and evaluation items in a rubric. However, conventional IRT models have two impractical assumptions: constant rater severity across all evaluation items in a rubric and an equal interval rating scale among evaluation items, which can decrease model fitting and ability measurement accuracy. To resolve this problem, we propose a new IRT model that introduces two parameters: (1) a rater-item interaction parameter representing the rater severity for each evaluation item and (2) an item-specific step-difficulty parameter representing the difference in rating scales among evaluation items. We demonstrate the effectiveness of the proposed model by applying it to actual data collected from a medical interview test conducted at Tokyo Medical and Dental University as part of a post-clinical clerkship OSCE. The experimental results showed that the proposed model was well-fitted to our OSCE data and measured ability accurately. Furthermore, it provided abundant information on rater and item characteristics that conventional models cannot, helping us to better understand rater and item properties., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Uto et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

The proposed smart system for Student Performance Assessment (SPA) is a system that evaluates students' knowledge and skill attainment in a specific course by measuring their achievements of the Course Learning Outcomes (CLOs). The instructor defines the aspects, weights, and rating scale used by SPA to analyze each course. The system calculates the average of students' marks in each learning outcome and compares them with the CLO targets and scores to determine the effectiveness of the teaching and learning methods used. The system uses facts and rules extracted from the course syllabus and Bloom's Taxonomy to build its knowledge base. This paper presents the development of the SPA inference engine, which is used to find CLO targets based on the course level. The inference engine uses efficient procedures and a prediction process to determine the correct target and score, providing a reliable and understandable methodology for reasoning about the information in the knowledge base and formulating conclusions. SPA is a highly responsive and intelligent system that can be a valuable tool for measuring students' achievements. Its characteristics include high performance, reliability, and intelligibility, and its combination of cognitive systems and cognitive theory has led to remarkable progress in measuring student performance. Limitations include dependency on accurate course content and initial setup time, potential bias in CLO weight assignments, challenges in integrating SPA with existing institutional databases, the need for continuous updates to the knowledge base to reflect curriculum changes, and potential resistance from educators to adopt new technologies. Future improvements could involve adaptive learning integrations, enhanced user interfaces, and broader applicability across diverse educational settings., (© 2024. The Author(s).)

Background: This study aimed to answer the research question: How reliable is ChatGPT in automated essay scoring (AES) for oral and maxillofacial surgery (OMS) examinations for dental undergraduate students compared to human assessors?, Methods: Sixty-nine undergraduate dental students participated in a closed-book examination comprising two essays at the National University of Singapore. Using pre-created assessment rubrics, three assessors independently performed manual essay scoring, while one separate assessor performed AES using ChatGPT (GPT-4). Data analyses were performed using the intraclass correlation coefficient and Cronbach's α to evaluate the reliability and inter-rater agreement of the test scores among all assessors. The mean scores of manual versus automated scoring were evaluated for similarity and correlations., Results: A strong correlation was observed for Question 1 (r = 0.752-0.848, p < 0.001) and a moderate correlation was observed between AES and all manual scorers for Question 2 (r = 0.527-0.571, p < 0.001). Intraclass correlation coefficients of 0.794-0.858 indicated excellent inter-rater agreement, and Cronbach's α of 0.881-0.932 indicated high reliability. For Question 1, the mean AES scores were similar to those for manual scoring (p > 0.05), and there was a strong correlation between AES and manual scores (r = 0.829, p < 0.001). For Question 2, AES scores were significantly lower than manual scores (p < 0.001), and there was a moderate correlation between AES and manual scores (r = 0.599, p < 0.001)., Conclusion: This study shows the potential of ChatGPT for essay marking. However, an appropriate rubric design is essential for optimal reliability. With further validation, the ChatGPT has the potential to aid students in self-assessment or large-scale marking automated processes., (© 2024. The Author(s).)

Objectives: ChatGPT (OpenAI, San Francisco, CA) has shown impressive results across various medical examinations, but its performance in kidney pathology is not yet established. This study evaluated proficiencies of GPT-4 Vision (GPT-4V), an updated version of the platform with the ability to analyze images, on kidney pathology questions and compared its responses with those of nephrology trainees., Methods: Thirty-nine questions (19 text-based questions and 20 with various kidney biopsy images) designed specifically for the training of nephrology fellows were employed., Results: GPT-4V displayed comparable accuracy rates in the first and second runs (67% and 72%, respectively, P = .50). The aggregated accuracy, however-particularly, the consistent accuracy-of GPT-4V was lower than that of trainees (72% and 67% vs 79%). Both GPT-4V and trainees displayed comparable accuracy in responding to image-based and text-only questions (55% vs 79% and 81% vs 78%, P = .11 and P = .67, respectively). The consistent accuracy in image-based, directly asked questions for GPT-4V was 29%, much lower than its 88% consistency on text-only, directly asked questions (P = .02). In contrast, trainees maintained similar accuracy in directly asked image-based and text-based questions (80% vs 77%, P = .65). Although the aggregated accuracy for correctly interpreting images was 69%, the consistent accuracy across both runs was only 39%. The accuracy of GPT-4V in answering questions with correct image interpretation was significantly higher than for questions with incorrect image interpretation (100% vs 0% and 100% vs 33% for the first and second runs, P = .001 and P = .02, respectively)., Conclusions: The performance of GPT-4V in handling kidney pathology questions, especially those including images, is limited. There is a notable need for enhancement in GPT-4V proficiency in interpreting images., (© The Author(s) 2024. Published by Oxford University Press on behalf of American Society for Clinical Pathology. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.)

Aim: This paper aims to report on two modifications made to improve the measurement precision of the Ms. Olsen test. Specifically, three items were added to the Ms. Olsen test and an extended scoring was applied to some items., Design: The competence assessment had a quantitative cross-sectional design comprising of test results from 111 Registered Nurses in four municipalities in South-Eastern Norway., Methods: The Rasch model was applied to evaluate the measurement properties in four versions: the Ms. Olsen test with 19 items and the same with three added items, and the Ms. Olsen test with 19 items with an expanded scoring for the original and expanded version, respectively., Results: The person separation indexes were improved from 0.50 to 0.62. Other measurement properties were not alternating between the four versions; all had shortcomings in terms of targeting (person measure means 3.02-3.87) and unidimensionality (% t-test >5% 9.01%-13.51%)., Conclusion: The clinical relevance and relatively short time spent on testing makes the Ms. Olsen test a reasonable choice and a step in the right direction for assessing competence as a means of targeting continuous professional development of nurses throughout their career. Nevertheless, depending on what kind of decisions are to be made, reliability might still be too low and further development is suggested., Public Contribution: The initiative for the competence assessment came from nursing leaders in the four municipalities involved. The municipalities were represented in all stages of the research process through co-author LS, that is, design, data collection, data analysis, writing of the manuscript as well as dissemination of the results to the four municipalities., Implications for Practice: The Ms. Olsen test exemplifies a move towards objective assessments in nursing. Precise and reliable measurements are essential to support the ongoing professional development of nurses., (© 2024 The Author(s). International Journal of Older People Nursing published by John Wiley & Sons Ltd.)

Background: The American Association of Colleges of Nursing (AACN) and National Organization of Nurse Practitioner Faculties (NONPF) emphasize the role of clinical reasoning in nurse practitioner (NP) competencies. Evidence-based clinical reasoning is vital to patient safety. Collaborative technology tools can aid in assessing progress towards achieving clinical reasoning competency., Purpose/aims: The purpose of this article is to describe an electronic, collaborative learning framework to teach and assess second year NP students in systematically selecting and eliminating diagnoses and forming treatment plans., Design/methods: Post gap analysis, the collaborative learning framework was created. This visual, collaborative resource was scaffolded across two sequential advanced NP second year clinical synthesis courses and embedded with evolving case studies. Students identified pertinent positives and negatives from the history, physical, and diagnostic findings. Each student developed a unique differential diagnosis and plan of care and critiqued their peers., Result/findings: The tool exceeded expectations. Faculty were able to visualize data, provide clarification on interpretation of data and pharmacology, and grade in small groups., Conclusion: The collaborative learning framework provided real-time visualization of students' work in clinical reasoning. It was easy to use and integrate into second year NP courses to meet learning objectives and assess clinical reasoning competency., Competing Interests: Declaration of competing interest We have no conflict of interest to disclose, (Published by Elsevier Inc.)

Purpose: We not only developed a clinical practice program for the assessment and feedback vis-à-vis medical students' medical records but also evaluated the effectiveness of this program via a self-assessment of medical students' competence in writing medical records pre- and post-program., Methods: In 2022, 74 third-year medical students were divided into four groups and participated in a 2-week program. The students' medical records were graded on a scale ranging from 1 to 3 daily, and the mean scores for 2 weeks were compared. Pre- and post-program, the students' self-assessment survey was conducted., Results: The mean scores increased from 1.30 in the first week to 2.14 in the second week. The mean score of self-assessment showed significant improvements, increasing from 2.43 to 4.00 for medical record, 2.64 to 4.08 for write present illness, 2.08 to 3.89 for initial orders, 2.35 to 4.34 for signature, and 2.38 to 3.97 for consent (all p&lt;0.001)., Conclusion: We found that providing students with real-time assessment and feedback on their medical records increased their skills and confidence in medical records writing.

Purpose: This study aimed to investigate the difference in satisfaction and learning benefits between e-portfolios compared to paper portfolios during clinical practice in medical schools., Methods: Utilization of and satisfaction with e-portfolios among 40 third-year medical students in the medicine department of Ajou University School of Medicine was collected using an online survey in December 2020. The collected data were analyzed using descriptive statistics and an analysis of variance., Results: Students perceived that e-portfolios were highly beneficial for consistently documenting activities during clinical practice, when compared to paper-based portfolios (mean±standard deviation [SD]=2.60±1.22). However, the least rated aspect was that e-portfolios require less time than paper-based portfolios (mean±SD=1.80±1.14). Additionally, among the various clinical practice courses using e-portfolios, the highest satisfaction was observed with the fewest content items in the e-portfolio., Conclusion: To maximize the potential benefits of e-portfolios, improvements in implementation and usability are essential. Additionally, for effective utilization of e-portfolios in clinical practice, it is necessary to clearly define students' required competencies and ultimate goals, and structure content accordingly.

Abstract: Competency-based education, pivotal in nursing programs, emphasizes learning methods over knowledge accumulation, integrating ongoing assessments for performance and outcomes. The American Association College of Nursing introduced competency-based education, urging educators to explore diverse assessment methods. This article explores the implementation of mock oral boards as an assessment method to synchronize performance standards with competencies in nursing education, specifically in preparing nurse practitioner students for clinical rotations., Competing Interests: Competing interests: The authors report no conflicts of interest., (Copyright © 2024 American Association of Nurse Practitioners.)

Objectives: The coronavirus disease 2019 pandemic catalyzed a rapid shift toward remote learning in medicine. This study hypothesized that using videos on adverse events and patient safety event reporting systems could enhance education and motivation among healthcare professionals, leading to improved performance on quizzes compared with those exposed to standard, in-person lectures., Methods: Participants were randomly assigned to a group both watching the video and attending an in-person lecture or a group that received only the in-person lecture in this study performed in 2022. Surveys gathered demographic information, tested knowledge, and identified barriers to reporting adverse events., Results: A total of 83 unique participants responded to the survey out of the 130 students enrolled (64%; 83/130). Among the students completing all of the surveys, the group who watched the Osmosis video had a higher average quiz score (6.46/7) than the lecture group (6.31/7) following the first intervention. Only 25% of respondents agreed or strongly agreed that they knew what to include in a patient safety report and only 10% agreed or strongly agreed that they knew how to access the reporting system., Conclusions: This study suggests virtual preclass video learning can be a beneficial tool to complement traditional lecture-based learning in medical education. Further research is needed to determine the efficacy of long-term video interventions in adverse events.

Objectives: Hypertension (HTN) affects nearly half of US adults. Our multi-institutional survey revealed that Internal Medicine residents lack proficiency in advanced HTN topics. We developed a curriculum to address knowledge gaps in these topics and aimed to assess the effects of the curriculum on residents' confidence, desire for future training, and knowledge in advanced HTN topics., Methods: HTN experts taught four advanced topics in HTN: conducting a workup for secondary HTN, managing HTN in chronic kidney disease, managing HTN in patients who are or may become pregnant, and managing hypertensive urgency (severe asymptomatic HTN) in the outpatient setting. The setting of the curriculum was an ambulatory educational half-day, during which residents rotated through small-group sessions dedicated to each HTN topic. We developed pre-, immediate post-, and 8 weeks postcurriculum surveys assessing residents' confidence and desire for future training in the four topics (4-point Likert scales), and multiple-choice quizzes to assess changes in knowledge. We used repeated-measures analysis of variance to compare means between time points for surveys and quizzes., Results: A total of 112 Internal Medicine residents participated in the curriculum. The mean confidence scores for all four topics increased from 1.79 to 2.61 precurriculum to 2.90 to 3.45 immediately postcurriculum (all P < 0.001) and remained higher (2.53-3.18) than precurriculum at 8 weeks postcurriculum (all P < 0.02). The mean desire for future training scores decreased from 2.74 to 2.96 precurriculum to 2.06 to 2.36 immediately postcurriculum (all P < 0.001 except for managing HTN in patients who are or may become pregnant, which was P = 0.17) and remained lower (2.08-2.36) than precurriculum at 8 weeks postcurriculum (all P ≤ 0.003). The mean knowledge score increased from 48% precurriculum to 62% immediate postcurriculum ( P < 0.001) and remained higher (55%) than precurriculum at 8 weeks postcurriculum ( P = 0.015)., Conclusions: A curriculum on advanced HTN topics produced durable gains in confidence and knowledge and partially satisfied the desire for future learning among Internal Medicine residents.

Background Recent advancements, including image processing capabilities, present new potential applications of large language models such as ChatGPT (OpenAI), a generative pretrained transformer, in radiology. However, baseline performance of ChatGPT in radiology-related tasks is understudied. Purpose To evaluate the performance of GPT-4 with vision (GPT-4V) on radiology in-training examination questions, including those with images, to gauge the model's baseline knowledge in radiology. Materials and Methods In this prospective study, conducted between September 2023 and March 2024, the September 2023 release of GPT-4V was assessed using 386 retired questions (189 image-based and 197 text-only questions) from the American College of Radiology Diagnostic Radiology In-Training Examinations. Nine question pairs were identified as duplicates; only the first instance of each duplicate was considered in ChatGPT's assessment. A subanalysis assessed the impact of different zero-shot prompts on performance. Statistical analysis included χ 2 tests of independence to ascertain whether the performance of GPT-4V varied between question types or subspecialty. The McNemar test was used to evaluate performance differences between the prompts, with Benjamin-Hochberg adjustment of the P values conducted to control the false discovery rate (FDR). A P value threshold of less than.05 denoted statistical significance. Results GPT-4V correctly answered 246 (65.3%) of the 377 unique questions, with significantly higher accuracy on text-only questions (81.5%, 159 of 195) than on image-based questions (47.8%, 87 of 182) (χ 2 test, P < .001). Subanalysis revealed differences between prompts on text-based questions, where chain-of-thought prompting outperformed long instruction by 6.1% (McNemar, P = .02; FDR = 0.063), basic prompting by 6.8% ( P = .009, FDR = 0.044), and the original prompting style by 8.9% ( P = .001, FDR = 0.014). No differences were observed between prompts on image-based questions with P values of .27 to >.99. Conclusion While GPT-4V demonstrated a level of competence in text-based questions, it showed deficits interpreting radiologic images. © RSNA, 2024 See also the editorial by Deng in this issue.

Stop the Bleed® is an emergency response course that offers a certificate of completion but no ending assessment. The purpose of this educational study was to develop and test the Response Readiness Tool (RRT) that measures learning of participants after taking Stop the Bleed®. The study used a pre-/post-test design to measure knowledge and attitudes, and a post-test only for skills. Participants were recruited from existing Stop the Bleed® courses. Knowledge was measured with a 10-item questionnaire, attitude was measured with five Likert-style questions, and skills were measured by observing simulated tourniquet placement. Ninety-five participants were recruited over two semesters. There was a significant difference in the knowledge and attitude scores indicating participants' improvement in learning and attitude toward responding. Internal consistency reliability of scores showed moderate reliability with Cronbach's Alpha of 0.73 and McDonald's Omega of 0.75. A positive correlation was found between expected proficiency and pre-test indicating the tool had construct validity. RRT provides Stop the Bleed® instructors with an instrument to measure knowledge, skills, and attitude of participants after taking Stop the Bleed®. Statistics show moderate reliability and validity; however, larger samples are needed for full psychometric testing. This publication presents the revised tool after the completion of this study., (© 2024 Wiley Periodicals LLC.)

Abstract: Ensuring students are prepared for the Physician Assistant National Certification Exam (PANCE) is an institutional priority for all physician assistant programs. While many factors predicting PANCE performance have been investigated, exploration and discussions of interventions to improve PANCE performance are limited. Repeated exposure to board-style questions has been shown to improve board scores in similar populations. Currently, existing question banks cannot be used to generate secure summative examinations, and commercial question banks may perpetuate inequitable access among physician assistant (PA) students. To address this issue, the University of Oklahoma Physician Assistant program used a community of practice model to create a mock board exam writing group. The group was created to facilitate the development of 9 PANCE-style exams to bolster students' clinical preparedness and PANCE performance. These writing groups also provided a platform for junior faculty to receive feedback and guidance from senior colleagues, thus facilitating and promoting mentorship. This article highlights the potential benefits of mock board exam writing groups in PA education and provides insight into their development and implementation., Competing Interests: The authors declare no conflict of interest., (Copyright © 2024 PA Education Association.)

The overwhelming majority of states require pharmacists to pass the Multistate Pharmacy Jurisprudence Exam (MPJE) prior to licensure, and student performance in the pharmacy law course is the biggest predictor of success on the MPJE. However, wide variation exists regarding the structure and delivery of the pharmacy law course. Evidence-based learning strategies are well described in the literature, including: (1) practice testing, (2) distributed practice, (3) interleaved practice, (4) self-explanation, and (5) interrogative elaboration, and an opportunity may exist to implement them into the pharmacy law course. We translated these five evidence-based learning principles into practice for pharmacy law educators with specific examples. Faculty directing the pharmacy law course may consider implementing evidence-based teaching strategies into their pharmacy law course to further support student success., Competing Interests: Declaration of competing interest Funding: This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Abstract: With artificial intelligence (AI) rapidly advancing, advanced practice nurses must understand and use it responsibly. Here, we describe an assignment in which Doctor of Nursing Practice (DNP) students learned to use generative text AI. Using our program and course outcomes, developed from the 2021 American Association of Colleges of Nursing (AACN) Essentials competency for DNP students to learn and use AI, we reviewed the literature seeking examples using ChatGPT for the DNP informatics course. No published examples existed to guide us toward infusing a ChatGPT assignment into the course. We developed a novel assignment that included a guide for students on how to use ChatGPT. Students were given time before the assignment to learn the AI/chatbot technology. They were then given the assignment and grading rubric. The assignment was to develop a tool for their current or future practice using ChatGPT. During the course faculty debrief, we learned that few students had questions and the assignment was clear. We also learned that students who sought to develop straightforward, uncomplicated patient tools succeeded with the technology. Those who sought to create something for complex patients had more challenges. Nursing education and practice will be influenced by the increasing prevalence of AI. This manuscript outlines an AI-based assignment for graduate nursing education intended for the students to become familiar with current AI and best practices for patient care. The assignment was well received by students. We plan to use it again in the next course offering., Competing Interests: Competing interests: The authors report no conflicts of interest., (Copyright © 2024 American Association of Nurse Practitioners.)

Purpose: To determine whether students' self-reported race/ethnicity and sex were associated with grades earned in 7 core clerkships. A person-centered approach was used to group students based on observed clerkship grade patterns. Predictors of group membership and predictive bias by race/ethnicity and sex were investigated., Method: Using data from 6 medical student cohorts at Johns Hopkins University School of Medicine (JHUSOM), latent class analysis was used to classify students based on clerkship grades. Multinomial logistic regression was employed to investigate if preclerkship measures and student demographic characteristics predicted clerkship performance-level groups. Marginal effects for United States Medical Licensing Exam (USMLE) Step 1 scores were obtained to assess the predictive validity of the test on group membership by race/ethnicity and sex. Predictive bias was examined by comparing multinomial logistic regression prediction errors across racial/ethnic groups., Results: Three clerkship performance-level groups emerged from the data: low, middle, and high. Significant predictors of group membership were race/ethnicity, sex, and USMLE Step 1 scores. Black or African American students were more likely (odds ratio [OR] = 4.26) to be low performers than White students. Black or African American (OR = 0.08) and Asian students (OR = 0.41) were less likely to be high performers than White students. Female students (OR = 2.51) were more likely to be high performers than male students. Patterns of prediction errors observed across racial/ethnic groups showed predictive bias when using USMLE Step 1 scores to predict clerkship performance-level groups., Conclusions: Disparities in clerkship grades associated with race/ethnicity were found among JHUSOM students, which persisted after controlling for USMLE Step 1 scores, sex, and other preclerkship performance measures. Differential predictive validity of USMLE Step 1 exam scores and systematic error predictions by race/ethnicity show predictive bias when using USMLE Step 1 scores to predict clerkship performance across racial/ethnic groups., (Copyright © 2024 the Association of American Medical Colleges.)

Background: Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation., Methods: Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis., Results: Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment., Conclusion: This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery., (© 2024. The Author(s).)

Background and Purpose: Specifications grading is a mastery-based grading approach to unlock student potential and empower students to focus on learning goals while receiving and acting on meaningful feedback. Within specifications grading, bundles are created to group assignments and assessments. Based on student achievement within each bundle, overall course grade is determined. This article describes the development and implementation of a specifications grading schema in a required skills-based course series, along with lessons learned., Educational Activity and Setting: In a longitudinal course series with both a didactic and lab component, specifications grading was utilized for determination of the overall course grade. Key components of the specifications grading schema were defined by assignment bundles. Assignment bundles aligned with knowledge and skills taught and assessed in each course and also included summative capstone assessments. Each bundle was assigned a numeric grade linked to a letter grade which determined the students' final grade in the course., Findings: Following first course offerings, several changes to the specifications grading schema were made to improve tracking of assignments and activities, to improve consistency across courses, and to aid in final course grade determination. All quizzes were changed to optional, formative quizzes to encourage student accountability. Additional changes were made to the processes of capstone remediation and reassessment, which led to changes in language of the grading schema., Summary: Developing and implementing specifications grading was a crucial first step in building a required skills-based course series, which led to further refinement and improvement for future course offerings., Competing Interests: Declaration of competing interest The authors have no financial disclosures to declare and no conflicts of interest to report. The authors have no financial interests and have not received any funding for anything mentioned in this manuscript., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background and Objectives: As application to residency programs becomes increasingly competitive, educational leaders face growing student concern about imprecise clinical assessments and clerkship grades., Methods: As part of a large annual survey of family medicine clerkship directors (FMCDs), 10 questions were disseminated in May 2023 about perceived levels of imprecise assessments by faculty. We aimed to determine to what extent respondents felt their institution's evaluation system propagated inaccurate grading., Results: A total of 52% of 169 FMCDs responded to the survey. Of these, 7% of respondents were completely confident that their preceptors would give two students of identical competence the same clinical evaluation rating. FMCDs estimated that an average of 38% of their preceptors inaccurately rate student performance. Most clerkships use an Honors/High Pass/Pass/Fail grading system. We found that 51% of FMCDs prefer to use a different grading paradigm than they currently use. We asked FMCDs to estimate the percentage of students that expressed concern over inaccurate preceptor ratings. Grading systems with more tiers were associated with a higher percentage of concerned students., Conclusions: Clerkship grades are widely used by residency program directors to classify and differentiate student applicants. We identified a significant concern from FMCDs that clinical evaluation ratings can vary greatly. Given the high stakes and perceived inaccuracy of clerkship grading, we recommend continued investigation into the appropriate weighing and usage of clinical evaluations. Continued exploration is recommended to develop grading paradigms centered on criterion-based assessment.

Introduction: The retention of resuscitation skills is a widespread concern, with a rapid decay in competence frequently following training. Meanwhile, training programmes continue to be disconnected with real-world expectations and assessment designs remain in conflict with the evidence for sustainable learning. This study aimed to evaluate a programmatic assessment pedagogy which employed entrustment decision and the principles of authentic and sustainable assessment (SA)., Methods: We conducted a prospective sequential explanatory mixed methods study to understand and address the sustainable learning challenges faced by final-year undergraduate paramedic students. We introduced a programme of five authentic assessments based on actual resuscitation cases, each integrating contextual elements that featured in these real-life events. The student-tutor consensus assessment (STCA) tool was configured to accommodate an entrustment scale framework. Each test produced dual student led and assessor scores. Students and assessors were surveyed about their experiences with the assessment methodologies and asked to evaluate the programme using the Ottawa Good Assessment Criteria., Results: Eighty-four students participated in five assessments, generating dual assessor-only and student-led results. There was a reported mean score increase of 9% across the five tests and an 18% reduction in borderline or below scores. No statistical significance was observed among the scores from eight assessors across 420 unique tests. The mean student consensus remained above 91% in all 420 tests. Both student and assessor participant groups expressed broad agreement that the Ottawa criteria were well-represented in the design, and they shared their preference for the authentic methodology over traditional approaches., Conclusion: In addition to confirming local sustainability issues, this study has highlighted the validity concerns that exist with conventional resuscitation training designs. We have successfully demonstrated an alternative pedagogy which responds to these concerns, and which embodies the principles of SA, quality in assessment practice, and the real-world expectations of professionals., (© 2024 The Authors. Journal of Evaluation in Clinical Practice published by John Wiley & Sons Ltd.)

Background: While multiple studies have tested the ability of large language models (LLMs), such as ChatGPT, to pass standardized medical exams at different levels of training, LLMs have never been tested on surgical sub-specialty examinations, such as the American Shoulder and Elbow Surgeons (ASES) Maintenance of Certification (MOC). The purpose of this study was to compare results of ChatGPT 3.5, GPT-4, and fellowship-trained surgeons on the 2023 ASES MOC self-assessment exam., Methods: ChatGPT 3.5 and GPT-4 were subjected to the same set of text-only questions from the ASES MOC exam, and GPT-4 was additionally subjected to image-based MOC exam questions. Question responses from both models were compared against the correct answers. Performance of both models was compared to corresponding average human performance on the same question subsets. One sided proportional z-test were utilized to analyze data., Results: Humans performed significantly better than Chat GPT 3.5 on exclusively text-based questions (76.4% vs. 60.8%, P = .044). Humans also performed significantly better than GPT 4 on image-based questions (73.9% vs. 53.2%, P = .019). There was no significant difference between humans and GPT 4 in text-based questions (76.4% vs. 66.7%, P = .136). Accounting for all questions, humans significantly outperformed GPT-4 (75.3% vs. 60.2%, P = .012). GPT-4 did not perform statistically significantly betterer than ChatGPT 3.5 on text-only questions (66.7% vs. 60.8%, P = .268)., Discussion: Although human performance was overall superior, ChatGPT demonstrated the capacity to analyze orthopedic information and answer specialty-specific questions on the ASES MOC exam for both text and image-based questions. With continued advancements in deep learning, LLMs may someday rival exam performance of fellowship-trained surgeons., (Copyright © 2024 Journal of Shoulder and Elbow Surgery Board of Trustees. Published by Elsevier Inc. All rights reserved.)

Purpose: With the popularization of ChatGPT (Open AI, San Francisco, California, United States) in recent months, understanding the potential of artificial intelligence (AI) chatbots in a medical context is important. Our study aims to evaluate Google Gemini and Bard's (Google, Mountain View, California, United States) knowledge in ophthalmology., Methods: In this study, we evaluated Google Gemini and Bard's performance on EyeQuiz, a platform containing ophthalmology board certification examination practice questions, when used from the United States (US). Accuracy, response length, response time, and provision of explanations were evaluated. Subspecialty-specific performance was noted. A secondary analysis was conducted using Bard from Vietnam, and Gemini from Vietnam, Brazil, and the Netherlands., Results: Overall, Google Gemini and Bard both had accuracies of 71% across 150 text-based multiple-choice questions. The secondary analysis revealed an accuracy of 67% using Bard from Vietnam, with 32 questions (21%) answered differently than when using Bard from the US. Moreover, the Vietnam version of Gemini achieved an accuracy of 74%, with 23 (15%) answered differently than the US version of Gemini. While the Brazil (68%) and Netherlands (65%) versions of Gemini performed slightly worse than the US version, differences in performance across the various country-specific versions of Bard and Gemini were not statistically significant., Conclusion: Google Gemini and Bard had an acceptable performance in responding to ophthalmology board examination practice questions. Subtle variability was noted in the performance of the chatbots across different countries. The chatbots also tended to provide a confident explanation even when providing an incorrect answer., (© 2024. The Author(s), under exclusive licence to The Royal College of Ophthalmologists.)

Physical therapy students must learn about heart transplantation. They need to know how to care for these patients' emotions and needs. The study aimed to compare the effectiveness of a narrative photography (NP) program and a traditional learning (TL) program in physical therapy students' knowledge, satisfaction, empathy, and moral sensitivity. A two-armed assessor-blinded randomized controlled trial was carried out. One hundred and seventeen physical therapy students participated in the study. They were divided into two groups: (i) NP group (n = 56) and (ii) TL group (n = 61). At the end of the program, NP group's knowledge increased when compared with the TL group (p = 0.02). 90.57% of the sample was very satisfied/satisfied with the NP method, and 88.68% felt that NP helped them to understand the importance of considering subjective realities. In conclusion, NP improved knowledge and satisfaction compared with the TL group. These results suggest that NP may be a useful method to improve the academic outcomes of physical therapy students in the heart transplantation field; thus, NP may be considered a teaching-learning methodology of choice in physical therapy students., (© 2024 The Author(s). Nursing & Health Sciences published by John Wiley & Sons Australia, Ltd.)

Teaching physiology can be challenging as students are initially required to understand basic and abstract concepts. Thus students typically view physiology as a "difficult" subject and place an emphasis on rote learning and memorization. Here, we attempted to address this knowledge gap by introducing a pedagogical intervention into the neurophysiology lesson plan of first-year medical and health physiology students at the University of La Réunion. This intervention aimed to better link abstract concepts (e.g., saltatory conduction) and a pathological disorder (multiple sclerosis), together with a discussion of a specific therapeutic intervention (fampridine). Students were required to complete readings (focused on neurophysiology aspects) and two online quizzes before two scheduled in-person lectures. They could also pose questions on a dedicated online forum. Thereafter, the in-person lectures discussed questions posted on the online forum, provided feedback on poorly answered questions (from the online quizzes), and dealt with questions posed by students attending classes. Student feedback regarding the pedagogic intervention was assessed by an anonymous online survey. This survey revealed that the pedagogical intervention was positively received. For example, 94% of respondents agreed the course was well developed, while 80% indicated that the pedagogical intervention was beneficial in terms of their understanding of basic and abstract neurophysiology concepts. Together, this pedagogical intervention was enthusiastically received by the students who better understood how basic nerve physiology concepts fit into the broader context and that such an understanding can result in the development and the roll-out of unique therapeutic interventions for multiple sclerosis. NEW & NOTEWORTHY First-year physiology students can find the subject challenging, struggling to understand abstract concepts without any context. To address this, we introduced a pedagogical intervention for first-year medical and health physiology students that aimed to link abstract concepts and a pathological disorder, together with a discussion of a specific therapeutic intervention. This pedagogical intervention was well received by first-year physiology students who better understood how basic nerve physiology concepts can be applied within the clinical setting.

Objectives: Effectively assessing dental students' knowledge and skills is an important part of their education. Assessment techniques available to teachers, have become more diverse as new technologies are introduced. This study aimed to investigate what dental educators thought about current and future assessment techniques., Methods: This was a mixed method study; quantitative data being collected at an annual dental educators' conference, via on-screen questions requesting: i) demographic information; ii) type of student assessment currently used; iii) impact of Covid-19 on student assessment, and iv) future perceptions of assessment. Questions were posed in real time using the 'Vevox' platform. Descriptive statistical analysis was employed. Qualitative data were collected on 'post-its' and analysed thematically., Results: 101 participants attended the study. 59.1% (n = 60) had >11 years' experience. 64% (n = 55) considered their summative and formative assessments were effective in developing student learning. Few participants reported students' involvement in creating their own assessments (n = 8; 9.75%). 41.68% (n = 36) participants agreed it was a good idea for students to be involved in developing assessments, but only 9.75% (n = 8) reported it was already happening. Qualitative data indicated a reluctance to wholeheartedly embrace technology (including AI) when planning assessments. Most participants were not in favour of Co-creation or the use of some technologies., Conclusions: Participants, reflected on the challenges of adopting a universal assessment approach. Dental educators should be given the institutional support and autonomy to implement the best assessment methods for their students. Most of this cohort of experienced dental educators reported that assessments facilitated student learning., Clinical Significance: The role of assessment for dental students is essential as dental educators must ensure that dental students graduate with appropriate knowledge and technical skills to be safe when in general dental practice. The use of various assessment techniques needs to reflect the diversity of skills the students need to demonstrate., Competing Interests: Declaration of competing interest None of the authors have any conflict of interest, during the data collection, preparation, writing and production of this manuscript. No fees were involved in this research project., (Copyright © 2024 The Author(s). Published by Elsevier Ltd.. All rights reserved.)

Abstract: Letters of reference (LORs) are a common component of the application process for residency training programs. With the United States Medical Licensing Examination Step 1 transitioning to pass/fail grading and with the increasing use of holistic review, the potential role of LORs is rising in importance. Among some key benefits are the ability to provide a broader and more holistic view of applicants, which can include highlighting elements of experiences or skills that could be missed in their application, as well as providing a third-party assessment of the applicant external to their rotation experiences. However, LORs also face issues, including variation in quality, challenges with comparability, and risk of bias. In this article, the authors discuss the unique benefits, limitations, and best practice recommendations for LORs in academic medicine. The authors also discuss future directions, including the role of artificial intelligence, unblinded, and co-created LORs., (Copyright © 2024 the Association of American Medical Colleges.)

It is essential for modern medical students to continuously enhance their clinical thinking abilities. This study aims to evaluate the efficacy of the combined World Café discussion and case-based learning (CBL) approach within the clinical thinking training course. The clinical thinking training course incorporated the combined World Café discussion and CBL approach. The assessment of the accuracy and rationality of clinical symptoms, medical examination, pathological processes, diagnostic results, diagnostic basis, and drug use was conducted through case-related queries. Feedback from students and instructors regarding the teaching content, teaching process, and teaching effect was gathered through questionnaires. The findings indicate that the students achieved high marks in all assessed areas, including clinical symptoms, medical examination, pathological processes, diagnostic results, diagnostic basis, and drug use. The feedback from students and instructors on the teaching content, teaching process, and teaching effect was positive. Medical educators can use our findings to implement the combined World Café discussion and CBL mode to enhance student engagement. NEW & NOTEWORTHY The combined World Café discussion and case-based learning approach was implemented in the clinical thinking training course. Students' scores for clinical symptoms, medical examination, pathological process, diagnostic results, diagnostic basis, and drug use were all excellent. Feedback from both students and teachers on the teaching content, teaching process, and teaching effect was positive.

In physiology education, students must learn to recognize and construct causal explanations. This challenges students, in part, because causal explanations in biology manifest in different varieties. Unlike other natural sciences, causal mechanisms in physiology support physiological functions and reflect biological adaptations. Therefore, students must distinguish between questions that prompt a proximate or an ultimate explanation. In the present investigation, we aimed to determine how these different varieties of student knowledge coordinate within students' written explanations. Prior research in science education demonstrates that students present specific challenges when distinguishing between proximate and ultimate explanations: students appear to conflate the two or construct other nonmechanistic explanations. This investigation, however, demonstrates that analytic frameworks can distinguish between students' proximate and ultimate explanations when they are provided explanatory scaffolds that contextualize questions. Moreover, these scaffolds and prompts help students distinguish between physiological functions and the cellular and molecular mechanisms that underpin them. Together, these findings deliver insight into the context-sensitive nature of student knowledge in physiology education and offer an analytic framework for identifying and characterizing student knowledge in physiology. NEW & NOTEWORTHY Why ask why? How questions posed in physiology task students with developing their mechanistic reasoning. Why questions sometimes undermine this reasoning. Prior research, however, also illustrates that framing the context of a question explicitly supports students in distinguishing between question types. We further illustrate how providing such context in the form of explanatory scaffolds and prompts allows students to tap different and useful varieties of knowledge when constructing written explanations.

Objective: To identify factors influencing patient responses in potentially sensitive situations that might lead to embarrassment (defined by politeness theory (PT) as positive face-threatening acts [FTAs]) or a sense of imposition (defined by PT as negative FTAs) during Objective Structured Clinical Examinations (OSCEs) and to assess the participant's ability to mitigate such situations., Methods: Nineteen OSCE video recordings of 10 pharmacy trainees interacting with mock patients were examined using the PT framework. All relevant participants' speech acts were coded and quantified into type of FTAs and the mitigation strategies used. Patient (assessor) responses were classified then quantified into preferred responses (ie, quick response) vs dispreferred (ie, delayed or hesitant responses) using conversation analysis. The chi-square test was used to identify any association between relevant variables according to predefined hypotheses using SPSS version 27., Results: A total of 848 FTAs were analyzed. Participants failed to meet patient face needs in 32.4% of positive FTAs, in 11.5% of negative FTAs, and 44.4% of positive and negative FTAs. Although patients disclosing information about any inappropriate lifestyle behavior (as per OSCE scripts) expressed these via dispreferred mannerisms, participants were less likely to provide patients with reassurance when patient face needs were challenged in this way (68.2% of these dispreferred responses were not given reassuring feedback) than when they were maintained., Conclusion: Improving educational programs to include the context of patient face needs and conversational strategies for properly dealing with highly sensitive situations was suggested as a way to equip trainees with the skills to effectively build rapport with patients., Competing Interests: Declaration of Competing Interest None declared., (Copyright © 2024 American Association of Colleges of Pharmacy. Published by Elsevier Inc. All rights reserved.)

Demonstration of competency is an essential component of nursing orientation in the practice environment. Standardization of the assessment with a validated instrument supports high-quality care and patient safety. This study engaged six nurse experts to establish content validation for a newly developed competency-based assessment framed by the domains of the national standard for nursing education. Two rounds of rigorous review resulted in a scale content validity index universal agreement calculation of .891, indicating that all items were evaluated as relevant for inclusion in this comprehensive measure of registered nurse clinical competency in the medical-surgical specialty., (Copyright © 2024 Wolters Kluwer Health, Inc. All rights reserved.)