Search

The Extended matching Questions (EMQs), or R-type questions, are format of selected-response. The validity evidence for this format is crucial, but there have been reports of misunderstandings about validity. It is unclear what kinds of evidence should be presented and how to present them to support their educational impact. This review explores the pattern and quality of reporting the sources of validity evidence of EMQs in health professions education, encompassing content, response process, internal structure, relationship to other variables, and consequences. A systematic search in the electronic databases including MEDLINE via PubMed, Scopus, Web of Science, CINAHL, and ERIC was conducted to extract studies that utilize EMQs. The framework for a unitary concept of validity was applied to extract data. A total of 218 titles were initially selected, the final number of titles was 19. The most reported pieces of evidence were the reliability coefficient, followed by the relationship to another variable. Additionally, the adopted definition of validity is mostly the old tripartite concept. This study found that reporting and presenting validity evidence appeared to be deficient. The available evidence can hardly provide a strong validity argument that supports the educational impact of EMQs. This review calls for more work on developing a tool to measure the reporting and presenting validity evidence.

Assessment influences how students define their priorities and their commitment to the learning process. Assessment strategies can empower students to actively engage in metacognitive processes, fostering cross-curricular competencies. Mastery of these competencies not only enhances deep and meaningful learning but also prepares learners for the challenges of the ever-evolving knowledge field. However, developing evaluative judgement, the ability to critically and autonomously judge the quality of one's own work and that of others, is essential but challenging. The purpose of this study was to design and assess an online educational experience for Galenic Pharmacy students ( N  = 339) during the 2021-2022 academic year of the Degree of Pharmacy. Beyond content acquisition, the primary goal was to foster evaluative judgement as a pivotal component of the 'learning to learn' competence. A complex task with iterative deliverables was proposed, using peer assessment as the central tool for the development of evaluative judgement. Students were required to give presentations on methods of administering medicines and had iterative deliverables. They underwent multiple rounds of peer feedback. The task as well as peer assessment process were compulsory for all the students. The participating students voluntarily answered an ad hoc online questionnaire in relation to their perception of the overall experience of peer assessment. The outcomes showcased the positive impacts of peer assessment based on the roles of assessor and assessed. An improvement in feedback quality was observed from one iteration to another, and an enhancement of critical judgement was evident. Enhancing assessment literacy might be essential for both educators and students. For educators, this would allow them to set criteria more aligned with competencies, whereas students might place higher value on these practices and actively engage with the learning process. Such engagement is crucial for promoting lifelong autonomous and self-regulated learning.

Introduction: Addressing systemic bias in medical school assessment is an urgent task for medical education. This paper outlines recommendations on topic areas for further research on systemic bias, developed from a workshop discussion at the 2023 annual meeting of the Society of Directors of Research in Medical Education., Materials and Methods: During the workshop, directors engaged in small-group discussions on guidelines to address bias in assessment practices following a proposed categorization of 'Do's,' 'Don'ts,' and 'Don't knows' and listed their insights using anonymous sticky notes, which were shared and discussed with the larger group of participants. The authors performed a content analysis of the notes through deductive and inductive coding. We reviewed and discussed our analysis to reach consensus., Results: The workshop included 31 participants from 28 institutions across the US and Canada, generating 51 unique notes. Participants identified 23 research areas in need of further study. The inductive analysis of proposed research areas revealed four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches., Discussion: While limited to data from a single workshop, the results offered perspectives about areas for further research shared by a group of directors of medical education research units from diverse backgrounds. The workshop produced valuable insights into the need for more evidence-based interventions that promote more equitable assessment practices grounded in real-world situations and that attenuate the effects of bias.

The COVID-19 pandemic triggered transformations in academic medicine, rapidly adopting remote teaching and online assessments. Whilst virtual environments show promise in evaluating medical knowledge, their impact on examiner workload is unclear. This study explores examiner's workload during different European Diploma in Anaesthesiology and Intensive Care Part 2 Structured Oral Examinations formats. We hypothesise that online exams result in lower examiner's workload than traditional face-to-face methods. We also investigate workload structure and its correlation with examiner characteristics and marking performance. In 2023, examiner's workload for three examination formats (face-to-face, hybrid, online) using the NASA TLX instrument was prospectively evaluated. The impact of examiner demographics, candidate scoring agreement, and examination scores on workload was analysed. The overall NASA TLX score from 215 workload measurements in 142 examiners was high at 59.61 ± 14.13. The online examination had a statistically higher workload (61.65 ± 12.84) than hybrid but not face-to-face. Primary contributors to workload were mental and temporal demands, and effort. Online exams were associated with elevated frustration. Male examiners and those spending more time on exam preparation experienced a higher workload. Multiple diploma specialties and familiarity with European Diploma in Anaesthesiology and Intensive Care exams were protective against high workload. Perceived workload did not impact marking agreement or examination scores across all formats. Examiners experience high workload. Online exams are not systematically associated with decreased workload, likely due to frustration. Despite workload differences, no impact on examiner's performance or examination scores was found. The hybrid examination mode, combining face-to-face and online, was associated with a minor but statistically significant workload reduction. This hybrid approach may offer a more balanced and efficient examination process while maintaining integrity, cost savings, and increased accessibility for candidates.

To offset grade inflation, many clerkships combine faculty evaluations with objective assessments including the Medical Examiners Subject Examination (NBME-SE) or Objective Structured Clinical Examination (OSCE), however, standardized methods are not established. Following a curriculum transition removing faculty clinical evaluations from summative grading, final clerkship designations of fail (F), pass (P), and pass-with-distinction (PD) were determined by combined NBME-SE and OSCE performance, with overall PD for the clerkship requiring meeting this threshold in both. At the time, 90% of students achieved PD on the Internal Medicine (IM) OSCE resulting in overall clerkship grades primarily determined by the NBME-SE. The clerkship sought to enhance the OSCE to provide a more thorough objective clinical skills assessment, offset grade inflation, and reduce the NBME-SE primary determination of the final clerkship grade. The single-station 43-point OSCE was enhanced to a three-station 75-point OSCE using the Reporter-Interpreter-Manager-Educator (RIME) framework to align patient encounters with targeted assessments of progressive skills and competencies related to the clerkship rotation. Student performances were evaluated pre- and post-OSCE enhancement. Student surveys provided feedback about the clinical realism of the OSCE and the difficulty. Pre-intervention OSCE scores were more tightly clustered ( SD  = 5.65%) around a high average performance with scores being highly negatively skewed. Post-intervention OSCE scores were more dispersed ( SD  = 6.88%) around a lower average with scores being far less skewed resulting in an approximately normal distribution. This lowered the total number of students achieving PD on the OSCE and PD in the clerkship, thus reducing the relative weight of the NMBE-SE in the overall clerkship grade. Student response was positive, indicating the examination was fair and reflective of their clinical experiences. Through structured development, OSCE assessment can provide a realistic and objective measurement of clinical performance as part of the summative evaluation of students.

For diversity to exist in the medical graduate workforce, students from all backgrounds should have equitable opportunities of employment. Specialties have utilized a minimal threshold for USMLE Step 1 score when screening applicants for residency interviews. The OHSU SOM class of 2021 completed a 14-question voluntary survey on their Step 1 score and the following non-modifiable risk factors: Adverse Childhood Experience score (ACEs), sex, gender, Underrepresented in Medicine status (URiM), family income during adolescence, highest degree held by a guardian, discrimination experience during medical school, federal/state assistance use, and rural versus urban primary home. Descriptive statistics and unadjusted risk ratios were applied to study the relation between Step 1 score and non-modifiable risk factors as well as certain non-modifiable risk factors and ACEs ≥ 3. The mean Step 1 score was 230 (213, 247). Of the students, 28.2% identified ACEs ≥ 3, 13.6% were considered URiM, and 65.4% were female. URiM were 2.34 (1.30, 4.23),females were 2.77 (1.06-7.29), and those who experienced discrimination in medical school were 4.25 (1.85, 9.77) times more likely to have ACEs ≥ 3. Students who had ACEs ≥ 3 were 3.58 (1.75, 7.29) times less likely to meet a minimal threshold for residency interviews of 220. These are the first results to demonstrate a relationship between Step 1 score and ACEs. Those who identified as URiM, females, and those who experienced discrimination in medical school were at a higher risk of ACEs of ≥ 3. Step 1 transitioned to pass/fail in January 2022. However, the first application cycle that residencies will see pass/fail scoring is 2023-2024, and fellowships will continue to see scored Step 1 until, at the earliest, the 2026-2027 application cycle. These data contribute to a foundation of research that could apply to Step 2CK testing scores, and help to inform decisions about the diversity and equity of the residency interview process.

Background: Video-recordings review of patient encounters is reported to improve the clinical performance of medical students. However, evidence on specific remediation strategies or outcomes are lacking. We aimed to implement videorecording-based remediation of standardized patient encounters among medical students, combined with preceptor one-on-one feedback or peer group discussion, and evaluate the effectiveness of the two remediation methods using objective structured clinical examination (OSCE)., Methods: Following standardized patient encounters, 107 final-year medical students were divided into two groups based on different remediation methods of video review: (1) precepted video review with preceptor feedback ( N  = 55) and (2) private video review and subsequent peer group discussion under supervision ( N  = 52). All students underwent twelve-stations of OSCE both before and after the video review. Students' pre- and post-remediation OSCE scores, self-efficacy level in patient encounters, and level of educational satisfaction with each method were assessed and compared between different video-based remediation methods to evaluate their respective effects., Results: After remediation, the total and subcomponent OSCE scores, such as history taking, physical examination, and patient - physician interaction (PPI), among all students increased significantly. Post-remediation OSCE scores showed no significant difference between two remediation methods (preceptor module, 79.6 ± 4.3 vs. peer module, 79.4 ± 3.8 in the total OSCE score). Students' self-efficacy levels increased after remediation in both modules (both p-value <0.001), with no difference between the two modules. However, students' satisfaction level was higher in the preceptor module than in the peer module (80.1 ± 17.7 vs. 59.2 ± 25.1, p-value <0.001). Among students with poor baseline OSCE performance, a prominent increase in PPI scores was observed in the preceptor-based module., Conclusion: Video-based remediation of patient encounters, either through preceptor review with one-on-one feedback or through private review with peer discussion, was equally effective in improving the OSCE scores and self-efficacy levels of medical students. Underperforming students can benefit from precepted video reviews for building PPI.

While objective clinical structured examination (OSCE) is a worldwide recognized and effective method to assess clinical skills of undergraduate medical students, the latest Ottawa conference on the assessment of competences raised vigorous debates regarding the future and innovations of OSCE. This study aimed to provide a comprehensive view of the global research activity on OSCE over the past decades and to identify clues for its improvement. We performed a bibliometric and scientometric analysis of OSCE papers published until March 2024. We included a description of the overall scientific productivity, as well as an unsupervised analysis of the main topics and the international scientific collaborations. A total of 3,224 items were identified from the Scopus database. There was a sudden spike in publications, especially related to virtual/remote OSCE, from 2020 to 2024. We identified leading journals and countries in terms of number of publications and citations. A co-occurrence term network identified three main clusters corresponding to different topics of research in OSCE. Two connected clusters related to OSCE performance and reliability, and a third cluster on student's experience, mental health (anxiety), and perception with few connections to the two previous clusters. Finally, the United States, the United Kingdom, and Canada were identified as leading countries in terms of scientific publications and collaborations in an international scientific network involving other European countries (the Netherlands, Belgium, Italy) as well as Saudi Arabia and Australia, and revealed the lack of important collaboration with Asian countries. Various avenues for improving OSCE research have been identified: i) developing remote OSCE with comparative studies between live and remote OSCE and issuing international recommendations for sharing remote OSCE between universities and countries; ii) fostering international collaborative studies with the support of key collaborating countries; iii) investigating the relationships between student performance and anxiety.

Chat Generative Pretrained Transformer (ChatGPT) is a natural language processing tool created by OpenAI. Much of the discussion regarding artificial intelligence (AI) in medicine is the ability of the language to enhance medical practice, improve efficiency and decrease errors. The objective of this study was to analyze the ability of ChatGPT to answer board-style cardiovascular medicine questions by using the Medical Knowledge Self-Assessment Program (MKSAP).The study evaluated the performance of ChatGPT (versions 3.5 and 4), alongside internal medicine residents and internal medicine and cardiology attendings, in answering 98 multiple-choice questions (MCQs) from the Cardiovascular Medicine Chapter of MKSAP. ChatGPT-4 demonstrated an accuracy of 74.5 %, comparable to internal medicine (IM) intern (63.3 %), senior resident (63.3 %), internal medicine attending physician (62.2 %), and ChatGPT-3.5 (64.3 %) but significantly lower than cardiology attending physician (85.7 %). Subcategory analysis revealed no statistical difference between ChatGPT and physicians, except in valvular heart disease where cardiology attending outperformed ChatGPT (p = 0.031) for version 3.5, and for heart failure (p = 0.046) where ChatGPT-4 outperformed senior resident. While ChatGPT shows promise in certain subcategories, in order to establish AI as a reliable educational tool for medical professionals, performance of ChatGPT will likely need to surpass the accuracy of instructors, ideally achieving the near-perfect score on posed questions., Competing Interests: Declaration of competing interest The authors report no relationships that could be construed as a conflict of interest., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Background: Large language models (LLMs), including ChatGPT (Chat Generative Pretrained Transformer), a popular, publicly available LLM, represent an important innovation in the application of artificial intelligence. These systems generate relevant content by identifying patterns in large text datasets based on user input across various topics. We sought to evaluate the performance of ChatGPT in practice test questions designed to assess knowledge competency for pediatric emergency medicine (PEM)., Methods: We evaluated the performance of ChatGPT for PEM board certification using a popular question bank used for board certification in PEM published between 2022 and 2024. Clinicians assessed performance of ChatGPT by inputting prompts and recording the software's responses, asking each question over 3 separate iterations. We calculated correct answer percentages (defined as correct in at least 2/3 iterations) and assessed for agreement between the iterations using Fleiss' κ., Results: We included 215 questions over the 3 study years. ChatGPT responded correctly to 161 of PREP EM questions over 3 years (74.5%; 95% confidence interval, 68.5%-80.5%), which was similar within each study year (75.0%, 71.8%, and 77.8% for study years 2022, 2023, and 2024, respectively). Among correct responses, most were answered correctly on all 3 iterations (137/161, 85.1%). Performance varied by topic, with the highest scores in research and medical specialties and lower in procedures and toxicology. Fleiss' κ across the 3 iterations was 0.71, indicating substantial agreement., Conclusion: ChatGPT provided correct answers to PEM responses in three-quarters of cases, over the recommended minimum of 65% provided by the question publisher for passing. Responses by ChatGPT included detailed explanations, suggesting potential use for medical education. We identified limitations in specific topics and image interpretation. These results demonstrate opportunities for LLMs to enhance both the education and clinical practice of PEM., Competing Interests: Disclosure: The authors declare no conflict of interest., (Copyright © 2024 Wolters Kluwer Health, Inc. All rights reserved.)

Objective Structured Clinical Examinations (OSCE) is an objective assessment approach that enables the evaluation of students' practical skills by simulating real clinical scenarios. This scale is specifically designed to evaluate the attributes and characteristics of the OSCE from the perspective of health sciences students, thereby filling a crucial gap in the existing assessment methods. This research conducted a methodological design involving 220 students from the faculties of medicine and nursing between 2022 and 2023. Data were gathered using a "Descriptive Form" and "OSCEPS." Factor analyses were performed for research and validation to ascertain the scale's factorial structure. The factorial analysis was used to test the structural validity of the scale. A structure consisting of 19 items was developed. The Cronbach's alpha coefficient was 0.943. Fit indices resulting from the study were RMSEA = 0.084; GFI = 0.839; CFI = 0.902; IFI = 0.902; RFI = 0.824; NFI = 0.843; TLI = 0.889. This study proves that the OSCEPS is a highly reliable and valid scale with strong psychometric properties. It effectively assesses health sciences students' perceptions of OSCE, thereby enhancing the credibility and usefulness of this assessment tool., (© 2024 The Author(s). Nursing & Health Sciences published by John Wiley & Sons Australia, Ltd.)

The advent of artificial intelligence (AI), particularly large language models (LLMs) like ChatGPT and Gemini, has significantly impacted the educational landscape, offering unique opportunities for learning and assessment. In the realm of written assessment grading, traditionally viewed as a laborious and subjective process, this study sought to evaluate the accuracy and reliability of these LLMs in evaluating the achievement of learning outcomes across different cognitive domains in a scientific inquiry course on sports physiology. Human graders and three LLMs, GPT-3.5, GPT-4o, and Gemini, were tasked with scoring submitted student assignments according to a set of rubrics aligned with various cognitive domains, namely "Understand," "Analyze," and "Evaluate" from the revised Bloom's taxonomy and "Scientific Inquiry Competency." Our findings revealed that while LLMs demonstrated some level of competency, they do not yet meet the assessment standards of human graders. Specifically, interrater reliability (percentage agreement and correlation analysis) between human graders was superior as compared to between two grading rounds for each LLM, respectively. Furthermore, concordance and correlation between human and LLM graders were mostly moderate to poor in terms of overall scores and across the pre-specified cognitive domains. The results suggest a future where AI could complement human expertise in educational assessment but underscore the importance of adaptive learning by educators and continuous improvement in current AI technologies to fully realize this potential. NEW & NOTEWORTHY The advent of large language models (LLMs) such as ChatGPT and Gemini has offered new learning and assessment opportunities to integrate artificial intelligence (AI) with education. This study evaluated the accuracy of LLMs in assessing an assignment from a course on sports physiology. Concordance and correlation between human graders and LLMs were mostly moderate to poor. The findings suggest AI's potential to complement human expertise in educational assessment alongside the need for adaptive learning by educators.

The significance of assessment in driving learning is undeniable, particularly in oral assessments like viva voce where students express themselves verbally. However, traditional viva voce examinations' challenges include reliability, objectivity, content validity, time constraints, varying question difficulty levels, inconsistency in questioning among students, the influence of the "halo" effect, and issues with the test environment. Thus the study aimed to evaluate the effectiveness of Objective, Integrated, Clustered Examination Software (VOICES) compared to traditional viva voce methods. Using a mixed-method design with 103 students, questions were framed using VOICES software created by the National Informatics Center (NIC), Chhattisgarh State Center, Government of India, and students responded through voice messages. Faculty graded students based on personalized videos created by the app. Then, feedback was given, and a focused group discussion was conducted for the students. Traditional viva voce examinations were conducted as part of internal assessments, and the marks obtained from these assessments were compared to evaluate their effectiveness in comparison to the VOICES app. There was no significant difference in the marks obtained using both methods indicating that viva voce using an oral assessment app is equally effective. Students responded positively to the VOICES method. The formative oral assessment conducted through the use of the VOICES app successfully addresses the challenges typically associated with traditional viva voce examinations. NEW & NOTEWORTHY Traditional viva voce examination encounters challenges including reliability, objectivity, content validity, time constraints, varying question difficulty levels, inconsistency in questioning among students, the influence of the "halo" effect, and issues with the test environment. Formative oral assessment conducted through the use of the Objective, Integrated, Clustered Examination Software (VOICES) app developed by the National Informatics Center (NIC) successfully addresses the challenges typically associated with traditional viva voce examinations.

Large language models (LLMs), such as ChatGPT and Bard, have shown potential in various medical applications. This study aimed to evaluate the performance of LLMs, specifically ChatGPT and Bard, in pathology by comparing their performance with those of pathology trainees, and to assess the consistency of their responses. We selected 150 multiple-choice questions from 15 subspecialties, excluding those with images. Both ChatGPT and Bard were tested on these questions across three separate sessions between June 2023 and January 2024, and their responses were compared with those of 16 pathology trainees (8 junior and 8 senior) from two hospitals. Questions were categorized into easy, intermediate, and difficult based on trainee performance. Consistency and variability in LLM responses were analyzed across three evaluation sessions. ChatGPT significantly outperformed Bard and trainees, achieving an average total score of 82.2% compared to Bard's 49.5%, junior trainees' 45.1%, and senior trainees' 56.0%. ChatGPT's performance was notably stronger in difficult questions (63.4%-68.3%) compared to Bard (31.7%-34.1%) and trainees (4.9%-48.8%). For easy questions, ChatGPT (83.1%-91.5%) and trainees (73.7%-100.0%) showed similar high scores. Consistency analysis revealed that ChatGPT showed a high consistency rate of 80%-85% across three tests, whereas Bard exhibited greater variability with consistency rates of 54%-61%. While LLMs show significant promise in pathology education and practice, continued development and human oversight are crucial for reliable clinical application., Competing Interests: Declaration of competing interest None. All authors declare that they have no financial or personal relationships with other people or organizations that could inappropriately influence their work., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background: Self-assessment (SA) is increasingly recognized not only as an assessment method but also as a vital learning activity that enables learners to identify their strengths and limitations, fostering a crucial skill for lifelong learning. Despite its acknowledged significance, there remains a gap in understanding SA for and as learning, rather than merely of learning. This review, therefore, explores two primary questions: (1) What factors inform the implementation of medical students' SA practices? (2) How does SA practices contribute to various educational outcomes?, Methods: A systematic review was conducted across five databases, employing a combination of keywords pertinent to SA and medical education. Our selection criteria deliberately excluded articles that solely focused on the accuracy of SA or used SA exclusively as a measure for other outcome variables. As a result, 39 studies met our inclusion criteria and were analyzed for this review. Qualitative narrative synthesis was used to summarize the informing factors. Furthermore, Kirkpatrick's model was employed to categorize and summarize the effects of SA activities across various educational outcome levels, including reactions, learning, and behavioral changes., Results: Our review reveals that key factors influencing students' SA includes feedback from various stakeholders, peer assessment, the format of SA, and both learner and teacher training. Among the 39 studies examined, the majority demonstrated positive effects of SA on aspects such as attitudes and skills. However, a minority found no significant associations, with these outcomes frequently linked to contextual variables and how SA was implemented., Conclusions: Our review does not focus on SA accuracy or its role as an assessment method; instead it delves into the role of SA as an integral learning practice. We explored its influencing factors and its impact across various outcomes. The findings indicate that for SA to effectively enhance student learning outcomes, it must be implemented with ample support, clear guidelines, and within contexts that encourage feedback and reflection.

Purpose: Competency-based medical education relies on equitable assessment. This study examined the influence of faculty and trainee gender on assessments of internal medicine (IM) resident performance over time., Method: A longitudinal analysis of clinical performance assessments from 7 U.S. IM residency programs (July 2014-June 2019) was conducted. Core competency scores (patient care [PC], medical knowledge [MK], practice-based learning and improvement [PBLI], systems-based practice [SBP], professionalism [PROF], and interpersonal and communication skills [ICS]) were standardized across programs. Cross-classified mixed-effects linear regression evaluated the relationship between gender and standardized competency scores within training programs, while adjusting for multiple variables including IM In Training Examination percentile rank., Results: Data included 9,346 evaluations by 1,011 faculty (552 [55%] men, 459 [45%] women) for 664 residents (358 [54%] men, 306 [46%] women). Initially, women residents' scores were significantly lower than men's in PC (estimated difference [standard error], -0.097 [0.033]; P = .004), MK (-0.145 [0.034], P < .001), and PBLI (-0.090 [0.040], P = .022). PC, MK, PBLI, and SBP scores increased more over time for women residents than men (PC: 0.050 [0.015], P = .001; MK: 0.052 [0.015], P = .001; PBLI: 0.036 [0.018], P = .048; SBP: 0.036 [0.016], P = .027). PROF and ICS scores were comparable across gender. There was a significant interaction between faculty gender and postgraduate year (PGY) across all competencies but none between resident gender, faculty gender, and PGY, indicating that men and women faculty rated residents differently over time but were consistent in how they rated men and women residents., Conclusions: Gender-based assessment differences were variable across competencies and time. Women residents had lower scores initially but greater gains in "hard skill" (MK, PC, and PBLI) than in "soft skill" (ICS and PROF) competencies, suggesting assessment inequities. Efforts to ensure equitable assessment are needed., (Copyright © 2024 Written work prepared by employees of the Federal Government as part of their official duties is, under the U.S. Copyright Act, a “work of the United States Government” for which copyright protection under Title 17 of the United States Code is not available. As such, copyright does not extend to the contributions of employees of the Federal Government.)

Objective: To identify if there is a relationship between the placement of standalone pharmacy law courses within the PharmD curriculum and Multistate Pharmacy Jurisprudence Examination (MPJE) first-time pass rates., Methods: Colleges of pharmacy were identified using the MPJE Passing Rates for 2019-2022 Graduates found on the National Association of Boards of Pharmacy (NABP) website. Characteristics of the pharmacy law content delivery within the curriculum were extracted from the program, Pharmacy College Application Service, American Association of Colleges of Pharmacy (AACP), and NABP websites. Pharmacy programs with standalone law courses, MPJE pass rates reported by NABP, and data that could be obtained via publicly available sources were included. To standardize between three year and four-year programs, law course delivery within the curriculum was measured as number of semesters (fall, spring, or summer) before graduation., Results: One hundred nine schools met the inclusion criteria. Linear path analysis revealed no relationship between the number of semesters a law course was scheduled before graduation and 4-year average first-time MPJE pass rates and 4-year average all-time MPJE pass rates., Conclusion: The findings did not show that earlier placement of pharmacy law courses predicted MPJE first-time pass rates. However, a strong correlation existed between NAPLEX and MPJE pass rates, suggesting NAPLEX performance may indicate overall licensure exam preparedness. Notable differences in pass rates were observed between public and private pharmacy programs, highlighting the need to investigate program characteristics impacting exam success. Further research is warranted to identify predictive factors for MPJE outcomes., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objective: Pharmacists are often the last line of defense from medical errors caused by inaccurate calculations. Effective teaching and assessment of pharmaceutical calculations is essential in preparing students for successful pharmacy careers. This study aimed to elucidate the potential benefit of self-testing practice questions on final examination performance in a first-year pharmaceutical calculations course., Methods: One-hundred and sixteen students across the class of 2026 and 2027 were given access to 110 online practice calculation questions eight days prior to the final examination. Retrospective analysis using Pearson's Correlation Coefficient and an Unpaired t-test was used to assess the effect of self-study practice questions on exam performance., Results: A correlation between higher quiz scores and enhanced final examination scores was observed for both the class of 2026 and 2027. A greater number of attempts on practice quiz questions correlated with a higher score on the final examination for the class of 2026, but not the class of 2027. Also, an earlier first access date was associated with higher final examination scores specifically for the class of 2026., Conclusion: This retrospective study was conducted to evaluate the use of practice calculation questions on final examination performance, and results reveal that the utilization of practice calculation questions positively correlates with improved final examination performance, notably observed in the class of 2026 but not in 2027. These findings suggest the potential efficacy of this preparatory method across various pharmaceutical courses and other calculation-based disciplines internationally., Competing Interests: Declaration of competing interest There are no financial disclosures or conflicts of interest., (Copyright © 2024 The Authors. Published by Elsevier Inc. All rights reserved.)

An educational escape room is a form of game-based learning that has been shown to improve students' learning experience. A cardiovascular escape room was designed to promote the development of transversal skills and help students retain learned content knowledge in an integrated pharmacology and therapeutic unit in a Pharmacy course., Objective: This study aims to design, pilot, and evaluate the students' self-perceived transversal skills, learning experience and knowledge acquisition from a cardiovascular educational escape room., Methods: The model pharmacy at Curtin University was used as the room that students were challenged to escape. Students were required to acquire a four-digit safe code, with each digit derived from four separate briefcases containing different puzzles and clues. This code enabled students to unlock a lifesaving medication to successfully escape the room and rescue a fictional patient who had collapsed outside of the pharmacy. Each session was followed by a debrief to reflect on the learning experience., Results: Fifty-four students participated in the study. Eight out of twelve groups successfully escaped the room. There was negligible difference in the score of the pre- and post-knowledge assessments. All students strongly agreed/agreed that the escape room was an effective way to review the cardiovascular content covered in the lecture and tutorial and that the activity should be continued for future students., Conclusion: Students perceived the education escape room as a valuable learning exercise to help them retain content knowledge and develop transversal skills in the learning of integrated pharmacology and therapeutic within a Pharmacy course., Competing Interests: Declaration of competing interest The author(s) declare that they have no competing interests., (Copyright © 2024 The Authors. Published by Elsevier Inc. All rights reserved.)

Formative assessment is a valuable tool in medical education as it allows students to reflect on their own learning and make necessary corrections. The use of innovative techniques in formative assessment can make it more engaging and effective for students, particularly utilizing electronic tools. A word cloud is a visual representation of data that is created using text inputs from a group of individuals. The resulting image shows the most frequently used words in a larger font, allowing quick understanding of the common themes and concepts in the group's responses. Mentimeter is an online platform for interactive presentations where students can provide their responses to a question, which can then be visualized as a word cloud. The study aims to explore the benefits and drawbacks of using word cloud formation as a formative assessment technique in the classroom, utilizing the online platform Mentimeter. NEW & NOTEWORTHY The current study was conducted to present a novel formative assessment classroom technique using Mentimeter. The benefit of this novel technique is that it is easy to implement, quick, anonymous, and strengthens student's conceptual learning. It also gives the teacher an opportunity to give feedback and stress commonly committed mistakes.

Assessing student mastery is often done by using exams. Inevitably, some students will complete remediation, which may include exam retakes. This method provides students an additional opportunity to take an exam that assesses the same objectives as the original exam, while using different questions. Although this form of remediation increases exam scores, it is also possible that students who prepare for an exam retake adversely affect preparation for an upcoming exam. Therefore, the purpose of this study was to determine whether studying for and taking an exam retake affected preparation for the next exam. We hypothesized that students who completed an exam retake would have lower than expected scores on their next exam. This study utilized data collected over five semesters from 467 students enrolled in a 200-level introductory human physiology class; 159 students (34.0%) completed at least one exam retake. Students who retook an exam increased their original exam score by 6.1% (SD 13.9). These findings suggest that retaking an exam leads to better outcomes, which could be explained by students improving their study habits or test-taking skills, which would help them perform better on future exams. NEW & NOTEWORTHY Exam retakes provide students with an additional opportunity to demonstrate mastery of learning objectives. However, this preparation might adversely affect performance on subsequent exams. This study suggests that students who choose to prepare for and take an exam retake not only improve their original exam score but show a larger improvement on subsequent exam performance than those students who did not take an exam retake.

The teaching of physiology plays a crucial role in the education of health care professionals. However, traditional approaches to physiology classes in undergraduate health courses in Brazil often result in passive student participation. Research has shown that active methodologies are more effective in the learning process. In this study, we introduce the game "Who Am I?-Cellular Signal Transduction Edition" as an educational tool. This game follows a popular format with well-known rules and aims to enhance understanding of basic concepts related to hormones, cell signaling, and the functioning of the endocrine system. Our findings demonstrate that the game improves student knowledge and fosters enthusiasm and active engagement among participants. Additionally, student feedback has indicated a high level of appreciation for the game. By incorporating active learning strategies and a gamified approach, "Who Am I?-Cellular Signal Transduction Edition" provides a practical and enjoyable way of teaching physiology. This innovative educational tool has the potential to revolutionize physiology instruction. Demonstrating significant improvement in students' understanding, the game underscores its efficacy in enhancing knowledge acquisition and comprehension of cellular signaling and endocrine physiology topics. NEW & NOTEWORTHY We developed "Who Am I?-Cellular Signal Transduction Edition" to assist students in comprehending concepts of cellular signal transduction. This simple and cost-effective tool is perfect for educational settings with limited resources, and it encourages active learning for both small and large groups. Pre- and posttests have shown that it effectively enhances knowledge of hormonal actions and cellular signaling. Positive feedback from students emphasizes its value in reinforcing understanding and improving classroom engagement, making it a promising educational tool.

In educational and psychological assessments, benefiting from back random response (BRR) is a major type of rapid guessing in misfitting item score patterns. Person-fit statistics (PFS) based on cumulative sum (CUSUM) and change point analysis (CPA) from statistical process control (SPC) are better than other PFS for detecting aberrant response. In this study, we developed new person-fit statistics based on three algorithms from CPA procedure and CUSUM method for detection of person misfit with dichotomous or polytomous items. By means of simulated data, the effectiveness of the new statistics to detect test-takers with BRR was investigated., (© 2024. The Psychonomic Society, Inc.)

Background: Traditionally, the assessment of student nursing skills relies heavily on lecturer evaluations. However, lecturers, as the primary evaluators, may be influenced by their individual subjective preferences, experience, and knowledge background. This limitation needs thorough consideration to improve students' assessment skills and ensure that assessment tasks are more objective and accurate. Peer assessment appears to take it into account. Currently, improving the quality of peer assessment in Basic Nursing skills remains a challenge for educators., Aim: To assess the effectiveness of peer assessment as applied to the basic nursing skills examination process and suggestions for improvement, to understand nursing students' perceptions of it, and to explore the feasibility of peer assessment as an alternative to single faculty assessment as perceived by nursing students., Design: Qualitative research., Settings: Conducted in a clinical skills simulation laboratory at a university in Wuxi, Jiangsu Province, China., Participants: 23 Second-year nursing students applying peer assessment to Basic Nursing course exams., Methods: Used face-to-face, semi-structured interviews, and thematic analysis to analyze data., Results: Three major themes and nine subthemes were identified: (1) Positive sentiments towards peer evaluation (e.g., enhancing the learning capacity, increasing emotional intelligence, and improving the objectivity and fairness of nursing skills assessment); (2) Challenges encountered in the peer evaluation process (e.g., inconsistent scoring criteria, exacerbate anxiety, and triggering team conflict); (3) Suggestions for improving peer evaluation (e.g., applying the design of peer assessment exam content, incorporation of lecturer evaluation, and refinement and materialization of marking criteria)., Conclusions: In this study, while the positive impact is evident, the challenges faced by nursing students and future recommendations for peer assessment application are critical. This study provides valuable insights into the potential of peer assessment to enhance nursing education and provides recommendations for optimizing peer assessment strategies for meaningful integration into Nursing Education practice., Competing Interests: Declaration of competing interest The authors declare that they have no competing interests, including financial and non-financial interests., (Copyright © 2024. Published by Elsevier Ltd.)

This study aimed to evaluate the performance for answering the Japanese medical physicist examination and providing the benchmark of knowledge about medical physics in language-generative AI with large language model. We used questions from Japan's 2018, 2019, 2020, 2021 and 2022 medical physicist board examinations, which covered various question types, including multiple-choice questions, and mainly focused on general medicine and medical physics. ChatGPT-3.5 and ChatGPT-4.0 (OpenAI) were used. We compared the AI-based answers with the correct ones. The average accuracy rates were 42.2 ± 2.5% (ChatGPT-3.5) and 72.7 ± 2.6% (ChatGPT-4), showing that ChatGPT-4 was more accurate than ChatGPT-3.5 [all categories (except for radiation-related laws and recommendations/medical ethics): p value < 0.05]. Even with the ChatGPT model with higher accuracy, the accuracy rates were less than 60% in two categories; radiation metrology (55.6%), and radiation-related laws and recommendations/medical ethics (40.0%). These data provide the benchmark for knowledge about medical physics in ChatGPT and can be utilized as basic data for the development of various medical physics tools using ChatGPT (e.g., radiation therapy support tools with Japanese input)., Competing Interests: Declarations. Conflicts of interest: Inoue is employees of Elith, inc. Ethical approval: There are no human subjects in this article and informed consent is not applicable., (© 2024. The Author(s), under exclusive licence to Japanese Society of Radiological Technology and Japan Society of Medical Physics.)

Background: The incorporation of digital health technologies in undergraduate teaching, such as the MyDispense program, provides a simulated opportunity to support students' competency development in dispensing tasks., Objective: This study aimed to evaluate the impact of using MyDispense, an online pharmacy simulation platform, on pharmacy students' self-reported reaction, learning, and accuracy in performing dispensing tasks., Methodology: 16 MyDispense cases simulating real-practice scenarios from prescription dispensing and verification to patient counseling were created for 55 professional year 1 pharmacy students as part of a professional skills course. A pre-post survey, nestled on Kirkpatrick's Model of Training Evaluation, was conducted on participants in the cohort to evaluate how the incorporation of MyDispense supported their development of dispensing tasks. A virtual formative assessment was also done using MyDispense to compare students' self-reported accuracy with faculty assessment scores to evaluate how MyDispense can support students' knowledge and metacognitive abilities in dispensing., Results: The study showed an increase in students' confidence after using MyDispense, which led to an increase in the "learning" level of Kirkpatrick's Model. However, small changes were observed in other levels and sublevels of Kirkpatrick's Model. A notable discrepancy was observed between students' self-assessment scores and faculty assessment scores, with students overestimating their performances. Correlation analysis showed a weak relationship between students' self-assessment scores and the learning level of Kirkpatrick's Model., Conclusion: This study highlights the positive impact of MyDispense on pharmacy students' confidence in dispensing. However, virtual simulation training may be best incorporated when students have exposure to experiential training placements, to maximize the learning outcomes and knowledge in dispensing processes., Competing Interests: Declaration of competing interest The authors declare no conflict of interest., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background: This review aimed to analyze the effectiveness of the teaching and assessment methodologies used to develop fundamental clinical skills in pharmacist who are training in patient care., Methods: SCOPUS, EMBASE, ERIC, and PubMed were searched for original studies that featured randomized controlled trials as the study design published until March 2024. The search and extraction process followed PRISMA Guidelines., Results: The database search resulted in 2954 articles, of which 14 met the inclusion criteria. Four studies developed and tested interactive web-based software as the teaching methodologies. Eight studies applied simulation to their teaching and/or evaluation strategies. Two articles used high fidelity simulation, and the remaining studies used standardized patients associated with other teaching and evaluation techniques. The simulation methodologies were more effective than the conventional ones in three studies. In the other studies, the interventions were as effective or better than the control, albeit there no meaningful differences between the methods. In the studies that focused on the assessment methods, immediate feedback was preferred by students over delayed feedback. Additionally, the tested assessment tool, General Level Framework, proposed a pragmatic assessment from which the individual's training needs were identified., Conclusion: Few studies involved the objective quantification of learning beyond pre- and post-intervention knowledge tests. Proper assessment in pharmaceutical education requires expansion beyond the administration of student satisfaction, self-efficacy research tools, and knowledge assessments, and should encompass an examination of clinical performance and critical thinking., Competing Interests: Declaration of competing interest The authors declare that they have no conflicts interests. The funders had no role in the study design, data collection and analysis, decision on the publication or preparation of the manuscript., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Introduction: Doctor of Pharmacy programs are charged with developing students' empathy by the 2016 Accreditation Council for Pharmacy Education (ACPE) Standard 3 and the 2022 Curriculum Outcomes and Entrustable Professional Activities (COEPA). Although empathy is essential to optimal patient care, its subjective nature makes it challenging to teach and therefore literature is lacking on best teaching practices. The authors of this paper describe a novel simulated approach to elicit and assess empathy in a pharmacy classroom. This study evaluated the impact of a decision-making game in a pharmacy skills lab course on the development of students' empathy using a validated empathy scale., Methods: This is a cohort-based quality improvement project in which third year pharmacy students participated in a 3-h classroom empathy game experience that simulated a month in a patient's life including issues related to the cycle of poverty. Prior to the game, students completed a voluntary, anonymous baseline demographics survey. They also completed a pre- and post-survey of the validated empathy tool, the Kiersma-Chen Empathy Scale (KCES-R), to assess change in the empathy score following the decision-making game. Students also provided narrative comments in the post-survey. Statistical tests used included descriptive statistics for demographic data, Shapiro-Wilk test of normality, and Wilcoxon Signed-Rank test for survey scores (SPSS Version 29)., Results: Pharmacy students (n = 37) showed an overall increase in composite KCES-R scores after participating in the empathy game class session (z = -5.071, p < 0.001). The scores of each of the 14 KCES-R items also increased after the learning experience (p < 0.05). Students' narrative comments were all positive and indicated that the activity offered new insights on self-perceived empathy development., Conclusion: The empathy game simulation was a successful approach to increase empathy scores in third-year pharmacy students., Competing Interests: Declaration of competing interest The authors declare no potential conflicts of interest with respect to the study, authorship and/or publication of this article. No funding was received to support the research project nor the preparation of this manuscript., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objective: "Test-enhanced learning" is a strategy utilized to impact knowledge retention. This study aimed to assess the effect of the frequency of spaced retrieval using repeat testing on knowledge depreciation in a Doctor of Pharmacy (PharmD) Program., Methods: The study randomly assigned second-year pharmacy students to one of three retrieval practice groups: (1) two retrieval practice assessments (2 and 4 months after the course), (2) one retrieval practice assessment (4 months after the course), or (3) no retrieval practice. A final assessment was administered to all participants 6 months after the course. The retrieval assessments and final assessment consisted of five multiple-choice questions on asthma pharmacotherapy., Results: In total, 94 participants were included in the study, with 32 in Group 1 and 31 in both Groups 2 and 3. All three groups performed similarly on the final assessment regardless of the frequency of spaced retrieval. While Group 1 scored numerically higher than Group 2 on the 4-month retrieval practice assessment and did not impact performance on the final assessment. In Group 1, final assessment scores were decreased compared to the two previous retrieval assessments., Conclusion: Regardless of the frequency of spaced retrieval following the academic course where asthma pharmacotherapy was initially learned, retrieval practice using multiple choice assessments, with the gap times of this study, did not impact asthma pharmacotherapy knowledge depreciation among second-year student pharmacists., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background and Purpose: A Medication Therapy Management (MTM) encounter allows pharmacists to gather information to make appropriate assessments and recommendations regardless of the patient's language needs and the encounter environment utilized. With the goal of improving comfort and confidence in providing MTM services, working with an interpreter, and providing care in a virtual environment, students participated in a virtual MTM activity with English and non-English speaking patients., Educational Activity and Setting: In groups, students completed two MTM encounters during a virtual skills laboratory. Students were asked to complete a pre- and post-lab survey to gauge their confidence before and after completing these encounters. One encounter was with an English-speaking patient, and one was with a non-English speaking patient which utilized an interpreter for communication. Both encounters were completed using a virtual platform., Findings: As a result of these activities, students reported improved confidence in providing MTM services to a patient who speaks a language different than themselves and managing patients using a virtual platform. Students self-identified that teamwork and pre-encounter preparation were essential for effective MTM services., Summary: Students found completing virtual MTM was a beneficial way to improve comfort and confidence with navigating virtual platforms, gathering medication information, and working with interpreters., Competing Interests: Declaration of competing interest The authors have no financial disclosures or other conflicts of interest to report., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background: While previous research has emphasized the importance of personal beliefs (expectancy-value theories) for achievement-motivated behaviour, it lacks the integration of temporal factors that are also discussed as important drivers of achievement-motivated behaviour. Temporal Motivation Theory (TMT) combines both approaches in a formalized manner., Aims: Although TMT is supported by empirical studies with self-reported academic procrastination, it has not been tested on actual achievement-motivated behaviour., Materials & Methods: We evaluated the predictive power of the TMT on N = 2351 learning days of 127 psychology students' self-regulated examination preparation for statistics over the course of one semester using logfile data of an e-learning system., Results: The proposed TMT score, incorporating expectancy and value beliefs, sensivitiy to delay, and actual time till examination predicted students' achievement-motivated behaviour significantly., Discussion: Further analyses revealed that not the trait compositions of the TMT, but the temporal proximity of the statistics examination was the main driver of this association., Conclusion: The results have important implications for understanding the factors that shape students' motivation to learn and subsequent academic success in actual learning situations. Thus, research should continue to take situational aspects, especially the temporal proximity of goals more into account., (© 2024 The Author(s). British Journal of Educational Psychology published by John Wiley & Sons Ltd on behalf of British Psychological Society.)

Physiology concepts, such as acid-base balance, may be difficult for students to understand. Systems modeling, a cognitive tool, allows students to visualize their mental model of the problem space to enhance learning and retention. We performed a within-subjects three-period randomized control comparison of systems modeling versus written discussion activities in an undergraduate asynchronous online Anatomy and Physiology II course. Participants ( n = 108) were randomized to groups with differing treatment orders across three course units: endocrine, immune, and acid-base balance. Participants demonstrated content understanding through either constructing systems modeling diagrams (M) or written discussion posts (W) in a MWM, MMW, or WMM sequence. For each of three units, student performance was assessed on 6 standardized multiple-choice questions embedded within a 45-question exam. The same 6 questions per unit, 18 questions in total, were again assessed on the 75-question final exam. The groups demonstrated no significant difference in performance in the endocrine unit exam [mean difference (MD) = -0.036]. However, the modeling group outperformed the writing group in the immune unit exam (MD = 0.209) and widened the gap in the acid-base balance unit exam (MD = 0.243). On the final exam, performance was again higher for the modeling group on acid-base balance content, as mean difference increased to 0.306 despite the final exam content for acid-base balance being significantly more difficult compared to other units [modeling: F (2) = 29.882, P < 0.001; writing: F (2) = 25.450, P < 0.001]. These results provide initial evidence that participation in systems modeling activities may enhance student learning of difficult physiology content as evidenced by improved multiple-choice question performance. NEW & NOTEWORTHY Physiology educators often intuitively utilize systems thinking and modeling while teaching difficult concepts. Guiding students in development of their own systems modeling skills by enhancing their visualization of their mental model of the problem space may improve performance on multiple-choice test questions.

Changes in the national examination structures have renewed interest in the development of test-taking strategies for professional students in the health sciences. It is often assumed that these high-achieving students have developed proficient test-taking skills and abilities before admittance. However, the assessments in these programs and for national licensure require an elevated level of reasoning and integration with clinical concepts. It was hypothesized that the implementation of an immersive test-taking strategies program would improve dental student perceptions of their abilities. A "toolbox" of four methods was implemented, which included 1 ) an introductory video that provided students with 10 tips for approaching high-level exam questions; 2 ) problem solution videos interspersed with class practice problems to walk students independently through rationales; 3 ) collaborative group assessments in which students implemented the strategies in teams to prepare for exams; and 4 ) unit exam debriefings to review the question rationales. Although all methods were positively reviewed on surveys, students indicated that the problem solution videos and the collaborative group assessments were more helpful and improved their test-taking skills significantly more than the other strategies ( P < 0.01, 1-way ANOVA and Bonferroni post hoc test). Students felt they had developed strong test-taking strategies (average of 4.21 on a 5-point scale, SD 0.76) and felt more prepared for the Integrated National Board Dental Examination (4.48, SD 0.66). These results suggest that a multipronged approach with frequent opportunities to practice test-taking strategies can improve student perceptions of their ability to master high-level and integrated assessment questions. NEW & NOTEWORTHY Professional students in the health sciences may need to improve their test-taking skills for high-level and integrated assessments. A toolbox of strategies was implemented in a Dental Physiology course that included an introductory video, problem solution videos for practice problems, collaborative group assessments, and unit exam debriefings. The students reported that the strategies were helpful, improving their preparation and test-taking strategies for summative exams such as the Integrated National Board Dental Examination.

Cognitive diagnostic models (CDMs) are a popular family of discrete latent variable models that model students' mastery or deficiency of multiple fine-grained skills. CDMs have been most widely used to model categorical item response data such as binary or polytomous responses. With advances in technology and the emergence of varying test formats in modern educational assessments, new response types, including continuous responses such as response times, and count-valued responses from tests with repetitive tasks or eye-tracking sensors, have also become available. Variants of CDMs have been proposed recently for modeling such responses. However, whether these extended CDMs are identifiable and estimable is entirely unknown. We propose a very general cognitive diagnostic modeling framework for arbitrary types of multivariate responses with minimal assumptions, and establish identifiability in this general setting. Surprisingly, we prove that our general-response CDMs are identifiable under Q -matrix-based conditions similar to those for traditional categorical-response CDMs. Our conclusions set up a new paradigm of identifiable general-response CDMs. We propose an EM algorithm to efficiently estimate a broad class of exponential family-based general-response CDMs. We conduct simulation studies under various response types. The simulation results not only corroborate our identifiability theory, but also demonstrate the superior empirical performance of our estimation algorithms. We illustrate our methodology by applying it to a TIMSS 2019 response time dataset., (© 2024. The Author(s), under exclusive licence to The Psychometric Society.)

For essay-writing tests, challenges arise when scores assigned to essays are influenced by the characteristics of raters, such as rater severity and consistency. Item response theory (IRT) models incorporating rater parameters have been developed to tackle this issue, exemplified by the many-facet Rasch models. These IRT models enable the estimation of examinees' abilities while accounting for the impact of rater characteristics, thereby enhancing the accuracy of ability measurement. However, difficulties can arise when different groups of examinees are evaluated by different sets of raters. In such cases, test linking is essential for unifying the scale of model parameters estimated for individual examinee-rater groups. Traditional test-linking methods typically require administrators to design groups in which either examinees or raters are partially shared. However, this is often impractical in real-world testing scenarios. To address this, we introduce a novel method for linking the parameters of IRT models with rater parameters that uses neural automated essay scoring technology. Our experimental results indicate that our method successfully accomplishes test linking with accuracy comparable to that of linear linking using few common examinees., (© 2024. The Author(s).)

Background: Medical students in the U.S. must demonstrate urgent and emergent care competence before graduation. Urgent and emergent care competence involves recognizing, evaluating and initiating management of an unstable patient. High-fidelity (HF) simulation can improve urgent and emergent care skills, but because it is resource intense, alternative methods are needed., Study Objective: Our primary purpose was to use program evaluations to compare medical student experiences with HF and virtual reality (VR) simulations as assessment platforms for urgent and emergent care skills., Methods: During their emergency medicine clerkship, students at The Ohio State University College of Medicine must demonstrate on HF manikins, competence in recognizing and initiating care of a patient requiring urgent or emergent care. Students evaluated these simulations on a five-point quality scale and answered open-ended questions about simulation strengths and weaknesses. Faculty provided feedback on student competence in delivering urgent or emergent care. In 2022, we introduced VR as an alternative assessment platform. We used Wilcoxon Signed Ranks and Boxplots to compare ratings of HF to VR and McNemar Test to compare competence ratings. Comments were analyzed with summative content analysis or thematic coding., Results: We received at least one evaluation survey from 160 of 216 (74.1%) emergency medicine clerkship students. We were able to match 125 of 216 (57.9%) evaluation surveys for students who completed both. Average ratings of HF simulations were 4.6 of 5, while ratings of VR simulations were slightly lower at 4.4. Comments suggested that feedback from both simulation platforms was valued. Students described VR as novel, immersive, and good preparation for clinical practice. Constructive criticism identified the need for additional practice in the VR environment. Student performance between platforms was significantly different with 91.7% of students achieving competence in HF, but only 65.5% in VR ( p ≤.001, odds-ratio = 5.75)., Conclusion: VR simulation functions similarly to HF for formative assessment of urgent and emergent care competence. However, using VR simulation for summative assessment of urgent and emergent care competence must be considered with caution because students require considerable practice and acclimation to the virtual environment.

Introduction: Many educators have adopted the Promoting Excellence and Reflective Learning in Simulation (PEARLS) model to guide debriefing sessions in simulation-based learning. The PEARLS Debriefing Checklist (PDC), a 28-item instrument, and the PEARLS Debriefing Adherence Rubric (PDAR), a 13-item instrument, assess facilitator adherence to the model. The aims of this study were to collect evidence of concurrent validity and to evaluate their unique strengths., Methods: A review of 130 video recorded debriefings from a synchronous high-fidelity mannequin simulation event involving third-year medical students was undertaken. Each debriefing was scored utilizing both instruments. Internal consistency was determined by calculating a Cronbach's α. A Pearson correlation was used to evaluate concurrent validity. Discrimination indices were also calculated., Results: Cronbach's α values were 0.515 and 0.714 for the PDAR and PDC, respectively, with ≥0.70 to ≤0.90 considered to be an acceptable range. The Pearson correlation coefficient for the total sum of the scores of both instruments was 0.648, with a values between ±0.60 and ±0.80 considered strong correlations. All items on the PDAR had positive discrimination indices; 3 items on the PDC had indices ≤0, with values between -0.2 and 0.2 considered unsatisfactory. Four items on both instruments had indices >0.4, indicating only fair discrimination between high and low performers., Conclusions: Both instruments exhibit unique strengths and limitations. The PDC demonstrated greater internal consistency, likely secondary to having more items, with the tradeoff of redundant items and laborious implementation. Both had concurrent validity in nearly all subdomains. The PDAR had proportionally more items with high discrimination and no items with indices ≤0. A revised instrument incorporating PDC items with high reliability and validity and removing those identified as redundant or poor discriminators, the PDAR 2, is proposed., Competing Interests: The authors declare no conflict of interest., (Copyright © 2024 Society for Simulation in Healthcare.)

With the growing attention on large-scale educational testing and assessment, the ability to process substantial volumes of response data becomes crucial. Current estimation methods within item response theory (IRT), despite their high precision, often pose considerable computational burdens with large-scale data, leading to reduced computational speed. This study introduces a novel "divide- and-conquer" parallel algorithm built on the Wasserstein posterior approximation concept, aiming to enhance computational speed while maintaining accurate parameter estimation. This algorithm enables drawing parameters from segmented data subsets in parallel, followed by an amalgamation of these parameters via Wasserstein posterior approximation. Theoretical support for the algorithm is established through asymptotic optimality under certain regularity assumptions. Practical validation is demonstrated using real-world data from the Programme for International Student Assessment. Ultimately, this research proposes a transformative approach to managing educational big data, offering a scalable, efficient, and precise alternative that promises to redefine traditional practices in educational assessments., Competing Interests: Declarations. Conflict of interest: Each author signed a form for conflict of interest of potential conflict of interest. No authors reported any financial or other conflict of interest in relation to the work described. The author(s) declared no potential conflict of interest with respect to the research, authorship, and/or publication of this article Ethical approval: The authors affirm having followed professional ethical guidelines in preparing this work. These guidelines include obtaining informed consent from human participants, maintaining ethical treatment and respect for the rights of human or animal participants, and ensuring the privacy of participants and their data, such as ensuring that individual participants cannot be identified in reported results or from publicly available original or archival data. Data availability: The real data set was derived from the following resources available in the public domain: http://www.oecd.org/pisa ., (© 2024. The Author(s), under exclusive licence to The Psychometric Society.)

Background: Training of standardized patients (SPs) for national high-stakes OSCE helps to ensure a reliable assessment of student performance in various clinical scenarios. However, workflow protocols to train SPs vary. Medical schools adopt specific measures to ensure standardization., Approach: We present a development workflow of the SPs' training framework for high-stakes OSCE using a volunteer SPs' bank. Our approach was guided by the social learning theory. Three educators and 17/20 (85%) members of our volunteer SPs' bank worked in a collaborative partnership on the construction of pedagogical content of the training framework comprising three 2-hour sessions. Since SPs have to demonstrate acquired behaviors, intended learning outcomes used the words "apply", "perform" and "participate.", Evaluation: A principal part of the evaluation was the achievement of intended learning outcomes by the SPs during 3 formative OSCEs. Seventeen SPs, 356 fourth year medical students and 60 examiners participated. Quantitative and qualitative data were collected by post-session questionnaires and analyzed using descriptive statistics and thematic analysis. Twelve examiners evaluated a mean of 29.7+/-0.14 SD patient-student encounters. In total, 15/16 SPs (93.8%) considered the contact with students as easy and 4 SPs (31%) reported the experience as stressful. Two themes emerged from the free-text comments: "Gaining experience as SP" and "Concerns for evaluated students.", Implication: The proposed SPs' training framework for high-stakes OSCE may be useful for other medical disciplines and health professions initiating SP-based assessment programs. The strategy of development and evaluation are outlined to guide a successful application of the curriculum standards., (© 2024 The Author(s). The Clinical Teacher published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Purpose: Observed assessments are integral to medical education but may be biased against structurally marginalized communities. Current understanding of assessment bias is limited because studies have focused on single specialties, levels of training, or social identity characteristics (SIDCs). This scoping review maps studies investigating bias in observed assessments in medical education arising from trainees' observable SIDCs at different medical training levels, with consideration of medical specialties, assessment environments, and assessment tools., Method: MEDLINE, Embase, ERIC, PsycINFO, Scopus, Web of Science Core Collection, and Cochrane Library were searched for articles published between January 1, 2008, and March 15, 2023, on assessment bias related to 6 observable SIDCs: gender (binary), gender nonconformance, race and ethnicity, religious expression, visible disability, and age. Two authors reviewed the articles, with conflicts resolved by consensus or a third reviewer. Results were interpreted through group review and informed by consultation with experts and stakeholders., Results: Sixty-six of 2,920 articles (2.3%) were included. These studies most frequently investigated graduate medical education [44 (66.7%)], used quantitative methods [52 (78.8%)], and explored gender bias [63 (95.5%)]. No studies investigated gender nonconformance, religious expression, or visible disability. One evaluated intersectionality. SIDCs were described inconsistently. General surgery [16 (24.2%)] and internal medicine [12 (18.2%)] were the most studied specialties. Simulated environments [37 (56.0%)] were studied more frequently than clinical environments [29 (43.9%)]. Bias favoring men was found more in assessments of intraoperative autonomy [5 of 9 (55.6%)], whereas clinical examination bias often favored women [15 of 19 (78.9%)]. When race and ethnicity bias was identified, it consistently favored White students., Conclusions: This review mapped studies of gender, race, and ethnicity bias in the medical education assessment literature, finding limited studies on other SIDCs and intersectionality. These findings will guide future research by highlighting the importance of consistent terminology, unexplored SIDCs, and intersectionality., (Copyright © 2024 the Association of American Medical Colleges.)

Context: One central consideration in health professions education (HPE) is to ensure we are making sound and justifiable decisions based on the assessment instruments we use on health professionals. To achieve this goal, HPE assessment researchers have drawn on Kane's argument-based framework to ascertain the validity of their assessment tools. However, the original four-inference model proposed by Kane - frequently used in HPE validation research - has its limitations in terms of what each inference entails and what claims and sources of backing are housed in each inference. The under-specification in the four-inference model has led to inconsistent practices in HPE validation research, posing challenges for (i) researchers who want to evaluate the validity of different HPE assessment tools and/or (ii) researchers who are new to test validation and need to establish a coherent understanding of argument-based validation., Methods: To address these identified concerns, this article introduces the expanded seven-inference argument-based validation framework that is established practice in the field of language testing and assessment (LTA). We explicate (i) why LTA researchers experienced the need to further specify the original four Kanean inferences; (ii) how LTA validation research defines each of their seven inferences and (iii) what claims, assumptions and sources of backing are associated with each inference. Sampling six representative validation studies in HPE, we demonstrate why an expanded model and a shared disciplinary validation framework can facilitate the examination of the validity evidence in diverse HPE validation contexts., Conclusions: We invite HPE validation researchers to experiment with the seven-inference argument-based framework from LTA to evaluate its usefulness to HPE. We also call for greater interdisciplinary dialogue between HPE and LTA since both disciplines share many fundamental concerns about language use, communication skills, assessment practices and validity in assessment instruments., (© 2024 The Author(s). Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Introduction: This study compares pharmacy students' performance using face-to-face (FTF) team-based learning (TBL) vs. virtual TBL across multiple courses and different academic levels while accounting for student demographic and academic factors., Methods: The study included pharmacy students from different academic levels (P1-P3) who were enrolled in three didactic courses taught using FTF TBL and virtual TBL. Multiple generalized linear models (GLMs) were performed to compare students' performance on individual readiness assurance tests (iRATs), team readiness assurance tests (tRATs), team application exercises (tAPPs), summative exams, and total course scores using FTF TBL vs. virtual TBL, adjusting for students' age, sex, race, and cumulative grade point average (cGPA)., Results: The study involved a total of 356 pharmacy students distributed across different academic levels and learning modalities: P1 students [FTF TBL ( n  = 26), virtual TBL ( n  = 42)], P2 students [FTF TBL ( n  = 77), virtual TBL ( n  = 71)], and P3 students [FTF TBL ( n  = 65), virtual TBL ( n  = 75)]. In the P1 cohort, the virtual group had higher iRAT and tRAT scores but lower tAPP scores than the FTF TBL group, with no significant differences in summative exams or total course scores. For P2 students, the virtual TBL group had higher iRAT and tRAT scores but lower summative exam scores and total course scores than the FTF TBL group, with no significant differences in tAPP scores. In the P3 student group, the virtual TBL group had higher iRAT, tRAT, tAPP, summative exam, and total course scores than the FTF TBL group., Conclusions: Students' performance in virtual TBL vs. FTF TBL in the pharmacy didactic curriculum varies depending on the course content, academic year, and type of assessment.

Background: The aim of this study was to describe the proficiency of ChatGPT (GPT-4) on certification style exams from the Canadian Association of Medical Radiation Technologists (CAMRT), and describe its performance across multiple exam attempts., Methods: ChatGPT was prompted with questions from CAMRT practice exams in the disciplines of radiological technology, magnetic resonance (MRI), nuclear medicine and radiation therapy (87-98 questions each). ChatGPT attempted each exam five times. Exam performance was evaluated using descriptive statistics, stratified by discipline and question type (knowledge, application, critical thinking). Light's Kappa was used to assess agreement in answers across attempts., Results: Using a passing grade of 65 %, ChatGPT passed the radiological technology exam only once (20 %), MRI all five times (100 %), nuclear medicine three times (60 %), and radiation therapy all five times (100 %). ChatGPT's performance was best on knowledge questions across all disciplines except radiation therapy. It performed worst on critical thinking questions. Agreement in ChatGPT's responses across attempts was substantial within the disciplines of radiological technology, MRI, and nuclear medicine, and almost perfect for radiation therapy., Conclusion: ChatGPT (GPT-4) was able to pass certification style exams for radiation technologists and therapists, but its performance varied between disciplines. The algorithm demonstrated substantial to almost perfect agreement in the responses it provided across multiple exam attempts. Future research evaluating ChatGPT's performance on standardized tests should consider using repeated measures., (Copyright © 2024. Published by Elsevier Inc.)

Background: Student-completed course evaluations are often included in university faculty promotion and tenure packets but may not be the best metrics for student learning. This juxtaposition creates a challenge - the metric faculty first want to improve is student learning rather than student perception, but course changes improving student learning do not always result in higher course evaluation scores. Additionally, making course changes can be time consuming for faculty. The objective of this project was to assess the changes made to a course, the time spent making those changes, and the impact of the changes on course evaluation scores and student performance., Educational Activity: A single course focused on management with a business plan as the primary course project was revised in Fall 2023; the goal of revisions was to address poor course evaluation scores and negative student perceptions of the course in as little faculty time as possible while maintaining quality of student work. Primary changes centered around the business plan project and a shift in course communication. Evaluation of the changes occurred through mid-semester survey and discussion and end-of-semester survey for the 2023 cohort, and comparing course evaluations and student work products between the 2022 and 2023 cohorts., Critical Analysis of the Educational Activity: Course evaluation scores were improved and student feedback was more positive for the revised Fall 2023 course. The implemented changes did not negatively affect student performance as quality of the business plan project was maintained across both semesters. Instructor transparency and responsiveness through the mid-semester survey and discussion may have positively influenced course evaluations scores. Overall, the course changes did not take a significant amount of faculty time, but the desired outcome of improving course evaluation scores was recognized., Competing Interests: Declaration of competing interest The authors declare no conflicts of interest. The authors have no funding sources to declare., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Artificial intelligence (AI) has gained massive interest with the public release of the conversational AI "ChatGPT," but it also has become a matter of concern for academia as it can easily be misused. We performed a quantitative evaluation of the performance of ChatGPT on a medical physiology university examination. Forty-one answers were obtained with ChatGPT and compared to the results of 24 students. The results of ChatGPT were significantly better than those of the students; the median (IQR) score was 75% (66-84%) for the AI compared to 56% (43-65%) for students ( P < 0.001). The exam success rate was 100% for ChatGPT, whereas 29% ( n = 7) of students failed. ChatGPT could promote plagiarism and intellectual laziness among students and could represent a new and easy way to cheat, especially when evaluations are performed online. Considering that these powerful AI tools are now freely available, scholars should take great care to construct assessments that really evaluate student reflection skills and prevent AI-assisted cheating. NEW & NOTEWORTHY The release of the conversational artificial intelligence (AI) ChatGPT has become a matter of concern for academia as it can easily be misused by students for cheating purposes. We performed a quantitative evaluation of the performance of ChatGPT on a medical physiology university examination and observed that ChatGPT outperforms medical students obtaining significantly better grades. Scholars should therefore take great care to construct assessments crafted to really evaluate the student reflection skills and prevent AI-assisted cheating.

Background: The assessment of undergraduate medical students consists of a written component and an advanced clinical competency test that evaluates the students' skills. The Objective Structured Clinical Examinations (OSCE) were only implemented in most medical schools in the last ten years, similar in many developing countries. It was first used with other clinical assessment methods. This study was designed to investigate how medical students in the fourth and sixth grades and examiners perceived the unique format, features, and quality of the Grand Objective Structured Clinical Examinations (GOSCE), a novel method for assessing students. The aim was to understand how this innovative method, distinct from traditional assessment methods, prepares students for real-world scenarios where they encounter different specialty cases. Additionally, the study aimed to understand the significant role and challenges faced by the Faculty of General Medicine at Koya University in implementing this type of OSCE and to identify potential improvements to enhance the quality of the assessment process., Methods: Following the administration of two distinct GOSCE examinations for the fourth and sixth stages, feedback forms were created using Google Forms and distributed to students and examiners. The feedback forms covered all aspects of the examination process. The responses were then rigorously analyzed using the scientific tool SPSS, allowing for a comparison between students from both stages and the faculty and external examiners., Results: In the survey of 104 undergraduate students (78% of attendants), the majority (84n,80%) rated the fairness of the assessment as good, which is the primary goal of GOSCE. A large percentage (89n,85.6%) agreed that the examination covered a wide range of clinical skills. Eighty students (76.85%) believed the patients cooperated and found the findings precise; on the other hand, The obtained responses from 24 out of 33 faculty staff members (75%) and 29 out of 35 external examiners (82%) strongly agreed that the examination adequately covered a wide variety of clinical abilities. Only two of 68 examiners (2.9%) believed the examination needed more to provide a learning opportunity., Conclusion: With the insightful suggestions for improvement provided by the participants, GOSCE has a promising potential to evolve into a fair, objective clinical tool for assessing medical students, instilling a sense of optimism for its future development., Competing Interests: Declarations. Ethical approval and consent to participate: Our study, entitled “Perception of undergraduate medical students and examiners towards grand objective structured clinical examination,” was reviewed by the Institutional Review Board (IRB) of the Faculty of General Medicine, Koya University. The IRB reviewed the study on July 23, 2024, with the reference code and determined that formal ethics approval or a reference number was not required. This decision was based on the fact that the research posed minimal risk to participants and focused solely on their educational experiences without involving any intervention or collecting sensitive personal data. Consent for publication: Not applicable. Clinical trial number: Not applicable. Competing interests: The authors declare no competing interests., (© 2024. The Author(s).)

Background: This study aimed to evaluate the performance of GPT-3.5, GPT-4, GPT-4o and Google Bard on the United States Medical Licensing Examination (USMLE), the Professional and Linguistic Assessments Board (PLAB), the Hong Kong Medical Licensing Examination (HKMLE) and the National Medical Licensing Examination (NMLE)., Methods: This study was conducted in June 2023. Four LLMs (Large Language Models) (GPT-3.5, GPT-4, GPT-4o and Google Bard) were applied to four medical standardized tests (USMLE, PLAB, HKMLE and NMLE). All questions are multiple-choice questions and were sourced from the question banks of these examinations., Results: In USMLE step 1, step 2CK and Step 3, there are accuracy rates of 91.5%, 94.2% and 92.7% provided from GPT-4o, 93.2%, 95.0% and 92.0% provided from GPT-4, 65.6%, 71.6% and 68.5% provided from GPT-3.5, and 64.3%, 55.6%, 58.1% from Google Bard, respectively. In PLAB, HKMLE and NMLE, GPT-4o scored 93.3%, 91.7% and 84.9%, GPT-4 scored 86.7%, 89.6% and 69.8%, GPT-3.5 scored 80.0%, 68.1% and 60.4%, and Google Bard scored 54.2%, 71.7% and 61.3%. There was significant difference in the accuracy rates of four LLMs in the four medical licensing examinations., Conclusion: GPT-4o performed better in the medical licensing examinations than other three LLMs. The performance of the four models in the NMLE examination needs further improvement., Clinical Trial Number: Not applicable., Competing Interests: Declarations. Ethics approval and consent to participate: Not applicable. Informed consent: Informed consent were not required. Institutional review board: This study does not involve human or animal subjects and therefore did not require approval from the Institutional Ethical Board. Competing interests: The authors declare no competing interests., (© 2024. The Author(s).)

Objective: This study aimed to investigate the performance of generative pre-trained transformer-4 (GPT-4) on the Certification Test for Mental Health Management and whether tuned prompts could improve its performance., Methods: This study used a 3 × 2 factorial design to examine the performance according to test difficulty (courses) and prompt conditions. We prepared 200 multiple-choice questions (600 questions overall) for each course using the Certification Test for Mental Health Management (levels I-III) and essay questions from the level I test for the previous four examinations. Two conditions were used: a simple prompt condition using the questions as prompts and tuned prompt condition using techniques to obtain better answers. GPT-4 (gpt-4-0613) was adopted and implemented using the OpenAI API., Results: The simple prompt condition scores were 74.5, 71.5, and 64.0 for levels III, II, and I, respectively. The tuned and simple prompt condition scores had no significant differences (Odds ratio = 1.03, 95% Confidence interval; 0.65-1.62, p = 0.908). Incorrect answers were observed in the simple prompt condition because of the inability to make choices, whereas no incorrect answers were observed in the tuned prompt condition. The average score for the essay questions under the simple prompt condition was 22.5 out of 50 points (45.0%)., Conclusion: GPT-4 had a sufficient knowledge network for occupational mental health, surpassing the criteria for levels II and III tests. For the level I test, which required the ability to describe more advanced knowledge accurately, GPT-4 did not meet the criteria. External information may be needed when using GPT-4 at this level. Although the tuned prompts did not significantly improve the performance, they were promising in avoiding unintended outputs and organizing output formats. UMIN trial registration: UMIN-CTR ID = UMIN000053582.

Background: Traditional assessments often lack flexibility, personalized feedback, real-world applicability, and the ability to measure skills beyond rote memorization. These may not adequately accommodate diverse learning styles and preferences, nor do they always foster critical thinking or creativity. The inclusion of Artificial Intelligence (AI), especially Generative Pre-trained Transformers, in medical education marks a significant shift, offering both exciting opportunities and notable challenges for authentic assessment practices. Various fields, including anatomy, physiology, pharmacy, dentistry, and pathology, are anticipated to employ the metaverse for authentic assessments increasingly. This innovative approach will likely enable students to engage in immersive, project-based learning experiences, facilitating interdisciplinary collaboration and providing a platform for real-world application of knowledge and skills., Methods: This commentary paper explores how AI, authentic assessment, and Student-as-Partners (SaP) methodologies can work together to reshape assessment practices in medical education., Results: The paper provides practical insights into effectively utilizing AI tools to create authentic assessments, offering educators actionable guidance to enhance their teaching practices. It also addresses the challenges and ethical considerations inherent in implementing AI-driven assessments, emphasizing the need for responsible and inclusive practices within medical education. Advocating for a collaborative approach between AI and SaP methodologies, the commentary proposes a robust plan to ensure ethical use while upholding academic integrity., Conclusion: Through navigating emerging assessment paradigms and promoting genuine evaluation of medical knowledge and proficiency, this collaborative effort aims to elevate the quality of medical education and better prepare learners for the complexities of clinical practice., (© The Author(s) 2024. Published by Oxford University Press on behalf of Fellowship of Postgraduate Medicine. All rights reserved. For permissions, please e-mail: journals.permissions@oup.com.)

The increasing use of technology in health care and health professions education is an invitation to examine how digital sources of evidence are used in making assessment claims. In this paper, we describe how four sets of terms-primary and secondary data; structured and unstructured data; development and use; and deterministic and generative-can aid in examining how data from digital sources are used in evaluating what learners know and can do. Drawing on multiple examples, this paper shows how the four sets of terms can help both developers and users of technology-based assessment systems., Competing Interests: Dr. Krumm receives research support from the Society for Improving Medical Professional Learning (SIMPL). Dr. George is the Executive Director of the Society for Improving Medical Professional Learning (SIMPL). This position is unpaid. Dr. Barone is a former employee of NBME. Dr. Smirnova received compensation from the Kern Institute for the Transformation of Medical Education, Medical College of Wisconsin, for her time helping coordinate the Next Era in Assessment meeting in Milwaukee, Wisconsin., (Copyright: © 2024 The Author(s).)