Search

Copyright of Japanese Journal of Educational Research / Kyoikugaku Kenkyu is the property of Japanese Educational Research Association and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Les examens cliniques à objectifs structurés (ECOS) évaluent les performances professionnelles en milieu simulé. À la suite de leur intégration dans la réforme du 2e cycle des études médicales (R2C), cette modalité pédagogique a été implémentée en France. Ce travail étudie la variabilité des notes des étudiants aux ECOS ainsi que leur reproductibilité inter-évaluateurs. Cette étude rétrospective monocentrique a porté sur plusieurs sessions de circuits d'ECOS évaluatifs réalisées entre janvier 2022 et juin 2023. Les variables recueillies étaient : la famille de situation de départ, le domaine de compétence, la présence d'un participant standardisé pour les stations ; le sexe et le statut professionnel pour les évaluateurs ; les notes obtenues à chaque station (globales, compétences cliniques, compétences de communication), le nombre de circuits ECOS préalablement réalisés et les notes facultaires pour les étudiants. La variabilité de la note globale était expliquée principalement par le facteur station (79,7 %, IC95 % 77,4–82,0). Le facteur étudiant et le facteur parcours expliquaient respectivement 7,5 % [12,9–20,2] et < 0,01 % [2,10-13–2,10-9]. Le coefficient de corrélation intra-classe inter-évaluateurs était de 87,2 % [86,4–87,9] pour la note globale. Les caractéristiques stations (situation de départ, domaine) et évaluateurs (sexe, statut) étaient significativement associées aux variations de notes. Cette première étude de la variabilité des notes de circuits d'ECOS en France montre une bonne reproductibilité inter-évaluateurs, cependant influencée par les caractéristiques de la station. Pour standardiser les circuits, les variabilités liées au domaine des stations sont donc à considérer. Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility. This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students. The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and < 0.01% [2.10-13; 2.10-9] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations. This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well. [ABSTRACT FROM AUTHOR]

As diagnostic classification models become more widely used in large‐scale operational assessments, we must give consideration to the methods for estimating and reporting reliability. Researchers must explore alternatives to traditional reliability methods that are consistent with the design, scoring, and reporting levels of diagnostic assessment systems. In this article, we describe and evaluate a method for simulating retests to summarize reliability evidence at multiple reporting levels. We evaluate how the performance of reliability estimates from simulated retests compares to other measures of classification consistency and accuracy for diagnostic assessments that have previously been described in the literature, but which limit the level at which reliability can be reported. Overall, the findings show that reliability estimates from simulated retests are an accurate measure of reliability and are consistent with other measures of reliability for diagnostic assessments. We then apply this method to real data from the Examination for the Certificate of Proficiency in English to demonstrate the method in practice and compare reliability estimates from observed data. Finally, we discuss implications for the field and possible next directions. [ABSTRACT FROM AUTHOR]

Introduction: Objective Structured Clinical Examinations (OSCEs) assess professional performance in a simulated environment. Following their integration into the reform of the 2nd cycle of medical studies (R2C), this pedagogical modality was implemented in France. This study investigates the variability of students' OSCE scores, as well as their inter-rater reproducibility., Methods: This single-center retrospective study covered several sessions of evaluative OSCE circuits conducted between January 2022 and June 2023. Variables collected were: baseline situation family, competency domain, presence of a standardized participant for stations; gender and professional status for evaluators; scores (global, clinical and communication skills), number of previously completed OSCE circuits and faculty scores for students., Results: The variability of the overall score was explained mainly (79.7%, CI95% [77.4; 82.0]) by the station factor. The student factor and the circuit factor explained 7.5% [12.9; 20.2] and<0.01% [2.10 -13 ; 2.10 -9 ] respectively. The inter-rater intra-class correlation coefficient was 87.2% [86.4; 87.9] for the global score. Station characteristics (starting situation, domain) and evaluator characteristics (gender, status) were significantly associated with score variations., Conclusion: This first study on the variability of OSCE circuit scores in France shows good reproducibility with influence of station characteristics. In order to standardize circuits, variability linked to the domain competency should be considered as well., (Copyright © 2024 The Author(s). Published by Elsevier Masson SAS.. All rights reserved.)

The upgrade educational information technology triggered by COVID‐19 has shaped a new educational order and new educational forms. As a result, traditional educational measurement is now facing a systematic transformation, that is, from the Assessment of Learning (AoL) to Assessment for Learning (AfL), and finally to Assessment as Learning (AaL). By showing the initiative and the flexibility of large‐scale online learning during the COVID‐19 pandemic, this paper examines the evolution from AoL to AaL from four aspects: participants, test form, multivariate data for process‐based measurement, and measurement models for multivariate data. We propose multidisciplinary corporations in the field, which integrate education, psychology, and information technology into the theories and methods of educational measurement. Further, we emphasize that the validity, ethics, and fairness of the measurement should also be considered the critical issues. Researchers and practitioners of educational measurement must persist in pursuing the substantive significance of measurement and provide unique experience and guidance for the theoretical and practical development of educational assessment in this great revolution. [ABSTRACT FROM AUTHOR]