1. Measuring and correcting staff variability in large-scale OSCEs
- Author
-
Skerdi Haviari, Christian de Tymowski, Nelly Burnichon, Cédric Lemogne, Martin Flamant, Philippe Ruszniewski, Saja Bensaadi, Gregory Mercier, Hasséne Hamaoui, Université Paris Cité OSCE study group, Tristan Mirault, Albert Faye, and Donia Bouzid
- Subjects
OSCE ,Score variability ,Inter-rater variability ,Special aspects of education ,LC8-6691 ,Medicine - Abstract
Abstract Context Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed. Methods Data originated from 3 OSCEs undergone by 900-student classes of 5th- and 6th-year medical students at Université Paris Cité in the 2022-2023 academic year. Sessions had five stations each, and one of the three sessions was scored by consensus by two raters (rather than one). We report OSCEs' longitudinal consistency for one of the classes and staff-related and student variability by session. We also propose a statistical method to adjust for inter-rater variability by deriving a statistical random student effect that accounts for staff-related and station random effects. Results From the four sessions, a total of 16,910 station scores were collected from 2615 student sessions, with two of the sessions undergone by the same students, and 36, 36, 35 and 20 distinct staff teams in each station for each session. Scores had staff-related heterogeneity (p
- Published
- 2024
- Full Text
- View/download PDF