Search

Your search keyword '"Smadja, Nathan Peiffer"' showing total 3 results
Start Over Author "Smadja, Nathan Peiffer"
3 results on '"Smadja, Nathan Peiffer"'

1. Measuring and correcting staff variability in large-scale OSCEs.

Author

Haviari, Skerdi, de Tymowski, Christian, Burnichon, Nelly, Lemogne, Cédric, Flamant, Martin, Ruszniewski, Philippe, Bensaadi, Saja, Mercier, Gregory, Hamaoui, Hasséne, Thy, Michael, Pellat, Anna, Soulat, Gilles, Dinh, Alexy Tran, Ferré, Valentine, Smadja, Nathan Peiffer, Deniau, Benjamin, Oules, Bénédicte, Nguyen, Yann, Khider, Lina, and Soumagne, Thibaud

Subjects
MEDICAL students, SCHOOL year, CONFIDENCE intervals
Abstract

Context: Objective Structured Clinical Examinations (OSCEs) are an increasingly popular evaluation modality for medical students. While the face-to-face interaction allows for more in-depth assessment, it may cause standardization problems. Methods to quantify, limit or adjust for examiner effects are needed. Methods: Data originated from 3 OSCEs undergone by 900-student classes of 5th- and 6th-year medical students at Université Paris Cité in the 2022-2023 academic year. Sessions had five stations each, and one of the three sessions was scored by consensus by two raters (rather than one). We report OSCEs' longitudinal consistency for one of the classes and staff-related and student variability by session. We also propose a statistical method to adjust for inter-rater variability by deriving a statistical random student effect that accounts for staff-related and station random effects. Results: From the four sessions, a total of 16,910 station scores were collected from 2615 student sessions, with two of the sessions undergone by the same students, and 36, 36, 35 and 20 distinct staff teams in each station for each session. Scores had staff-related heterogeneity (p<10-15), with staff-level standard errors approximately doubled compared to chance. With mixed models, staff-related heterogeneity explained respectively 11.4%, 11.6%, and 4.7% of station score variance (95% confidence intervals, 9.5-13.8, 9.7-14.1, and 3.9-5.8, respectively) with 1, 1 and 2 raters, suggesting a moderating effect of consensus grading. Student random effects explained a small proportion of variance, respectively 8.8%, 11.3%, and 9.6% (8.0-9.7, 10.3-12.4, and 8.7-10.5), and this low amount of signal resulted in student rankings being no more consistent over time with this metric, rather than with average scores (p=0.45). Conclusion: Staff variability impacts OSCE scores as much as student variability, and the former can be reduced with dual assessment or adjusted for with mixed models. Both are small compared to unmeasured sources of variability, making them difficult to capture consistently. [ABSTRACT FROM AUTHOR]

Published
2024
Full Text
View/download PDF

3. eOSCE stations live versus remote evaluation and scores variability

Author

Bouzid, Donia, Mullaert, Jimmy, Ghazali, Aiham, Ferré, Valentine Marie, Mentré, France, Lemogne, Cédric, Ruszniewski, Philippe, Faye, Albert, Dinh, Alexy Tran, Mirault, Tristan, Smadja, Nathan Peiffer, Muller, Léonore, Pierrotin, Laure Falque, Thy, Michael, Assadi, Maksud, Yung, Sonia, de Tymowski, Christian, Le Hingrat, Quentin, Eyer, Xavier, Wicky, Paul Henri, Oualha, Mehdi, Houdouin, Véronique, Jabre, Patricia, Vodovar, Dominique, Burgio, Marco Dioguardi, Zucman, Noémie, Tsopra, Rosy, Tazi, Asmaa, Ressaire, Quentin, Nguyen, Yann, Girard, Muriel, Frachon, Adèle, Depret, François, Pellat, Anna, de Masson, Adèle, Azais, Henri, de Castro, Nathalie, Jeantrelle, Caroline, Javaud, Nicolas, Malmartel, Alexandre, de Margerie, Constance Jacquin, Chousterman, Benjamin, Fournel, Ludovic, Holleville, Mathilde, Blanche, Stéphane, Infection, Anti-microbiens, Modélisation, Evolution (IAME (UMR_S_1137 / U1137)), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPCité)-Université Sorbonne Paris Nord, Institut de psychiatrie et neurosciences de Paris (IPNP - U1266 Inserm), Institut National de la Santé et de la Recherche Médicale (INSERM)-Université Paris Cité (UPCité), Hôpital Beaujon [AP-HP], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP), AP-HP Hôpital universitaire Robert-Debré [Paris], CIC Hôpital Bichat, AP-HP - Hôpital Bichat - Claude Bernard [Paris], Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Institut National de la Santé et de la Recherche Médicale (INSERM)-UFR de Médecine, Hôpital Européen Georges Pompidou [APHP] (HEGP), Assistance publique - Hôpitaux de Paris (AP-HP) (AP-HP)-Hôpitaux Universitaires Paris Ouest - Hôpitaux Universitaires Île de France Ouest (HUPO), Health data- and model- driven Knowledge Acquisition (HeKA), Inria de Paris, Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre de Recherche des Cordeliers (CRC (UMR_S_1138 / U1138)), École Pratique des Hautes Études (EPHE), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Sorbonne Université (SU)-Université Paris Cité (UPCité)-École Pratique des Hautes Études (EPHE), Université Paris sciences et lettres (PSL)-Université Paris sciences et lettres (PSL)-Institut National de la Santé et de la Recherche Médicale (INSERM)-Sorbonne Université (SU)-Université Paris Cité (UPCité), and Centre de Recherche des Cordeliers (CRC (UMR_S_1138 / U1138))

Subjects
[SHS.EDU]Humanities and Social Sciences/Education, General Medicine, Education
Abstract

Background Objective structured clinical examinations (OSCEs) are known to be a fair evaluation method. These recent years, the use of online OSCEs (eOSCEs) has spread. This study aimed to compare remote versus live evaluation and assess the factors associated with score variability during eOSCEs. Methods We conducted large-scale eOSCEs at the medical school of the Université de Paris Cité in June 2021 and recorded all the students’ performances, allowing a second evaluation. To assess the agreement in our context of multiple raters and students, we fitted a linear mixed model with student and rater as random effects and the score as an explained variable. Results One hundred seventy observations were analyzed for the first station after quality control. We retained 192 and 110 observations for the statistical analysis of the two other stations. The median score and interquartile range were 60 out of 100 (IQR 50–70), 60 out of 100 (IQR 54–70), and 53 out of 100 (IQR 45–62) for the three stations. The score variance proportions explained by the rater (ICC rater) were 23.0, 16.8, and 32.8%, respectively. Of the 31 raters, 18 (58%) were male. Scores did not differ significantly according to the gender of the rater (p = 0.96, 0.10, and 0.26, respectively). The two evaluations showed no systematic difference in scores (p = 0.92, 0.053, and 0.38, respectively). Conclusion Our study suggests that remote evaluation is as reliable as live evaluation for eOSCEs.

Published
2022
Full Text
View/download PDF

Catalog

Books, media, physical & digital resources
See catalog results