Search

Although problem-based learning (PBL) has been widely used in medical schools, few studies have attended to the assessment of PBL processes using validated instruments. This study examined reliability and validity for an instrument assessing PBL performance in four domains: Problem Solving, Use of Information, Group Process, and Professionalism. Two cohorts of medical students (N = 310) participated in the study, with 2 years of PBL evaluation data extracted from archive rated by a total of 158 faculty raters. Analyses based on generalizability theory were conducted for reliability examination. Validity was examined through following the Standards for Educational and Psychological Testing to evaluate content validity, response processes, construct validity, predictive validity, and the relationship to the variable of training. For construct validity, correlations of PBL scores with six other outcome measures were examined, including Medical College Admission Test, United States Medical Licensing Examination (USMLE) Step 1, National Board of Medical Examiners (NBME) Comprehensive Basic Science Examination, NBME Comprehensive Clinical Science Examination, Clinical Performance Examination, and USMLE Step 2 Clinical Knowledge. Predictive validity was examined by using PBL scores to predict five medical school outcomes. The highest percentage of PBL total score variance was associated with students (60%), indicating students in the study differed in their PBL performance. The generalizability and dependability coefficients were moderately high (Ep[superscript 2] = 0.68, ? = 0.60), showing the instrument is reliable for ranking students and identifying competent PBL performers. The patterns of correlations between PBL domain scores and the outcome measures partially support construct validity. PBL performance ratings as a whole significantly (p < 0.01) predicted all the major medical school achievements. The second year PBL scores were significantly higher than those of the first year, indicating a training effect. Psychometric findings provided support for reliability and many aspects of validity of PBL performance assessment using the instrument.

To determine the direction and extent to which medical student scores (as observed by small-group tutors) on four problem-based-learning-related domains change over nine consecutive blocks during a two-year period (Domains: Problem Solving/Use of Information/Group Process/Professionalism). Latent growth curve modeling is used to analyze performance trajectories in each domain of two cohorts of 1st and 2nd year students (n = 296). Slopes of the growth trajectories show similar linear increments in the first three domains. Further analysis revealed relative strong individual variability in initial scores but not in their later increments. Professionalism, on the other hand, shows low variability and has very small, insignificant slope increments. In this study, we showed that the learning domains (Problem Solving, Use of Information, and Group Process) observed during PBL tutorials are not only related to each other but also develop cumulatively over time. Professionalism, in contrast to the other domains studied, is less affected by the curriculum suggesting that this represents a stable characteristic. The observation that the PBL tutorial has an equal benefit to all students is noteworthy and needs further investigation.