Establishing the Validity of Licensing Examination Scores

BACKGROUND: The Comprehensive Osteopathic Medical Licensure Examination (COMLEX-USA) Level 2–Cognitive Examination (CE) and the Comprehensive Osteopathic Medical Achievement Test (COMAT) are administered to similar populations (third- and fourth-year osteopathic students) at similar points in time. Examining the relationship between scores on the 2 assessments that measure similar constructs ultimately supports the validity of both. OBJECTIVE: The purpose of this study is to provide empirical evidence of the concurrent and predictive validity of COMAT and COMLEX-USA Level 2-CE. METHODS: In 2018, first-attempt scores on Level 2-CE were aggregated from June 2015 to May 2018 and matched with first-attempt scores on each COMAT clinical subject. We conducted correlational analyses between performance on COMAT and Level 2-CE, and COMAT scores and Level 2-CE discipline subscores. Additionally, we used multivariate regression to analyze the predictive relationship between performance on all COMAT clinical subjects and Level 2-CE. RESULTS: The results from correlational analyses indicated statistically significant, positive associations between COMAT and Level 2-CE scores (r = 0.49–0.68, P < .0001), and statistically significant, but slightly weaker relationships between COMAT scores and Level 2-CE discipline subscores (r = 0.31–0.60, P < .0001). Furthermore, results from the multiple regression indicated that scores on COMAT explained 68% of the variance in Level 2-CE scores, and that COMAT internal medicine and emergency medicine were weighted more heavily than other specialties. CONCLUSIONS: The findings from this study can inform assessment practices by supporting the use of COMAT for osteopathic medical schools that do not administer COMAT.