CanMEDS Competencies in Family Medicine Residents: Can Criterion-Based Assessment Improve the Quality of Teacher Feedback?

Background: The Université Laval family medicine program has developed an innovative computerized tool called the criterion-based Competency Assessment Tool (CAT), currently undergoing validity assessment. Methods: This study followed a qualitative design assessing written comments collected in the assessment reports from the cohorts before and after the implementation of the CAT (npre = 200, npost = 200) in order to ascertain the tool's consequence validity. A deductive thematic content analysis was performed and pre- and post-implementation cohorts were compared. Findings: Overall feedback quality does not appear to have changed between cohorts. When analyzing CanMEDS roles separately, each is covered more often, but related comments appear to be less specific. The new report also seems to enable the teacher to tell more with the same number of words. Conclusions: Perhaps since the items are complete, exhaustive, and detailed enough to be self-explanatory, the tool helps the teacher to cover a wider area of competencies without the need to add many details with narrative comments. Consequence validity does not seem to have been substantially affected by changes in the family medicine resident's competency assessment, but the results do not support the contention that comment quality has improved either.