A Tool to Assess Family Medicine Residents' Patient Encounters Using Secure Messaging

Background Secure messages exchanged between patients and family medicine residents via an electronic health record (EHR) could be used to assess residents' clinical and communication skills, but the mechanism is not well understood. Objective To design and test a secure messaging competency assessment for family medicine residents in a patient-centered medical home (PCMH). Methods Using the existing literature and evidence-based guidelines, we designed an assessment tool to evaluate secure messaging competency for family medicine residents training in a PCMH. Core faculty performed 2-stage validity and reliability testing (n = 2 and n = 9, respectively). A series of randomly selected EHR secure messages (n = 45) were assessed from a sample of 10 residents across all years of training. Results The secure message assessment tool provided data on a set of competencies and a framework for resident feedback. Assessment showed 10% (n = 2) of residents at the novice level, 50% (n = 10) as progressing, and 40% (n = 8) as proficient. The most common deficiencies for residents' secure messages related to communication rather than clinical competencies (n = 37 [90%] versus n = 4 [10%]). Interrater reliability testing ranged from 60% to 78% agreement and 20% to 44% disagreement. Disagreement centered on interpersonal communication factors. After 2 stages of testing, the assessment using residents' secure messages was incorporated into our existing evaluation process. Conclusions Assessing family medicine residents' secure messaging for patient encounters closed an evaluation gap in our family medicine program, and offered residents feedback on their clinical and communication skills in a PCMH.