Proficiency in identifying, managing and communicating medical errors: feasibility and validity study assessing two core competencies

Background Communication skills and professionalism are two competencies in graduate medical education that are challenging to evaluate. We aimed to develop, test and validate a de novo instrument to evaluate these two competencies. Methods Using an Objective Standardized Clinical Examination (OSCE) based on a medication error scenario, we developed an assessment instrument that focuses on distinctive domains [context of discussion, communication and detection of error, management of error, empathy, use of electronic medical record (EMR) and electronic medical information resources (EMIR), and global rating]. The aim was to test feasibility, acceptability, and reliability of the method. Results Faculty and standardized patients (SPs) evaluated 56 trainees using the instrument. The inter-rater reliability of agreement between faculty was substantial (Fleiss k = 0.71) and intraclass correlation efficient was excellent (ICC = 0.80). The measured agreement between faculty and SPs evaluation of resident was lower (Fleiss k = 0.36). The instrument showed good conformity (ICC = 0.74). The majority of the trainees (75 %) had satisfactory or higher performance in all six assessed domains and 86 % found the OSCE to be realistic. Sixty percent reported not receiving feedback on EMR use and asked for subsequent training. Conclusion An OSCE-based instrument using a medical error scenario can be used to assess competency in professionalism, communication, using EMRs and managing medical errors. Electronic supplementary material The online version of this article (doi:10.1186/s12909-016-0755-5) contains supplementary material, which is available to authorized users.