Search

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014. Inter-rater reliability (% exact agreement and kappa) was calculated for five standardized patient cases administered in a local graduation competency examination. Generalizability studies were conducted to examine the overall reliability. Qualitative data were collected through surveys and a rater-debriefing meeting. The overall inter-rater reliability (weighted kappa) was 0.79 (Documentation = 0.63, Differential Diagnosis = 0.90, Justification = 0.48, and Workup = 0.54). The majority of score variance was due to case specificity (13%) and case-task specificity (31%), indicating differences in student performance by case and by case-task interactions. Variance associated with raters and its interactions were modest (<5%). Raters felt that justification was the most difficult task to score and that having case and level-specific scoring guidelines during training was most helpful for calibration. The overall inter-rater reliability indicates high level of confidence in the consistency of note scores. Designs for scoring notes may optimize reliability by balancing the number of raters and cases.

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Purpose: There is a paucity of evaluation forms specifically developed and validated for outpatient settings. The purpose of this study was to develop and validate an instrument specifically for evaluating outpatient teaching, to provide reliable and valid ratings for individual and group feedback to faculty, and to identify outstanding teachers in that setting. Method: By literature reviews and pilot studies at the Faculties of Health Sciences, McMaster University (Canada) and Aga Khan University (AKU-Pakistan), a 15-item instrument, Student Evaluation of Teaching in Outpatient Clinics (SETOC), was created with five subscales: "Establishing Learning-Milieu, Clinical-Teaching, General-Teaching, Clinical-Competence, and Global-Rating." Seven-point Likert-type rating scales were used. Students also nominated three "best" outpatient teachers. Participants: 87 faculty members (79%) rated by all 224 third to fifth-year students (clerks) at outpatient departments of the AKU hospital over a one-year period. Analyses: Repeated measures generalizability studies, correlations, concurrent validity of SETOC scores with best teacher nominations. Results: Inter-rater G-coefficient and internal consistency of SETOC student ratings were 0.92 and 0.98. Average inter-item and inter-subscale correlations were 0.79 and 0.86. Comparing SETOC scores against "Best Teacher" nominations, sensitivity, specificity, positive and negative predictive values were greater than 0.84.Student ratings ranged from unsatisfactory (fourteen instructors) to outstanding (four instructors). Mean-scores for Learning-Milieu, Clinical-Teaching and General-Teaching were lower than those for Clinical-Competence and Global-Rating (p=0.000 for all). Conclusions: The SETOC elicited reliable and valid student ratings that can provide specific feedback to individual faculty with weak or outstanding teaching skills, and identify overall group shortcomings for faculty development.

Explored strengths and weaknesses of medical education reports by analyzing ratings and written comments by reviewers. Found that the main strengths noted in accepted manuscripts were importance or timeliness of problem studied, excellence of writing, and soundness of study design. Ignoring the literature, designing poor studies, choosing inappropriate instruments, and writing poor manuscripts are likely to be fatal flaws. (EV)

Outlines 13 priority research topics and 51 important research questions concerning medical education in ambulatory care settings, as defined by a distinguished group of medical educators, clinicians, and policymakers. Issues of theory binding, research priorities, and research design are discussed, and policy recommendations made for the development of valid measures of educational outcomes. (Author/MSE)

The diagnostic discourse of medical students and physicians in thinking-aloud protocols on paper cases was analyzed for evidence of semantic structure. Results show that structural semantics can be used to distinguish various levels of mental processing among novices as well as between novices and professionals. (MSE)

Patel and Groen's notion that semiotic analysis is a framework rather than a theory for the study of diagnostic reasoning is flawed. Semiotics, particularly structural semantics is a well-established theory which deals with several variables. Based on semiotic constructs, a theory of expertise is proposed which could be supported by medical data. This theory is more comprehensive and indept than the one proposed by Patel and Groen.