Search

When the safety of the public is at stake, it is particularly relevant for licensing and credentialing exam agencies to use defensible standard setting methods to categorize candidates into competence categories (e.g., pass/fail). The aim of this study was to gather evidence to support change to the Comprehensive Osteopathic Medical Licensing-USA Level 2-Performance Evaluation standard setting design and administrative process. Twenty-two video recordings of candidates assessed for clinical competence were randomly selected from the 2014-2015 Humanistic domain test score distribution ranging from the highest to lowest quintile of performance. Nineteen panelists convened at the same site to receive training and practice prior to generating judgments of qualified or not qualified performance to each of the twenty videos. At the end of training, one panel remained onsite to complete their judgments and the second panel was released and given 1 week to observe the same twenty videos and complete their judgments offsite. The two one-sided test procedure established equivalence between panel group means at the 0.05 confidence level, controlling for rater errors within each panel group. From a practical cost-effective and administrative resource perspective, results from this study suggest it is possible to diverge from typical panel groups, who are sequestered the entire time onsite, to larger numbers of panelists who can make their judgments offsite with little impact on judged samples of qualified performance. Standard setting designs having panelists train together and then allowing those to provide judgments yields equivalent ratings and, ultimately, similar cut scores.

Item does not contain fulltext Context specificity, or the variation in a participant's performance from one case, or situation, to the next, is a recognized problem in medical education. However, studies have not explored the potential reasons for context specificity in experts using the lens of situated cognition and cognitive load theories (CLT). Using these theories, we explored the influence of selected contextual factors on clinical reasoning performance in internal medicine experts. We constructed and validated a series of videotapes portraying different chief complaints for three common diagnoses seen in internal medicine. Using the situated cognition framework, we modified selected contextual factors--patient, encounter, and/or physician--in each videotape. Following each videotape, participants completed a post-encounter form (PEF) and a think-aloud protocol. A survey estimating recent exposure from their practice to the correct videotape diagnoses was also completed. The time given to complete the PEF was randomly varied with each videotape. Qualitative utterances from the think-aloud procedure were converted to numeric measures of cognitive load. Survey and cognitive load measures were correlated with PEF performance. Pearson correlations were used to assess relations between the independent variables (cognitive load, survey of experience, contextual factors modified) and PEF performance. To further explore context specificity, analysis of covariance (ANCOVA) was used to assess differences in PEF scores, by diagnosis, after controlling for time. Low correlations between PEF sections, both across diagnoses and within each diagnosis, were observed (r values ranged from -.63 to .60). Limiting the time to complete the PEF impacted PEF performance (r = .2 to .4). Context specificity was further substantiated by demonstrating significant differences on most PEF section scores with a diagnosis (ANCOVA). Cognitive load measures were negatively correlated with PEF scores. The presence of selected contextual factors appeared to influence diagnostic more than therapeutic reasoning (r = -.2 to -.38). Contextual factors appear to impact expert physician performance. The impact observed is consistent with situated cognition and CLT's predictions. These findings have potential implications for educational theory and clinical practice. 01 maart 2012

While checklists are often used to score standardized patient based clinical assessments, little research has focused on issues related to their development or the level of agreement with respect to the importance of specific items. Five physicians independently reviewed checklists from 11 simulation scenarios that were part of the former Educational Commission for Foreign Medical Graduate's Clinical Skills Assessment and classified the clinical appropriateness of each of the checklist items. Approximately 78% of the original checklist items were judged to be needed, or indicated, given the presenting complaint and the purpose of the assessment. Rater agreement was relatively poor with pairwise associations (Kappa coefficient) ranging from 0.09 to 0.29. However, when only consensus indicated items were included, there was little change in examinee scores, including their reliability over encounters. Although most checklist items in this sample were judged to be appropriate, some could potentially be eliminated, thereby minimizing the scoring burden placed on the standardized patients. Periodic review of checklist items, concentrating on their clinical importance, is warranted.

Although studies have been conducted to examine the effects of a variety of factors on the comparability of scores obtained from standardized patient examinations (SPE), little research has been conducted to specifically investigate the challenge of detecting drift in case difficulty estimates over time, particularly for large-scale, performance-based, assessments. The purpose of the current study was to investigate the use of a procedure to detect drift in the difficulty estimates for a large-scale, high stakes SPE. The results of this investigation suggest that, for particular performance tasks, there was some variation in mean scores over time. These findings indicate that, although it is feasible to create a bank of case-SP means and link scores back to these fixed estimates, special attention must be paid to the standardization of exam materials over time. This is essential to ensure comparability of scores and pass-fail decisions for candidates who are assessed on multiple test forms throughout the year.

Accumulating evidence to defend decisions based on scores from evaluations is an ongoing process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment (CSA®). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine's Mini-CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores.

Purpose: The purpose of this study was to explore possible performance differences in interpersonal skills (IPS) ratings as a function of candidate and standardized patient (SP) gender.

The use of experts to judge performance assessments is desirable because ratings of performances, carried out by experts in the content domain of the examination, are often considered to be the "gold standard." However, one drawback of using experts to rate performances is the high cost involved. A more economic alternative for scoring performance assessments entails using analytic scoring, which typically involves assigning points to individual traits present in the performance, and summing to arrive at a single score. This strategy is less costly, but may lack the richness of holistic scoring. This study investigates the use of regression-based techniques to predict expert judgments on a written performance task from a combination of analytic scores. Potentially, this will result in scores that approximate the richness of holistic ratings while maintaining the cost-effectiveness of analytic scoring. Results show that a substantial proportion of variance in expert judgments can be explained by the analytic scores, but that decisions based on actual expert judgments and the predicted expert judgments were not sufficiently consistent to warrant the substitution of one score for the other.

Purpose. Post-encounter written exercises (e.g., patient notes) have been included in clinical skills assessments that use standardized patients. The purpose of this study was to estimate the generalizability of the scores from these written exercises when they are rated by various trained health professionals, including physicians. Method. The patient notes from a 10 station clinical skills examination involving 10 first year emergency medicine residents were analytically scored by four rater groups: three physicians, three nurses, three fourth year medical students, three billing clerks. Generalizability analyses were used to partition the various sources of error variance and derive reliability-like coefficients for each group of raters. Results. The generalizability analyses indicated that case-to-case variability was a major source of error variance in the patient note scores. The variance attributable to the rater or to the rater by examinee interaction was negligible. This finding was consistent across the four rater groups. Generalizability coefficients in excess of 0.80 were achieved for each of the four sets of raters. Physicians did, however, produce the most dependable scores. Conclusion. There is little advantage, from a reliability perspective, in using more than one trained physician, or other health professional who is adequately trained to score the patient note. Measurement error is introduced primarily by case sampling variability. This suggests that, if required, increases in the generalizability of the patient note scores can be made through the addition of cases, and not the addition of raters.