Search

Background: To determine the psychometric characteristics of diagnostic justification scores based on the patient note format of the United States Medical Licensing Examination Step 2 Clinical Skills exam, which requires students to document history and physical findings, differential diagnoses, diagnostic justification, and plan for immediate workup., Method: End-of-third-year medical students at one institution wrote notes for five standardized patient cases in May 2013 (n = 180) and 2014 (n = 177). Each case was scored using a four-point rubric to rate each of the four note components. Descriptive statistics and item analyses were computed and a generalizability study done., Results: Across cases, 10% to 48% provided no diagnostic justification or had several missing or incorrect links between history and physical findings and diagnoses. The average intercase correlation for justification scores ranged from 0.06 to 0.16; internal consistency reliability of justification scores (coefficient alpha across cases) was 0.38. Overall, justification scores had the highest mean item discrimination across cases. The generalizability study showed that person-case interaction (12%) and task-case interaction (13%) had the largest variance components, indicating substantial case specificity., Conclusions: The diagnostic justification task provides unique information about student achievement and curricular gaps. Students struggled to correctly justify their diagnoses; performance was highly case specific. Diagnostic justification was the most discriminating element of the patient note and had the greatest variability in student performance across cases. The curriculum should provide a wide range of clinical cases and emphasize recognition and interpretation of clinically discriminating findings to promote the development of clinical reasoning skills.

Background: Availability of reliable, valid, and feasible workplace-based assessment (WBA) tools is important to allow faculty to make important and complex judgments about resident competence. The Minicard is a WBA direct observation tool designed to provide formative feedback while supporting critical competency decisions., Objective: The purpose of this study was to collect validity and feasibility evidence for use of the Minicard for formative assessment of internal medicine residents., Methods: We conducted a retrospective cohort analysis of Minicard observations from 2005-2011 in 1 institution to obtain validity evidence, including content (settings, observation rates, independent raters); response process (rating distributions across the scale and ratings by month in the program); consequences (qualitative assessment of action plans); and feasibility (time to collect observations)., Results: Eighty faculty observers recorded 3715 observations of 73 residents in the inpatient ward (43%), clinic (39%), intensive care (15%), and emergency department (3%) settings. Internal medicine residents averaged 28 (SD=8.4) observations per year from 9 (SD=4.1) independent observers. Minicards had an average of 5 (SD=5.1) discrete recorded observations per card. Rating distributions covered the entire rating scale, and increased significantly over the time in training. Half of the observations included action plans with action-oriented feedback, 11% had observational feedback, 9% had minimal feedback, and 30% had no recorded plan. Observations averaged 15.6 (SD=9.5) minutes., Conclusions: Validity evidence for the Minicard direct observation tool demonstrates its ability to facilitate identification of "struggling" residents and provide feedback, supporting its use for the formative assessment of internal medicine residents.

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals. Formal blueprinting was performed and content experts were used to develop the cases and rating instruments. Examiners underwent a frame-of-reference training session to prepare them for their rater role. Scores were compared by levels of training, experience, and to evaluation data from a non-procedural OSCE (IM-OSCE). Reliability was calculated using Generalizability analyses. Reliabilities for the technical and non-technical scores were 0.68 and 0.76, respectively. Third-year residents scored significantly higher than first-year residents on the technical (73.5 vs. 62.2%) and non-technical (83.2 vs. 75.1%) components of the PS-OSCE (p < 0.05). Residents who had performed the procedures more frequently scored higher on three of the five stations (p < 0.05). There was a moderate disattenuated correlation (r = 0.77) between the IM-OSCE and the technical component of the PS-OSCE scores. The PS-OSCE is a feasible method for assessing multiple competencies related to performing procedures and this study provides validity evidence to support its use as an in-training examination.

Objectives: Despite significant evidence supporting the use of three-option multiple-choice questions (MCQs), these are rarely used in written examinations for health professions students. The purpose of this study was to examine the effects of reducing four- and five-option MCQs to three-option MCQs on response times, psychometric characteristics, and absolute standard setting judgements in a pharmacology examination administered to health professions students., Methods: We administered two versions of a computerised examination containing 98 MCQs to 38 Year 2 medical students and 39 Year 3 pharmacy students. Four- and five-option MCQs were converted into three-option MCQs to create two versions of the examination. Differences in response time, item difficulty and discrimination, and reliability were evaluated. Medical and pharmacy faculty judges provided three-level Angoff (TLA) ratings for all MCQs for both versions of the examination to allow the assessment of differences in cut scores., Results: Students answered three-option MCQs an average of 5 seconds faster than they answered four- and five-option MCQs (36 seconds versus 41 seconds; p = 0.008). There were no significant differences in item difficulty and discrimination, or test reliability. Overall, the cut scores generated for three-option MCQs using the TLA ratings were 8 percentage points higher (p = 0.04)., Conclusions: The use of three-option MCQs in a health professions examination resulted in a time saving equivalent to the completion of 16% more MCQs per 1-hour testing period, which may increase content validity and test score reliability, and minimise construct under-representation. The higher cut scores may result in higher failure rates if an absolute standard setting method, such as the TLA method, is used. The results from this study provide a cautious indication to health professions educators that using three-option MCQs does not threaten validity and may strengthen it by allowing additional MCQs to be tested in a fixed amount of testing time with no deleterious effect on the reliability of the test scores., (© 2014 John Wiley & Sons Ltd.)

Summary Statement: Checklists for basic procedural skills require an approach to standard setting in which patient safety concerns are paramount. We suggest that faculty members identify dimensions such as patient or clinician safety, procedure outcome, and patient comfort that are essential for a given procedure and demand for essential checklist items a high level of mastery that is not compensated by performance of nonessential items. This approach differs from traditional standard setting methods developed for written examinations, which allow examinees to miss a given percentage of items regardless of the specific items missed. Application of cut scores based on this patient safety standard resulted in requiring a substantial number of students to retrain and retest; these students would have passed if the traditional Angoff method had been used. Only by demanding a high level of mastery in the simulation laboratory can we promote patient safety in the messy and unpredictable real world.

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving learners are more likely to use. The purpose of this study was to determine if limiting checklist items to clinically discriminating items and/or adding missing evidence-based items improved score reliability in an Internal Medicine residency OSCE. Six internists reviewed the traditional checklists of four OSCE stations classifying items as clinically discriminating or non-discriminating. Two independent reviewers augmented checklists with missing evidence-based items. We used generalizability theory to calculate overall reliability of faculty observer checklist scores from 45 first and second-year residents and predict how many 10-item stations would be required to reach a Phi coefficient of 0.8. Removing clinically non-discriminating items from the traditional checklist did not affect the number of stations (15) required to reach a Phi of 0.8 with 10 items. Focusing the checklist on only evidence-based clinically discriminating items increased test score reliability, needing 11 stations instead of 15 to reach 0.8; adding missing evidence-based clinically discriminating items to the traditional checklist modestly improved reliability (needing 14 instead of 15 stations). Checklists composed of evidence-based clinically discriminating items improved the reliability of checklist scores and reduced the number of stations needed for acceptable reliability. Educators should give preference to evidence-based items over non-evidence-based items when developing OSCE checklists.

Purpose: High-quality checklists are essential to performance test score validity. Prior research found that physical exam checklists of items that clinically discriminated between competing diagnoses provided more generalizable scores than all-encompassing thoroughness checklists. The purpose of this study was to compare validity evidence for clinically discriminating versus thoroughness checklists, hypothesizing that evidence would favor the former., Method: Faculty at four Chicago-area medical schools developed six standardized patient (SP) cases with checklists of about 20 items ("thoroughness [long] checklists"). Four clinicians identified a subset of items that clinically discriminated between competing diagnoses of each case ("clinically discriminating [short] checklists"). Cases were administered to 155 University of Illinois at Chicago fourth-year medical students during their 2011 Clinical Skills Examination (CSE). Validity evidence was compared for CSE scores based on thoroughness versus clinically discriminating checklist items., Results: Validity evidence favoring clinically discriminating checklists included response process: greater SP checklist accuracy (kappa = 0.75 for long and 0.84 for short checklists, P < .05); internal structure: better item discrimination (0.28 long, 0.42 short, P < .001); internal consistency reliability (0.80 long, 0.92 short); standard error of measurement (z score 8.87 long, 8.05 short); and generalizability (G = 0.504 long, 0.533 short). There were no significant differences overall in relevance ratings, item difficulty, or cut scores of long versus short checklist items., Conclusions: Limiting checklist items to those affecting diagnostic decisions resulted in better accuracy and psychometric indices. Thoroughness items performed without thinking do not reflect clinical reasoning ability and contribute construct-irrelevant variance to scores.

Background: Graduate medical education is undergoing a dramatic shift toward competency-based assessment of learners. Competency assessment requires clear definitions of competency and validated assessment methods. The purpose of this study is to identify criteria used by surgical educators to judge competence in Practice-Based Learning and Improvement (PBL&I) as demonstrated in learning portfolios., Methods: A total of 6 surgical learning and instructional portfolio entries served as documents to be assessed by 3 senior surgical educators. These faculty members were asked to rate and then identify criteria used to assess PBL&I competency. Individual interviews and group discussions were conducted, recorded, and transcribed to serve as the study dataset. Analysis was performed using qualitative methodology to identify themes for the purpose of defining competence in PBL&I. The assessment themes derived are presented with narrative examples to describe the progression of competency., Results: The collaborative coding process resulted in identification of 7 themes associated with competency in PBL&I related to surgical learning and instructional portfolio entries: (1) self-awareness regarding effect of actions; (2) identification and thorough description of learning goals; (3) cases used as catalyst for reflection; (4) reconceptualization with appropriate use and critique of cited literature; (5) communication skills/completeness of entry template; (6) description of future behavioral change; and (7) engagement in process--identifies as personally relevant., Conclusions: The identified themes are consistent with and complement other criteria emerging from reflective practice literature and experiential learning theory. This study provides a foundation for further development of a tool for assessing learner portfolios consistent with the Accreditation Council for Graduate Medical Education's Next Accreditation System requirements., (Copyright © 2014 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Purpose: This study examines validity evidence for the Patient Note Scoring Rubric, which was developed for a local graduation competency exam (GCE) to assess patient notes written in the new United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills format. The rubric was designed to measure three dimensions: Documentation, justified differential diagnosis (DDX), and Workup., Method: Analyses used GCE data from 170 fourth-year medical students who completed five standardized patient (SP) cases in May 2012. Five physician raters each scored all responses for one case. Internal structure was examined using correlations between dimensions and between cases; a generalizability study was also conducted. Relationship to other variables was examined by correlating patient note scores with SP encounter scores. Consequence was assessed by comparing pass-fail rates between the rubric and the previous global rating. Response process was examined using rater feedback., Results: Correlations between scores from different dimensions ranged between 0.33 and 0.44. Reliability of scores based on the phi coefficient was 0.43; 15 cases were required to reach a phi coefficient of 0.70. Evidence of case specificity was found. Documentation scores were moderately correlated with SP scores for data gathering (r = 0.47, P < .001). There was no meaningful change in pass-fail rates. Raters' feedback indicated that they required more training for scoring the DDX and Workup dimensions., Conclusions: There is initial validity evidence for use of this rubric to score local clinical exams that are based on the new USMLE patient note format.

Introduction: The purpose was to evaluate the Bookmark standard-setting method for use on a performance-based assessment in medical education., Methods: We compared cutscores for Aseptic performance assessment using the modified Angoff, Hofstee and modified Bookmark methods., Results: The Angoff produced a cutscore of 62%, SD=18 and a percent passing (pp)= 64%. The Hofstee cutscore was 71%, SD=7 and pp=46%. Bookmark mean cutscores were 65.9% SD=10.7 and pp=42% for advanced beginners; 83.6%, SD=9.2 and pp=17% for competent and the proficient category resulted in a cutscore of 96.4% SD=3.9 and pp=1%. Faculty judges found the Bookmark method to be an easy and acceptable method., Conclusions: The Bookmark method was acceptable to faculty, has reasonable quality metrics when compared to other methods and can be a practical tool for establishing standards in performance-based examinations. The Bookmark method could be useful for establishing multiple levels of competency using the Dreyfus criteria.

Background: Despite wide-spread endorsement of patient-centered communication (PCC) in health care, there has been little evidence that it leads to positive change in health outcomes. The lack of correlation may be due either to an overestimation of the value of PCC or to a measurement problem. If PCC measures do not capture elements of the interaction that determine whether the resulting care plan is patient-centered, they will confound efforts to link PCC to outcomes., Objective: To evaluate whether one widely used measure of PCC, the Roter Interaction Analysis System (RIAS), captures patient-centered care planning., Design: RIAS was employed in the coding of unannounced standardized patient (USP) encounters that were scripted so that the failure to address patient contextual factors would result in an ineffective plan of care. The design enabled an assessment of whether RIAS can differentiate between communication behavior that does and does not result in a care plan that takes into account a patient's circumstances and needs., Participants: Eight actors role playing four scripted cases (one African American and one Caucasian for each case) in 399 visits to 111 internal medicine attending physicians., Main Measures: RIAS measures included composites for physician utterance types and (in separate models) two different previously applied RIAS patient-centeredness summary composites. The gold standard comparison measure was whether the physician's treatment plan, as abstracted from the visit note, successfully addressed the patient's problem. Mixed effects regression models were used to evaluate the relationship between RIAS measures and USP measured performance, controlling for a variety of design features., Key Results: None of the RIAS measures of PCC differentiated encounters in which care planning was patient-centered from care planning in which it was not., Conclusions: RIAS, which codes each utterance during a visit into mutually exclusive and exhaustive categories, does not differentiate between conversations leading to and not leading to care plans that accommodate patients' circumstances and needs.

Introduction: Ventriculostomy is a neurosurgical procedure for providing therapeutic cerebrospinal fluid drainage. Complications may arise during repeated attempts at placing the catheter in the ventricle. We studied the impact of simulation-based practice with a library of virtual brains on neurosurgery residents' performance in simulated and live surgical ventriculostomies., Methods: Using computed tomographic scans of actual patients, we developed a library of 15 virtual brains for the ImmersiveTouch system, a head- and hand-tracked augmented reality and haptic simulator. The virtual brains represent a range of anatomies including normal, shifted, and compressed ventricles. Neurosurgery residents participated in individual simulator practice on the library of brains including visualizing the 3-dimensional location of the catheter within the brain immediately after each insertion. Performance of participants on novel brains in the simulator and during actual surgery before and after intervention was analyzed using generalized linear mixed models., Results: Simulator cannulation success rates increased after intervention, and live procedure outcomes showed improvement in the rate of successful cannulation on the first pass. However, the incidence of deeper, contralateral (simulator) and third-ventricle (live) placements increased after intervention. Residents reported that simulations were realistic and helpful in improving procedural skills such as aiming the probe, sensing the pressure change when entering the ventricle, and estimating how far the catheter should be advanced within the ventricle., Conclusions: Simulator practice with a library of virtual brains representing a range of anatomies and difficulty levels may improve performance, potentially decreasing complications due to inexpert technique.

Context: In a past study using unannounced standardised patients (USPs), substantial rates of diagnostic and treatment errors were documented among internists. Because the authors know the correct disposition of these encounters and obtained the physicians' notes, they can identify necessary treatment that was not provided and unnecessary treatment. They can also discern which errors can be identified exclusively from a review of the medical records., Objective: To estimate the avoidable direct costs incurred by physicians making errors in our previous study., Design: In the study, USPs visited 111 internal medicine attending physicians. They presented variants of four previously validated cases that jointly manipulate the presence or absence of contextual and biomedical factors that could lead to errors in management if overlooked. For example, in a patient with worsening asthma symptoms, a complicating biomedical factor was the presence of reflux disease and a complicating contextual factor was inability to afford the currently prescribed inhaler. Costs of missed or unnecessary services were computed using Medicare cost-based reimbursement data., Setting: Fourteen practice locations, including two academic clinics, two community-based primary care networks with multiple sites, a core safety net provider, and three Veteran Administration government facilities., Main Outcome Measures: Contribution of errors to costs of care., Results: Overall, errors in care resulted in predicted costs of approximately $174,000 across 399 visits, of which only $8745 was discernible from a review of the medical records alone (without knowledge of the correct diagnoses). The median cost of error per visit with an incorrect care plan differed by case and by presentation variant within case., Conclusions: Chart reviews alone underestimate costs of care because they typically reflect appropriate treatment decisions conditional on (potentially erroneous) diagnoses. Important information about patient context is often entirely missing from medical records. Experimental methods, including the use of USPs, reveal the substantial costs of these errors.

Purpose: Consultation evaluation tools with strong validity evidence are scarce. The goal of this study was to demonstrate evidence of the validity of a new checklist, the 5Cs model (contact, communicate, core question, collaboration, and closing the loop), used to assess simulated emergent telephone consultations between residents and attending physician consultants., Method: The authors developed items for the checklist based on a theoretical model, and content experts constructed two simulated emergent cases (one psychiatric and one surgical). In 2010, residents conducted consultations on the two cases by telephone with a standardized physician consultant, and three clinical faculty members rated the residents' consultations using the checklist. The authors analyzed the residents' checklist scores for response process validity, internal structure validity, and validity concerning the relationship to other variables., Results: The authors established the checklist's content validity through its development based on a theoretical model, literature, and expert input. Response process validity evidence includes interrater reliability based on an intraclass correlation of 0.94. Internal structure validity evidence includes a generalizability analysis yielding a phi coefficient of 0.89 with three raters across two cases and a decision study yielding a phi coefficient of 0.8 with a single rater and three cases. Relationship to other variables includes correlations of 0.59 and 0.79 between mean scores on a global rating scale and on the checklist for the two simulated cases., Conclusions: Multiple sources of validity evidence support the use of the 5Cs model as a tool to assess telephone consultations.

Objectives: The objective was to evaluate whether a standardized consultation model in the emergency department (ED), the 5 Cs of Consultation (Contact, Communicate, Core Question, Collaboration, and Closing the Loop), would improve physicians' ability to relay appropriate information and communicate successfully during a consultation., Methods: This was a prospective, randomized study at a large, academic, urban, tertiary care medical center in Chicago. Forty-three emergency medicine (EM) and EM/internal medicine (EM/IM) residents were randomized into two groups, an intervention group and an unstructured group, stratified by postgraduate year (PGY). Intervention group participants received an interactive educational session on the 5 Cs of Consultation, a standardized consultation model. Intervention and unstructured groups placed two simulated consultation phone calls, based on pretested simulated patient cases, to a standardized consultant. Three raters, naive to the consultation model and blinded to group assignments, individually assessed recordings of each call using a seven-item, five-point global rating scale (GRS). Finally, an attending surgeon and an attending psychiatrist each rated respective cases using a single global rating to provide validity evidence for the scale., Results: Residents trained with the 5 Cs model communicated significantly better, regardless of PGY and clinical case. The intervention group had significantly higher mean GRS scores than the unstructured group (4.1 vs. 3.5, F(1,39) = 33.5, p < 0.0001). Secondary analysis of the recordings suggested that encounters with more 5 Cs behaviors tended to receive higher GRS scores., Conclusions: A standardized educational model increased the effectiveness of consultation communication from the ED. Residents trained with the 5 Cs of Consultation scored better on consultation assessments compared with untrained residents. Training programs should consider adopting standardized consultation models., (© 2012 by the Society for Academic Emergency Medicine.)

Introduction: The physical examination is an essential clinical competence for all physicians. Most medical schools have students who learn the physical examination maneuvers using a head-to-toe approach. However, this promotes a rote approach to the physical exam, and it is not uncommon for students later on to fail to appreciate the meaning of abnormal findings and their contribution to the diagnostic reasoning process. The purpose of the project was to develop a model teaching session for the hypothesis-driven physical examination (HDPE) approach in which students could practice the physical examination in the context of diagnostic reasoning., Methods: We used an action research methodology to create this HDPE model by developing a teaching session, implementing it over 100 times with approximately 700 students, conducting internal reflection and external evaluations, and making adjustments as needed., Results: A model nine-step HDPE teaching session was developed, including: (1) orientation, (2) anticipation, (3) preparation, (4) role play, (5) discussion-1, (6) answers, (7) discussion-2, (8) demonstration and (9) reflection., Discussions and Conclusions: A structured model HDPE teaching session and tutor guide were developed into a workable instructional intervention. Faculty members are invited to teach the physical examination using this model.

Background: Over years, surgical training is changing and years of tradition are being challenged by legal and ethical concerns for patient safety, work hour restrictions, and the cost of operating room time. Surgical simulation and skill training offer an opportunity to teach and practice advanced techniques before attempting them on patients. Simulation training can be as straightforward as using real instruments and video equipment to manipulate simulated "tissue" in a box trainer. More advanced virtual reality (VR) simulators are now available and ready for widespread use. Early systems have demonstrated their effectiveness and discriminative ability. Newer systems enable the development of comprehensive curricula and full procedural simulations., Methods: A PubMed review of the literature was performed for the MESH words "Virtual reality, "Augmented Reality", "Simulation", "Training", and "Neurosurgery". Relevant articles were retrieved and reviewed. A review of the literature was performed for the history, current status of VR simulation in neurosurgery., Results: Surgical organizations are calling for methods to ensure the maintenance of skills, advance surgical training, and credential surgeons as technically competent. The number of published literature discussing the application of VR simulation in neurosurgery training has evolved over the last decade from data visualization, including stereoscopic evaluation to more complex augmented reality models. With the revolution of computational analysis abilities, fully immersive VR models are currently available in neurosurgery training. Ventriculostomy catheters insertion, endoscopic and endovascular simulations are used in neurosurgical residency training centers across the world. Recent studies have shown the coloration of proficiency with those simulators and levels of experience in the real world., Conclusion: Fully immersive technology is starting to be applied to the practice of neurosurgery. In the near future, detailed VR neurosurgical modules will evolve to be an essential part of the curriculum of the training of neurosurgeons.

Purpose: To identify the qualities and skills of exemplary and ideal pediatric hospitalist educators., Method: The authors conducted a prospective, multi-institutional qualitative study from November 2008 through January 2009 in which they interviewed pediatric hospitalists who were identified as exemplary educators at three academic pediatric residency programs. They then conducted focus groups with residents and medical students who had recently worked with these hospitalists. Qualitative analysis was used to identify themes., Results: All six hospitalists identified as exemplary participated. Among invited learners, 14/18 residents (78%) and 16/18 medical students (89%) participated. Together, the participants contributed 266 comments, which the authors categorized into 36 themes within the four domains of teaching skills, personal qualities, patient care skills, and role modeling. New qualities and skills--including self-reflection/insight, encouraging autonomy, time management, knowledge acquisition, and systems knowledge--and differences in perceptions among hospitalists, residents, and students were identified. Differences between the qualities and skills of actual exemplary hospitalist educators and perceptions of those of an ideal hospitalist educator were also identified., Conclusions: Pediatric hospitalists in academic residency programs have unique opportunities to significantly affect the education of medical students and residents. This study validates and expands on prior studies of the qualities and skills needed to be a successful hospitalist educator. Researchers and educators designing faculty development programs to train more successful hospitalist educators may wish to target these qualities and skills as well as the differences in medical student and resident needs.

The investigators used evidence based on response processes to evaluate and improve the validity of scores on the Patient-Centered Communication and Interpersonal Skills (CIS) Scale for the assessment of residents' communication competence. The investigators retrospectively analyzed the communication skills ratings of 68 residents at the University of Illinois at Chicago (UIC). Each resident encountered six standardized patients (SPs) portraying six cases. SPs rated the performance of each resident using the CIS Scale--an 18-item rating instrument asking for level of agreement on a 5-category scale. A many-faceted Rasch measurement model was used to determine how effectively each item and scale on the rating instrument performed. The analyses revealed that items were too easy for the residents. The SPs underutilized the lowest rating category, making the scale function as a 4-category rating scale. Some SPs were inconsistent when assigning ratings in the middle categories. The investigators modified the rating instrument based on the findings, creating the Revised UIC Communication and Interpersonal Skills (RUCIS) Scale--a 13-item rating instrument that employs a 4-category behaviorally anchored rating scale for each item. The investigators implemented the RUCIS Scale in a subsequent communication skills OSCE for 85 residents. The analyses revealed that the RUCIS Scale functioned more effectively than the CIS Scale in several respects (e.g., a more uniform distribution of ratings across categories, and better fit of the items to the measurement model). However, SPs still rarely assigned ratings in the lowest rating category of each scale.

Context: Diagnostic accuracy is maximised by having clinical signs and diagnostic hypotheses in mind during the physical examination (PE). This diagnostic reasoning approach contrasts with the rote, hypothesis-free screening PE learned by many medical students. A hypothesis-driven PE (HDPE) learning and assessment procedure was developed to provide targeted practice and assessment in anticipating, eliciting and interpreting critical aspects of the PE in the context of diagnostic challenges., Objectives: This study was designed to obtain initial content validity evidence, performance and reliability estimates, and impact data for the HDPE procedure., Methods: Nineteen clinical scenarios were developed, covering 160 PE manoeuvres. A total of 66 Year 3 medical students prepared for and encountered three clinical scenarios during required formative assessments. For each case, students listed anticipated positive PE findings for two plausible diagnoses before examining the patient; examined a standardised patient (SP) simulating one of the diagnoses; received immediate feedback from the SP, and documented their findings and working diagnosis. The same students later encountered some of the scenarios during their Year 4 clinical skills examination., Results: On average, Year 3 students anticipated 65% of the positive findings, correctly performed 88% of the PE manoeuvres and documented 61% of the findings. Year 4 students anticipated and elicited fewer findings overall, but achieved proportionally more discriminating findings, thereby more efficiently achieving a diagnostic accuracy equivalent to that of students in Year 3. Year 4 students performed better on cases on which they had received feedback as Year 3 students. Twelve cases would provide a reliability of 0.80, based on discriminating checklist items only., Conclusions: The HDPE provided medical students with a thoughtful, deliberate approach to learning and assessing PE skills in a valid and reliable manner.

An Objective Structured Clinical Examination (OSCE) is an effective method for evaluating competencies. However, scores obtained from an OSCE are vulnerable to many potential measurement errors that cases, items, or standardized patients (SPs) can introduce. Monitoring these sources of errors is an important quality control mechanism to ensure valid interpretations of the scores. We describe how one can use generalizability theory (GT) and many-faceted Rasch measurement (MFRM) approaches in quality control monitoring of an OSCE. We examined the communication skills OSCE of 79 residents from one Midwestern university in the United States. Each resident performed six communication tasks with SPs, who rated the performance of each resident using 18 5-category rating scale items. We analyzed their ratings with generalizability and MFRM studies. The generalizability study revealed that the largest source of error variance besides the residual error variance was SPs/cases. The MFRM study identified specific SPs/cases and items that introduced measurement errors and suggested the nature of the errors. SPs/cases were significantly different in their levels of severity/difficulty. Two SPs gave inconsistent ratings, which suggested problems related to the ways they portrayed the case, their understanding of the rating scale, and/or the case content. SPs interpreted two of the items inconsistently, and the rating scales for two items did not function as 5-category scales. We concluded that generalizability and MFRM analyses provided useful complementary information for monitoring and improving the quality of an OSCE.

Background: The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories., Method: Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively., Results: Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted., Conclusions: The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.

Purpose: To determine the psychometric properties and validity of an OSCE to assess the competencies of Practice-Based Learning and Improvement (PBLI) and Systems-Based Practice (SBP) in graduate medical education., Method: An eight-station OSCE was piloted at the end of a three-week Quality Improvement elective for nine preventive medicine and endocrinology fellows at Mayo Clinic. The stations assessed performance in quality measurement, root cause analysis, evidence-based medicine, insurance systems, team collaboration, prescription errors, Nolan's model, and negotiation. Fellows' performance in each of the stations was assessed by three faculty experts using checklists and a five-point global competency scale. A modified Angoff procedure was used to set standards. Evidence for the OSCE's validity, feasibility, and acceptability was gathered., Results: Evidence for content and response process validity was judged as excellent by institutional content experts. Interrater reliability of scores ranged from 0.85 to 1 for most stations. Interstation correlation coefficients ranged from -0.62 to 0.99, reflecting case specificity. Implementation cost was approximately $255 per fellow. All faculty members agreed that the OSCE was realistic and capable of providing accurate assessments., Conclusions: The OSCE provides an opportunity to systematically sample the different subdomains of Quality Improvement. Furthermore, the OSCE provides an opportunity for the demonstration of skills rather than the testing of knowledge alone, thus making it a potentially powerful assessment tool for SBP and PBLI. The study OSCE was well suited to assess SBP and PBLI. The evidence gathered through this study lays the foundation for future validation work.

The authors used a many-faceted Rasch measurement model to analyze rating data from a clinical skills assessment of 173 fourth-year medical students to investigate four types of rater errors: leniency, inconsistency, the halo effect, and restriction of range. Students performed six clinical tasks with 6 standardized patients (SPs) selected from a pool of 17 SPs. SPs rated the performance of each student in six skills: history taking, physical examination, interpersonal skills, communication technique, counseling skills, and physical examination etiquette. SPs showed statistically significant differences in their rating severity, indicating rater leniency error. Four SPs exhibited rating inconsistency. Four SPs restricted their ratings in high categories. Only 1 SP exhibited a halo effect. Administrators of objective structured clinical examinations should be vigilant for various types of rater errors and attempt to reduce or eliminate those errors to improve the validity of inferences based on objective structured clinical examination scores.

Background: Establishing credible, defensible, and acceptable passing scores for performance-type examinations in real-world settings is a challenge for health professions educators. Our purpose in this article is to provide step-by-step instructions with worked examples for 5 absolute standard-setting methods that can be used to establish acceptable passing scores for performance examinations such as Objective Structured Clinical Examinations or standardized patient encounters., Summary: All standards reflect the subjective opinions of experts. In this "how-to" article, we demonstrate procedures for systematically capturing these expert opinions using 5 research-based methods (Angoff, Ebel, Hofstee, Borderline Group, and Contrasting Groups). We discuss issues relating to selection of judges, use of performance data, and decision-making processes., Conclusions: Different standard-setting methods produce different passing scores; there is no "gold standard." The key to defensible standards lies in the choice of credible judges and in the use of a systematic approach to collecting their judgments. Ultimately, all standards are policy decisions.

Purpose: The authors describe the development and validation of an institution-wide, cross-specialty assessment of residents' communication and interpersonal skills, including related components of patient care and professionalism., Method: Residency program faculty, the department of medical education, and the Clinical Performance Center at the University of Illinois at Chicago College of Medicine collaborated to develop six standardized patient-based clinical simulations. The standardized patients rated the residents' performance. The assessment was piloted in 2003 for internal medicine and family medicine and was subsequently adapted for other specialties, including surgery, pediatrics, obstetrics-gynecology, and neurology. We present validity evidence based on the content, internal structure, relationship to other variables, feasibility, acceptability, and impact of the 2003 assessment., Results: Seventy-nine internal medicine and family medicine residents participated in the initial administration of the assessment. A factor analysis of the 18 communication scale items resulted in two factors interpretable as "communication" and "interpersonal skills." Median internal consistency of the scale (coefficient alpha) was 0.91. Generalizability of the assessment ranged from 0.57 to 0.82 across specialties. Case-specific items provided information about group-level deficiencies. Cost of the assessment was about $250 per resident. Once the initial cases had been developed and piloted, they could be adapted for other specialties with minimal additional effort, at a cost saving of about $1,000 per program., Conclusion: Centrally developed, institution-wide competency assessment uses resources efficiently to relieve individual programs of the need to "reinvent the wheel" and provides program directors and residents with useful information for individual and programmatic review.

Problem Statement and Purpose: The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance., Method: Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE)., Results: Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009)., Conclusion: Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.

Objective: The Accreditation Council for Graduate Medical Education (ACGME) has challenged program directors to assess their residents' core competencies, including communication and interpersonal skills (CIS). We report our institution's experience using a series of standardized patient encounters in an objective structured clinical examination (OSCE) to evaluate CIS in surgical residents., Methods: Standardized patients rated the residents' ability to maintain a patient-centered approach across 6 challenging communication tasks. Residents received verbal feedback from the patients after each encounter and completed a survey indicating their experience and comfort with each task. Individual and group reports documented resident competency and provided aggregate information for curriculum review. Formal grades were not assigned., Results: Twenty-two residents in 2 surgical residency programs piloted the assessment. The Generalizability of the assessment was 0.81. Scores of second- and third-year residents were not significantly different. Residents found the program to be helpful and able to assess their skills., Conclusions: The standardized patient-based OSCE is an effective method to assess communication and interpersonal skills and provides useful information for curriculum review.

Second-year medical students performed a head-to-toe screening physical examination on trained patient instructors (PIs) as a high-stakes examination before beginning clinical clerkships. PIs completed a 138-item checklist and instructed the students in the proper performance of any incorrectly executed or omitted items. To assess the accuracy of the PI ratings, 'monitor' PIs watching the encounter by video completed checklists for 11 retests of failed students in 2001 and for 28 randomly selected encounters in 2002. Checklist-level discrepancies measured by mean absolute checklist difference were 8% in 2001 and 9% in 2002. Case-level mean difference between raters in 2002 was -2.9%. The inter-rater reliability for a single rater, estimated by the intra-class correlation coefficient for total encounter scores, was 0.95. The assessment and feedback were well received by the students. The Head-to-Toe examination provides a reliable assessment of a student's ability to carry out a complete screening physical examination. Patient instructors can be trained to acceptable accuracy and reliability despite the length and complexity of the exercise, and can provide effective feedback and teaching targeted to individual student deficiencies.

Purpose: We describe the use of standardised students (SSs) in interdisciplinary faculty development programmes to improve clinical teaching skills. Standardised students are actual health professions students who are trained to portray a prototypical teaching challenge consistently across many encounters with different faculty participants., Methods: The faculty development programmes described focused on the skills of providing feedback and brief clinical teaching. At the beginning of each session, each participant was videotaped in encounters with 2 different SSs. Using microteaching (an instructional method in which learners view short segments of their own videotaped performance and discuss the tapes with a facilitator, consultant or other workshop participants), each group of participants and instructors reviewed the tapes and reflected on the encounters, providing immediate feedback to participants and modelling different approaches to the same teaching problem. The same process was repeated with more complicated scenarios after 2 weeks and again after 6 months offering reinforcement, further practice and more sophisticated development of the strategies learned. Participants completed post-session evaluations and a follow-up telephone survey., Results: A total of 36 faculty members from the colleges of medicine, dentistry, pharmacy and nursing participated in workshops in 2000-01. The workshops were rated as highly relevant to participants' teaching, and most participants reported that they had learned a great deal. Participants most appreciated reviewing the videotaped interactions, the feedback they received, the interactions with their colleagues, the interdisciplinary nature of the groups and the practical focus of the workshops., Conclusions: Standardised students provide a high fidelity, low risk, simulated environment in which faculty can reflect on and experiment with new teaching behaviours. Such encounters can enhance the effectiveness and impact of faculty development programmes to improve clinical teaching skills.

Standardized patients (SPs) were used in a 9-week Introduction to Psychodynamic Psychotherapy course for 11 first-year psychiatry residents. After 7 weeks of coursework, each resident conducted a simulated initial psychotherapy session. The SPs used were experienced in simulating psychiatric disorders and were free to use as much or as little personal history as they wished. Session ratings by the SPs afforded written feedback to the residents. The sessions were videotaped and selected segments viewed and discussed by the class. Residents kept their tapes and were encouraged to review them on their own and with their supervisors. Residents, SPs, and class instructor all rated the experience very positively. Standardized patients may be a useful adjunct to psychotherapy education. They can provide valuable learning opportunities without giving rise to concerns that complicate the videotaping of actual patients.

Objective: If faculty development programs are to have impact, we believe they should be made up of several self-reinforcing workshops that provide opportunities for behavior review, practice, reflection, and reinforcement within a context of interdisciplinary perspectives. A program was developed that supports these four activities and includes clinical faculty from medicine, dentistry, nursing, and pharmacy., Description: At the University of Illinois at Chicago we have focused our efforts to improve clinical teaching through a program of two parallel courses. One course focuses on the "one-minute teacher" approach to clinical teaching. We call this course "Teaching-on-the-Fly," reflecting what many of our faculty attendees have suggested they do. The second course is "Feedback Strategies." The two courses are identical in format, which is a series of three three-to-four-hour small-group workshops of not more than eight participants. The first two workshops in each series are titled "Teaching-on-the-Fly I and II" and "Feedback Strategies I and II," respectively. Workshop I is separated from Workshop II by two weeks. During this hiatus faculty are urged to practice the skills learned and to reflect on what worked and what did not work in Workshop I. Six months following both series' Workshop II, "Advanced Teaching-on-the-Fly" and "Advanced Feedback Strategies" workshops are held. Each workshop begins with videotaping of two separate teaching encounters in which each participant interacts with standardized students. Standardized student and clinical faculty interactions are meant to mimic typical teaching and feedback situations. Teaching encounters are limited to five minutes, maximizing the similarity to the stress and frenetic quality of many of today's clinical settings. In Workshop II and in the advanced workshops the standardized students offer one positive comment to the faculty immediately following their encounter. Students are taught how to give this feedback. Immediately after videotaping, faculty adjourn to a small-group discussion. Workshops consist of reflective discussions during the review of each participant's videotape, brief lectures, and responsive comments and instruction by the workshop leaders. Participants are asked to reflect on each workshop and to bring examples of problems and opportunities gained to succeeding workshops in the series., Discussion: Several theories support these clinical teaching workshops. (1) Outcomes research in continuing medical education suggests the need for ongoing reinforcement, which we do structurally through the three-session model. (2) We use a classical microteaching approach to develop insight and self-awareness. Each videotaped encounter is reviewed, stopped at key points, and discussed by the entire group. These discussions commonly open up after the workshop leaders ask questions such as, "What were you thinking there?" or "What were you trying to do?" or "What would you ask next?" (3) We emphasize the importance of knowing-in-action and the related reflection that guides action in practice. (4) The quality of the workshops is enhanced using standardized students, whom we carefully train and use repeatedly. At least two students have worked with us from their first years through their final clinical years. We are currently examining the program's impact through videotape review.

Purpose: To determine whether psychiatry program directors and residents agree on the characteristics most important in determining the quality of a residency program., Method: The authors carried out factor analyses of the results of two national surveys that asked participants to rate the importance of 41 items in determining the quality of residency programs: a 1997 survey completed by 180 psychiatry residents and a 1998 survey completed by 234 psychiatry program directors and rotation heads., Results: Residents' factors determining program quality were the interpersonal culture in the program, the curriculum, academic resources and opportunities, clinical resources and opportunities, and outcomes. Program directors' factors were program administration, curriculum and clinical resources, the quality of the institution, the supportiveness of the program, and individual preferences., Conclusions: Program directors and residents focus on different indicators of the quality of residency programs, differences that can be conceptualized as those between the "producer" and the "consumer" of the program. Four domains appear to underlie the results of the resident and faculty factor analyses: the context, content, culture, and consequences of the program. Multidimensional evaluations by multiple stakeholders may be the most appropriate way to evaluate the quality of residency programs. These results also strongly suggest that the interpersonal culture of a program should be assessed as part of its evaluation process.