Search

Differential rater function over time (DRIFT) and contrast effects (examiners' scores biased away from the standard of preceding performances) both challenge the fairness of scoring in objective structured clinical exams (OSCEs). This is important as, under some circumstances, these effects could alter whether some candidates pass or fail assessments. Benefitting from experimental control, this study investigated the causality, operation and interaction of both effects simultaneously for the first time in an OSCE setting.We used secondary analysis of data from an OSCE in which examiners scored embedded videos of student performances interspersed between live students. Embedded video position varied between examiners (early vs. late) whilst the standard of preceding performances naturally varied (previous high or low). We examined linear relationships suggestive of DRIFT and contrast effects in all within-OSCE data before comparing the influence and interaction of 'early' versus 'late' and 'previous high' versus 'previous low' conditions on embedded video scores.Linear relationships data did not support the presence of DRIFT or contrast effects. Embedded videos were scored higher early (19.9 [19.4-20.5]) versus late (18.6 [18.1-19.1], p 0.001), but scores did not differ between previous high and previous low conditions. The interaction term was non-significant.In this instance, the small DRIFT effect we observed on embedded videos can be causally attributed to examiner behaviour. Contrast effects appear less ubiquitous than some prior research suggests. Possible mediators of these finding include the following: OSCE context, detail of task specification, examiners' cognitive load and the distribution of learners' ability. As the operation of these effects appears to vary across contexts, further research is needed to determine the prevalence and mechanisms of contrast and DRIFT effects, so that assessments may be designed in ways that are likely to avoid their occurrence. Quality assurance should monitor for these contextually variable effects in order to ensure OSCE equivalence.

CONTEXT \ud Globally, primary health care is facing workforce shortages. Longer and higher-quality placements in primary care increase the likelihood of medical students choosing this specialty. However, the recruitment and retention of community primary care teachers are challenging. Relevant research was predominantly carried out in the 1990s. We seek to understand contemporary facilitators and barriers to general practitioner (GP) engagement with undergraduate education. Communities of practice (CoP) theory offers a novel conceptualisation, which may be pertinent in other community-based teaching settings.\ud \ud METHODS \ud Semi-structured interviews were undertaken with 24 GP teachers at four UK medical schools. We purposively sampled GPs new to teaching, established GP teachers and GPs who had recently stopped teaching. We undertook NVivo-assisted deductive and inductive thematic analysis of transcripts. We used CoP theory to interpret data. RESULTS Communities of practice theory illustrated that teachers negotiate membership of three CoPs: (i) clinical practice; (ii) the medical school, and (iii) teaching. The delivery of clinical care and teaching may be integrated or exist in tension. This can depend upon the positioning of the teaching and teacher as central or peripheral to the clinical CoP. Remuneration, workload, space and the expansion of GP trainee numbers impact on this. Teachers did not identify strongly as members of the medical school or a teaching community. Perceptions of membership were affected by medical school communication and support. The findings demonstrate gaps in medical school recruitment. \ud \ud CONCLUSIONS \ud This research demonstrates the marginalisation of primary care-based teaching and proposes a novel explanation rooted in CoP theory. Concepts including identity and membership may be pertinent to other community-based teaching settings. We recommend that medical schools review and broaden recruitment methods. Teacher retention may be improved by optimising the interface between medical schools and teachers, fostering a teaching community, increasing professional rewards for teaching involvement and altering medical school expectations of learning in primary care.

Context Medical students can have difficulty in distinguishing left from right. Many infamous medical errors have occurred when a procedure has been performed on the wrong side, such as in the removal of the wrong kidney. Clinicians encounter many distractions during their work. There is limited information on how these affect performance. Objectives Using a neuropsychological paradigm, we aim to elucidate the impacts of different types of distraction on left–right (LR) discrimination ability. Methods Medical students were recruited to a study with four arms: (i) control arm (no distraction); (ii) auditory distraction arm (continuous ambient ward noise); (iii) cognitive distraction arm (interruptions with clinical cognitive tasks), and (iv) auditory and cognitive distraction arm. Participants’ LR discrimination ability was measured using the validated Bergen Left–Right Discrimination Test (BLRDT). Multivariate analysis of variance was used to analyse the impacts of the different forms of distraction on participants’ performance on the BLRDT. Additional analyses looked at effects of demographics on performance and correlated participants’ self-perceived LR discrimination ability and their actual performance. Results A total of 234 students were recruited. Cognitive distraction had a greater negative impact on BLRDT performance than auditory distraction. Combined auditory and cognitive distraction had a negative impact on performance, but only in the most difficult LR task was this negative impact found to be significantly greater than that of cognitive distraction alone. There was a significant medium-sized correlation between perceived LR discrimination ability and actual overall BLRDT performance. ConclusionsDistraction has a significant impact on performance and multifaceted approaches are required to reduce LR errors. Educationally, greater emphasis on the linking of theory and clinical application is required to support patient safety and human factor training in medical school curricula. Distraction has the potential to impair an individual's ability to make accurate LR decisions and students should be trained from undergraduate level to be mindful of this.

Medical Education 2012: 46:586–592 Context Accreditation of medical education programmes is becoming increasingly prevalent worldwide, but beyond the face validity of these quality assurance methods, data linking accreditation to improved student outcomes are limited. Mexico and the Philippines both have voluntary systems of medical education accreditation and large numbers of students who voluntarily take components of the United States Medical Licensing Examination (USMLE). We investigated the examination performance of Mexican and Philippine citizens who attended medical schools in their home countries by medical school accreditation status. Methods The sample included 5045 individuals (1238 from Mexico, 3807 from the Philippines) who took at least one of the three USMLE components required for Educational Commission for Foreign Medical Graduates (ECFMG) certification. We also separately studied 2702 individuals who took all three examinations (589 from Mexico, 2113 from the Philippines). The chi-squared statistic was used to determine whether the associations between outcomes (first attempt pass rate on USMLE components and rate of ECFMG certification) and medical school accreditation (yes/no) were statistically significant. Results For the sample of registrants who took at least one USMLE component, first attempt pass rates on all USMLE components were higher for individuals attending accredited schools, although there were differences in pass rates among the components and between the two countries. The distinction was greatest for USMLE Step 1, for which attending an accredited school was associated with increases in first attempt pass rates of 15.9% for Mexican citizens and 29.2% for Philippine citizens. In registrants from the Philippines who took all three examinations, attending an accredited medical school was also associated with increased success in obtaining ECFMG certification. Conclusions These findings support the value and usefulness of accreditation in Mexico and the Philippines by linking accreditation to improved student outcomes.

Context The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic. Objectives This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills. Methods We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff. Results In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was > 70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station. Conclusions This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).

INTRODUCTION This paper seeks to describe the consultation strengths and weaknesses of senior medical students, the explicit and prioritised strategies for improvement utilised in student feedback, and curriculum developments informed by this work. METHODS Prospective, descriptive study of students on clinical placements in general practice. All were observed directly by 2 assessors in consultation with 5 patients in a general practice setting. Performance was judged against 5 categories of consultation competence and 35 component competences as contained in a modified version of the Leicester Assessment Package. Specific strategies for improvement were selected from a list of 69 previously formulated strategies. RESULTS Data from 1116 students were included. The consultation competences identified most frequently as strengths related to interpersonal skills, while weaknesses were mainly in the domain of clinical problem-solving. The median number of key strengths identified per student was 5, with 5 additional but lesser strengths. A median of 3 key and lesser weaknesses were identified. The average number of strategies selected to address an identified weakness was 1.2. Students rated the assessment process and its impact very positively. CONCLUSION The systematic assessment of the consultation competence of medical students by direct observation involving real patients is feasible and facilitates the 'educational diagnosis' of individuals and of their peer group. It has informed development of teaching and generated research hypotheses.

Introduction Professionalism is an important topic in medical education today. While much work has focused on defining professionalism and teaching medical students the appropriate interpersonal behaviours, relatively little research has looked at meaningful ways of assessing the relevant attributes. Method The Educational Commission for Foreign Medical Graduates (ECFMG®) clinical skills assessment (CSA®) uses standardised patients (SPs) to evaluate the readiness of graduates of international medical schools to enter accredited graduate training programmes in the USA. Doctor interpersonal skills, including professional qualities such as rapport, are evaluated as part of the CSA. Attentiveness, attitude and empathy, all facets of professional behaviour, are specifically targeted as part of the assessment. Results To date, over 35 000 candidates have been assessed, encompassing more than 370 000 individual SP encounters. Based on a 1-year cohort of examinees, the reliability of the individual professionalism-related component scores ranged from 0.61 to 0.70. Doctors who had graduated from medical school more recently, or were younger, generally obtained higher ratings. Professional qualities were only marginally related to measures of basic science and clinical science proficiency. Female candidates were rated significantly higher than male candidates on all dimensions. Conclusions While some professional behaviours are probably best measured using formats such as surveys, self-assessment and critical incident techniques, certain aspects of the domain can be reliably and validly measured in SP examinations.

What problems were addressed? In medical education, the adequate supervision of medical students is known to result in improved patient outcomes. However, this presents a challenge in clinical clerkships, in which faculty supervisors need to balance their time and resources. As such, the ability to provide indirect supervision to students – particularly in a setting in which the faculty member is not directly managing the student–patient encounter – is an important skill. Although minimal supervision of students presents a feasible solution to challenges in faculty supervision, it is unclear how such practices are perceived by patients and how they affect the learning experiences of students. To investigate this issue, this study examined patient and medical student perceptions of clinical teaching in a setting with minimal faculty supervision. What was tried? We performed a cross-sectional study with a sequential sample of obstetrics patients at an antenatal out-patient clinic. Questionnaires focusing on perceptions of students’ performance during the consultation were administered preand post-student consultation. In addition, patients were asked to write three words representing what they felt before and after the consultation. Students’ selfperceptions of their performance while consulting with patients were obtained through a questionnaire administered at the end of the process. An openended question requested students to write five words representing what they felt after consulting with patients with minimal clinical supervision. What lessons were learned? Of the 95 pregnant women studied, 95% knew that the student would perform routine antenatal care and would receive feedback from the supervisor following the consultation. They acknowledged that it was important for students to perform prenatal care as part of their training (85%). Before the consultation, 22% were not sure if they felt comfortable being examined by a medical student and 20% were unsure whether they could trust a student. After the consultation, 97% of patients felt comfortable about the examination and 99% felt they trusted the students. Of the pre-consultation descriptor words, ‘anxiety’ was the most frequently used word. ‘Calm’ and ‘satisfaction’ were the most frequently used post-consultation descriptors. Of the 77 students, 56% were in Year 5 of medical school and 44% were in Year 6. Students agreed that the ambulatory format with minimal faculty supervision encouraged their participation and engagement (82%). They felt that case discussions with faculty staff were useful (90%) and contributed to their learning (88%). The words ‘trust’, ‘secure’, ‘responsibility’ and ‘autonomy’ were the most representative of those used by students to express their feelings about meeting patients by themselves, supported by the outside supervision of the preceptor. Students expressed interest in knowing the opinions of the pregnant women about their performance in providing supportive medical care (92%) and confirmed that they would feel comfortable about being assessed by patients (84%). We observed positive patient feedback, and the perceptions of finalyear students regarding this out-patient clinical teaching method with minimal faculty supervision reinforced the potential to expand their autonomy. These findings may support additional assessments that incorporate patient perspectives into the overall clinical evaluation of students, which may improve the validity of the clinical assessment system.

Background The pivotal role of doctor–patient communication in effective health care delivery led the Educational Commission for Foreign Medical Graduates (ECFMG) to incorporate the assessment of interpersonal skills and spoken English proficiency into its Clinical Skills Assessment (CSA). Furthermore, it was decided that to pass the CSA, a candidate would need to meet or surpass defined performance standards for doctor–patient communication as a discrete component. This requirement, among others, is designed to ensure the readiness of graduates of foreign medical schools (FMGs) to enter postgraduate medical education programmes in the United States. Objective The primary focus of this study was to determine the extent to which performance in a simulated testing environment is related to performance in the clinical setting. Method Nurses were trained to rate the communication skills of residents from the patient’s perspective. A total of 43 first-year residents were evaluated. The survey ratings (n=225) were compared with the residents’ CSA communication scores. Results Corrected correlations between CSA ratings and those obtained from nurses ranged from 0·61 to 0·73. Conclusion This study provides evidence for the validity of the communication ratings provided by standardized patients. The reasonably strong associations between ratings obtained during testing and those obtained through observation of ‘real’ patient interactions suggest that external observers can provide accurate evaluations of doctor–patient communication.

Purpose The purpose of this study was to gather additional evidence for the validity and reliability of spoken English proficiency ratings provided by trained standardized patients (SPs) in high-stakes clinical skills examination. Method Over 2500 candidates who took the Educational Commission for Foreign Medical Graduates’ (ECFMG) Clinical Skills Assessment (CSA) were studied. The CSA consists of 10 or 11 timed clinical encounters. Standardized patients evaluate spoken English proficiency and interpersonal skills in every encounter. Generalizability theory was used to estimate the consistency of spoken English ratings. Validity coefficients were calculated by correlating summary English ratings with CSA scores and other external criterion measures. Mean spoken English ratings were also compared by various candidate background variables. Results The reliability of the spoken English ratings, based on 10 independent evaluations, was high. The magnitudes of the associated variance components indicated that the evaluation of a candidate’s spoken English proficiency is unlikely to be affected by the choice of cases or SPs used in a given assessment. Proficiency in spoken English was related to native language (English versus other) and scores from the Test of English as a Foreign Language (TOEFL). Discussion The pattern of the relationships, both within assessment components and with external criterion measures, suggests that valid measures of spoken English proficiency are obtained. This result, combined with the high reproducibility of the ratings over encounters and SPs, supports the use of trained SPs to measure spoken English skills in a simulated medical environment.

Objective To evaluate the use of a modified version of the Leicester Assessment Package (LAP) in the formative assessment of the consultation performance of medical students with particular reference to validity, inter-assessor reliability, acceptability, feasibility and educational impact. Design 180 third and fourth year Leicester medical students were directly observed consulting with six general practice patients and independently assessed by a pair of assessors. A total of 70 practice and 16 departmental assessors took part. Performance scores were subjected to generalizability analysis and students’ views of the assessment were gathered by questionnaire. Results Four of the five categories of consultation performance (Interviewing and history taking, Patient management, Problem solving and Behaviour and relationship with patients) were assessed in over 99% of consultations and Physical examination was assessed in 94%. Seventy-six percent of assessors reported that the case mix was ‘satisfactory’ and 20% that it was ‘borderline’; 85% of students believed it to have been satisfactory. Generalizability analysis indicates that two independent assessors assessing the performance of students across six consultations would achieve a reliability of 0·94 in making pass or fail decisions. Ninety-eight percent of students perceived that their particular strengths and weaknesses were correctly identified, 99% that they were given specific advice on how to improve their performance and 98% believed that the feedback they had received would have long-term benefit. Conclusions The modified version of the LAP is valid, reliable and feasible in formative assessment of the consultation performance of medical students. Furthermore, almost all students found the process fair and believed it was likely to lead to improvements in their consultation performance. This approach may also be applicable to regulatory assessment as it accurately identifies students at the pass/fail margin.

The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic.This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills.We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff. Results In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station.This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).

OBJECTIVE To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies. METHODS We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists. RESULTS We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised. CONCLUSIONS It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.