Search

Educational practice and educational research are not aligned with each other. Current educational practice heavily relies on information transmission or content delivery to learners. Yet evidence shows that delivery is only a minor part of learning. To illustrate the directions we might take to find better educational strategies, six areas of educational evidence are briefly reviewed. The flipped classroom idea is proposed to shift our expenditure and focus in education. All information delivery could be web distributed, thus creating more time for other more expensive educational strategies to support the learner. In research our focus should shift from comparing one curriculum to the other, to research that explains why things work in education and under which conditions. This may generate ideas for creative designers to develop new educational strategies. These best practices should be shared and further researched. At the same time attention should be paid to implementation and the realization that teachers learn in a way very similar to the people they teach. If we take the evidence seriously, our educational practice will look quite different to the way it does now.

In recent years, postgraduate assessment programmes around the world have embraced workplace-based assessment (WBA) and its related tools. Despite their widespread use, results of studies on the validity and reliability of these tools have been variable. Although in many countries decisions about residents' continuation of training and certification as a specialist are based on the composite results of different WBAs collected in a portfolio, to our knowledge, the reliability of such a WBA toolbox has never been investigated. Using generalisability theory, we analysed the separate and composite reliability of three WBA tools [mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), and multisource feedback (MSF)] included in a resident portfolio. G-studies and D-studies of 12,779 WBAs from a total of 953 residents showed that a reliability coefficient of 0.80 was obtained for eight mini-CEXs, nine DOPS, and nine MSF rounds, whilst the same reliability was found for seven mini-CEXs, eight DOPS, and one MSF when combined in a portfolio. At the end of the first year of residency a portfolio with five mini-CEXs, six DOPS, and one MSF afforded reliable judgement. The results support the conclusion that several WBA tools combined in a portfolio can be a feasible and reliable method for high-stakes judgements.

Supervisor assessments are critical for both formative and summative assessment in the workplace. Supervisor ratings remain an important source of such assessment in many educational jurisdictions even though there is ambiguity about their validity and reliability. The aims of this evaluation is to explore the: (1) construct validity of ward-based supervisor competency assessments; (2) reliability of supervisors for observing any overarching domain constructs identified (factors); (3) stability of factors across subgroups of contexts, supervisors and trainees; and (4) position of the observations compared to the established literature. Evaluated assessments were all those used to judge intern (trainee) suitability to become an unconditionally registered medical practitioner in the Australian Capital Territory, Australia in 2007-2008. Initial construct identification is by traditional exploratory factor analysis (EFA) using Principal component analysis with Varimax rotation. Factor stability is explored by EFA of subgroups by different contexts such as hospital type, and different types of supervisors and trainees. The unit of analysis is each assessment, and includes all available assessments without aggregation of any scores to obtain the factors. Reliability of identified constructs is by variance components analysis of the summed trainee scores for each factor and the number of assessments needed to provide an acceptably reliable assessment using the construct, the reliability unit of analysis being the score for each factor for every assessment. For the 374 assessments from 74 trainees and 73 supervisors, the EFA resulted in 3 factors identified from the scree plot, accounting for only 68 % of the variance with factor 1 having features of a "general professional job performance" competency (eigenvalue 7.630; variance 54.5 %); factor 2 "clinical skills" (eigenvalue 1.036; variance 7.4 %); and factor 3 "professional and personal" competency (eigenvalue 0.867; variance 6.2 %). The percent trainee score variance for the summed competency item scores for factors 1, 2 and 3 were 40.4, 27.4 and 22.9 % respectively. The number of assessments needed to give a reliability coefficient of 0.80 was 6, 11 and 13 respectively. The factor structure remained stable for subgroups of female trainees, Australian graduate trainees, the central hospital, surgeons, staff specialist, visiting medical officers and the separation into single years. Physicians as supervisors, male trainees, and male supervisors all had a different grouping of items within 3 factors which all had competency items that collapsed into the predefined "face value" constructs of competence. These observations add new insights compared to the established literature. For the setting, most supervisors appear to be assessing a dominant construct domain which is similar to a general professional job performance competency. This global construct consists of individual competency items that supervisors spontaneously align and has acceptable assessment reliability. However, factor structure instability between different populations of supervisors and trainees means that subpopulations of trainees may be assessed differently and that some subpopulations of supervisors are assessing the same trainees with different constructs than other supervisors. The lack of competency criterion standardisation of supervisors' assessments brings into question the validity of this assessment method as currently used.

Background: Although the literature suggests that reflection has a positive impact on learning, there is a paucity of evidence to support this notion., Aim: We investigated feedback and reflection in relation to the likelihood that feedback will be used to inform action plans. We hypothesised that feedback and reflection present a cumulative sequence (i.e. trainers only pay attention to trainees' reflections when they provided specific feedback) and we hypothesised a supplementary effect of reflection., Method: We analysed copies of assessment forms containing trainees' reflections and trainers' feedback on observed clinical performance. We determined whether the response patterns revealed cumulative sequences in line with the Guttman scale. We further examined the relationship between reflection, feedback and the mean number of specific comments related to an action plan (ANOVA) and we calculated two effect sizes., Results: Both hypotheses were confirmed by the results. The response pattern found showed an almost perfect fit with the Guttman scale (0.99) and reflection seems to have supplementary effect on the variable action plan., Conclusions: Reflection only occurs when a trainer has provided specific feedback; trainees who reflect on their performance are more likely to make use of feedback. These results confirm findings and suggestions reported in the literature.

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Educational research not only showed that student characteristics are of major importance for study success, but also that education does make a difference. Essentially, teaching is about stimulating students to invest time in learning and to use that time as effectively as possible. Assessment, goal-orientated work, and feedback have a major effect. The teacher is the key figure. With the aim to better understand teaching and learning, educational researchers usefindingsfrom other disciplines more and more often. A pitfall is to apply the findings of educational research without taking into consideration the context and the specific characteristics of students and teachers. Because of the large number offactors that influence the results ofeducation, educational science is referred as 'the hardest science of all'.

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

Introduction: For health professionals, the development of insight into their performance is vital for safe practice, professional development and self-regulation. This study investigates whether the development of dental trainees' insight, when provided with external feedback on performance, can be assessed using a single criterion on a simple global ratings form such as the Longitudinal Evaluation of Performance or Mini Clinical Evaluation Exercise., Methods: Postgraduate dental trainees (N = 139) were assessed using this tool on a weekly basis for 6 months. Regression analysis of the data was carried out using SPSS, and a short trainer questionnaire was implemented to investigate feasibility., Results: Ratings for insight were shown to increase with time in a similar manner to the growth observed in other essential skills. The gradient of the slope for growth of insight was slightly less than that of the other observed skills. Trainers were mostly positive about the new criterion assessing trainees' insight, although the importance of training for trainers in this process was highlighted., Discussion: Our data suggest that practitioners' insight into their performance can be developed with experience and regular feedback. However, this is most likely a complex skill dependent on a number of intrinsic and external factors., Conclusion: The development of trainees' insight into their performance can be assessed using a single criterion on a simple global ratings form. The process involves no additional burden on evaluators in terms of their time or cost, and promotes best practice in the provision of feedback for trainees., (© 2011 John Wiley & Sons A/S.)

Workplace learning in undergraduate medical education has predominantly been studied from a cognitive perspective, despite its complex contextual characteristics, which influence medical students' learning experiences in such a way that explanation in terms of knowledge, skills, attitudes and single determinants of instructiveness is unlikely to suffice. There is also a paucity of research which, from a perspective other than the cognitive or descriptive one, investigates student learning in general practice settings, which are often characterised as powerful learning environments. In this study we took a socio-cultural perspective to clarify how students learn during a general practice clerkship and to construct a conceptual framework that captures this type of learning. Our analysis of group interviews with 44 fifth-year undergraduate medical students about their learning experiences in general practice showed that students needed developmental space to be able to learn and develop their professional identity. This space results from the intertwinement of workplace context, personal and professional interactions and emotions such as feeling respected and self-confident. These forces framed students' participation in patient consultations, conversations with supervisors about consultations and students' observation of supervisors, thereby determining the opportunities afforded to students to mind their learning. These findings resonate with other conceptual frameworks and learning theories. In order to refine our interpretation, we recommend that further research from a socio-cultural perspective should also explore other aspects of workplace learning in medical education.

Even though rater-based judgements of clinical competence are widely used, they are context sensitive and vary between individuals and institutions. To deal adequately with rater-judgement unreliability, evaluating the reliability of workplace rater-based assessments in the local context is essential. Using such an approach, the primary intention of this study was to identify the trainee score variation around supervisor ratings, identify sampling number needs of workplace assessments for certification of competence and position the findings within the known literature. This reliability study of workplace-based supervisors' assessments of trainees has a rater-nested-within-trainee design. Score variation attributable to the trainee for each competency item assessed (variance component) were estimated by the minimum-norm quadratic unbiased estimator. Score variance was used to estimate the number needed for a reliability value of 0.80. The trainee score variance for each of 14 competency items varied between 2.3% for emergency skills to 35.6% for communication skills, with an average for all competency items of 20.3%; the "Overall rating" competency item trainee variance was 28.8%. These variance components translated into 169, 7, 17 and 28 assessments needed for a reliability of 0.80, respectively. Most variation in assessment scores was due to measurement error, ranging from 97.7% for emergency skills to 63.4% for communication skills. Similar results have been demonstrated in previously published studies. In summary, overall supervisors' workplace based assessments have poor reliability and are not suitable for use in certification processes in their current form. The marked variation in the supervisors' reliability in assessing different competencies indicates that supervisors may be able to assess some with acceptable reproducibility; in this case communication and possibly overall competence. However, any continued use of this format for assessment of trainee competencies necessitates the identification of what supervisors in different institutions can reliably assess rather than continuing to impose false expectations from unreliable assessments.

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

We reviewed the literature on instruments for work-based assessment in single clinical encounters, such as the mini-clinical evaluation exercise (mini-CEX), and examined differences between these instruments in characteristics and feasibility, reliability, validity and educational effect. A PubMed search of the literature published before 8 January 2009 yielded 39 articles dealing with 18 different assessment instruments. One researcher extracted data on the characteristics of the instruments and two researchers extracted data on feasibility, reliability, validity and educational effect. Instruments are predominantly formative. Feasibility is generally deemed good and assessor training occurs sparsely but is considered crucial for successful implementation. Acceptable reliability can be achieved with 10 encounters. The validity of many instruments is not investigated, but the validity of the mini-CEX and the 'clinical evaluation exercise' is supported by strong and significant correlations with other valid assessment instruments. The evidence from the few studies on educational effects is not very convincing. The reports on clinical assessment instruments for single work-based encounters are generally positive, but supporting evidence is sparse. Feasibility of instruments seems to be good and reliability requires a minimum of 10 encounters, but no clear conclusions emerge on other aspects. Studies on assessor and learner training and studies examining effects beyond 'happiness data' are badly needed.

This article presents lessons learnt from experiences with assessment of professional competence. Based on Miller's pyramid, a distinction is made between established assessment technology for assessing 'knows', 'knowing how' and 'showing how' and more recent developments in the assessment of (clinical) performance at the 'does' level. Some general lessons are derived from research of and experiences with the established assessment technology. Here, many paradoxes are revealed and empirical outcomes are often counterintuitive. Instruments for assessing the 'does' level are classified and described, and additional general lessons for this area of performance assessment are derived. These lessons can also be read as general principles of assessment (programmes) and may provide theoretical building blocks to underpin appropriate and state-of-the-art assessment practices., (Copyright © 2010 Elsevier Ltd. All rights reserved.)

Research on assessment in medical education has strongly focused on individual measurement instruments and their psychometric quality. Without detracting from the value of this research, such an approach is not sufficient to high quality assessment of competence as a whole. A programmatic approach is advocated which presupposes criteria for designing comprehensive assessment programmes and for assuring their quality. The paucity of research with relevance to programmatic assessment, and especially its development, prompted us to embark on a research project to develop design principles for programmes of assessment. We conducted focus group interviews to explore the experiences and views of nine assessment experts concerning good practices and new ideas about theoretical and practical issues in programmes of assessment. The discussion was analysed, mapping all aspects relevant for design onto a framework, which was iteratively adjusted to fit the data until saturation was reached. The overarching framework for designing programmes of assessment consists of six assessment programme dimensions: Goals, Programme in Action, Support, Documenting, Improving and Accounting. The model described in this paper can help to frame programmes of assessment; it not only provides a common language, but also a comprehensive picture of the dimensions to be covered when formulating design principles. It helps identifying areas concerning assessment in which ample research and development has been done. But, more importantly, it also helps to detect underserved areas. A guiding principle in design of assessment programmes is fitness for purpose. High quality assessment can only be defined in terms of its goals.

Background: Lack of faculty training is often cited as the main obstacle to post-graduate teaching in communication skills., Aims: To explore clinical supervisors' needs and perceptions regarding their role as communication skills trainers., Methods: Four focus group discussions were conducted with clinical supervisors from two in-patient and one out-patient medical services from the Geneva University Hospitals. Focus groups were audio taped, transcribed verbatim and analyzed in a thematic way using Maxqda software for qualitative data analysis., Results: Clinical supervisors said that they frequently addressed communication issues with residents but tended to intervene as rescuers, clinicians or coaches rather than as formal instructors. They felt their own training did not prepare them to teach communication skills. Other barriers to teach communication skills include lack of time, competing demands, lack of interest and experience on the part of residents, and lack of institutional priority given to communication issues. Respondents expressed a desire for experiential and reflective training in a work-based setting and emphasised the need for a non-judgmental learning atmosphere., Conclusions: Results suggest that organisational priorities, culture and climate strongly influence the degree to which clinical supervisors may feel comfortable to teach communication skills to residents. Attention must be given to these contextual factors in the development of an effective communication skills teaching program for clinical supervisors.

Introduction: Clinical skills centres (or Skillslabs) prepare students for patient-encounters. Evidence on teaching skills in these centres is lacking. What teaching skills do teachers view as effective in supporting the acquisition of physical examination skills in undergraduate medical training?, Method: Structured interviews of 10 teachers (1/3 of staff of Maastricht University, Skillslab) were conducted. Selection was based on even representation of age, years teaching experience, gender and previous experience at Maastricht University. A topic grid was used to ensure comparability. Interviews (average 70 min, range 33-95 min) were recorded and transcripts were analyzed independently by two researchers., Results: Teaching skills identified include the ability to adapt content of the training, level of depth and teaching method according to the needs of any particular group. Thorough comprehension of students' context (level of knowledge,prior experience and insight in the curriculum) is considered helpful. Explicitly inviting students to ask questions and providing relevant literature is seen to stimulate learning. Providing constructive feedback is essential, as is linking physical examination skills training to clinical situations. The ideal attitude includes appropriate dress and behaviour, as well as the use of humour. Affinity for teaching is regarded as the most important reason to work as a teacher., Conclusion: Desired characteristics for undergraduate skills teachers resemble findings in other teaching roles. Affinity for teaching and flexibility in teaching methods are novel findings.

Different lines of research have suggested that context is important in acting and learning in the clinical workplace. It is not clear how contextual information influences residents' constructions of the situations in which they participate. The category accessibility paradigm from social psychology appears to offer an interesting perspective for studying this topic. We explored the effect of activating medically irrelevant mental concepts in one context, so-called 'priming', on residents' interpretations as reflected in their judgments in another, work-related context. Obstetric-gynecologic residents participated in two unrelated-tasks experiments. In the first experiment residents were asked to indicate affect about a change in a routine procedure after performing an ostensibly unrelated 'priming' task which activated the concept of either ineffective coping or effective coping. The second experiment concerned residents' patient management decisions in a menorrhagia case after 'priming' with either action or holding off. Contextually activated mental concepts lead to divergent affective and cognitive evaluations in a subsequent medical context. Residents are not aware of this effect. The strength of the effect varies with residents' level of experience. Context influences residents' constructions of a work-related situation by activating mental concepts which in turn affect how residents experience situations. Level of experience appears to play a mediating role in this process.

High stakes postgraduate specialist certification examinations have considerable implications for the future careers of examinees. Medical colleges and professional boards have a social and professional responsibility to ensure their fitness for purpose. To date there is a paucity of published data about the reliability of specialist certification examinations and objective methods for improvement. Such data are needed to improve current assessment practices and sustain the international credibility of specialist certification processes. To determine the component and composite reliability of the Fellowship examination of the College of Physicians of South Africa, and identify strategies for further improvement, generalizability and multivariate generalizability theory were used to estimate the reliability of examination subcomponents and the overall reliability of the composite examination. Decision studies were used to identify strategies for improving the composition of the examination. Reliability coefficients of the component subtests ranged from 0.58 to 0.64. The composite reliability of the examination was 0.72. This could be increased to 0.8 by weighting all test components equally or increasing the number of patient encounters in the clinical component of the examination. Correlations between examination components were high, suggesting that similar parameters of competence were being assessed. This composite certification examination, if equally weighted, achieved an overall reliability sufficient for high stakes examination purposes. Increasing the weighting of the clinical component decreased the reliability. This could be rectified by increasing the number of patient encounters in the examination. Practical ways of achieving this are suggested.

Introduction: A key aspect of the success of a PBL curriculum is the effective implementation of its small group tutorials. Diversity among students participating in tutorials may affect the effectiveness of the tutorials and may require different implementation strategies., Aims: To determine how students from diverse backgrounds perceive the effectiveness of the processes and content of the PBL tutorials. This study also aims to explore the relationship between students' perceptions of their PBL tutorials and their gender, age, language, prior educational training, and secondary schooling., Materials/methods: Data were survey results from 244 first-year student-respondents at the Nelson Mandela School of Medicine at the University of KwaZulu-Natal in South Africa. Exploratory factor analysis was conducted to verify scale constructs in the questionnaire. Relationships between independent and dependent variables were investigated in an analysis of variance., Results: The average scores for the items measured varied between 3.3 and 3.8 (scale value 1 indicated negative regard and 5 indicated positive regard). Among process measures, approximately two-thirds of students felt that learning in a group was neither frustrating nor stressful and that they enjoyed learning how to work with students from different social and cultural backgrounds. Among content measures, 80% of the students felt that they learned to work successfully with students from different social and cultural groups and 77% felt that they benefited from the input of other group members. Mean ratings on these measures did not vary with students' gender, age, first language, prior educational training, and the types of schools they had previously attended., Discussion and Conclusion: Medical students of the University of KwaZulu-Natal, regardless of their backgrounds, generally have positive perceptions of small group learning. These findings support previous studies in highlighting the role that small group tutorials can play in overcoming cultural barriers and promoting unity and collaborative learning within diverse student groups.

Objective: To examine the increase in self-perceived clinical competence during a three-year postgraduate training in general practice and to explore the relation between the growth of self-perceived competence and several background variables., Design: Cohort, 1995-1998., Setting: Three-year Postgraduate Training for General practice in the Netherlands, Participants: All Dutch trainees who followed postgraduate training from September 1995 to September 1998 (N=191)., Intervention: We asked the trainees at the start and at the end of their postgraduate training to complete a questionnaire, which assessed their self-perceived knowledge, clinical skills and consultations skills. We collected information about potentially influencing background variables. Amongst these were variables such as: age, gender, prior medical experience, the effort someone has spent upon her/his education, insight in weak and strong areas of clinical competence and knowledge and skills levels., Main Outcome Measure: Self-perceived competence., Results: A total of 127 trainees completed both questionnaires (190 at the first administration and 128 at the second one). We found statistically significant growth of self-perceived clinical competence. Self-perceived consultation skills increased more than self-perceived knowledge and clinical skills. The afore mentioned background variables did not relate in any way with the growth of self-perceived clinical competence., Conclusion: This study shows that a 3-year postgraduate training in general practice enhances self-perceived clinical competence. However, we still do not know how to explain this improvement. Further study into the theoretical concept of self-assessment in medical education and into the factors contributing to the feeling of being competent, is required.

Objectives: The clinical learning environment is an influential factor in work-based learning. Evaluation of this environment gives insight into the educational functioning of clinical departments. The Postgraduate Hospital Educational Environment Measure (PHEEM) is an evaluation tool consisting of a validated questionnaire with 3 subscales. In this paper we further investigate the psychometric properties of the PHEEM. We set out to validate the 3 subscales and test the reliability of the PHEEM for both clerks (clinical medical students) and registrars (specialists in training)., Methods: Clerks and registrars from different hospitals and specialties filled out the PHEEM. To investigate the construct validity of the 3 subscales, we used an exploratory factor analysis followed by varimax rotation, and a cluster analysis known as Mokken scale analysis. We estimated the reliability of the questionnaire by means of variance components according to generalisability theory., Results: A total of 256 clerks and 339 registrars filled out the questionnaire. The exploratory factor analysis plus varimax rotation suggested a 1-dimensional scale. The Mokken scale analysis confirmed this result. The reliability analysis showed a reliable outcome for 1 department with 14 clerks or 11 registrars. For multiple departments 3 respondents combined with 10 departments provide a reliable outcome for both groups., Discussion: The PHEEM is a questionnaire measuring 1 dimension instead of the hypothesised 3 dimensions. The sample size required to achieve a reliable outcome is feasible. The instrument can be used to evaluate both single and multiple departments for both clerks and registrars.

Benchmarking lists are indicators of the quality of medical curricula. The measurement on which a list in order of merit is based must of course have validity, but this is generally the case. The irresponsible use of ranking lists is to be deplored, but that does not constitute a reason not to publish them. More competition in attracting the aspirant medical student, for example on the basis of ranking lists, would be a good thing. This can contribute to making the quality more visible. Lists in order of quality are therefore a useful aid to assist the aspirant student in choosing a school.

Context: There has been an increase in the number of medical schools implementing community-based educational (CBE) programmes. However, there are doubts whether CBE programmes are appropriately implemented. As a case study, the CBE programme of the Medical Faculty of Diponegoro University (MFDU) in Semarang, Indonesia was evaluated., Objectives: To acquire MFDU's students' opinions on their CBE programme as part of a comprehensive evaluation of that programme, and to generate recommendations for improvement of MFDU's CBE programme., Methods: Coles and Grant's model for curriculum evaluation was applied. This model recommends triangulation of data generated from comparison of the programme as designed "on paper", as implemented "in action" by the faculty, and "as experienced" by the students. To assess the curriculum as experienced by the students, direct participatory observation was performed and focus group discussions were conducted to collect students' opinions., Results: Students specifically signalled: (1) frequent overlap of lectures given during the CBE clerkship and in the previous part of the curriculum, (2) mismatch between their activities in the community and the community's felt health needs, (3) greater benefits of the CBE programme for Primary Health Care (PHC) centres and students than for target communities, and (4) incidental defective co-operation between students and health providers or community health workers., Conclusion: Students' opinions yielded more in-depth information on the CBE programme evaluated and facilitated formulation of recommendations for improvement of that CBE programme.

Introduction: A framework for the definition of generic objectives for community-based education (CBE) was developed for undergraduate medical programmes, particularly for developing countries. To probe the validity of the set of CBE objectives generated by this approach, opinions from a wider audience were sought., Method: Questionnaires were sent to 72 medical schools in 36 developing countries. Half of the addressees were randomly drawn from the list of institutional members of The Network: Toward Unity For Health (TUFH) and stratified according to developing countries. Another 36 medical schools were randomly drawn from non-Network: TUFH schools from the same country where the selected Network: TUFH addressee was located, or from a neighbouring country., Results: A total of 43 medical schools responded to the questionnaire (60% response rate), 31 out of the 36 addressed Network: TUFH members (86%) and 12 out of 36 addressed non-Network: TUFH schools (33%). Out of all 43 respondents 39 (91%) had implemented CBE in their curricula. Opinions of Network: TUFH and non-Network: TUFH schools on the framework and the generic objectives were not significantly different. Out of the 21 proposed objectives, 17 were scored as relevant by 75% or over of all responders and one out of the four objectives considered to be less relevant by the responders was deleted., Conclusion: A framework to develop generic CBE objectives and a derived set of 21 objectives were modified based on input by 43 medical schools residing in developing countries distributed all over the globe. The outcome is a validated set of 20 generic objectives for CBE programmes in developing countries.

Semi-structured interviews were conducted with external supervisors of international electives undertaken by Dutch undergraduate students, in order to gain insight into student learning processes during these electives. The interviews served to triangulate information on these learning processes that was obtained from students' self-reports. The results of the case study reported in this paper were largely consistent with findings from prior studies of international electives in which learning processes and sociocultural differences were examined: experiential learning processes appeared to dominate and sociocultural differences occasionally seemed to blur productive learning, especially when the differences between the national cultures of host country and student home country were substantial. It is recommended that students' experiential learning from international electives should be supplemented with 'guided' and 'self-directed' learning with a focus on the sociocultural dimension.

In the framework of the Bologna Process, internationalization co-ordinators of seven (out of eight) Dutch medical schools completed an electronic survey about internationalization-related aspects of the curriculum. Common features of internationalization in Dutch medical schools were: the numbers of outgoing students exceeded the numbers of incoming students, and most international programmes involved clinical training and research projects. We recommend that Dutch medical schools should pay more attention to 'Internationalization at Home' and focus on conditions that are conducive to participation by foreign students.

Background: Supervision and feedback are essential factors that contribute to the learning environment in the context of workplace learning and their frequency and quality can be improved. Assessment is a powerful tool with which to influence students' learning and supervisors' teaching and thus the learning environment., Objective: To investigate an in-training assessment (ITA) programme in action and to explore its effects on supervision and feedback., Design: A qualitative study using individual, semistructured interviews., Subjects and Setting: Eight students and 17 assessors (9 members of staff and 8 residents) in the internal medicine undergraduate clerkship at Vrije Universiteit Medical Centre, Amsterdam, the Netherlands., Results: The ITA programme in action differed from the intended programme. Assessors provided hardly any follow-up on supervision and feedback given during assessments. Although students wanted more supervision and feedback, they rarely asked for it. Students and assessors failed to integrate the whole range of competencies included in the ITA programme into their respective learning and supervision and feedback. When giving feedback, assessors rarely gave borderline or fail judgements., Discussion and Conclusion: If an ITA programme in action is to be congruent with the intended programme, the implementation of the programme must be monitored. It is also necessary to provide full information about the programme and to ensure this information is given repeatedly. Introducing an ITA programme that includes the assessment of several competencies does not automatically lead to more attention being paid to these competencies in terms of supervision and feedback. Measures that facilitate change in the learning environment seem to be a prerequisite for enabling the assessment programme to steer the learning environment.

There has been considerable change in the field of assessment of medical competence. At the moment, competency-orientated assessment, 'mini-CEX' (brief clinical evaluation exercises) and portfolios are quite popular. These methods are based on research findings indicating that medical competence can better be described as a collection of the complex tasks (so-called competencies) that a doctor must be able to perform than as the sum of knowledge, skills, problem-solving ability and attitudes. Mini-CEX represents a method for the assessment of medical competence reliably and validly in a practical setting. Using a portfolio, information on the student's competence can be collated and evaluated from various sources, including mini-CEX. As such, a portfolio has much in common with a patient chart.

Introduction: The Clinical Teaching Effectiveness Instrument (CTEI) was developed to evaluate the quality of the clinical teaching of educators. Its authors reported evidence supporting content and criterion validity and found favourable reliability findings. We tested the validity and reliability of this instrument in a European context and investigated its reliability as an instrument to evaluate the quality of clinical teaching at group level rather than at the level of the individual teacher., Methods: Students participating in a surgical clerkship were asked to fill in a questionnaire reflecting a student-teacher encounter with a staff member or a resident. We calculated variance components using the urgenova program. For individual score interpretation of the quality of clinical teaching the standard error of estimate was calculated. For group interpretation we calculated the root mean square error., Results: The results did not differ statistically between staff and residents. The average score was 3.42. The largest variance component was associated with rater variance. For individual score interpretation a reliability of > 0.80 was reached with 7 ratings or more. To reach reliable outcomes at group level, 15 educators or more were needed with a single rater per educator., Discussion: The required sample size for appraisal of individual teaching is easily achievable. Reliable findings can also be obtained at group level with a feasible sample size. The results provide additional evidence of the reliability of the CTEI in undergraduate medical education in a European setting. The results also showed that the instrument can be used to measure the quality of teaching at group level.

Context: A growing number of health professions schools have implemented programmes for community-based education (CBE) for their students. There are indications, however, that particularly in developing countries, CBE programmes are not always optimally implemented or sustained., Objective: To test the suitability of an established method for curriculum evaluation, combined with a set of generic objectives for CBE programmes, for evaluation of CBE programmes., Methods: As a case study, Coles and Grant's model for curriculum evaluation was applied to the CBE programme of the Medical Faculty of Diponegoro University (MFDU) in Semarang, Indonesia. Document analysis yielded information on the programme on paper; participatory observation and staff interviews on the programme in action. In addition, MFDU's CBE programme was evaluated against a set of generic objectives for CBE programmes recently designed by us., Results: MFDU has created great opportunities for its CBE programme in which, however, also significant weaknesses were revealed. (1) In the community, much time was spent on formal teaching; (2) Students' work in the community was not jointly identified with community members regarding the community's felt health needs; (3) There was rarely continuity, and evaluation or follow-up of the students' work in the community; and (4) No systematic programme evaluations are carried out., Discussion: This evaluation study showed shortcomings in the implementation of MFDU's CBE programme. The major weaknesses identified point at an underutilization of the opportunities and potential jeopardization of the facilities in the community. On the other hand, more time is needed in the CBE programme to establish the health needs to be addressed jointly with the community and to assess the impact of activities undertaken. A thorough review of the CBE programme, perhaps taking the outcomes of this study into account, could turn MFDU's CBE programme into a fine example for other medical schools in Indonesia and beyond., Conclusion: Coles and Grant's method for curriculum evaluation proved suitable for evaluation of a CBE programme in a developing country. After additional comparison with a reference list of objectives for CBE programmes, reasoned suggestions for programme can be made.

Background: Global performance rating is frequently used in clinical training despite its known psychometric drawbacks. Inter-rater reliability is low in undergraduate training but better in residency training, possibly because residency offers more opportunities for supervision. The low or moderate predictive validity of global performance ratings in undergraduate and residency training may be due to low or unknown reliability of both global performance ratings and criterion measures. In an undergraduate clerkship, we investigated whether reliability improves when raters are more familiar with students' work and whether validity improves with increased reliability of the predictor and criterion instrument., Methods: Inter-rater reliability was determined in a clerkship with more student-rater contacts than usual. The in-training assessment programme of the clerkship that immediately followed was used as the criterion measure to determine predictive validity., Results: With four ratings, inter-rater reliability was 0.41 and predictive validity was 0.32. Reliability was lower and validity slightly higher than similar results published for residency training., Conclusion: Even with increased student-rater interaction, the reliability and validity of global performance ratings were too low to warrant the usage of global performance ratings as individual assessment format. However, combined with other assessment measures, global performance ratings may lead to improved integral assessment.

The study was designed to illustrate how residents perceive the Mini Clinical Examination Exercise as an assessment tool and its influence on their approach to learning and studying. A phenomenographic approach was applied. All 16 residents from a cardiology training program in Buenos Aires were included. Results show that in all cases residents demonstrate an intrinsic interest in the subject matter. They show self-regulating strategies when required to select, relate and make critical appraisals of their own. They consistently demonstrate an aim to build a relationship between individual experience and their chosen topic. The residents feel comfortable because it melts with their routine. Residents find the Mini Clinical Examination Exercise to be a useful assessment tool with a favorable influence towards a constructive approach to study and learning.

Introduction: Various measures have been introduced to enhance learning experiences in clerkships, generally with limited success. This study evaluated the impact of a multifaceted approach on the effectiveness of learning in a surgical clerkship. In accordance with results obtained in continuing medical education, several interventions were introduced simultaneously. We compared students' evaluations of the traditional surgical clerkship with those of the restructured clerkship., Methods: Two consecutive cohorts of students were asked to complete a questionnaire about the quality and quantity of their learning experiences. Cohort 1 (n = 28) undertook the traditional clerkship and cohort 2 (n = 72) the restructured clerkship. A Mann-Whitney test was used to compare outcomes between the 2 cohorts., Results: There were few statistically significant differences between cohorts 1 and 2. Overall, quality indicators did not differ between the 2 cohorts., Discussion: A short-term multifaceted intervention led to a slight increase in the performance of clinical skills and a slight decrease in time spent on activities of limited educational value. The intervention may have been too brief to produce substantial effects. Future interventions should also target teachers, including trainees, in order to assess their opinions and address their educational needs.

Competences are becoming more and more prominent in undergraduate medical education. Workplace learning is regarded as crucial in competence learning. Assuming that effective learning depends on adequate supervision, feedback and assessment, the authors studied the occurrence of these three variables in relation to a set of clinical competences. They surveyed students at the end of their rotation in surgery, internal medicine or paediatrics asking them to indicate for each competence how often they had received observed and unobserved supervision, the seniority of the person who provided most of their feedback, and whether the competence was addressed in formal assessments. Supervision was found to be scarce and mostly unobserved. Senior staff did not provide much feedback, and assessment mostly targeted patient-related competences. For all variables, the variation between students exceeded that between disciplines. We conclude that conditions for adequate workplace learning are poorly met and that clerkship experiences show huge inter-student variation.

Rationale: The availability of a framework for the definition of generic objectives for community-based education (CBE) programmes may assist in the rational design of objectives for specific CBE programmes., Strategy: Factors impacting on community health from the perspective of a developing country were collected. Potential assistance from medical students to communities to improve their health status was determined. Competencies required in students to execute tasks in the community were defined and eventually educational objectives to develop these competencies in the students were established., Methods: Factors impacting on community health and activities of medical students in CBE programmes were identified by review of literature and Internet resources. Competencies desired for execution of tasks by students and educational objectives to develop these competencies were defined by us and checked against pertinent literature. A draft table representing the 4 elements of the framework was discussed by an international group of experts for external validation., Main Outcomes: A total of 26 factors impacting on community health were identified and clustered in 5 domains. Twenty-one generic objectives for CBE programmes were defined to develop the required competencies in students. Analogues of each of these 21 objectives were found in at least 1 publication specifying objectives for specific CBE programmes but none of these publications stated any objective not covered by our list of generic objectives., Conclusion: It proved possible to develop a framework to define generic objectives for CBE programmes. An example was elaborated from the perspective of a medical school in a developing country.

Purpose: The evidence suggests that a longitudinal training of communication skills embedded in a rich clinical context is most effective. In this study we evaluated the acquisition of communication skills under such conditions., Methods: In a longitudinal design the communication skills of a randomly selected sample of 25 trainees of a three-year postgraduate training programme for general practice were assessed at the start and at the end of training. Eight videotaped real life consultations were rated per measurement and per trainee, using the MAAS-Global scoring list. The results were compared with each other and with those of a reference group of 94 experienced GPs., Results: The mean score of the MAAS-Global was slightly increased at the end of training (2.4) compared with the start (2.2). No significant difference was found between the final results of the trainees and the reference group. According to the criteria of the rating scale the performance of both trainees and GPs was unsatisfactory., Conclusion: The results of this study indicate that communication skills do not improve in a three-year postgraduate training comprising both a rich clinical context and a longitudinal training of communication skills, and that an unsatisfactory level still exists at the end of training. Moreover, GPs do not acquire communication skills during independent practice as they perform comparably to the trainees. Further research into the measurement of communication skills, the teaching procedures, the role of the GP-trainer as a model and the influence of rotations through hospitals and the like, is required.

One of the major reasons for the shortage of donor organs is the high number of refusals by relatives. Studies have shown that the quality of communication with bereaved relatives influences whether to object or agree to organ and/or tissue donation. Breaking news of brain stem death, approaching relatives for permission to donate organs while also appropriately managing emotional reactions of relatives are complex tasks, which require knowledge of the domains as well as adequate skills to communicate information and understanding. In this study the effect of the European Donor Hospital Education Programme (EDHEP) on the self-efficacy of Intensive Care staff is evaluated. Self-efficacy scores significantly improved after attending EDHEP; an effect that was maintained at six month follow-up. EDHEP participants with high baseline scores on self-efficacy, maintained the increase at follow-up. EDHEP participants with low baseline scores on self-efficacy showed the greatest increase at the post-test. Increases in self-efficacy were significantly related to decreases in the perceived difficulty of requesting. Experience had a significant effect on both self-efficacy beliefs and perceived difficulty of requesting donation. As self-efficacy beliefs are perceived as better predictors for future behaviour than prior attainments, the results call for further research in this domain. The data indicate that training programmes should be tailored not only to working circumstances of participants, but should also take levels of experience and self-efficacy into account. Further study is necessary and the best way to proceed is to relate the outcomes of this study to behavioural outcomes.

This study evaluated the feasibility of two different scoring forms for assessing the clinical performance of residents in anaesthesiology. One of the forms had a checklist format including task-specific items and the other was a global rating form with general dimensions of competence including 'clinical skills', 'communication skills' and 'knowledge'. Thirty-two clinicians representing 25 (83%) of the 30 training hospitals in the country participated in the study. The clinicians were randomized into two groups, each of which used one of the scoring formats to assess a resident's performance in four simulated clinical scenarios on videotape. Clinicians' opinions about the appropriateness of the scoring forms were rated on a scale of 1-5. The checklist format was rated significantly higher compared with the global rating form (mean 4.6, 0.5 vs. mean 3.5, 1.4, p < 0.001). The inter-rater agreement regarding pass/fail decisions was poor irrespective of the scoring form used. This was explained by clinicians' leniency as assessors rather than by lack of vigilance in the observations or disagreements on standards for good performance.

Context: Simulation-based testing methods have been developed to meet the need for assessment procedures that are both authentic and well-structured. It is widely acknowledged that, although the authenticity of a procedure may be a contributing factor to its validity, authenticity alone never is a sufficient factor., Aim: In this paper we describe the mainstream development of various simulation-based approaches, with their strengths and weaknesses. The purpose is not to provide a review based on an extensive meta-analysis but to present crucial factors in the development of these methods and their implications for current and future developments., Method: The description of these simulation-based instruments uses a subdivision according to the layers of Miller's pyramid. Written and computer-based simulations are aimed at measuring the 'knows how' layer, observation-based techniques such as standardised patient-based examinations and objective structured clinical examinations target the 'shows how' layer and performance practice measures assess performance at the 'does' layer., Conclusion: In all simulations, case specificity was found to pose the most prominent threat to reliability, while too much structure threatened to trivialise the assessment. The conclusion is that authentic and reliable assessment is predicated on a wise balance between efficiency and adequate content sampling.

Little is known about the effectiveness of clinical education. A more educational structure is considered to be potentially beneficial. The following structured components were added to a surgical clerkship: logbooks, an observed student-patient encounter, individual appraisals, feedback on patient notes, and (case) presentations by students. The authors organized two focus-group sessions in which 19 students participated to explore their perceptions about effective clinical learning experiences and the newly introduced structured components. The analysis of the transcripts showed that observation and constructive feedback are key features of clinical training. The structured activities were appreciated and the results show the direction to be taken for further improvement. Learning experiences depended vastly on individual clinicians' educational qualities. Students experienced being on call, assisting in theatre and time for self-study as instructive elements. Recommended clerkship components are: active involvement of students, direct observation, selection of teachers, a positive learning environment and time for self-study.

Introduction: In order to gain more insight into learning in different traineeships we sought students' opinions on their experiences., Method: 24 students of Maastricht University, the Netherlands, were interviewed: 8 medical students, and 16 students in the mental health science programme of the Faculty of Health Sciences (8 research trainees (MHS-res.) and 8 trainees in mental health care practice (MHS-prac.). Students' perceptions about instructive, difficult and less instructive learning experiences were recorded on audiotape. The literal transcripts of the interviews were analysed and categorised., Results: The most frequently mentioned learning process was "learning by doing" followed by "actively overcoming gaps in knowledge and skills". These processes occurred with instructive and difficult experiences. Other processes were "learning by seeing things in practice" and "preparation and evaluation". Learning outcomes were categorised as learning about: working, professional competences and personal growth. Most frequently mentioned - in most cases with difficult experiences - were "professional knowledge and skills", "learning about personal growth" and "learning about working". The latter was mentioned most often by MHS-res. students. Medical students' responses suggested that they occasionally perceived the clerkship environment as stressful., Discussion and Conclusions: Although the small sample size precludes any firm conclusions, the overwhelming impression is that students prefer being actively involved in their learning process. In addition, traineeships appear to introduce students to professional working life the hard way. Investigating how teachers and supervisors can stimulate active learning and facilitate the introduction to professional practice might be subjects for fruitful further investigation.

The purpose of this paper is to identify the key theoretical and empirical considerations that should underpin effective training for supervisors. Decisions about the content of training courses are complex because there is no appropriate model of supervision in medicine. This paper argues that, in the absence of an explanatory model, effective training for supervisors should be based on existing relevant understandings about learning processes and models of supervision, together with relevant empirical data. It explores some useful models of supervision, considers helpful aspects of theories about learning from experience and identifies some relevant empirical findings to identify some key theoretical and empirical considerations that should underpin effective training for supervisors. This paper offers a framework, derived from both empirical and theoretical work, to guide the content of effective supervision training courses. It does not prescribe structure or organization of training but suggests a basis for devising specific courses for response to local needs.

Objectives: To assess, using standardised patients (SPs), how rheumatologists diagnose psoriatic arthritis, whether the diagnostic efficiency is influenced by specific characteristics of the rheumatologists, and to study the relationship with costs., Methods: Twenty three rheumatologists were each visited by one of two SPs (one male, one female) presenting as a patient with psoriatic arthritis. SPs remained incognito for all meetings for the duration of the study. Immediately after the encounter, SPs completed case-specific checklists on the medical content of the encounter. Information on ordered laboratory and imaging tests was obtained from each hospital., Results: Fourteen rheumatologists diagnosed psoriatic arthritis correctly. They inspected the skin for psoriatic lesions more often than those rheumatologists who established other diagnoses. Rheumatologists diagnosing psoriatic arthritis spent more on additional laboratory and imaging investigations. These were carried out after the diagnosis to confirm it and to record the extent and severity of the disease. No differences in type of practice, number of outpatients seen each week, working experience, or sex were found between rheumatologists who made the correct diagnosis and those who made other diagnoses. The correct diagnosis was more often missed by rheumatologists who saw the male SP, who presented with clear distal interphalangeal DIP joint arthritis only, causing confusion with osteoarthritis of the DIP joints., Conclusion: There is a considerable amount of variation in the delivery of care among rheumatologists who see an SP with psoriatic arthritis. Rheumatologists focusing too much on the most prominent features (DIP joint arthritis) sometimes seem to forget "the hidden (skin) symptoms".

Aim of the Study: The communication skills of students of the Dutch medical schools of Maastricht and Leiden were compared, to assess the effectiveness of these schools' different approaches to communication skills training. Both schools have a six-year undergraduate medical curriculum, divided into four preclinical years and two years of clinical clerkships. The Maastricht problem-based curriculum offers an integrated clinical skills training programme, including communication skills, which runs throughout the first four years. Communication skills training in Leiden is concentrated in courses in the preclinical phase, at the beginning of the clinical phase and preceding two clerkships., Method: Communication skills of fourth-year and sixth-year students (N = 161) of both universities were assessed using four OSCE stations in which students did entire consultations with standardised patients. Trained observers rated these consultations, using a checklist., Results: Maastricht students of both year groups obtained significantly higher checklist scores for their communication skills than their Leiden colleagues. The Leiden students' scores increase between years 4 and 6, whereas those of the Maastricht students showed no significant change., Discussion: The higher scores obtained by the Maastricht students indicate a greater overall effectiveness of a longitudinal, integrated approach compared with concentrated courses. Absence of formal training in the clinical phase in Maastricht leads to stabilisation of communication skills, whereas the increase in the Leiden students' scores between years 4 and 6 offers evidence that formal communication skills training during the clinical phase does pay off. These findings suggest that the preferred approach to communication skills training would be an integrated, longitudinal programme, which continues during the clinical years.

This article reviews consistent research findings concerning the assessment of clinical competence during the clerkship phase of the undergraduate medical training programme on issues of reliability, validity, effect on training programme and learning behaviour, acceptability and costs. Subsequently, research findings on the clinical clerkship as a learning environment are discussed demonstrating that the clinical attachment provides a rather unstructured educational framework. Five fundamental questions (why, what, when, how, who) are addressed to generate general suggestions for improving assessment on the basis of the evidence on assessment and clinical training. Good assessment requires a thoughtful compromise between what is achievable and what is ideal. It is argued that educational effects are eminently important in this compromise, particularly in the unstructured clinical setting. Maximizing educational effects can be achieved in combination with improvements of other measurement qualities of the assessment. Two concrete examples are provided to illustrate the recommended assessment strategies.