Search

Background: Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively., Methods: The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score., Results: A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument., Conclusion: There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research., (© 2023 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Context: Simulation-based medical education (SBME) is now ubiquitous at all levels of medical training. Given the substantial resources needed for SBME, economic evaluation of simulation-based programmes or curricula is required to demonstrate whether improvement in trainee performance (knowledge, skills and attitudes) and health outcomes justifies the cost of investment. Current literature evaluating SBME fails to provide consistent and interpretable information on the relative costs and benefits of alternatives., Content: Economic evaluation is widely applied in health care, but is relatively scarce in medical education. Therefore, in this paper, using a focus on SBME, we define economic evaluation, describe the key components, and discuss the challenges associated with conducting an economic evaluation of medical education interventions. As a way forward to the rigorous and state of the art application of economic evaluation in medical education, we outline the steps to gather the necessary information to conduct an economic evaluation of simulation-based education programmes and curricula, and describe the main approaches to conducting an economic evaluation., Conclusion: A properly conducted economic evaluation can help stakeholders (i.e., programme directors, policy makers and curriculum designers) to determine the optimal use of resources in selecting the modality or method of assessment in simulation. It also helps inform broader decision making about allocation of scarce resources within an educational programme, as well as between education and clinical care. Economic evaluation in medical education research is still in its infancy, and there is significant potential for state-of-the-art application of these methods in this area., (© 2017 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Objectives: Key-feature questions (KFQs) have been developed to assess clinical reasoning skills. The purpose of this paper is to review the published evidence on the reliability and validity of KFQs to assess clinical reasoning., Methods: A literature review was conducted by searching MEDLINE (1946-2012) and EMBASE (1980-2012) via OVID and ERIC. The following search terms were used: key feature; question or test or tests or testing or tested or exam; assess or evaluation, and case-based or case-specific. Articles not in English were eliminated., Results: The literature search resulted in 560 articles. Duplicates were eliminated, as were articles that were not relevant; nine articles that contained reliability or validity data remained. A review of the references and of citations of these articles resulted in an additional 12 articles to give a total of 21 for this review. Format, language and scoring of KFQ examinations have been studied and modified to maximise reliability. Internal consistency reliability has been reported as being between 0.49 and 0.95. Face and content validity have been shown to be moderate to high. Construct validity has been shown to be good using vector thinking processes and novice versus expert paradigms, and to discriminate between teaching methods. The very modest correlations between KFQ examinations and more general knowledge-based examinations point to differing roles for each. Importantly, the results of KFQ examinations have been shown to successfully predict future physician performance, including patient outcomes., Conclusions: Although it is inaccurate to conclude that any testing format is universally reliable or valid, published research supports the use of examinations using KFQs to assess clinical reasoning. The review identifies areas of further study, including all categories of evidence. Investigation into how examinations using KFQs integrate with other methods in a system of assessment is needed., (© 2014 John Wiley & Sons Ltd.)

Context: Emotion in medical education rests between the idealised and the invisible, sitting uneasily at the intersection between objective fact and subjective values. Examining the different ways in which emotion is theorised within medical education is important for a number of reasons. Most significant is the possibility that ideas about emotion can inform a broader understanding of issues related to competency and professionalism., Objectives: The current paper provides an overview of three prevailing discourses of emotion in medical education and the ways in which they activate particular professional expectations about emotion in practice., Methods: A Foucauldian critical discourse analysis of the medical education literature was carried out. Keywords, phrases and metaphors related to emotion were examined for their effects in shaping medical socialisation processes., Discussion: Despite the increasing recognition over the last two decades of emotion as 'socially constructed', the view of emotion as individualised is deeply embedded in our language and conceptual frameworks. The discourses that inform our emotion talk and practice as teachers and health care professionals are important to consider for the effects they have on competence and professional identity, as well as on practitioner and patient well-being. Expanded knowledge of how emotion is 'put to work' within medical education can make visible the invisible and unexamined emotion schemas that serve to reproduce problematic professional behaviours. For this discussion, three main discourses of emotion will be identified: a physiological discourse in which emotion is described as located inside the individual as bodily states which are universally experienced; emotion as a form of competence related to skills and abilities, and a socio-cultural discourse which calls on conceptions from the humanities and social sciences and directs our attention to emotion's function in social exchanges and its role as a social, political and cultural mediator., (© Blackwell Publishing Ltd 2013.)

Objectives: The current paper describes a model of learning that has been used to produce efficient learning, thus yielding greater retention of information and superior performance under stress. In this paper, the model is applied to the learning of technical skills., Structure: After a brief review of the learning-performance paradox and other relevant literature from the field of movement science, the benefits of challenge and adversity for learning are discussed in the context of a framework for learning known as the challenge point framework (CPF). The framework is based on laboratory and field studies of methods that have been shown to consistently enhance learning, and is used to model and generate insight into the relationships between practice protocols and the learning that results from them., Application: The practical application of the CPF to simulation-based medical education and training is described. Firstly, a simple conceptual model that utilises three key elements to adjust the functional difficulty of the tasks to be learned is outlined. Secondly, a number of assessment strategies that may be necessary to ensure that the trainee remains in the optimal learning zone are proposed. Thirdly, a practical example is used to demonstrate how to utilise this conceptual model to design simulation environments suitable for teaching an endotracheal intubation task to beginners and more advanced trainees., (© Blackwell Publishing Ltd 2012.)

Context:   Current debate in medical education focuses on the nature of 'competency-based medical education' (CBME) and whether or not it should be adopted. Many medical schools claim to run 'competency-based' curricula, but the structure of their programmes can differ radically. A review of the existing CBME literature reveals that little attention has been paid to defining the concept of competence. A straightforward examination of what is meant by the term 'competence' is noticeably missing from the literature, despite its impact on medical training., Objectives:   This paper aims to illustrate the varying conceptions of 'competence' by comparing and contrasting definitions provided in the health sciences education literature and discussing their respective impacts on medical education., Methods:   A systematic review of recent publications in medical education journals published in English and French was conducted to extract definitions of competence or, if definitions were not explicitly stated, to derive the authors' implicit conception of competence. A sample of 14 definitions from articles in the health sciences education field was studied using thematic analysis., Results:   There is agreement that competence is composed of knowledge, skills and other components. Although agreement about the nature of these other components is lacking, attitudes and values are suggested to be essential ingredients of competence. Furthermore, a clear divergence in conceptions of how a competent person utilises these components is apparent. One view specifies that competence involves selecting components according to specific situations, as required. A second view places greater emphasis on the synergy that results from the use of a combination of components in a given situation., Conclusions:   These conceptual distinctions have many implications for the way CBME is implemented. A conception of competence as the selection of components may lead to a greater emphasis, in a training setting, on the mastery of each component separately. A conception of competence as the use of a combination of components leads to greater emphasis on the synergy that results as they are deployed in clinical situations., (© Blackwell Publishing Ltd 2012.)

Context: Gender is increasingly regarded as an important factor in doctor-patient communication education. This review aims to assess if and how gender is addressed by current assessment instruments for communication skills in medical education., Methods: In 2009 at Radboud University Nijmegen Medical Centre, an online search was conducted in the bibliographic databases PubMed, PsycINFO and ERIC for references about communication skills assessment instruments designed to be completed by trained faculty staff and used in medical education. The search strategy used the following search terms: 'consultation skills'; 'doctor-patient communication'; 'physician-patient relations'; 'medical education'; 'instruments'; 'measurement', and 'assessment'. Papers published between January 1999 and June 2009 were included. The assessment instruments identified were analysed for gender-specific content., Results: The search yielded 21 communication skills assessment instruments. Only two of the 17 checklists obtained explicitly considered gender as a communication-related issue. Only six of 21 manuals considered gender in any way and none gave specific details to explain which aspects of communication behaviour should be assessed with regard to gender., Conclusions: Very few communication assessment instruments in medical education focus on gender. Nevertheless, interest exists in using gender in communication skills assessment. The criteria for and purpose of assessing gender in communication skills in medical education are yet to be clarified., (© Blackwell Publishing Ltd 2011.)

Objectives: The ability to innovate new solutions in response to daily workplace challenges is an important component of adaptive expertise. Exploring how to optimally develop this skill is therefore of paramount importance to education researchers. This is certainly no less true in health care, where optimal patient care is contingent on the continuous efforts of doctors and other health care workers to provide the best care to their patients through the development and incorporation of new knowledge. Medical education programmes must therefore foster the skills and attitudes necessary to engage future doctors in the systematic development of innovative problem solving. The aim of this paper is to describe the perceptions and experiences of medical students in their third and fourth years of training, and to explore their understanding of their development as adaptive experts., Methods: A sample of 25 medical students participated in individual 45-60-minute semi-structured interviews. Interviews were audiotaped, transcribed and entered into NVivo qualitative data analysis software to facilitate a thematic analysis. The analysis was both inductive, in that themes were generated from the data, and deductive, in that our data were meaningful when interpreted in the context of theories of adaptive expertise., Results: Participants expressed a general belief that, as learners in the health care system, exerting any effort to be innovative was beyond the scope of their responsibilities. Generally, students suggested that innovative practice was the prerogative of experts and an outcome of expert development centred on the acquisition of knowledge and experience., Conclusions: Students' perceptions of themselves as having no responsibility to be innovative in their learning process have implications for their learning trajectories as adaptive experts.

Objective: To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies., Methods: We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists., Results: We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised., Conclusions: It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.

Context: The move towards competence-based medical education has created a need for instruments that support and assess competence development. Portfolios seem suitable but mixed reports of their success are emerging., Methods: To examine the effectiveness of portfolios, we searched PubMed and EMBASE using the keyword 'portfol*', PsychInfo and ERIC using the keywords 'portfol*' and 'medical education' and references of retrieved papers for empirical studies on portfolios in all phases of medical education. Thirty of 1939 retrieved papers met the inclusion criteria and were analysed. Data were collated against the research question, number of subjects, design, setting, findings and limitations, purpose and content, mentoring and assessment. We analysed impact using a modified version of Kirkpatrick's hierarchy., Results: Because differences across studies precluded statistical meta-analysis, the data were analysed by context, goals and procedure. Positive effects were strongest in undergraduate education. Important factors for success were: clearly communicated goals and procedures; integration with curriculum and assessment; flexible structure; support through mentoring, and measures to heighten feasibility and reduce required time. Moderately good inter-rater reliability was reported and global criteria and discussions among raters were beneficial. Formative and summative assessment could be combined. Without assessment, portfolios were vulnerable to competition from other summative assessment instruments., Conclusions: For portfolios to be effective in supporting and assessing competence development, robust integration into the curriculum and tutor support are essential. Further studies should focus on the effectiveness and user-friendliness of portfolios, the merits of holistic assessment procedures, and the competences of an effective portfolio mentor.

Context: Many approaches to the study of expertise in medical education have their roots most strongly established in the traditional cognitive psychology literature. As such, they take a common approach to the construction of expertise and frame their questions in a common way. This paper reflects on a few of the paradigmatic assumptions that have 'come along for the ride' with the traditional cognitive approach, and explores what might have been left out as a consequence., Methods: We examine the operational definition of 'expert' as it has evolved using the traditional cognitive paradigm and we explore some alternative definitions and constructions of expert performance that have arisen in parallel education research paradigms. We address 3 inter-related aspects of expertise as manifested in the traditional cognitive approach: the construction of the expert as a (routine) diagnostician; the construction of the developmental process as the (automatic and un-reflective) accrual of resources through experience, and the construction of accrued knowledge as a relatively static resource that is subsequently used and built upon with further experience., Conclusions: We hope that, by highlighting these issues, we may begin to marry the strengths of the traditional cognitive paradigm with the strengths of these other paradigms and expand the scope of cognitive research in medical expertise.

Objectives: The aim of this review was to evaluate the effectiveness of medical skills laboratories or simulators. In particular, it aimed to determine if performance in medical skills laboratories is transferable to actual clinical performance and maintained over time., Methods: A range of databases was utilised to search for relevant papers published from 1998 to June 2006. Articles were included in the review if they met a number of criteria that included the evaluation of a skills laboratory or simulator for the purpose of procedural skills training, that participants were either undergraduate medical students or postgraduate medical trainees, and that the study used a randomised, controlled trial (RCT) research design in evaluation., Results: A total of 44 RCTs were identified for inclusion in the review. Overall, 32 (70%) studies reported that simulator training significantly improved procedural skills performance in comparison with standard or no training. Twenty (45%) RCTs assessed the transfer of simulator performance to clinical skills performance; however, 8 of these used animal models, not real patients. Only 2 studies assessed the maintenance of skills post-intervention, both at 4-month follow-up periods., Conclusions: Medical skills laboratories do lead to improvement in procedural skills compared with standard or no training at all when assessed by simulator performance and immediately post-training. However, there is a lack of well designed trials addressing the crucial issues of transferability to clinical practice and retention of skills over time. Further research must be carried out to address these matters if medical skills laboratories are to remain an integral component of medical education.

Background: No attempt has been made to give a systematic overview of the innovative use of unannounced or incognito standardised patients., Objective: To provide a systematic overview of all published studies with incognito standardised patients assessing doctors in actual practice situations using sound reliable and valid standardised patient methodology., Methods: Systematic literature search in Medline, Eric and Psyclit with piloted search terms, followed by retrieving additional papers from reference lists. All papers were scored with a predefined protocol-sheet using predefined inclusion and exclusion criteria., Results: Forty papers referring to 21 projects with incognito standardised patients were found and described in detail. The majority of studies were conducted in primary care and were descriptive., Discussion: There is a need for research on the additional value of the incognito standardised patient methodology and for a consensus on how to report on accuracy and consistency of standardised patients.

Background: There is increasing interest in the role of context in medical education, with the conjecture that learning in a clinical context may be helpful for later recall of knowledge. Although this may be true in a general sense, at a closer look it appears that the notion of context is not well substantiated in the medical education literature and that the concept is not clearly defined. Effects of context on learning appear to depend on type of learning task, the relationship or interaction between the context and the learning material, and motivational features of the context. Context is often implicitly regarded as a uniform concept but conceptual analysis shows that a distinction can be made in several dimensions., Results: In this paper, we identify 3 different dimensions of context: a physical dimension, representing the environmental characteristics; a semantic dimension, reflecting how well the context contributes to the learning task, and a commitment dimension, representing the amount of commitment (in terms of motivation and responsibility) that is generated by the context. On these dimensions, context can be ordered from reduced (providing few cues, little meaning, little commitment) to enriched (many cues, much meaning, high commitment)., Conclusion: This model can serve a dual purpose: first, to disentangle several aspects of educational contexts (e.g. as high in meaning but low in commitment), and second, to provide a theoretical framework to generate research on the influence of different contexts in education on students' learning.

Context: The evaluation of training is problematic and the evidence base inconclusive. This situation may arise for 2 main reasons: training is not understood as a complex intervention and, related to this, the evaluation methods applied are often overly simplistic., Method: This paper makes the case for construing training, especially in the field of specialist medical education, as a complex intervention. It also selectively reviews the available literature in order to match evaluative techniques with the demonstrated complexity., Conclusions: Construing training as a complex intervention can provide a framework for selecting the most appropriate methodology to evaluate a given training intervention and to appraise the evidence base for training fairly, choosing from among both quantitative and qualitative approaches and applying measurement at multiple levels of training impact.

The present paper briefly describes a growing trend in continuing professional development (CPD) for medical practitioners, examines some recent published contributions related to professionalism in medicine and describes the theoretical platform of a CPD framework based on medical professionalism that has been developed for use in a variety of medical specialties. The work arises from a project funded by The Commonwealth Department of Health and Ageing and conducted by the Royal Australian and New Zealand College of Obstetricians and Gynaecologists on behalf of the Committee of Presidents of Medical Colleges. Overall, the aim of the project was to design and develop a framework that can be considered 'best practice' in the field of medical specialist CPD, and which could be applicable to a wide variety of medical specialties and practice environments. The project was informed by available literature in areas such as the ways in which medical practitioners learn and the effectiveness of CPD, as well as consultations with a variety of stakeholders in the health-care field, including medical boards, medical defence organisations, health complaints organisations, specialist medical colleges and practising specialists. Overwhelmingly, the primary purpose of specialists participating in ongoing CPD was, according to the project consultations conducted, the maintenance, development and improvement of skills that enable consumers of healthcare to be confident of the level of care that they receive. Coupled with the need to recognise CPD as contributing to the wider context of professional practice, the resulting framework is underpinned by a theoretical platform based on the concept of medical professionalism. It is this theoretical platform, which articulates the nature of and responsibilities associated with the practice of specialist medicine, that enables a CPD framework to be constructed that can accommodate a wide range of specialist disciplines and practice settings, while also framing ongoing education expectations from the viewpoint of a range of stakeholders, rather than a single perspective; that of practitioners themselves.

Background: Anatomy learning is generally seen as essential to medicine, and exposure to cadavers is generally seen as essential to anatomy learning around the world. Few voices dissenting from these propositions can be identified., Aims: This paper aims to consider arguments relating to the use of cadavers in anatomy teaching, and to describe the rationale behind the decision of a new UK medical school not to use cadaveric material., Discussion: First, the background to use of cadavers in anatomy learning is explored, and some general educational principles are explored. Next, arguments for the use of human cadaveric material are summarised. Then, possible arguments against use of cadavers, including educational principles as well as costs, hazards and practicality, are considered. These are much less well explored in the existing literature. Next, the rationale behind the decision of a new UK medical school not to use cadaveric material is indicated, and the programme of anatomy teaching to be employed in the absence of the use of human remains is described. Curriculum design and development, and evaluation procedures, are briefly described. Issues surrounding pathology training by autopsy, and postgraduate training in surgical anatomy, are not addressed in this paper., Future Directions: Evidence relating to the effect on medical learning by students not exposed to cadavers is scant, and plainly opportunities will now arise through our programme to gather such evidence. We anticipate that this discussion paper will contribute to an ongoing debate, in which virtually all previous papers on this topic have concluded that use of cadavers is essential to medical learning.

The traditional clinical examination has been shown to have serious limitations in terms of its validity and reliability. The OSCE provides some answers to these limitations and has become very popular. Many variants on the original OSCE format now exist and much research has been done on various aspects of their use. Issues to be addressed relate to organization matters and to the quality of the assessment. This paper focuses particularly on the latter with respect to ways of ensuring content validity and achieving acceptable levels of reliability. A particular concern has been the demonstrable need for long examinations if high levels of reliability are to be achieved. Strategies for reducing the practical difficulties this raises are discussed. Standard setting methods for use with OSCEs are described.

Objective: This paper outlines the considerations to be made when establishing a clinical skills learning facility., Considerations: Establishing a clinical skills learning facility is a complex project with many possible options to be considered. A number of professional groups, undergraduate or postgraduate, may be users. Their collaboration can have benefits for funding, uses and promotion of interprofessional education. Best evidence and educational theory should underpin teaching and learning. The physical environment should be flexible to allow a range of clinical settings to be simulated and to facilitate a range of teaching and learning methods, supported by computing and audio-visual resources. Facilities should be available to encourage self-directed learning. The skills programme should be designed to support the intended learning outcomes and be integrated within the overall curriculum, including within the assessment strategy. Teaching staff may be configured in a number of ways and may be drawn from a variety of backgrounds. Appropriate staff development will be required to ensure consistency and quality of teaching with monitoring and evaluation to assure appropriate standards. Patients can also play a role, not only as passive teaching material, but also as teachers and assessors. Clinical, diagnostic and therapeutic equipment will be required, as will models and manikins. The latter will vary from simple part task trainers to highly sophisticated human patient simulators. Care must be taken when choosing equipment to ensure it matches specified requirements for teaching and learning., Conclusion: Detailed planning is required across a number of domains when setting up a clinical skills learning facility.

Purpose: Despite numerous calls for reform over several decades, medical educators have been unable to address many significant challenges. Potentially, employing new metaphors and looking at the teaching and learning of medicine in a new way will facilitate the development of creative solutions., Main Findings: In this paper we propose the metaphor of medicine as a performing art. Building on this metaphor, string music education is compared to medical education., Principal Conclusions: Looking to string education as a model, suggestions for reorganisation of learning experiences, academic structure and assessment are discussed. Medical educators are encouraged to think about the challenges they face in creative ways. By looking outside traditional medical education, solutions may be found to new and old educational dilemmas.

Background: Surgical skills are required by a wide range of health care professionals. Tasks range from simple wound closure to highly complex diagnostic and therapeutic procedures. Technical expertise, although essential, is only one component of a complex picture. By emphasising the importance of knowledge and attitudes, this article aims to locate the acquisition of surgical skills within a wider educational framework., Simulators: Simulators can provide safe, realistic learning environments for repeated practice, underpinned by feedback and objective metrics of performance. Using a simple classification of simulators into model-based, computer-based or hybrid, this paper summarises the current state of the art and describes recent technological developments. Advances in computing have led to the establishment of precision placement and simple manipulation simulators within health care education, while complex manipulation and integrated procedure simulators are still in the development phase., Evaluation: Tension often exists between the design and evaluation of surgical simulations. A lack of high quality published data is compounded by the difficulties of conducting longitudinal studies in such a fast-moving field. The implications of this tension are discussed., The Wider Context: The emphasis is now shifting from the technology of simulation towards partnership with education and clinical practice. This highlights the need for an integrated learning framework, where knowledge can be acquired alongside technical skills and not in isolation from them. Recent work on situated learning underlines the potential for simulation to feed into and enrich everyday clinical practice.

Background: This paper demystifies clinical perception by explaining its mechanisms, using insights from neuroscience and evolutionary biology. Clinical diagnosis begins with rapid recognition using our imaging, perceptual (but non-verbal) brain, followed by guided search using our slower, verbal, reasoning brain. Experiential cognition can be (more or less) achieved by integrating these two ways of knowing. Perceptual expertise requires alertness and persistence to ensure clinical accuracy. Each clinician, as a self-aware participant-observer (SAPO) keeping track of what they're thinking 'as it happens', can study their perceptual accuracy, pattern matching, interpretation, motivation and judgement.

Background: The use of portfolios can potentially provide flexibility in the summative assessment of doctors in practice. An assessment system should reflect and reinforce the active and planned professional development goals of individual doctors. This paper discusses some of the issues involved in developing such a system., Results: To provide a complete picture of an individual doctor's practice, we suggest that a portfolio should encompass: (1) evidence covering all three domains of patient care, personal development and context management; (2) evidence that the person continuously undertakes critical assessment of their own performance, identifies and prioritises areas requiring enhanced performance and takes action to improve them as appropriate; (3) evidence that has been generated by assessments that are acceptably reliable, and (4) evidence which, taken in its entirety, is sufficient, valid, current and authentic. We include a suggested outline of the components of such a portfolio and suggest some criteria to determine the effectiveness of learning cycles. Portfolio reliability and validity requires sufficient evidence on which to base a judgement combined with reliable processes., Conclusion: Carefully specified portfolios can contribute to a system that ensures all doctors take an active part in identifying and meeting their own learning needs. Such a system, if properly implemented, would have a greatly beneficial impact on continuous quality improvement for the profession in general.

The assessment of the performance of doctors in practice is becoming more widely accepted. While there are many potential purposes for such assessments, sometimes the consequences of the assessments will be 'high stakes'. In these circumstances, any of the many elements of the assessment programme may potentially be challenged. These assessment programmes therefore need to be robust, fair and defensible, taken from the perspectives of consumer, assessee and assessor. In order to inform the design of defensible programmes for assessing practice performance, a group of education researchers at the 10th Cambridge Conference adopted a project management approach to designing practice performance assessment programmes. This paper describes issues to consider in the articulation of the purposes and outcomes of the assessment, planning the programme, the administrative processes involved, including communication and preparation of assessees. Examples of key questions to be answered are provided, but further work is needed to test validity.

Context: Basic surgical skills are needed throughout the medical profession, but current training is haphazard and unpredictable. There is increasing pressure to provide transparency about training and performance standards. There is a clear need for inexperienced learners to build a framework of basic skills before carrying out surgical procedures on patients. Effective learning of a skill requires sustained deliberate practice within a cognitive framework, and simulation offers an opportunity for safe preparation., Objectives: This paper presents a new approach to basic surgical skills training, where tuition using a specially designed computer program is combined with structured practice using simulated tissue models. This approach to teaching has evolved from practical experience with surgical skills training in workshops., Methods: Pilot studies with 72 first-year medical students highlighted the need for separate programs for teaching and for self-directed learning. The authors developed a training approach in the light of this experience. Subsequent in-depth observational and interview studies examined (a) individual teaching sessions between surgical teachers and learners (five consultant surgeons and five senior house officers) and (b) group teaching sessions with general practitioners (14 participants in three group interviews). Further work has resulted in a self-directed learning program., Conclusions: Qualitative analysis of observational and interview data provides strong preliminary support for the effectiveness of this approach. The response of teachers and learners was extremely positive. The combination of information (presented by computer) and practice of psychomotor skill (using simulated tissue models) could be extended to other surgical and practical skills.

Context: A fundamental premise of medical education is that faculty should educate trainees, that is, students and residents, to provide high quality patient care. Yet, there is little research on the effect of medical education on patient outcomes., Objective: A content analysis of leading medical education journals was performed to determine the primary foci of medical education research, using a three- dimensional outcomes research framework based on the paradigm of health services outcomes research., Data Sources: All articles in three medical education journals (Academic Medicine, Medical Education, and Teaching and Learning in Medicine) from 1996 to 1998 were reviewed. Papers presented at the Research in Medical Education conference at the Association of American Medical Colleges annual meeting during the same period, and published as Academic Medicine supplements, were also analysed., Study Selection: Only data-driven articles were selected for analysis; thus editorials and abstracts were excluded., Data Extraction: Each article was categorized according to primary participant (i.e. trainee, faculty, provider and patient), outcome (performance, satisfaction, professionalism and cost), and level of analysis (geographic, system, institution and individual(s))., Data Synthesis: A total of 599 articles were analysed. Trainees were the most frequent participants studied (68.9%), followed by faculty (19.4%), providers (8.1%) and patients (3.5%). Performance was the most common outcome measured (49.4%), followed by satisfaction (34.1%). Cost was the focus of only 2.3% of articles and patient outcomes accounted for only 0.7% of articles., Conclusions: Medical education research is dominated by assessment of trainee performance followed by trainee satisfaction. Leading journals in medical education contain little information concerning the cost and products of medical education, that is, provider performance and patient outcomes. The study of these medical education outcomes represents an important challenge to medical education researchers.

Increasingly, courses in communication skills are being incorporated into medical training. In order for communication skills to be effectively maintained in post-training medical practice, they must be taught within an appropriate clinical context. The present paper describes and provides rationale for seven criteria by which to select clinical issues which are appropriate foci for communication skills courses. The criteria are: (1) the issue must be one which is encountered frequently in clinical practice; (2) the issue must be associated with a high burden of illness; (3) there must be evidence that practitioners need to improve skills for dealing with the issue; (4) there must be an intervention, of which communication skills are an integral component, that is demonstrably effective for dealing with the clinical issue; (5) the intervention must represent a cost-effective means of dealing with the issue; (6) the intervention must be acceptable to doctors and be able to be incorporated into routine medical practice; (7) the intervention must be acceptable to patients. Examples of clinical issues which fit these criteria are given in the paper and include smoking, hazardous alcohol consumption, non-adherence to treatment instructions, overdue cervical screening, inappropriate diet, recovery from medical interventions, and breaking bad news to patients.

This paper reports a study which compared the performance of different groups of students and doctors on identical and equivalent tests set in an objective-type format and in a free-response format. The tests were designed to ensure that the content was relevant to clinical practice at the hospital intern level. In all test situations candidates' scores were significantly higher in the objective tests than in the free-response tests. This difference was greater for the more junior and less competent students than for the more competent doctors. The cueing effect of the options was thought to be the main factor responsible for the difference in performance. The results of a questionnaire survey demonstrated that students were aware of the deficiencies in multiple-choice tests. A large majority of the students believed that the free-response tests gave a more accurate assessment of their clinical ability. It was found that in these tests, aimed at measuring aspects of clinical competence, multiple-choice questions appeared to overestimate the candidate's ability to an extent that made them less suitable than free-response questions for this purpose. It was also found that free-response tests, of the type used in this study, provide a suitable alternative to multiple-choice tests for use in the written section of clinical examinations. It was concluded that the written component of the final examination in the medical course should have a preponderance of free-response items over multiple-choice items.