Search

Having teaching staff with didactic qualifications in university teaching leads to a measurable improvement in academic skills among students. Previous recommendations on the type and scope of medical didactic qualification measures primarily apply to teaching staff at university and in-patient settings. The situation of primary care medicine, which often employs external lecturers and whose teaching takes place to a considerable extent in decentralized training facilities (teaching practices) is not adequately addressed. Taking into account a survey on the status quo at higher education institutions for General Practice in Germany, recommendations for minimum standards are made, based on national and international recommendations on the content and scope of medical didactic qualification measures. These recommendations include preliminary work by the Personnel and Organizational Development in Teaching (POiL) Committee of the Society for Medical Education (GMA), the MedicalTeachingNetwork (MDN), the Society of University Teaching Staff in General Medicine (GHA) as well as the experiences of the committee members, who hail from the field of general medicine, internal medicine and pediatrics amongst others., Competing Interests: The authors declare that they have no competing interests., (Copyright © 2020 Böhme et al.)

Introduction: Multiple-choice-examinations are still fundamental for assessment in medical degree programs. In addition to content related research, the optimization of the technical procedure is an important question. Medical examiners face three options: paper-based examinations with or without computer support or completely electronic examinations. Critical aspects are the effort for formatting, the logistic effort during the actual examination, quality, promptness and effort of the correction, the time for making the documents available for inspection by the students, and the statistical analysis of the examination results., Methods: Since three semesters a computer program for input and formatting of MC-questions in medical and other paper-based examinations is used and continuously improved at Wuerzburg University. In the winter semester (WS) 2009/10 eleven, in the summer semester (SS) 2010 twelve and in WS 2010/11 thirteen medical examinations were accomplished with the program and automatically evaluated. For the last two semesters the remaining manual workload was recorded., Results: The cost of the formatting and the subsequent analysis including adjustments of the analysis of an average examination with about 140 participants and about 35 questions was 5-7 hours for exams without complications in the winter semester 2009/2010, about 2 hours in SS 2010 and about 1.5 hours in the winter semester 2010/11. Including exams with complications, the average time was about 3 hours per exam in SS 2010 and 2.67 hours for the WS 10/11., Discussion: For conventional multiple-choice exams the computer-based formatting and evaluation of paper-based exams offers a significant time reduction for lecturers in comparison with the manual correction of paper-based exams and compared to purely electronically conducted exams it needs a much simpler technological infrastructure and fewer staff during the exam.

Objective: To understand the degree of local medical students' clinical practice skills, evaluate teaching achievements and reveal weaknesses by analyzing the test papers and grades, so as to provide scientific advices for reform in clinical education., Methods: Test papers of 721 copies of traditional Chinese medical qualification examination (practical skills examination) were collected from the examination district of Shanghai in 2009. The papers were analyzed by using the principle and method of teaching statistics. The difficulty and reliability coefficients, discrimination and scores were evaluated., Results: The reliability coefficient of the paper was 0.527, the difficulty coefficients of different parts differed from 0.582 to 0.828 and the discrimination ranged from 0.401 to 0.753. The average score was 75.519 with the standard deviation of 11.073. The examinees had high scores in Chinese medicine training, including acupoint location and methods of acupuncture and moxibustion, but low scores in practice and display of Western medicine., Conclusion: The reliability coefficient, discrimination and the difficulty are all preferable. The students got a satisfying average while they had a shortage of ability in practice and display of Western medicine.

Background: Computer-based testing (CBT) has become increasingly popular as a testing modality in under- and postgraduate medical education. Since 2004, our medical school has utilised CBT to conduct 2 papers for the third- and final-year assessments - Paper 3, with 30 multiple choice questions featuring clinical vignettes, and the modified essay question (MEQ) paper., Aims: To obtain feedback from final-year students on their preferred mode of testing for Paper 3 and MEQ components of the Medicine track examination, and the reasons underlying their preferences., Methods: An online survey was carried out on 213 final-year undergraduates, in which they were asked to provide feedback on Paper 3 and MEQ papers. Students were asked if they thought that the CBT format was preferable to the pen-and-paper (PNP) format for Paper 3 and the MEQ, and why., Results: One hundred and fourteen out of 213 (53.5%) students completed the online survey. For Paper 3, 91 (79.8%) felt that CBT was preferable to PNP, 11 (9.6%) preferred the PNP format and 12 (10.5%) were unsure. For the MEQ, 62 (54.4%) preferred CBT over PNP, 30 (26.3%) preferred the PNP format and 22 (19.3%) were unsure. Reasons given to explain preference for CBT over PNP for Paper 3 included independence from seating position, better image quality (as images were shown on personal computer screens instead of projected onto a common screen) and the fact that CBT allowed them to proceed at their own pace. For the MEQ, better image quality, neater answer scripts and better indication of answer length in CBT format were cited as reasons for their preference., Conclusions: Our survey indicated that whereas the majority of students preferred CBT over PNP for Paper 3, a smaller margin had the same preference for the MEQ.

Purpose: Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance., Method: In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability., Results: Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment., Conclusions: G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills., (Copyright © 2021 by the Association of American Medical Colleges.)

Introduction: A workplace-based assessment (WBA) is a learning recording device that is widely used in medical education globally. Although entrenched in medical curricula, and despite a substantial body of literature exploring them, it is not yet fully understood how WBAs play out in practice. Adopting a constructivist standpoint, we examine these assessments, in the workplace, using principles based upon naturalist inquiry, drawing from a theoretical framework based on Goffman's dramaturgical analogy for the presentation of self, and using qualitative research methods to articulate what is happening as learners complete them., Methods: Learners were voluntarily recruited to participate in the study from a single teaching hospital. Data were generated, in-situ, through observations with field notes and audiovisual recording of WBAs, along with accompanying interviews with learners., Results: Data from six learners was analysed to reveal a set of general principles-the WBA playbook. These four principles were tacit, unwritten, unofficial and learners applied them to complete their WBA proformas: (1) maintain the impression of progression, (2) manage the authenticity of the individual proforma, (3) avoid losing face with the assessor and (4) complete the proforma in an effort-efficient way. By adhering to these principles, learners expressed their understanding of their social position in their world at that time the documents were created., Discussion: This paper recognises the value of the WBA as a lived experience, and of the WBA document as a social space, where learners engage in a social performance before the readers of the proforma. Such an interpretation better represents what happens as learners undergo and record WBAs in the real-world, recognising WBAs as learner-centred, learner-driven, meaning-making phenomena. In this way, as a record of interpretation and meanings, the subjective nature of the WBA process is a strength to be harnessed, rather than a weakness to be glossed over., (© 2023 Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Clerkship grades (like money) are a social construct that function as the currency through which value exchanges in medical education are negotiated between the system's various stakeholders. They provide a widely recognizable and efficient medium through which learner development can be assessed, tracked, compared, and demonstrated and are commonly used to make decisions regarding progression, distinction, and selection for residency. However, substantial literature has demonstrated how grades imprecisely and unreliably reflect the value of learners. In this article, the authors suggest that challenges with clerkship grades are fundamentally tied to their role as currency in the medical education system. Associations are drawn between clerkship grades and the history of the U.S. economy; 2 major concepts are highlighted: regulation and stock prices. The authors describe the history of these economic concepts and how they relate to challenges in clerkship grading. Using lessons learned from the history of the U.S. economy, the authors then propose a 2-step solution to improve upon grading for future generations of medical students: (1) transition from grades to a federally regulated competency-based assessment model and (2) development of a departmental competency letter that incorporates competency-based assessments rather than letter grades and meets the needs of program directors., (Copyright © 2020 by the Association of American Medical Colleges.)

Introduction: Many institutions require candidates to achieve a minimum number of OSCE stations passed (MNSP) in addition to the aggregate pass mark. The stated rationale is usually that this conjunctive standard prevents excessive degrees of compensation across an assessment. However, there is a lack of consideration and discussion of this practice in the medical education literature., Methods: We consider the motivations for the adoption of the MNSP from the assessment designer perspective, outlining potential concerns about the complexity of what the OSCE is trying to achieve, particularly around the blueprinting process and the limitations of scoring instruments. We also introduce four potential methods for setting an examinee-centred MNSP standard, and highlight briefly the theoretical advantages and disadvantages of these approaches., Discussion and Conclusion: There are psychometric arguments for and against the limiting of compensation in OSCEs, but it is clear that many stakeholders value the application of an MNSP standard. This paper adds to the limited literature on this important topic and notes that current MNSP practices are often problematic in high stakes settings. More empirical work is needed to develop understanding of the impact on pass/fail decision-making of the proposed standard setting methods developed in this paper.

Background: Taking a medical history and performing a physical examination represent basic medical skills. However, numerous national and international studies show that medical students and physicians-to-be demonstrate substantial deficiencies in the proper examination of individual organ systems. Aim: The objective of this study was to conduct a randomized controlled pilot study to see if, in the context of a bedside clinical examination course in internal medicine, an additional app-based blended-learning strategy resulted in (a) higher satisfaction, better self-assessments by students when rating their history-taking skills (b1) and their ability to perform physical examinations (b2), as well as (c) higher multiple-choice test scores at the end of the course, when compared to a traditional teaching strategy. Methods: Within the scope of a bedside course teaching the techniques of clinical examination, 26 students out of a total of 335 students enrolled in the 2012 summer semester and 2012/2013 winter semester were randomly assigned to two groups of the same size. Thirteen students were in an intervention group (IG) with pre- and post-material for studying via an app-based blended-learning tool, and another 13 students were in a control group (CG) with the usual pre- and post-material (handouts). The IG was given an app specifically created for the history-taking and physical exam course, an application program for smartphones enabling them to view course material directly on the smartphone. The CG received the same information in the form of paper-based notes. Prior to course begin, all of the students filled out a questionnaire on sociodemographic data and took a multiple-choice pretest with questions on anamnesis and physical examination. After completing the course, the students again took a multiple-choice test with questions on anamnesis and physical examination. Results: When compared to the CG, the IG showed significantly more improvement on the multiple-choice tests after taking the clinical examination course (p=0.022). This improvement on the MC tests in the IG significantly correlated with the amount of time spent using the app (Spearman's rho=0.741, p=0.004). Conclusion: When compared to conventional teaching, an app-based blended-learning approach leads to improvement in test scores, possibly as a result of more intensive preparation for and review of the clinical examination course material., Competing Interests: The authors declare that they have no competing interests., (Copyright © 2021 Sonne et al.)

The uncertainty in all spheres of higher education due to the COVID-19 pandemic has had an unprecedented impact on teaching-learning and assessments in medical colleges across the globe. The conventional ways of assessment are now neither possible nor practical for certifying medical graduates. This has necessitated thoughtful considerations in making adjustments to the assessment system, with most institutions transitioning to online assessments that so far have remained underutilized. Programmatic assessment encourages the deliberate and longitudinal use of diverse assessment methods to maximize learning and assessment and at present can be utilized optimally as it ensures the collection of multiple low-stake assessment data which can be aggregated for high-stake pass/fail decisions by making use of every opportunity for formative feedback to improve performance. Though efforts have been made to introduce programmatic assessment in the competency-based undergraduate curriculum, transitioning to online assessment can be a potential opportunity if the basic tenets of programmatic assessment, choice of online assessment tools, strategies, good practices of online assessments and challenges are understood and explored explicitly for designing and implementing online assessments. This paper explores the possibility of introducing online assessment with face-to-face assessment and structuring a blended programmatic assessment in competency-based medical education., Competing Interests: None

Objective The comprehensive test is adaptable to medical integration education and is an effective way to evaluate students' performance and assess the effects of teaching reform. Given the special characteristics of the comprehensive test, this paper is designed to apply item response theory to immunology test question analysis in the comprehensive test of basic medicine, illustrating the method of applying the theory to the medical test evaluation and elaborating its role and significance in quality control of medical comprehensive test. Methods The software e-irt and SPSS were used to analyze 180 immunology test questions from the comprehensive test of basic medicine for six years. All parameters were calculated and then equated to produce characteristic curve and information function curve. Results The immunology test questions from the comprehensive test of basic medicine was of high reliability, but of lower difficulty value and discrimination value than the overall level of the comprehensive test paper. The test parameters and examinees' ability showed significant differences among the six years. The quality of certain immunology test questions did not meet the ideal standard, which is required to be analyzed. Conclusion Medical integration education facilitates the process of our students acquiring immunology knowledge. The item response theory can be effectively applied to comparing the papers, questions and examinees' abilities among the years on the same scale, thus presenting the existing problems in terms of question quality, teaching process and examines' abilities, improving the quality of test questions, and facilitating the establishment of a standardized examination database.

Background: There are plenty of options for evaluating medical students and medical residents' clinical skills. Objective structured clinical evaluations (OSCEs) have emerged as a powerful and reliable tool for assessing multiple cognition domains of clinical expertise. In the same way as OSCEs have emerged to assess clinical skills, objective structured teaching evaluations (OSTEs) have come to light as promising and unbiased interventions for evaluating the act of clinical teaching., Design and Setting: Narrative review developed at Universidade Federal de Uberlândia, Brazil., Methods: We searched the literature regarding OSTEs using the MEDLINE (via PubMed) and LILACS (viaBiblioteca Virtual em Saude) databases. The SciELO library was also searched for Brazilian papers. Systematic reviews, reviews and randomized controlled trials specifically assessing how OSTEs performed in relation to development of academic staff and medical residents were then selected., Results: Our search retrieved 178 papers, of which 40 were considered eligible for intensive review. Most of the studies selected reported positive effects from OSTE activities. However, there was little quantitative data to gauge the impact of OSTEs on improvement of teaching skills., Conclusions: Considering that OSCEs have become a widely used tool for assessing medical students' and residents' clinical skills, it is high time to incorporate OSTEs for evaluating teaching skills in Brazil. Encouraging data to support implementation of this assessment tool in this country is available from abroad. The net benefit from this would possibly encompass medical students, residents and academic staff, through bringing awareness about the importance of excelling in teaching skills.

Introduction: The first computerised national ranking exam (cNRE) in Medicine was introduced in June 2016 for 8214 students. It was made of 18 progressive clinical cases (PCCs) with multiple choice questions (MCQs), 120 independent MCQs and 2 scientific articles to criticize. A lack of mark discrimination grounded the cNRE reform. We aimed to assess the discrimination of the final marks after this first cNRE., Methods: A national Excel ® file gathering overall statistics and marks were transmitted to the medical faculties after the cNRE. The mean points deviation between two papers and the percentage of points ranking 75% of students allowed us to analyse marks' discrimination., Results: The national distribution sigmoid curve of the marks is superimposable with previous NRE in 2015. In PCCs, 72% of students were ranked in 1090 points out of 7560 (14%). In independents MCQs, 73% of students were ranked in 434 points out of 2160 (20%). In critical analysis of articles, 75% of students were ranked in 225 points out of 1080 (21%). The above percentages of students are on the plateau of each discrimination curve for PCCs, independent MCQs and critical analysis of scientific articles., Conclusion: The cNRE reduced equally-ranked students compared to 2015, with a mean deviation between two papers of 0.28 in 2016 vs 0.04 in 2015. Despite the new format introduced by the cNRE, 75% of students are still ranked in a low proportion of points that is equivalent to previous NRE in 2015 (between 15 et 20% of points)., (Copyright © 2018 Société Nationale Française de Médecine Interne (SNFMI). Published by Elsevier Masson SAS. All rights reserved.)

Docimology has allowed the development of evaluative processes assuring valid, reliable and objective assessments. It was adopted within the faculty of Medicine of Tunis since  2007. The aim of this study was to analyze the docimological survey results of hematology-oncology exams, to evaluate the interest of this analysis in the elaboration of exams and the construction of an item bank, and propose some corrections  in order to improve assessment.     Methods :We have analyzed the hematology-oncology exams of SCMS1 (Second cycle of Medical Studies 1) from educational year 2008-2009 to 2013-2014. The data input was already done with Excel. The test includes 4 disciplines (Hematology, Oncology, Genetics and the Anatomic Pathology). We have calculated docimological parameters allowing global analysis, by discipline and by item. Results : A total of 3281 papers and 1004 questions were analyzed. The average success rate per year was 91,54% ± 7,12. The highest average success rate was found in hematology (80,51% ± 10,18). The lowest rate was found in the anatomic pathology (51,61% ± 23,76). The average rate of students succeeding the test without having average note in hematology was 5,36%. It was 42,29% in the anatomic pathology. Average difficulty index was 0,57 ± 0,05. Items analysis showed that 38,04% were easy and 19,02% were difficult. Average discrimination index was 0,25 ± 0,02. Discrimination was very good in 20,51% of items and good in 17,13%. Useless and bad discrimination items were about 40,53%. The average of Cronbach Alpha coefficient was 0,84 ± 0,03, showing a good internal-consistency. Conclusion :This study allowed an objective evaluation of "contributive disciplines" in multidisciplinary evaluation and showed the interest of integrating questions. Question analysis with teachers would be important to reevaluate and improve these items.

Background Excellence in clinical reasoning is one of the most important outcomes of medical education programs, but assessing learners' reasoning to inform corrective feedback is challenging and unstandardized. Methods The Society to Improve Diagnosis in Medicine formed a multi-specialty team of medical educators to develop the Assessment of Reasoning Tool (ART). This paper describes the tool development process. The tool was designed to facilitate clinical teachers' assessment of learners' oral presentation for competence in clinical reasoning and facilitate formative feedback. Reasoning frameworks (e.g. script theory), contemporary practice goals (e.g. high-value care [HVC]) and proposed error reduction strategies (e.g. metacognition) were used to guide the development of the tool. Results The ART is a behaviorally anchored, three-point scale assessing five domains of reasoning: (1) hypothesis-directed data gathering, (2) articulation of a problem representation, (3) formulation of a prioritized differential diagnosis, (4) diagnostic testing aligned with HVC principles and (5) metacognition. Instructional videos were created for faculty development for each domain, guided by principles of multimedia learning. Conclusions The ART is a theory-informed assessment tool that allows teachers to assess clinical reasoning and structure feedback conversations.

Background: Medical education is a dynamic process that will continuously evolve to respond to changes in the foundations of medicine, the clinical practice of medicine and in health systems science., Purpose: In this paper, we review how assessing learning in such a dynamic environment requires comprehensive flexible and adaptable methodological approaches designed to assess knowledge attainment and transfer, clinical skills/competency development, and ethical/professional behavior. Adaptive assessments should measure the learner's ability to observe where changes in health care delivery are needed and how to implement them. Balancing formative and summative assessments will promote reflective learning so that each student will reach her/his highest potential. From the programmatic perspective, measuring the design and delivery of instruction in relation to students? efforts to achieve competency will improve learning and foster continuous professional development of faculty and advance the science of learning., Approach: We describe how two medical schools are approaching adaptive assessment, including using portfolio systems that encompass teaching and learning experiences while offering real-time longitudinal tracking of digital data toward improving learning and provide curricula continuous improvement cycles. Using latest technologies, portfolios produce actionable data displays with precise guidance for learning and program development.

An essential component of expertise is a clinician's ability to adapt to uncertain, complex, or novel situations while maintaining their competence in routine situations. Adaptive expertise provides a framework for understanding and developing experts who have the skills to effectively balance and support these dimensions of work using both procedural and conceptual knowledge. It is important for educators to understand that often the training which fosters adaptive expertise does not require new tools or approaches, but rather a reconceptualization of training using many of the same instruction and assessment formats already available. The twelve tips discussed in this paper showcase ways in which education can be transformed to support the development of adaptive expertise including the significance of instruction that combines various forms for knowledge, the value of productive struggle, and shifting the design of assessments to support learning and performance beyond retention and direct application.

Easily accessible and interpretable performance data constitute critical feedback for learners that facilitate informed self-assessment and learning planning. To provide this feedback, there has been a proliferation of educational dashboards in recent years. An educational (learner) dashboard systematically delivers timely and continuous feedback on performance and can provide easily visualized and interpreted performance data. In this paper, we provide practical tips for developing a functional, user-friendly individual learner performance dashboard and literature review of dashboard development, assessment theory, and users' perspectives. Considering key design principles and maximizing current technological advances in data visualization techniques can increase dashboard utility and enhance the user experience. By bridging current technology with assessment strategies that support learning, educators can continue to improve the field of learning analytics and design of information management tools such as dashboards in support of improved learning outcomes.

Despite the variety of assessment tools available, multiple choice questions (MCQs) still play an integral part in examinations at both a national and speciality board level. MCQs have a number of methodological advantages yet their strength is related to the quality of the question posed. Specifically, there has been a move towards the MCQ testing a taxonomically higher order concept of integration-interpretation and problem solving. This paper focuses on question development and the potential pitfalls to avoid., (Copyright © 2017 European Society for Vascular Surgery. Published by Elsevier Ltd. All rights reserved.)

Competency-based medical education (CBME) is an approach to the design of educational systems or curricula that focuses on graduate abilities or competencies. It has been adopted in many jurisdictions, and in recent years an explosion of publications has examined its implementation and provided a critique of the approach. Assessment in a CBME context is often based on observations or judgments about an individual's level of expertise; it emphasizes frequent, direct observation of performance along with constructive and timely feedback to ensure that learners, including clinicians, have the expertise they need to perform entrusted tasks. This paper explores recent developments since the publication in 2010 of Holmboe and colleagues' description of CBME assessment. Seven themes regarding assessment that arose at the second invitational summit on CBME, held in 2013, are described: competency frameworks, the reconceptualization of validity, qualitative methods, milestones, feedback, assessment processes, and assessment across the medical education continuum. Medical educators interested in CBME, or assessment more generally, should consider the implications for their practice of the review of these emerging concepts.

Background Tablet computers have emerged as a tool to capture, process and store data in examinations, yet evidence relating to their acceptability and usefulness in assessment is limited. Methods We performed an observational study to explore opinions and attitudes relating to tablet computer use in recording performance in a final year objective structured clinical examination at a single UK medical school. Examiners completed a short questionnaire encompassing background, forced-choice and open questions. Forced choice questions were analysed using descriptive statistics and open questions by framework analysis. Results Ninety-two (97% response rate) examiners completed the questionnaire of whom 85% had previous use of tablet computers. Ninety per cent felt checklist mark allocation was 'very/quite easy', while approximately half considered recording 'free-type' comments was 'easy/very easy'. Greater overall efficiency of marking and resource savings were considered the main advantages of tablet computers, while concerns relating to technological failure and ability to record free type comments were raised. Discussion In a context where examiners were familiar with tablet computers, they were preferred to paper checklists, although concerns were raised. This study adds to the limited literature underpinning the use of electronic devices as acceptable tools in objective structured clinical examinations.

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

The aim of this study was to follow up a sample of physicians who began core medical training (CMT) in 2009. This paper examines the long-term validity of CMT and GP selection methods in predicting performance in the Membership of Royal College of Physicians (MRCP(UK)) examinations. We performed a longitudinal study, examining the extent to which the GP and CMT selection methods (T1) predict performance in the MRCP(UK) examinations (T2). A total of 2,569 applicants from 2008-09 who completed CMT and GP selection methods were included in the study. Looking at MRCP(UK) part 1, part 2 written and PACES scores, both CMT and GP selection methods show evidence of predictive validity for the outcome variables, and hierarchical regressions show the GP methods add significant value to the CMT selection process. CMT selection methods predict performance in important outcomes and have good evidence of validity; the GP methods may have an additional role alongside the CMT selection methods., (© Royal College of Physicians 2017. All rights reserved.)

Over the last two decades, Objective Structured Clinical Examination (OSCE) has become an increasingly important part of psychiatry education and assessment in the Australian context. A reappraisal of the evidence base regarding the use of OSCE in psychiatry is therefore timely. This paper reviews the literature regarding the use of OSCE as an assessment tool in both undergraduate and postgraduate psychiatry training settings. Suitable articles were identified using the search terms 'psychiatry AND OSCE' in the ERIC (educational) and PubMed (healthcare) databases and grouped according to their predominant focus: (1) the validity of OSCEs in psychiatry; (2) candidate preparation and other factors impacting on performance; and (3) special topics. The literature suggests that the OSCE has been widely adopted in psychiatry education, as a valid and reliable method of assessing psychiatric competencies that is acceptable to both learners and teachers alike. The limited evidence base regarding its validity for postgraduate psychiatry examinations suggests that more research is needed in this domain. Despite any shortcomings, OSCEs are currently ubiquitous in all areas of undergraduate and postgraduate medicine and proposing a better alternative for competency-based assessment is difficult. A critical question is whether OSCE is sufficient on its own to assess high-level consultancy skills, and aspects of professionalism and ethical practice, that are essential for effective specialist practice, or whether it needs to be supplemented by additional testing modalities.

Context: There is a growing body of research investigating assessor judgments in complex performance environments such as OSCE examinations. Post hoc analysis can be employed to identify some elements of "unwanted" assessor variance. However, the impact of individual, apparently "extreme" assessors on OSCE quality, assessment outcomes and pass/fail decisions has not been previously explored. This paper uses a range of "case studies" as examples to illustrate the impact that "extreme" examiners can have in OSCEs, and gives pragmatic suggestions to successfully alleviating problems., Method and Results: We used real OSCE assessment data from a number of examinations where at station level, a single examiner assesses student performance using a global grade and a key features checklist. Three exemplar case studies where initial post hoc analysis has indicated problematic individual assessor behavior are considered and discussed in detail, highlighting both the impact of individual examiner behavior and station design on subsequent judgments., Conclusions: In complex assessment environments, institutions have a duty to maximize the defensibility, quality and validity of the assessment process. A key element of this involves critical analysis, through a range of approaches, of assessor judgments. However, care must be taken when assuming that apparent aberrant examiner behavior is automatically just that.

Introduction: It is known that test-centered methods for setting standards in knowledge tests (e.g. Angoff or Ebel) are problematic, with expert judges not able to consistently predict the difficulty of individual items. A different approach is the Cohen method, which benchmarks the difficulty of the test based on the performance of the top candidates., Methods: This paper investigates the extent to which Ebel (and also Cohen) produces a consistent standard in a knowledge test when comparing between adjacent cohorts. The two tests are linked using common anchor items and Rasch analysis to put all items and all candidates on the same scale., Results: The two tests are of a similar standard, but the two cohorts are different in their average abilities. The Ebel method is entirely consistent across the two years, but the Cohen method looks less so, whilst the Rasch equating itself has complications - for example, with evidence of overall misfit to the Rasch model and change in difficulty for some anchor items., Conclusion: Based on our findings, we advocate a pluralistic and pragmatic approach to standard setting in such contexts, and recommend the use of multiple sources of information to inform the decision about the correct standard.

This paper takes on a list of the 10 most wanted test cheaters comparable with 'most wanted' lists used by law enforcement agencies to bring attention to the issue of test cheating in medical education. The research provides an overview of test cheating in modern medical education, presents a typology of common cheaters, and provides guidelines for the prevention and detection of cheating., (© 2016 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Objectives: Challenges in healthcare demand great leadership. In response, leadership training programs have been developed within academic medical centers, business schools, and healthcare organizations; however, we are unaware of any well-developed programs for physicians-in-training., Methods: To address this gap, we developed a two-day leadership development course for chief residents (CRs) at the Cleveland Clinic, framed around the concept of emotional intelligence. This paper describes our five-year experience with the CRs leadership program., Results: Since inception, 105 CRs took the course; 81 (77%) completed before-and-after evaluations. Participants indicated that they had relatively little prior knowledge of the concepts that were presented and that the workshop greatly enhanced their familiarity with leadership competencies. Qualitative analysis of open-ended responses indicated that attendees valued the training, especially in conflict resolution and teamwork, and indicated specific action plans for applying these skills. Furthermore, the workshop spurred some participants to express plans to learn more about leadership competencies., Conclusions: This study extends prior experience in offering an emotional intelligence-based leadership workshop for CRs. Though the program is novel, further research is needed to more fully understand the impact of leadership training for CRs and for the institutions and patients they serve., (© The Royal Australian and New Zealand College of Psychiatrists 2016.)

Background: Single-best answer (SBA) questions are widely used for assessment in medical schools; however, often clinical staff have neither the time nor the incentive to develop high-quality material for revision purposes. A student-led approach to producing formative SBA questions offers a potential solution., Methods: Cardiff University School of Medicine students created a bank of SBA questions through a previously described staged approach, involving student question-writing, peer-review and targeted senior clinician input. We arranged questions into discrete tests and posted these online. Student volunteer performance on these tests from the 2012/13 cohort of final-year medical students was recorded and compared with the performance of these students in medical school finals (knowledge and objective structured clinical examinations, OSCEs). In addition, we compared the performance of students that participated in question-writing groups with the performance of the rest of the cohort on the summative SBA assessment. Often clinical staff have neither the time nor the incentive to develop high-quality material for revision purposes, Results: Performance in the end-of-year summative clinical knowledge SBA paper correlated strongly with performance in the formative student-written SBA test (r = ~0.60, p <0.01). There was no significant correlation between summative OSCE scores and formative student-written SBA test scores. Students who wrote and reviewed questions scored higher than average in the end-of-year summative clinical knowledge SBA paper., Conclusion: Student-written SBAs predict performance in end-of-year SBA examinations, and therefore can provide a potentially valuable revision resource. There is potential for student-written questions to be incorporated into summative examinations., (© 2015 John Wiley & Sons Ltd.)

Introduction: The use of group work assessment in medical education is becoming increasingly important to assess the competency of collaborator. However, debate continues on whether this does justice to individual development and assessment. This paper focuses on assessing the individual component within group work., Method: An integrative literature review was conducted and complemented with a survey among representatives of all medical schools in the Netherlands to investigate current practices., Results: The 14 studies included in our review show that an individual component is mainly assessed by peer assessment of individual contributions. Process and product of group work were seldom used separately as criteria. The individual grade is most often based on a group grade and an algorithm to incorporate peer grades. The survey provides an overview of best practices and recommendations for implementing group work assessment., Discussion: The main pitfall when using peer assessment for group work assessment lies in differentiating between the group work process and the resulting product of the group work. Hence, clear criteria are needed to avoid measuring only effort. Decisions about how to weigh assessment of the product and peer assessment of individual contribution should be carefully made and based on predetermined learning goals.

Statement: Simulation-based medical education (SBME) offers opportunities for trainees to learn how to perform procedures and to be assessed in a safe environment. However, SBME research studies often lack robust evidence to support the validity of the interpretation of the results obtained from tools used to assess trainees' skills. The purpose of this paper is to describe how a validity framework can be applied when reporting and interpreting the results of a simulation-based assessment of skills related to performing procedures. The authors discuss various sources of validity evidence because they relate to SBME. A case study is presented.

Background: The primary goal of any assessment of students is to provide valid and reliable evaluations of students' knowledge and skills as well as provision of accurate feedback to students about their performance. Contrary to best practice guidelines for development of multiple choice questions (MCQs), however, items used within medical schools are often flawed. This disappoints students and discourages examiners from using in-house MCQ databases. Vetting and reviewing items can improve the quality of MCQs. In this paper, we describe our approach to standardize the format used for MCQ assessment and provide recommendations for quality enhancement of high-stakes assessment., Methods: A collaborative enhancement-focused vetting and review approach to development of high quality single best answer MCQs has been described., Results: Implementation of a collaborative strategy to blueprint, vet, review and standard set MCQ items for high stakes examinations can effectively contribute to assessment quality assurance. Similarly, shared responsibility for post examination analyses of items may reveal the psychometric properties of items in need of improvement and contribute to closure of the assessment outcomes feedback loop., Discussion: Devolving responsibility for implementation of assessment processes as an integral part of educational practices and values can maximize reliability and standards of assessment processes. We contend that while logistics and time constraints are of concern to busy faculty members, judicious utilization of resources to develop well-written MCQ items are well worth the effort to produce reliable and valid examinee scores. An enhancement-focused approach can be institutionally rewarding and lead to improved quality of high stakes assessments.

This paper presents perspectives and controversies surrounding the use of milestones to assess competency in outcomes-based medical education. Global perspectives (Canada, Europe, and the United States) and developments supporting their rationales are discussed. In Canada, there is a significant movement away from conceptualizing competency based on time, and a move toward demonstration of specific competencies. The success of this movement may require complex (rather than reductionist) milestones that reflect students' progression through complexity and context and a method to narrate their journey. European countries (United Kingdom, France, and Germany) have stressed the complexity associated with time and milestones for medical students to truly achieve competence. To meet the changing demands of medicine, they view time as actually providing students with knowledge and exposure to achieve various milestones. In the United States, milestones are based on sampling throughout professional development to initiate lifelong learning. However, the use of milestones may not imply overall competence (reductionism). Milestones must be developed alongside outcomes-based curriculum with use of faculty and competency committees. The perspectives outlined in this paper underscore emerging challenges for implementing outcomes-based medical education and call for new conceptualizations of competence.

Background: Biomedical Informatics (BMI) education in medical schools is developing a sound curricular base, but there are few published reports of their educational usefulness. The goal of this paper is to assess knowledge change and satisfaction in medical students after a BMI curriculum., Methods: The National Autonomous University of México Faculty of Medicine (UNAM) recently implemented a curricular reform that includes two BMI sequential courses (BMI-1 and BMI-2). The research design was one-group pretest-posttest. An objective test with evidence of validity was used for knowledge measurement. A satisfaction questionnaire was applied at the end of the courses. Two-tailed paired Student's t-tests were applied, comparing knowledge scores in the pre and post-test for each course., Results: The study included student cohorts during two consecutive academic years. The 2013 BMI-1 course (n = 986 students) knowledge pretest score was 43.0 ± 8.6 (mean percent correct ± SD), and the post-test score was 57.7 ± 10.3 (p < 0.001); the 2014 BMI-1 (n = 907) pretest score was 43.7 ± 8.5, and the post-test was 58.1 ± 10.5 (p < 0.001). The 2012 BMI-2 course (n = 683) pretest score was 26.3 ± 7.9, the post-test score was 44.3 ± 13.3 (p < 0.001); the 2013 BMI-2 (n = 926) pretest score was 27.5 ± 7.5, and the post-test was 42.0 ± 11.0 (p < 0.001). The overall opinion of the students regarding the course was from good to excellent, with a response rate higher than 90%. The satisfaction questionnaires had high reliability (Cronbach's alpha of 0.93)., Conclusions: The study shows a significant increase in BMI knowledge after an educational intervention in four medical student cohorts, and an overall positive evaluation by the students. Long-term follow-up is needed, as well as controlled studies of BMI educational interventions using performance endpoints.

Abstract The shift to using outcomes-based competency frameworks in medical education in many countries around the world requires educators to find ways to assess multiple competencies. Contemporary medical educators recognize that a competent trainee not only needs sound biomedical knowledge and technical skills, they also need to be able to communicate, collaborate and behave in a professional manner. This paper discusses methodological challenges of assessment with a particular focus on the CanMEDS Roles. The paper argues that the psychometric measures that have been the mainstay of assessment practices for the past half-century, while still valuable and necessary, are not sufficient for a competency-oriented assessment environment. New assessment approaches, particularly ones from the social sciences, are required to be able to assess non-Medical Expert (Intrinsic) roles that are situated and context-bound. Realist and ethnographic methods in particular afford ways to address the challenges of this new assessment. The paper considers the theoretical and practical bases for tools that can more effectively assess non-Medical Expert (Intrinsic) roles.

The 16th International Ottawa Conference/Canadian Conference on Medical Education (2014) featured a keynote deconstructing the rising discourse of competence-as-reflection in medical education. This paper, an elaborated version of the presentation, is an investigation into the theoretical roots of the diverse forms of reflective practice that are being employed by medical educators. It also raises questions about the degree to which any of these practices is compatible with assessment.

Background: Exams are essential components of medical students' knowledge and skill assessment during their clinical years of study. The paper provides a retrospective analysis of validity evidence for the internal medicine component of the written and clinical exams administered in 2012 and 2013 at King Abdulaziz University's Faculty of Medicine., Methods: Students' scores for the clinical and written exams were obtained. Four faculty members (two senior members and two junior members) were asked to rate the exam questions, including MCQs and OSCEs, for evidence of content validity using a rating scale of 1-5 for each item. Cronbach's alpha was used to measure the internal consistency reliability. Correlations were used to examine the associations between different forms of assessment and groups of students., Results: A total of 824 students completed the internal medicine course and took the exam. The numbers of rated questions were 320 and 46 for the MCQ and OSCE, respectively. Significant correlations were found between the MCQ section, the OSCE section, and the continuous assessment marks, which include 20 long-case presentations during the course; participation in daily rounds, clinical sessions and tutorials; the performance of simple procedures, such as IV cannulation and ABG extraction; and the student log book. Although the OSCE exam was reliable for the two groups that had taken the final clinical OSCE, the clinical long- and short-case exams were not reliable across the two groups that had taken the oral clinical exams. The correlation analysis showed a significant linear association between the raters with respect to evidence of content validity for both the MCQ and OSCE, r = .219 P < .001 and r = .678 P < .001, respectively, and r = .241 P < .001 and r = .368 P = .023 for the internal structure validity, respectively. Reliability measured using Cronbach's alpha was greater for assessments administered in 2013., Conclusion: The pattern of relationships between the MCQ and OSCE scores provides evidence of the validity of these measures for use in the evaluation of knowledge and clinical skills in internal medicine. The OSCE exam is more reliable than the short- and long-case clinical exams and requires less effort on the part of examiners and patients.

Context: Performance assessments, such as workplace-based assessments (WBAs), represent a crucial component of assessment strategy in medical education. Persistent concerns about rater variability in performance assessments have resulted in a new field of study focusing on the cognitive processes used by raters, or more inclusively, by assessors., Methods: An international group of researchers met regularly to share and critique key findings in assessor cognition research. Through iterative discussions, they identified the prevailing approaches to assessor cognition research and noted that each of them were based on nearly disparate theoretical frameworks and literatures. This paper aims to provide a conceptual review of the different perspectives used by researchers in this field using the specific example of WBA., Results: Three distinct, but not mutually exclusive, perspectives on the origins and possible solutions to variability in assessment judgements emerged from the discussions within the group of researchers: (i) the assessor as trainable: assessors vary because they do not apply assessment criteria correctly, use varied frames of reference and make unjustified inferences; (ii) the assessor as fallible: variations arise as a result of fundamental limitations in human cognition that mean assessors are readily and haphazardly influenced by their immediate context, and (iii) the assessor as meaningfully idiosyncratic: experts are capable of making sense of highly complex and nuanced scenarios through inference and contextual sensitivity, which suggests assessor differences may represent legitimate experience-based interpretations., Conclusions: Although each of the perspectives discussed in this paper advances our understanding of assessor cognition and its impact on WBA, every perspective has its limitations. Following a discussion of areas of concordance and discordance across the perspectives, we propose a coexistent view in which researchers and practitioners utilise aspects of all three perspectives with the goal of advancing assessment quality and ultimately improving patient care., (© 2014 John Wiley & Sons Ltd.)

Medical education relies heavily on assessment formats that require raters to assess the competence and skills of learners. Unfortunately, there are often inconsistencies and variability in the scores raters assign. To ensure the scores from these assessment tools have validity, it is important to understand the underlying cognitive processes that raters use when judging the abilities of their learners. The goal of this paper, therefore, is to contribute to a better understanding of the cognitive processes used by raters. Representative findings from the social judgment and decision making, cognitive psychology, and educational measurement literature will be used to enlighten the underpinnings of these rater-based assessments. Of particular interest is the impact judgments referred to as first impressions (or thin slices) have on rater-based assessments. These are judgments about people made very quickly and based on very little information. A narrative review will provide a synthesis of research in these three literatures (social judgment and decision making, educational psychology, and cognitive psychology) and will focus on the underlying cognitive processes, the accuracy and the impact of first impressions on rater-based assessments. The application of these findings to the types of rater-based assessments used in medical education will then be reviewed. Gaps in understanding will be identified and suggested directions for future research studies will be discussed.

In this paper we are challenging six traditional notions about assessment that are unhelpful when designing 'assessment for learning'-programmes for competency-based education. We are arguing for the following: Reductionism is not the only way to assure rigour in high-stakes assessment; holistic judgements can be equally rigorous. Combining results of assessment parts only because they are of the same format (like different stations in an OSCE) is often not defensible; instead there must be a logically justifiable combination. Numbers describe the quality of the assessment. Therefore, manipulating the numbers is usually not the best way to improve its quality. Not every assessment moment needs to be a decision moment, disconnecting both makes combining summative and formative functions of assessment easier. Standardisation is not the only route to equity. Especially with diverse student groups tailoring is more equitable than standardisation. The most important element to standardise is the quality of the process and not the process itself. Finally, most assessment is too much focussed on detecting deficiencies and not on valuing individual student differences. In competency-based education--especially with a focus on learner orientation--this 'deficiency-model' is not as well aligned as a 'differences-model'.

The utilization of competencies in medical education is relatively recent. In 1999 the United States Accreditation Council for Graduate Medical Education (ACGME) established six main competencies. Since then, the American Board of Psychiatry and Neurology have approved a specific list of competencies for their specialities in each of the ACGME's core competency areas. Assessment of competencies in both medical students and residents can be achieved through such methods as structured case discussion, direct observation, simulation, standardized patients, and 360-degree assessments, etc. Each assessment methodology has specific applications in the discipline of psychiatry. This paper reviews the different methods for assessing competencies with specific examples in psychiatric education. It is not intended as a comprehensive review of all assessment methods, but to provide examples and strategies to guide psychiatric educators in their practice. Students and residents were intentionally separated because there are differences in the teaching goals and objectives, and thus in the assessment purposes and design. Students are general, undifferentiated physicians-in-training who need to learn about psychiatric nosology, examinations, and treatment. Residents are mental health professionals who need more in-depth supervision in order to hone skills in all the specialized areas that arise in psychiatric practices, making supervision a vital part of residency programs.

Objectives: Remediation is usually offered to medical students and doctors in training who underperform on written or clinical examinations. However, there is uncertainty and conflicting evidence about the effectiveness of remediation. The aim of this systematic review was to synthesise the available evidence to clarify how and why remediation interventions may have worked in order to progress knowledge on this topic., Methods: The MEDLINE, EMBASE, CINAHL (Cumulative Index to Nursing and Allied Health Literature), ERIC (Educational Resources Information Centre), Web of Science and Scopus databases were searched for papers published from 1984 to April 2012, using the search terms 'remedial teaching', 'education', 'medical', 'undergraduate'/or 'clinical clerkship'/or 'internship and residency', 'at risk' and 'struggling'. Only studies that included an intervention, then provided retest data, and reported at least one outcome measure of satisfaction, knowledge, skills or effects on patients were eligible for inclusion. Studies of practising doctors were excluded. Data were abstracted independently in duplicate for all items. Coding differences were resolved through discussion., Results: Thirty-one of 2113 studies met the review criteria. Most studies were published after 2000 (n=24, of which 12 were published from 2009 onwards), targeted medical students (n=22) and were designed to improve performance on an immediately subsequent examination (n=22). Control or comparison groups, conceptual frameworks, adequate sample sizes and long-term follow-up measures were rare. In studies that included long-term follow-up, improvements were not sustained. Intervention designs tended to be highly complex, but their design or reporting did not enable the identification of the active components of the remedial process., Conclusions: Most remediation interventions in medical education focus on improving performance to pass a re-sit of an examination or assessment and provide no insight into what types of extra support work, or how much extra teaching is critical, in terms of developing learning. More recent studies are generally of better quality. Rigorous approaches to developing and evaluating remediation interventions are required., (© Blackwell Publishing Ltd 2013.)

Objectives: The objective was to critically appraise and highlight medical education research studies published in 2011 that were methodologically superior and whose outcomes were pertinent to teaching and education in emergency medicine (EM)., Methods: A search of the English language literature in 2011 querying PubMed, Scopus, Education Resources Information Center (ERIC), and PsychInfo identified EM studies that used hypothesis-testing or observational investigations of educational interventions. Six reviewers independently ranked all publications based on 10 criteria, including four related to methodology, that were chosen a priori to standardize evaluation by reviewers. This method was used previously to appraise medical education published in 2008, 2009, and 2010., Results: Forty-eight educational research papers were identified. Comparing the literature of 2011 to that of 2008 through 2010, the number of published educational research papers meeting the criteria increased over time from 30, to 36, to 41, and now to 48. Five medical education research studies met the a priori criteria for inclusion as exemplary and are reviewed and summarized in this article. The number of funded studies remained fairly stable over the past 3 years, at 13 (2008), 16 (2009), 9 (2010), and 13 (2011). As in past years, research involving the use of technology accounted for almost half (n = 22) of the publications. Observational study designs accounted for 28 of the papers, while nine studies featured an experimental design., Conclusions: Forty-eight EM educational studies published in 2011 and meeting the criteria were identified. This critical appraisal reviews and highlights five studies that met a priori quality indicators. Current trends and common methodologic pitfalls in the 2011 papers are noted., (© 2013 by the Society for Academic Emergency Medicine.)

Context:   Major changes in thinking about validity have occurred during the past century, shifting the focus in thinking from the validity of the test to the validity of test score interpretations. These changes have resulted from the 'new' thinking about validity in which construct validity has emerged as the central or unifying idea of validity today. Construct validity was introduced by Cronbach and Meehl in the mid-1950s in an attempt to address the validity of those many psychological concepts that have no clear referent in reality. To do this, construct validity theory required a nomological network--an elaborate theoretical network of constructs and observations connected by scientific laws--to validate the constructs. However, nomological networks are hard to come by and none that would do the job required by construct validity has been forthcoming to date. Thus, the current construct validity approach has retreated to one of simply 'interpretation and argument', but this seems to be too general to tie down the constructs in the way a nomological network would do to give credibility to the validity of the construct. As a result, the concept of validity seems to have been watered down and the credibility of validity claims weakened., Objectives:   The purpose of this paper is to encourage a discussion of the use of construct validity in medical education, and to suggest that test developers and users reconsider the use of abstract theoretical constructs that have no referent apart from theory., Methods:   We present a critical review of these concerns about construct validity and provide for contrast a brief overview of a recently proposed view of measurement based on scientific realism and causality analysis., (© Blackwell Publishing Ltd 2012.)

Context: Historically, assessments have often measured the measurable rather than the important. Over the last 30 years, however, we have witnessed a gradual shift of focus in medical education. We now attempt to teach and assess what matters most. In addition, the component parts of a competence must be marshalled together and integrated to deal with real workplace problems. Workplace-based assessment (WBA) is complex, and has relied on a number of recently developed methods and instruments, of which some involve checklists and others use judgements made on rating scales. Given that judgements are subjective, how can we optimise their validity and reliability?, Methods: This paper gleans psychometric data from a range of evaluations in order to highlight features of judgement-based assessments that are associated with better validity and reliability. It offers some issues for discussion and research around WBA. It refers to literature in a selective way. It does not purport to represent a systematic review, but it does attempt to offer some serious analyses of why some observations occur in studies of WBA and what we need to do about them., Results and Discussion: Four general principles emerge: the response scale should be aligned to the reality map of the judges; judgements rather than objective observations should be sought; the assessment should focus on competencies that are central to the activity observed, and the assessors who are best-placed to judge performance should be asked to participate., (© Blackwell Publishing Ltd 2012.)

An often reported challenge of evidence-based medicine concerns increasing use of clinical guidelines in practice. One of the proposed improvements is to promote guidelines by presenting them in conjunction with virtual patients. Three approaches to linking clinical guidelines to virtual patients are presented in this paper: (1) guidelines as a source for generating virtual patients; (2) guidelines hyper-flowchart as a virtual patient progress indicator; (3) guidelines flowchart reconstruction as a learning activity in virtual patient systems. The scenarios have been preliminarily evaluated using two demonstrator applications: Bit Pathways and CASUS. Challenges and direction for further development are proposed.

Introduction: Audience response systems (ARS) are increasingly being used to heighten participants' involvement. Knowledge of technical and pedagogical challenges is, however, limited. The purpose of this paper is to evaluate ARS as a tool for 1) evaluation, 2) knowledge testing, 3) attention raising and 4) discussion stimulation., Material and Methods: ARS was used 33 times at four different courses. Data include voting results, observations, questionnaires and interviews., Results: A total of 215 participants and 12 teachers were included. The majority of the participants found ARS suitable for course evaluation. The teachers found it useful for obtaining the results immediately and thereby for receiving feedback on their own teaching. The participants and the teachers found ARS suitable for knowledge testing. ARS was used as an instrument to increase activity and attention. The system was found to increase the level of concentration and the interactivity. ARS was used to initiate discussions. The participants found that the questions could be a good starting point for discussion. The teachers found it challenging to comment on answers. Our experiences are that thorough planning and preparation is needed for the successful implementation of ARS., Conclusion: Our experiences indicate that ARS is suitable for course evaluation. Overall, we find ARS a valuable technology that may stimulate discussion and support learning, but teachers need to be technically and pedagogically well prepared to use the tool. The use of ARS does not in itself entail that the quality of the teaching increases.