Search

Abstract Background Systematic reviews on simulation training effectiveness have pointed to the need to adhere to evidence-based instructional design (ID) guidelines. ID guidelines derive from sound cognitive theories and aim to optimize complex learning (integration of knowledge, skills, and attitudes) and learning transfer (application of acquired knowledge and skills in the workplace). The purpose of this study was to explore adherence to ID guidelines in simulation training programs for dealing with postpartum hemorrhage (PPH), a high-risk situation and the leading cause of maternal mortality worldwide. Methods A total of 40 raters analyzed simulation training programs as described in 32 articles. The articles were divided into four subsets of seven articles and one subset of four articles. Each subset was judged by seven to ten raters on adherence to ID guidelines. The 5-point Likert score rating scale was based on Merrill’s First Principles of Instruction and included items relating to key ID features categorized into five subscales: authenticity, activation of prior knowledge, demonstration, application, and integration/transfer. The authors searched for articles published in English between January 2007 and March 2017 in PubMed, Eric, and Google Scholar and calculated the mean Likert-scale score, per subscale, and interrater reliability (IRR). Results The mean Likert-scale scores calculated for all subscales were < 3.00. For the number of raters used to judge the papers in this study (varying between 7 and 10), the IRR was found to be excellent for the authenticity and integration/transfer subscales, good-to-excellent for the activation of prior knowledge and application subscales, and fair-to-good for the demonstration subscale. Conclusion The results demonstrate a paucity of the description of adherence to evidence-based ID guidelines in current simulation trainings for a high-risk situation such as PPH.

Handover between colleagues is a complex task. The problem is that handovers are often inadequate because they are not structured according to theoretically grounded guidelines. Based on the cognitive load theory, we suggest that allowing a clarifying dialogue and thereby optimizing germane cognitive load enhances the information quality and diagnostic accuracy at handover, but may prolong handover duration. We also expect that mentioning key information first and thus decreasing intrinsic cognitive load improves information quality and diagnostic accuracy. We developed two representative paediatric cases for presentation in a factorial 2 × 2 design. Sixth-year medical students (N = 80) were randomly assigned to one of four groups that differed with regard to how the case histories were delivered to them (chronological order versus key information mentioned first) and direction of information exchange (unidirectional versus a clarifying dialogue). The receivers of the handover were asked to write a report of the cases and suggest the best diagnosis. Dependent variables were information quality of the written report (Information score), quality of the diagnosis (Diagnostic accuracy score) and the time it took to deliver the written handover case report (Handover report duration). Seen through the lens of cognitive load theory, allowing a clarifying dialogue at handover, and thus optimizing the germane cognitive load, significantly increased the Information score (p < 0.0005), Diagnostic accuracy score (< 0.05) and Handover report duration (p < 0.001).

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments--rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically "the locus of regulation of learning" (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use--and, consequently, the effectiveness--of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Abstract Background Mentoring plays a pivotal role in workplace-based learning, especially in the medical realm. Organising a formal mentoring programme can be labor and time intensive and generally impractical in resource constrained medical schools with limited numbers of mentors. Hence, informal mentoring offers a valuable alternative, but will be more likely to be effective when mentors and protégés share similar views. It is therefore important to gain more insight into factors influencing perceptions of informal mentoring. This study aims to explore mentors and protégés’ perceptions of informal mentoring and how these vary (or not) with gender, age and the duration of the relationship. Method We administered an Informal Mentor Role Instrument (IMRI) to medical practitioners and academics from Egypt, Pakistan and Saudi Arabia. The questionnaire was developed for the study from other validated instruments. It contained 39 items grouped into 7 domains: acceptance, counselling, friendship, parenting, psychological support, role modelling and sociability. Results A total of 103 mentors and 91 protégés completed the IMRI. Mentors had a better appreciation for the interpersonal aspects of informal mentoring than protégés, especially regarding acceptance, counselling and friendship. Moreover, being older and engaged in a longer mentoring relationship contributed to more positive perceptions of interpersonal aspects of mentoring, regardless of one’s role (mentor or protégé). Conclusion It can be concluded that the expectations of mentors and protégés differed regarding the content and aim of the interpersonal characteristics of their mentoring relationship. We recommend mentors and protégés to more explicitly exchange their expectations of the informal mentoring relationship, as typically practiced in formal mentoring. Additionally, in our study, seniority and lasting relationships seem crucial for good informal mentoring. It appears beneficial to foster lasting informal mentoring relationships and to give more guidance to younger mentors.

Abstract Background Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a ‘backward chaining’ approach to selection, based on the competences of a ‘good doctor’. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. Methods We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. Results Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p

Abstract Background In postgraduate training, there is a need to continuously assess the learning and working conditions to optimize learning. Students or trainees respond to the learning climate as they perceive it. The Dutch Residency Educational Climate Test (D-RECT) is a learning climate measurement tool with well-substantiated validity. However, it was originally designed for Dutch postgraduate trainees and it remains to be shown whether extrapolation to non-Western settings is viable. The dual objective of this study was to revalidate D-RECT outside of a Western setting and to evaluate the factor structure of a recently revised version of the D-RECT containing 35 items. Methods We invited Filipino internal medicine residents from 96 hospitals to complete the revised 35-item D-RECT. Subsequently, we performed a confirmatory factor analysis to check the fit of the 9 scale model of the revised 35-item D-RECT. Inter-rater reliability was assessed using generalizability theory. Results Confirmatory factor analysis unveiled that the factor structure of the revised 35-item D-RECT provided a reasonable fit to the Filipino data, after removal of 7 items. Five to seven evaluations of individual residents were needed per scale to obtain a reliable result. Conclusion Even in a non-Western setting, the D-RECT exhibited psychometric validity. This study validated the factor structure of the revised 35-item D-RECT after some modifications. We recommend that its application be extended to other Asian countries and specialties.

Formula scoring (FS) is the use of a don't know option (DKO) with subtraction of points for wrong answers. Its effect on construct validity and reliability of progress test scores, is subject of discussion. Choosing a DKO may not only be affected by knowledge level, but also by risk taking tendency, and may thus introduce construct-irrelevant variance into the knowledge measurement. On the other hand, FS may result in more reliable test scores. To evaluate the impact of FS on construct validity and reliability of progress test scores, a progress test for radiology residents was divided into two tests of 100 parallel items (A and B). Each test had a FS and a number-right (NR) version, A-FS, B-FS, A-NR, and B-NR. Participants (337) were randomly divided into two groups. One group took test A-FS followed by B-NR, and the second group test B-FS followed by A-NR. Evidence for impaired construct validity was sought in a hierarchical regression analysis by investigating how much of the participants' FS-score variance was explained by the DKO-score, compared to the contribution of the knowledge level (NR-score), while controlling for Group, Gender, and Training length. Cronbach's alpha was used to estimate NR and FS-score reliability per year group. NR score was found to explain 27 % of the variance of FS [F(1,332) = 219.2, p < 0.0005], DKO-score, and the interaction of DKO and Gender were found to explain 8 % [F(2,330) = 41.5, p < 0.0005], and the interaction of DKO and NR 1.6 % [F(1,329) = 16.6, p < 0.0005], supporting our hypothesis that FS introduces construct-irrelevant variance into the knowledge measurement. However, NR-scores showed considerably lower reliabilities than FS-scores (mean year-test group Cronbach's alphas were 0.62 and 0.74, respectively). Decisions about FS with progress tests should be a careful trade-off between systematic and random measurement error.

Abstract Background Items must be relevant to ensure item quality and test validity. Since “item relevance” has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement. Methods Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group. Results The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p 0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p

Clinical clerks learn more than they are taught and not all they learn can be measured. As a result, curriculum leaders evaluate clinical educational environments. The quantitative Dundee Ready Environment Measure (DREEM) is a "de facto" standard for that purpose. Its 50 items and 5 subscales were developed by consensus. Reasoning that an instrument would perform best if it were underpinned by a clearly conceptualized link between environment and learning as well as psychometric evidence, we developed the mixed methods Manchester Clinical Placement Index (MCPI), eliminated redundant items, and published validity evidence for its 8 item and 2 subscale structure. Here, we set out to compare MCPI with DREEM. 104 students on full-time clinical placements completed both measures three times during a single academic year. There was good agreement and at least as good discrimination between placements with the smaller MCPI. Total MCPI scores and the mean score of its 5-item learning environment subscale allowed ten raters to distinguish between the quality of educational environments. Twenty raters were needed for the 3-item MCPI training subscale and the DREEM scale and its subscales. MCPI compares favourably with DREEM in that one-sixth the number of items perform at least as well psychometrically, it provides formative free text data, and it is founded on the widely shared assumption that communities of practice make good learning environments.

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

The drive to quality-manage medical education has created a need for valid measurement instruments. Validity evidence includes the theoretical and contextual origin of items, choice of response processes, internal structure, and interrelationship of a measure's variables. This research set out to explore the validity and potential utility of an 11-item measurement instrument, whose theoretical and empirical origins were in an Experience Based Learning model of how medical students learn in communities of practice (COPs), and whose contextual origins were in a community-oriented, horizontally integrated, undergraduate medical programme. The objectives were to examine the psychometric properties of the scale in both hospital and community COPs and provide validity evidence to support using it to measure the quality of placements. The instrument was administered twice to students learning in both hospital and community placements and analysed using exploratory factor analysis and a generalizability analysis. 754 of a possible 902 questionnaires were returned (84% response rate), representing 168 placements. Eight items loaded onto two factors, which accounted for 78% of variance in the hospital data and 82% of variance in the community data. One factor was the placement learning environment, whose five constituent items were how learners were received at the start of the placement, people's supportiveness, and the quality of organisation, leadership, and facilities. The other factor represented the quality of training--instruction in skills, observing students performing skills, and providing students with feedback. Alpha coefficients ranged between 0.89 and 0.93 and there were no redundant or ambiguous items. Generalisability analysis showed that between 7 and 11 raters would be needed to achieve acceptable reliability. There is validity evidence to support using the simple 8-item, mixed methods Manchester Clinical Placement Index to measure key conditions for undergraduate medical students' experience based learning: the quality of the learning environment and the training provided within it. Its conceptual orientation is towards Communities of Practice, which is a dominant contemporary theory in undergraduate medical education.

Although other web-based approaches to assessment of professional behaviour have been studied, no publications studying the potential advantages of a web-based instrument versus a classic, paper-based method have been published to date. This study has two research goals: it focuses on the quantity and quality of comments provided by students and their peers (two researchers independently scoring comments as correct and incorrect in relation to five commonly used feedback rules (and resulting in an aggregated score of the five scores) on the one, and on the feasibility, acceptability and perceived usefulness of the two approaches on the other hand (using a survey). The amount of feedback was significantly higher in the web-based group than in the paper based group for all three categories (dealing with work, others and oneself). Regarding the quality of feedback, the aggregated score for each of the three categories was not significantly different between the two groups, neither for the interim, nor for the final assessment. Some, not statistically significant, but nevertheless noteworthy trends were nevertheless noted. Feedback in the web-based group was more often unrelated to observed behaviour for several categories for both the interim and final assessment. Furthermore, most comments relating to the category "Dealing with oneself" consisted of descriptions of a student's attendance, thereby neglecting other aspects of personal functioning. The survey identified significant differences between the groups for all questionnaire items regarding feasibility, acceptability and perceived usefulness in favour of the paper-based form. The use of a web-based instrument for professional behaviour assessment yielded a significantly higher number of comments compared to the traditional paper-based assessment. Unfortunately, the quality of the feedback obtained by the web-based instrument as measured by several generally accepted feedback criteria did not parallel this increase. (Contains 5 tables and 1 figure.)

Little is known about the influence of language background in problem-based learning (PBL) tutorial groups on group processes and students' academic achievement. This study investigated the relationship between language background, secondary school score, tutorial group processes, and students' academic achievement in PBL. A validated tutorial group effectiveness questionnaire was administered to undergraduate medical students in a PBL curriculum at the Nelson R. Mandela School of Medicine (NRMSM) in South Africa. Although 58 percent of the students did not speak English as their first language, the tutorials were in English. Furthermore, secondary school scores differed strongly due to inadequate resources between secondary schools. A path analysis was conducted to test a causal model in which the two independent variables were English as the First Language (EFL) and secondary school scores. These variables were assumed to influence the process variables (cognitive, motivational, and demotivational group processes). Input and process variables were assumed to influence the two output variables, being overall group productivity and students' academic achievement. All data were analyzed at the individual student level (N = 387). A very good model fit was found (CMIN/DF = 0.68, GFI = 1.00, TLI = 1.02, CFI = 1.00, RMSEA = 0.00). EFL and secondary school scores positively affected students' academic achievement (respectively beta = 0.24 and beta = 0.16). EFL negatively affected motivational group processes (beta = -0.22). Cognitive group processes positively influenced overall group productivity (beta = 0.31) and so did motivational group processes (beta = 0.27). Demotivational group processes negatively predicted academic achievement and overall group productivity (beta = -0.15, and beta = -0.25). The model resulted in an R-square of 0.15 and 0.45 for academic achievement and overall group productivity, respectively. EFL and secondary school scores had a positive effect on academic achievement. Cognitive and motivational group processes had a positive effect on overall group productivity, while EFL negatively impacted motivational group processes. We recommend English language development courses to be formally included into curricula to enhance student learning.

Conditions such as muscular dystrophies (MDs) that affect both cardiac and skeletal muscles would benefit from therapeutic strategies that enable regeneration of both of these striated muscle types. Protocols have been developed to promote induced pluripotent stem cells (iPSCs) to differentiate toward cardiac or skeletal muscle; however, there are currently no strategies to simultaneously target both muscle types. Tissues exhibit specific epigenetic alterations; therefore, source-related lineage biases have the potential to improve iPSC-driven multilineage differentiation. Here, we determined that differential myogenic propensity influences the commitment of isogenic iPSCs and a specifically isolated pool of mesodermal iPSC-derived progenitors (MiPs) toward the striated muscle lineages. Differential myogenic propensity did not influence pluripotency, but did selectively enhance chimerism of MiP-derived tissue in both fetal and adult skeletal muscle. When injected into dystrophic mice, MiPs engrafted and repaired both skeletal and cardiac muscle, reducing functional defects. Similarly, engraftment into dystrophic mice of canine MiPs from dystrophic dogs that had undergone TALEN-mediated correction of the MD-associated mutation also resulted in functional striatal muscle regeneration. Moreover, human MiPs exhibited the same capacity for the dual differentiation observed in murine and canine MiPs. The findings of this study suggest that MiPs should be further explored for combined therapy of cardiac and skeletal muscles., Introduction Induced pluripotent stem cells (iPSCs) represent a promising contribution to regenerative medicine (1). Despite the regulatory hurdles and safety issues involved, reprogramming patients' cells into iPSCs for autologous cell [...]

In the complex practice of today's blended learning, educators need to be able to evaluate both online and face-to-face communication in order to get the full picture of what is going on in blended learning scenarios. The aim of this study was to investigate the reliability and feasibility of a practical instrument for analysing face-to-face communication that is in alignment with tools for analysing online communication. We examined whether an existing coding scheme for online communication could be modified to evaluate face-to-face communication. We investigated the reliability, validity and usability of a modified version of the Transcript Analysis Tool (TAT) in analysing communication among students during small-group sessions comprising presentations by students and discussion. The modified TAT proved to be effective in obtaining a detailed description of students' participation in the communication and of the focus of the communication. Evaluation was more efficient when direct video analysis was used than when a transcript of the same video was analysed. Direct analysis took about 2 h for 20 min of video recording and had an inter-rater reliability of [kappa] = 0.56. The modified TAT appears to be an effective and efficient instrument to analyse face-to-face communication in small group sessions. Its combined use with an evaluation instrument for online learning may create a powerful evaluation instrument for blended learning. Further studies are needed to establish the value of such an instrument. (Contains 6 tables and 3 figures.)

In this paper, staff members' perceptions about the organisational culture are measured. The questions addressed are: what are their opinions about the current and preferred organisational culture? Are there differences between the current and preferred situation? Do the perceptions differ per department? The Organisational Culture Assessment Instrument was sent to staff involved in 18 different departments within Dutch universities of applied science. The results demonstrated that both a flexibility- and control-oriented culture were moderately experienced in practice. In addition, significant differences were found between the current and preferred situation as well as a significant variation over departments for the current situation. Staff members in general, prefer a flexibility-oriented culture to a control-oriented culture. These findings imply that the organisational culture in many departments is not yet fully in line with the staff members' preferences. (Contains 2 tables and 3 figures.)

In a previous study, we established that compared to a written case, a video case enhances observable cognitive processes in the verbal interaction in a postgraduate problem-based learning format. In a new study we examined non-observable cognitive processes using a stimulated recall procedure alongside a reanalysis of the data from the first study. We examined the development of shared cognition as reflected in collaborative concept link formation, an approach to connecting a series of concepts related to a particular topic. Eleven paediatric residents were randomly allocated to two groups. After both analysing the same written case vignette, one group watched a video of the case in the vignette and the other group read a written description of the video. Both groups then reanalysed the vignette. After the group sessions, time-logged transcripts were made of the verbal interaction in both groups and all residents individually took part in a stimulated recall procedure. Causal reasoning concept links were labelled as individual or collaborative depending on whether they originated from individual residents or were directly elicited by verbal utterances from others. The video led to a significantly increased frequency ratio (after intervention: before intervention) of collaborative concept links but did not affect the frequency of individual concept links. This novel process approach to chronological registration of concept link formation offered additional evidence that shared cognition by means of co-elaboration of concept formation is stimulated by the use of patient video recordings in small group learning.

Progress testing provides data on the growth of students' knowledge over the course of the curriculum obtained from the results of all students in the curriculum on periodical similar tests pitched at end-of-curriculum level. Since 2001, three medical schools have jointly constructed and administered four progress tests annually. All students in the 6-year undergraduate curricula of these schools take the same tests resulting in 24 distinct measurements per academic year (four tests for six student year groups), which may be used to compare performance between and within schools. Because single point measurements had proven unreliable, we devised a method to use cumulative information to compare schools' test performance. This cumulative deviation method involves calculation of the deviations of schools' scores from the cross-institutional average score for 24 measurement moments in 1 year. The current study shows that it appears to be feasible to use a combination of the cumulative deviation method and trend analysis for subdomains of medical knowledge to detect strengths and weaknesses in knowledge development in medical curricula. We illustrate the method by applying it to data from 16 consecutive progress tests administered to all students (4,300) of three medical schools in the academic years 2001/2002 through 2004/2005.

Background: To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges' expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges' item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges. Method: Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion. Results and conclusions: This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the "minimally competent student", but who would also be capable of answering all the items.

Sequential testing is applied to reduce costs in SP-based tests (OSCEs). Initially, all candidates take a screening test consisting of a part of the OSCE. Candidates who fail the screen sit the complete test, whereas those who pass the screen are qualified as a pass of the complete test. The procedure may result in a reduction of testing resources, but at the cost of false positives (candidates who pass the screen but would fail the complete test). In a diagnostic test an optimum cutpoint is obtained by minimizing the weighted sum of false negatives and false positives using Receiver Operator Characteristic (ROC) analysis. However, in the sequential procedure there are no false negatives, because the result of the complete test is considered the final outcome. Several authors ignored this difference and applied classic ROC analysis to a sequential test. Earlier, we doubted the validity of the procedure and proposed to set the cutpoint by minimizing the loss, defined as the weighted sum of the screen negatives and the false positives. Recently, Regehr and Colliver (2003) argued that under certain theoretically derived conditions the use of the loss formula in sequential testing is functionally identical to using classic ROC analysis. The current study shows (a) the conclusion of Regehr and Colliver is based on assumptions that can be challenged, and (b) ROC analysis indeed can be used in sequential testing, but only if the procedure is modified according to the results of a corresponding loss function analysis.

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

Background: One of the methods used for standard setting is the borderline regression method (BRM). This study aims to assess the reliability of BRM when the pass-fail standard in an objective structured clinical examination (OSCE) was calculated by averaging the BRM standards obtained for each station separately. Materials and Methods: In nine stations of the OSCE with direct observation the examiners gave each student a checklist score and a global score. Using a linear regression model for each station, we calculated the checklist score cut-off on the regression equation for the global scale cut-off set at 2. The OSCE pass-fail standard was defined as the average of all station′s standard. To determine the reliability, the root mean square error (RMSE) was calculated. The R2 coefficient and the inter-grade discrimination were calculated to assess the quality of OSCE. Results: The mean total test score was 60.78. The OSCE pass-fail standard and its RMSE were 47.37 and 0.55, respectively. The R2 coefficients ranged from 0.44 to 0.79. The inter-grade discrimination score varied greatly among stations. Conclusion: The RMSE of the standard was very small indicating that BRM is a reliable method of setting standard for OSCE, which has the advantage of providing data for quality assurance.

Developed a method to estimate the cutscore precisions for empirical standard-setting methods by using resampling. Illustrated the method with two actual datasets consisting of 86 Dutch medical residents and 155 Canadian medical students taking objective structured clinical examinations. Results show the applicability of the method. (SLD)

Compared four standard-setting procedures for an objective structure clinical examination (OSCE) in medical education. Applied Angoff, borderline, relative, and holistic procedures to the data used to establish a cutoff score for a pass/fail decision. The Angoff and borderline procedures gave similar results; however, the relative and holistic methods gave widely divergent results. (Author/DB)

Sequential, standardized patient-based tests were given to medical students at the University of Maastricht (Netherlands); the first test was a screening test that evaluated the efficiency/validity of cutoff scores. Findings indicated that stringent pass/fail cutoff scores on the screening test produced optimal results. Fewer than 0.2 percent of students who passed the screening test failed the complete test. (DB)

General practitioners (GPs) are increasingly called upon to identify patients at risk for hereditary cancers, and their genetic competencies need to be enhanced. This article gives an overview of a research project on how to build effective educational modules on genetics, assessed by randomized controlled trials (RCTs), reflecting the prioritized educational needs of primary care physicians. It also reports on an ongoing study to investigate long-term increase in genetic consultation skills (1-year follow-up) and interest in and satisfaction with a supportive website on genetics among GPs. Three oncogenetics modules were developed: an online Continuing Professional Development (G-eCPD) module, a live genetic CPD module, and a "GP and genetics" website (huisartsengenetica.nl) providing further genetics information applicable in daily practice. Three assessments to evaluate the effectiveness (1-year follow-up) of the oncogenetic modules were designed: 1.An online questionnaire on self-reported genetic competencies and changes in referral behaviour, 2.Referral rates from GPs to clinical genetics centres and 3.Satisfaction questionnaire and visitor count analytics of supportive genetics website. The setting was Primary care in the Netherlands and three groups of study participants were included in the reported studies:. Assessment 1. 168 GPs responded to an email invitation and were randomly assigned to an intervention or control group, evaluating the G-eCPD module (n = 80) or the live module (n = 88). Assessment 2. Referral rates by GPs were requested from the clinical genetics centres, in the northern and southern parts of the Netherlands (Amsterdam and Maastricht), for the two years before (2010 [n = 2510] and 2011 [n = 2940]) and the year after (2012 [n = 2875]) launch of the oncogenetics CPD modules and the website. Assessment 3. Participants of the website evaluation were all recruited online. When they visited the website during the month of February 2013, a pop-up invitation came up. Of the 1350 unique visitors that month, only 38 completed the online questionnaire. Main outcomes measure showed long-term (self-reported) genetic consultation skills (i.e. increased genetics awareness and referrals to clinical genetics centres) among GPs who participated in the oncogenetic training course, and interest in and satisfaction with the supportive website. 42 GPs (52%) who previously participated in the G-eCPD evaluation study and 50 GPs (57%) who participated in the live training programme responded to the online questionnaire on long-term effects of educational outcome. Previous RCTs showed that the genetics CPD modules achieved sustained improvement of oncogenetic knowledge and consultation skills (3-months follow-up). Participants of these RCTs reported being more aware of genetic problems long term; this was reported by 29 GPs (69%) and 46 GPs (92%) participating in the G-eCPD and live module evaluation studies, respectively (Chisquare test, p