Search

There is an expectation that health professions schools respond to priority societal health needs. This expectation is largely based on the underlying assumption that schools are aware of the priority needs in their communities. This paper demonstrates how open-access, pan-national health data can be used to create a reliable health index to assist schools in identifying societal needs and advance social accountability in health professions education. Using open-access data, a psychometric evaluation was conducted to examine the reliability and validity of the Canadian Health Indicators Framework (CHIF) conceptual model. A non-linear confirmatory factor analysis (CFA) on 67 health indicators, at the health-region level (n = 97) was used to assess the model fit of the hypothesized 10-factor model. Reliability analysis using McDonald's Omega were conducted, followed by Pearson's correlation coefficients. Findings from the non-linear CFA rejected the original conceptual model structure of the CHIF. Exploratory post hoc analyses were conducted using modification indices and parameter constraints to improve model fit. A final 5-factor multidimensional model demonstrated superior fit, reducing the number of indicators from 67 to 32. The 5-factors included: Health Conditions (8-indicators); Health Functions (6-indicators); Deaths (5-indicators); Non-Medical Health Determinants (7-indicators); and Community & Health System Characteristics (6-indicators). All factor loadings were statistically significant (p < 0.001) and demonstrated excellent internal consistency ([omega]>0.95). Many schools struggle to identify and measure socially accountable outcomes. The process highlighted in this paper and the indices developed serve as starting points to allow schools to leverage open-access data as an initial step in identifying societal needs.

One aim of vocational education is educating students for occupations by fostering the development of students' capacities to become successful practitioners. During their education students are usually afforded work experience. When this is the case, students learn both at school and at the workplace. Learning at the workplace requires assessment, but this differs from assessment at school because of where (at the workplace), how (through participation) and what students learn (a process of belonging, becoming and being). At the workplace, students are usually assigned an educator who takes on the dual role of educator and assessor. This review takes a sociocultural perspective on learning at the workplace and from this perspective brings together what is already known about how workplace educators assess student performance through a qualitative systematic review. Our analysis aimed for narrative synthesis using iterative thematic analysis. The results depict workplace assessment as manifested in day-to-day work and shaped by relationships. Workplace educators are engaged in a continuous process of assessment-related interactions. They prefer using criteria that are embedded in the norms and values of their vocational community, rather than criteria prescribed by school. Workplace assessment requires negotiated criteria and truly collaborative assessment practices. These practices can be purposefully co-designed and require close communication between school and work. This review shows that assessment of workplace performance in vocational education can be conceptualised as a practice that is shaped by the specific workplace in which it is embedded. From this perspective assessment can be explicated and acknowledged, and as a consequence be further conceptualised and researched in both assessment research and vocational education research.

The assessment of workplace learning by educators at the workplace is a complex and inherently social process, as the workplace is a participatory learning environment. We therefore propose seeing assessment as a process of judgment embedded in a community of practice and to this purpose use the philosophy of inferentialism to unravel the judgment process of workplace educators by seeing it as an interrelated system of judgments, actions and reasons. Focussing on the unfolding of a process, we applied a longitudinal holistic case study design. Results show that educators are engaged in a constant judgment process during which they use multiple and adaptive frames of reference when forming their judgment about students. They construct an overarching image of students that develops throughout the placement, and their judgments about students go hand in hand with their actions relating to fostering independent practice.

Purpose To create medical school service regions and examine national in-region graduate retention patterns across the medical education continuum and into professional practice as one approach to advancing social accountability in medical education.Methods Medical school service regions were created in Canada using publicly available data and mapped using Geographic Information System (GIS) software. Population size and density for each service region were calculated using census data. Retrospective data of medical graduates who completed their medical degrees between 2001–2015 (n = 19,971) were obtained from a centralized data repository and used to analyze in-region retention rates by medical specialty across the training continuum and five years into professional practice.Results Spatial inequities were observed across medical school service regions. Graduate retention patterns also varied across service region groups and medical specialties. Quebec (86.5%) and Ontario (80.4%) had above-average retention rates across the medical education continuum. Family medicine had the highest retention rates from undergraduate to postgraduate training (81.9%), while psychiatry had the highest retention rate across the training continuum and into professional practice (71.2%). The Alberta and British Columbia service region group demonstrated high retention rates across the training continuum and into professional practice and medical specialties, except for retention from undergraduate to postgraduate medical education.Conclusion This study highlights the importance of considering both medical specialty and practice location of graduates when planning and retaining the physician workforce. The observed retention patterns among graduates are a critical aspect of addressing societal needs and represent an intermediate step towards achieving health equity. Furthermore, graduate retention patterns serve as an outcome measure for schools to demonstrate their commitment to social accountability. Tracking and monitoring graduate outcomes may lead schools to actively collaborate with government agencies responsible for healthcare policy, which may ultimately improve physician workforce planning and promote more equitable healthcare access.

Absolute ability/achievement does not explain gender differences in educational trajectories, but the role of comparative advantage (i.e., being better in one subject compared to another) has received much less attention. To study this, longitudinal data collected among 1,352 individuals (age 15-16) in upper secondary education in the Netherlands are used. Multinomial path analyses showed that compared to girls, boys are on average 15% more likely to enter the most male-typical trajectory and 16% less likely to enter the most female-typical trajectory. Although having a comparative advantage affects trajectory choices, it does not explain gender differences in trajectory choices in secondary education.

Objectives:. Sirolimus is one of the first oral drugs for safe and effective treatment of patients with vascular malformations, and may save (expensive) interventions. Therefore the objective was to investigate the cost-effectiveness of sirolimus in vascular malformations. Methods:. This cost-effectiveness analysis from a medical and societal perspective was performed alongside a clinical phase IIB effectiveness of sirolimus in patients with vascular malformations trial. Utility and quality-adjusted life years (QALY) were calculated using the short-form six-dimension in adults and pediatric quality of life inventory scores to express effectiveness in children. Total costs included medical and productivity costs. The net monetary benefit was calculated, given a range of willingness to pay (WTP) values within societies with nationalized health care. Results:. After 6 months on sirolimus, the pediatric quality of life inventory increased by a mean of 12.10 points (95% confidence interval [CI], 7.64–16.37) in children, and the QALY increased by 0.07 (95% CI, 0.03–0.11) in adults. After 6 months on sirolimus, total costs per child were numerically increased by €110.88 (95% CI, −€1786.17–€2165.95). Per adult, sirolimus resulted in a numerical decrease of mean total cost of €426.84 (95% CI, −€2831.38–€1414.60). In adults, there was a 95% chance that sirolimus treatment would be cost-effective with a WTP of €50k per QALY gained. Conclusion:. This first pharmacoeconomic analysis shows a substantial improvement in utility and health-related quality of life due to sirolimus treatment in patients with vascular malformations. It shows a high chance that sirolimus will be cost-effective in adults within the limits of acceptable WTP values within societies with nationalized health care. In children, sirolimus may become cost-effective in the future due to a possible decrease in interventions.

Introduction: The mini-Clinical Evaluation Exercise (CEX) is meant to provide on the spot feedback to trainees. We hypothesised that an ultra-short assessment tool with just one global entrustment scale (micro-CEX) would encourage faculty to provide better feedback compared to the traditional multiple domain mini-CEX. Methods: 59 pairs of faculty and trainees from internal medicine completed both the 7-item mini-CEX and a micro-CEX and were surveyed regarding their perceptions of the 2 forms. Wordcount and specificity of the feedback was assessed. Participants were subsequently interviewed to elicit their views on factors affecting the utility of the CEX. Results: Quantity and quality of feedback increased with the micro-CEX compared to the mini-CEX. Wordcount increased from 9.5 to 17.5 words, and specificity increased from 1.6 to 2.3 on a 4-point scale, p < 0.05 in both cases. Faculty and residents both felt the micro-CEX provided better assessment and feedback. The micro-CEX, but not the mini-CEX, was able to discriminate between residents in different years of training. The mini-CEX showed a strong halo effect between different domains of scoring. In interviews, ease of administration, immediacy of assessment, clarity of purpose, structuring of desired feedback, assessor-trainee pairing and alignment with trainee learning goals were identified as important features to optimize utility of the (mini or micro or both) CEX. Conclusions: Simplifying the assessment component of the CEX frees faculty to concentrate on feedback and this improves both quantity and quality of feedback. How the form is administered on the ground impacts its practical utility.

Lidwine B Mokkink,1,2 Iris Eekhout,1– 3 Maarten Boers,1,2 Cees PM van der Vleuten,4 Henrica CW de Vet1,2 1Amsterdam UMC, Vrije Universiteit Amsterdam, Epidemiology and Data Science, Amsterdam, the Netherlands; 2Amsterdam Public Health Research Institute, Amsterdam, the Netherlands; 3Child Health, Netherlands Organisation for Applied Scientific Research, Leiden, the Netherlands; 4Department of Educational Development and Research, School of Health Professions Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, Maastricht, the NetherlandsCorrespondence: Lidwine B Mokkink, Amsterdam UMC, AMC, J1B -225, Meibergdreef 9, Amsterdam, 1105 AZ, the Netherlands, Tel +31 20 44 44474, Email w.mokkink@amsterdamumc.nlAbstract: Reliability and measurement error are measurement properties that quantify the influence of specific sources of variation, such as raters, type of machine, or time, on the score of the individual measurement. Several designs can be chosen to assess reliability and measurement error of a measurement. Differences in design are due to specific choices about which sources of variation are varied over the repeated measurements in stable patients, which potential sources of variation are kept stable (ie, restricted), and about whether or not the entire measurement instrument (or measurement protocol) was repeated or only part of it. We explain how these choices determine how intraclass correlation coefficients and standard errors of measurement formulas are built for different designs by using Venn diagrams. Strategies for improving the measurement are explained, and recommendations for reporting the essentials of these studies are described. We hope that this paper will facilitate the understanding and improve the design, analysis, and reporting of future studies on reliability and measurement error of measurements.Keywords: reliability, measurement error, classical test theory, generalizability theory, intraclass correlation coefficient, standard error of measurement

Abstract Background Systematic reviews on simulation training effectiveness have pointed to the need to adhere to evidence-based instructional design (ID) guidelines. ID guidelines derive from sound cognitive theories and aim to optimize complex learning (integration of knowledge, skills, and attitudes) and learning transfer (application of acquired knowledge and skills in the workplace). The purpose of this study was to explore adherence to ID guidelines in simulation training programs for dealing with postpartum hemorrhage (PPH), a high-risk situation and the leading cause of maternal mortality worldwide. Methods A total of 40 raters analyzed simulation training programs as described in 32 articles. The articles were divided into four subsets of seven articles and one subset of four articles. Each subset was judged by seven to ten raters on adherence to ID guidelines. The 5-point Likert score rating scale was based on Merrill’s First Principles of Instruction and included items relating to key ID features categorized into five subscales: authenticity, activation of prior knowledge, demonstration, application, and integration/transfer. The authors searched for articles published in English between January 2007 and March 2017 in PubMed, Eric, and Google Scholar and calculated the mean Likert-scale score, per subscale, and interrater reliability (IRR). Results The mean Likert-scale scores calculated for all subscales were < 3.00. For the number of raters used to judge the papers in this study (varying between 7 and 10), the IRR was found to be excellent for the authenticity and integration/transfer subscales, good-to-excellent for the activation of prior knowledge and application subscales, and fair-to-good for the demonstration subscale. Conclusion The results demonstrate a paucity of the description of adherence to evidence-based ID guidelines in current simulation trainings for a high-risk situation such as PPH.

IntroductionThe shift toward an assessment for learning culture includes assessment quality criteria that emphasise the learning process, such as transparency and learning impact, in addition to the traditional validity and reliability criteria. In practice, the quality of the assessment depends on how the criteria are interpreted and applied. We explored how educators perceive and achieve assessment quality, as well as how they perceive assessment impact upon student learning.MethodsWe employed a qualitative research approach and conducted semi-structured interviews with 37 educators at one Dutch research university. The data were subsequently analysed using a template analysis.ResultsThe findings indicate that educators predominantly perceive and achieve assessment quality through traditional criteria. The sampled curricular stakeholders largely perceived assessment quality at the course level, whilst few specified programme-level quality criteria. Furthermore, educators perceived the impact of assessment on student learning in two distinct ways: as a source of information to monitor and direct student learning, and as a tool to prompt student learning.DiscussionThe shift toward a culture of assessment for learning is not entirely reflected in educators’ current perceptions. The study’s findings set the stage for better assessment quality and alignment with an assessment for learning culture.

Objective(s):. To elaborate expert consensus patient pathways to guide patients and physicians toward efficient diagnostics and management of patients with venous malformations. Methods:. VASCERN-VASCA (https://vascern.eu/) is a European network of multidisciplinary centers for Vascular Anomalies. The Nominal Group Technique was used to establish the pathways. Two facilitators were identified: one to propose initial discussion points and draw the pathways, and another to chair the discussion. A dermatologist (AD) was chosen as first facilitator due to her specific clinical and research experience. The draft was subsequently discussed within VASCERN-VASCA monthly virtual meetings and annual face-to-face meetings. Results:. The Pathway starts from the clinical suspicion of a venous type malformation (VM) and lists the clinical characteristics to look for to support this suspicion. Strategies for subsequent imaging and histopathology are suggested. These aim to inform on the diagnosis and to separate the patients into 4 subtypes: (1) sporadic single VMs or (2) multifocal, (3) familial, multifocal, and (4) combined and/or syndromic VMs. The management of each type is detailed in subsequent pages of the pathway, which are color coded to identify sections on (1) clinical evaluations, (2) investigations, (3) treatments, and (4) associated genes. Actions relevant to all types are marked in separate boxes, including when imaging is recommended. When definite diagnoses have been reached, the pathway also points toward disease-specific additional investigations and recommendations for follow up. Options for management are discussed for each subtype, including conservative and invasive treatments, as well as novel molecular therapies. Conclusion:. The collaborative efforts of VASCERN-VASCA, a network of the 9 Expert Centers, has led to a consensus Diagnostic and Management Pathways for VMs to assist clinicians and patients. It also emphasizes the role of multidisciplinary expert centers in the management of VM patients. This pathway will become available on the VASCERN website (http://vascern.eu/).

Abstract Background Collaborative learning is a group learning approach in which positive social interdependence within a group is key to better learning performance and future attitudes toward team practice. Recent attempts to replace a face-to-face environment with an online one have been developed using information communication technology. However, this raises the concern that online collaborative learning (OCL) may reduce positive social interdependence. Therefore, this study aimed to compare the degree of social interdependence in OCL with face-to-face environments and clarify aspects that affect social interdependence in OCL. Methods We conducted a crossover study comparing online and face-to-face collaborative learning environments in a clinical reasoning class using team-based learning for medical students (n = 124) in 2021. The participants were randomly assigned to two cohorts: Cohort A began in an online environment, while Cohort B began in a face-to-face environment. At the study’s midpoint, the two cohorts exchanged the environments as a washout. The participants completed surveys using the social interdependence in collaborative learning scale (SOCS) to measure their perceived positive social interdependence before and after the class. Changes in the mean SOCS scores were compared using paired t-tests. Qualitative data related to the characteristics of the online environment were obtained from the focus groups and coded using thematic analysis. Results The matched-pair tests of SOCS showed significant progression between pre- and post-program scores in the online and face-to-face groups. There were no significant differences in overall SOCS scores between the two groups. Sub-analysis by subcategory showed significant improvement in boundary (discontinuities among individuals) and means interdependence (resources, roles, and tasks) in both groups, but outcome interdependence (goals and rewards) improved significantly only in the online group. Qualitative analysis revealed four major themes affecting social interdependence in OCL: communication, task-sharing process, perception of other groups, and working facilities. Conclusions There is a difference in the communication styles of students in face-to-face and online environments, and these various influences equalize the social interdependence in a face-to-face and online environment.

IntroductionVascular malformations are rare congenital anomalies of the vascular system, which can involve the capillaries, veins, arteries, lymphatics, or a combination of vessel types. Patients with vascular malformations experience an impaired health-related quality of life (HRQoL) because of their symptoms (e.g., pain, swelling, and bleeding) and psychosocial distress. Sirolimus is an effective drug used in the medical treatment of these patients; however, relatively little is known about the effect of sirolimus on specific changes in the HRQoL domains and its magnitude.MethodsThe magnitude of change (effect size) following intervention is more informative to clinical practitioners than statistically significant but clinically unimportant changes; therefore, this study aimed to examine the magnitude and meaningfulness of change in the HRQoL of children and adults with vascular malformations following sirolimus treatment using low target levels.ResultsIn total, 50 patients with vascular malformations (19 children, 31 adults) were included in this study. These patients experienced a lower HRQoL than the general population, with the adults reporting a significantly lower score in almost all domains. A 6-month sirolimus treatment improved the HRQoL in 29 patients, including 77.8% of the children (Pediatric Quality of Life Inventory score [PedsQL]) and 57.7% of the adults (Short Form 36 [SF-36]). The effect sizes of sirolimus for each SF-36/PedsQL domain ranged from 0.19 to 1.02. The clinically relevant moderate magnitude of changes was seen in the domains of the children's reports: “Physical functioning” and “Social functioning” and in the domains of the parent reports: “Social functioning,” “School functioning,” and “Psychosocial.” A high-magnitude change was seen in the domains “Emotional functioning” and “Psychosocial” in the children's reports and “Physical functioning” in the parent reports. In addition, the moderate magnitude of changes was also seen in the adults SF-36: in all domains except for “Role limitations—physical problems,” “Role limitations—emotional problems,” and “General health perception.”ConclusionWe believe this is the first study showing the magnitude of change in HRQoL after sirolimus treatment in patients with vascular malformations. Before treatment, these patients experienced an impaired HRQoL compared with the general Dutch population. A 6-month sirolimus treatment with low target levels led to moderate-to-high clinically relevant changes in multiple domains, which significantly improved the HRQoL.Clinical trial registrationhttps://clinicaltrials.gov/ct2/show/NCT03987152?cond=Vascular+Malformations&cntry=NL&city=Nijmegen&draw=2&rank=1, identifier: NCT03987152.

Background Preclinical studies have firmly established the CD47-signal-regulatory protein (SIRP)α axis as a myeloid immune checkpoint in cancer, and this is corroborated by available evidence from the first clinical studies with CD47 blockers. However, CD47 is ubiquitously expressed and mediates functional interactions with other ligands as well, and therefore targeting of the primarily myeloid cell-restricted inhibitory immunoreceptor SIRPα may represent a better strategy.Method We generated BYON4228, a novel SIRPα-directed antibody. An extensive preclinical characterization was performed, including direct comparisons to previously reported anti-SIRPα antibodies.Results BYON4228 is an antibody directed against SIRPα that recognizes both allelic variants of SIRPα in the human population, thereby maximizing its potential clinical applicability. Notably, BYON4228 does not recognize the closely related T-cell expressed SIRPγ that mediates interactions with CD47 as well, which are known to be instrumental in T-cell extravasation and activation. BYON4228 binds to the N-terminal Ig-like domain of SIRPα and its epitope largely overlaps with the CD47-binding site. BYON4228 blocks binding of CD47 to SIRPα and inhibits signaling through the CD47-SIRPα axis. Functional studies show that BYON4228 potentiates macrophage-mediated and neutrophil-mediated killing of hematologic and solid cancer cells in vitro in the presence of a variety of tumor-targeting antibodies, including trastuzumab, rituximab, daratumumab and cetuximab. The silenced Fc region of BYON4228 precludes immune cell-mediated elimination of SIRPα-positive myeloid cells, implying anticipated preservation of myeloid immune effector cells in patients. The unique profile of BYON4228 clearly distinguishes it from previously reported antibodies representative of agents in clinical development, which either lack recognition of one of the two SIRPα polymorphic variants (HEFLB), or cross-react with SIRPγ and inhibit CD47-SIRPγ interactions (SIRPAB-11-K322A, 1H9), and/or have functional Fc regions thereby displaying myeloid cell depletion activity (SIRPAB-11-K322A). In vivo, BYON4228 increases the antitumor activity of rituximab in a B-cell Raji xenograft model in human SIRPαBIT transgenic mice. Finally, BYON4228 shows a favorable safety profile in cynomolgus monkeys.Conclusions Collectively, this defines BYON4228 as a preclinically highly differentiating pan-allelic SIRPα antibody without T-cell SIRPγ recognition that promotes the destruction of antibody-opsonized cancer cells. Clinical studies are planned to start in 2023.

The way quality of assessment has been perceived and assured has changed considerably in the recent 5 decades. Originally, assessment was mainly seen as a measurement problem with the aim to tell people apart, the competent from the not competent. Logically, reproducibility or reliability and construct validity were seen as necessary and sufficient for assessment quality and the role of human judgement was minimised. Later, assessment moved back into the authentic workplace with various workplace-based assessment (WBA) methods. Although originally approached from the same measurement framework, WBA and other assessments gradually became assessment processes that included or embraced human judgement but based on good support and assessment expertise. Currently, assessment is treated as a whole system problem in which competence is evaluated from an integrated rather than a reductionist perspective. Current research therefore focuses on how to support and improve human judgement, how to triangulate assessment information meaningfully and how to construct fairness, credibility and defensibility from a systems perspective. But, given the rapid changes in society, education and healthcare, yet another evolution in our thinking about good assessment is likely to lurk around the corner.

Abstract Background Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student’s performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. Methods In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner’s interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. Results The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student’s performance and quality of narrative feedback. Conclusion The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these “difficult” portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Abstract Background Rubrics are frequently used to assess competencies in outcome-based medical education (OBE). The implementation of assessment systems using rubrics is usually realised through years of involvement in projects with various stakeholders. However, for countries or specialities new to OBE, faster and more simplified processes are required. In March 2019, Japan introduced nine competencies and generic rubrics of competencies for medical residents. We explored the local adaptation of these generic rubrics and its consequences for assessors. Methods The study followed three steps. First, we locally adapted the generic rubrics. This was followed by conducting mixed-method research to explore the effect of the local adaptation. In step two, we examined the correlations between the scores in the locally adapted assessment sheets for supervising doctors and generic rubrics. In step three, we conducted interviews with supervising doctors. The study was conducted in the General Internal Medicine Department of Nagoya University, Japan. In the first step, doctors in the Medical Education Center and other medical departments, clerks, and residents participated. Supervising doctors in the General Internal Medicine Department participated in the second and third steps. Results A locally adapted assessment system was developed and implemented in seven months. The scores of the generic rubrics and the adapted assessment tool completed by the supervising doctors showed good correlations in some items as opposed to others, assessed mainly with other tools. Participant interviews revealed that local adaptation decreased their cognitive load leading to consistent ratings, increased writing of comments, and promoting reflection on instruction. Conclusions This adaptation process is a feasible way to begin the implementation of OBE. Local adaptation has advantages over direct use of generic rubrics.

Abstract Background In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular. Methods We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained. Results The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development. Conclusions To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.

Low-stakes assessments are theorised to stimulate and support self-regulated learning. They are feedback-, not decision-oriented, and should hold little consequences to a learner based on their performance. The use of low-stakes assessment as a learning opportunity requires an environment in which continuous improvement is encouraged. This may be hindered by learners' perceptions of assessment as high-stakes. Teachers play a key role in learners' assessment perceptions. By investigating assessment perceptions through an interpersonal theory-based perspective of teacher-learner relationships, we aim to better understand the mechanisms explaining the relationship between assessment and learning within medical education. First, twenty-six purposefully selected learners, ranging from undergraduates to postgraduates in five different settings of programmatic assessment, were interviewed about their assessment task perception. Next, we conducted a focussed analysis using sensitising concepts from interpersonal theory to elucidate the influence of the teacher-learner relationship on learners' assessment perceptions. The study showed a strong relation between learners' perceptions of the teacher-learner relationship and their assessment task perception. Two important sources for the perception of teachers' agency emerged from the data: positional agency and expert agency. Together with teacher's communion level, both types of teachers' agency are important for understanding learners' assessment perceptions. High levels of teacher communion had a positive impact on the perception of assessment for learning, in particular in relations in which teachers' agency was less dominantly exercised. When teachers exercised these sources of agency dominantly, learners felt inferior to their teachers, which could hinder the learning opportunity. To utilise the learning potential of low-stakes assessment, teachers are required to stimulate learner agency in safe and trusting assessment relationships, while carefully considering the influence of their own agency on learners' assessment perceptions. Interpersonal theory offers a useful lens for understanding assessment relationships. The Interpersonal Circumplex provides opportunities for faculty development that help teachers develop positive and productive relationships with learners in which the potential of low-stakes assessments for self-regulated learning is realised.

The relationship between clinician teachers and their students is of major importance in medical education. However, there is little known about the effects on clinicians when conflict occurs with their students. What do clinicians perceive to be major causes of these conflicts? How do they react when and after conflict occurs? A phenomenological inquiry exploring the lived experience of 12 clinician teachers in medical schools was performed. The clinicians were selected using purposeful sampling and snowballing techniques. The interviews revolved around discussions based on episodes of conflict with medical students that the clinicians considered significant. The analysis and emergent themes were partially constructed around and informed by theories of conflict, and conflict management. A number of themes emerged from this study. Clinicians experienced that significant psychological and behavioural problems of students had a dominant impact on the likelihood and severity of conflict; these conflicts had a significant emotional impact on clinicians; though the responses to conflict varied, "avoidance" was a mechanism commonly used by clinicians and thus the assessment of attitudinal and behavioural professional issues in the workplace was problematic. This study shows how the clinician perspective to challenging student/clinician encounters impacts on the quality of education they are able to provide. We recommend medical schools consider these issues when designing their programs in order to develop and maintain clinician-teacher engagement and participation.

Most courses in higher education finish with one or more assessments which commonly all have to be passed. In these courses, student learning is commonly measured using conventional classroom tests, therefore test preparation is a common task for students. In higher education, compared with students’ prior education, they are faced with a more complex curriculum and have to perform their studies with less guidance and limited resources. Therefore, effective and efficient test preparation is important. A strategy to help students study effectively in the context of test preparation is to make the appropriate control decisions, for instance to cease test preparation on specific content and (re) study other subjects that need attention. These control decisions are an important psychological aspect of the test preparation study process. We conducted a qualitative study on how students made control decisions in a test preparation period for a knowledge test in Educational Sciences. The study was conducted with students of a teacher training program at a University of Applied Sciences in the Netherlands. Results show that different progressions of learning judgments and the self-efficacy of students led to two different saturations. This in turn led to students making either no, inaccurate, or accurate control decisions. This article discusses the impact and practical implications of these insights.

Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption--just like PBL--will take many years ahead of us.

Introduction: Previous studies indicate that professional identity formation (PIF), the formation of a self-identity with the internalised values and norms of professionalism, may influence self-regulated learning (SRL). However, it remains unclear whether a PIF-oriented intervention can improve SRL in clinical education. The aim of this study was to explore whether a PIF-oriented mentoring platform improves SRL in a clinical clerkship. Methods: A mixed-methods study was conducted. Forty-one students in a community-based clinical clerkship (CBCC) used a PIF-oriented mentoring platform. They articulated the values and norms of professionalism in a professional identity essay, elaborated on future professional self-image, and reflected on their current compared to future selves. They made a study plan while referring to PIF-based self-reflection and completed it. The control group of 41 students completed CBCC without the PIF-oriented mentoring platform. Changes in SRL between the two groups were quantitatively compared using the Motivated Strategies for Learning Questionnaire. We explore how PIF elements in the platform affected SRL by qualitative analysis of questionnaire and interview data. Results: A moderate improvement in intrinsic goal orientation (p = 0.005, ε2 = 0.096) and a mild improvement in critical thinking (p = 0.041, ε2 = 0.051) were observed in the PIF-oriented platform group. Qualitative analysis revealed that the PIF-oriented platform fostered professional responsibility as a key to expanding learning goals. Gaining authentic knowledge professionally fostered critical thinking, and students began to elaborate knowledge in line with professional task processes. Conclusion: A PIF-oriented mentoring platform helped students improve SRL during a clinical clerkship.

In programmatic assessment (PA), an arrangement of different assessment methods is deliberately designed across the entire curriculum, combined and planned to support both robust decision-making and student learning. In health sciences education, evidence about the merits and pitfalls of PA is emerging. Although there is consensus about the theoretical principles of PA, programs make diverse design choices based on these principles to implement PA in practice, fitting their own contexts. We therefore need a better understanding of how the PA principles are implemented across contexts—within and beyond health sciences education. In this study, interviews were conducted with teachers/curriculum designers representing nine different programs in diverse professional domains. Research questions focused on: (1) design choices made, (2) whether these design choices adhere to PA principles, (3) student and teacher experiences in practice, and (4) context-specific differences between the programs. A wide range of design choices were reported, largely adhering to PA principles but differing across cases due to contextual alignment. Design choices reported by almost all programs include a backbone of learning outcomes, data-points connected to this backbone in a longitudinal design allowing uptake of feedback, intermediate reflective meetings, and decision-making based on a multitude of data-points made by a committee and involving multi-stage procedures. Contextual design choices were made aligning the design to the professional domain and practical feasibility. Further research is needed in particular with regard to intermediate-stakes decisions.

BackgroundEvaluation of patients' health care experiences is central to measuring patient-centred care. However, different instruments tend to be used at the hospital or departmental level but rarely both, leading to a lack of standardization of patient experience measures.ObjectiveTo validate the Consumer Quality Index (CQI) Inpatient Hospital Care for use on both department and hospital levels.DesignUsing cross-sectional observational data, we investigated the internal validity of the questionnaire using confirmatory factor analyses (CFA), and the generalizability of the questionnaire for use at the department and hospital levels using generalizability theory.Setting and participants22924 adults hospitalized for ≥24 hours between 1 January 2013 and 31 December 2014 in 23 Dutch hospitals (515 department evaluations).Main variableCQI Inpatient Hospital Care questionnaire.ResultsCFA results showed a good fit on individual level (CFI=0.96, TLI=0.95, RMSEA=0.04), which was comparable between specialties. When scores were aggregated to the department level, the fit was less desirable (CFI=0.83, TLI=0.81, RMSEA=0.06), and there was a significant overlap between communication with doctors and explanation of treatment subscales. Departments and hospitals explained ≤5% of total variance in subscale scores. In total, 4-8 departments and 50 respondents per department are needed to reliably evaluate subscales rated on a 4-point scale, and 10 departments with 100-150 respondents per department for binary subscales.Discussion and conclusionsThe CQI Inpatient Hospital Care is a valid and reliable questionnaire to evaluate inpatient experiences in Dutch hospitals provided sufficient sampling is done. Results can facilitate meaningful comparisons and guide quality improvement activities in individual departments and hospitals.

Objectives At the very start of medical residency training, entrustment of clinical tasks may be a major challenge, on which current scientific knowledge is scarce. This study therefore aimed to gain insight into the process of supervisors’ decision making underlying the entrustment of clinical tasks at the start of one-on-one supervisor–trainee working relationships.Setting This study was performed in a general practice (GP) training department in the Netherlands.Participants For this study, we recruited supervisor–trainee pairs who were just about to start the first year of GP residency training. Of 10 eligible supervisor–trainee pairs, 4 participated.Design We used a qualitative, social-constructivist research approach. Data from naturalistic non-participant observations and semistructured interviews with supervisors in four GP practices were triangulated and analysed using a thematic analysis approach.Results Supervisors’ early entrustment decisions were based on generic trainee qualities such as self-reflexivity, knowing one’s limitations and asking for help in time, rather than on task-specific performance. At the start of residency training, supervisors’ primary concern was to create a safe working and learning environment in which trainees could and would ask for timely supervision while being entrusted with challenging tasks. Supervisors used idiosyncratic entrustment strategies that were influenced by their propensity to trust, previous experiences with trainees, and their perspective on learning and teaching.Conclusion Entrustment decisions require high levels of safety for all stakeholders involved. Especially at the beginning of supervisor–trainee relationships, establishing an educational alliance in which entrustment is the subject of ongoing trainee–supervisor conversations is essential to achieve and maintain and optimal balance between trainee learning and patient safety. Additional research may further our understanding of early entrustment decision making and the role of generic trainee qualities in different settings.

Formative assessment is considered as one of the most effective interventions to support teacher decision-making and improve education and student learning. However, formative assessment does not always meet these expectations. In order to be effective, formative assessment activities should be consciously and coherently planned aligned with other aspects of the curriculum and the decisions teachers wish to make based on these activities. While there is sufficient support for teachers to design formative assessment activities, no guidelines exist to help them tie these different activities together in an effective way. To support teachers in designing formative assessment plans informing formative decision-making, this study focused on the creation of a set of design principles. These design principles for formative assessment plans were formulated based on expert interviews and subsequently evaluated by future users. The result is a set of eight design principles that can be used and validated in educational practice.

Abstract Background Problem-based learning (PBL) is classified as a collaborative learning approach, wherein students learn while contributing meaning to experiences and interactions with others. An important theoretical fundament of PBL is social interdependence theory (SIT) because positive social interdependence within a group has been found to be key to better learning performance and future attitudes towards team practice. However, most previous studies in health professions education focused on cognitive outcomes, and few studies have focused on collaborative behaviors in PBL groups. The lack of this empirical insight makes implementation of PBL difficult, especially in contexts where there is limited experience with collaborative learning. Therefore, the aim of this study was to elucidate what promotes or hinders positive social interdependence and how the attributes work during PBL. Methods We conducted four focus groups among clinical year medical students (n = 26) who participated in PBL tutorials in the formal curriculum. We asked semi-structured questions that corresponded with the overall concept of SIT. We analyzed the transcript using constructivist grounded theory and developed a model to explain contextual attributes that promote or hinder positive social interdependence in PBL. Results Two contextual attributes of “academic inquisition” and “desire for efficiency” affect social interdependence among a student group in PBL. Academic inquisition is students’ desire to engage in their academic learning, and desire for efficiency is students’ attitude toward learning as an imposed duty and desire to complete it as quickly as possible. These attributes are initially mutually conflicting and constructing social interdependence through multiple steps including inquisition from a case, seeking efficient work, sharing interest in problem solving, expecting mutual contributions, and complementing learning objectives. Conclusion These findings will contribute to understanding collaborative learning environments in PBL and may help explain contexts where PBL is less successful. The model can also be used as a tool to support innovation of PBL as collaborative learning.

Summary: Background: The OVAMA (Outcome Measures for VAscular MAlformations) project determined quality of life (QoL) as a core outcome domain for evaluating treatment effect in vascular malformations. To correctly evaluate treatment effect on QoL, patient-reported outcome measures (PROMs) are needed that are responsive to changes. In children with vascular malformations, we explored if two widely used PROMs were responsive to changes: the Pediatric Quality of Life Inventory (PedsQL) and the Children's Dermatology Life Quality Index (CDLQI). Methods: In an international multicenter prospective study, conservatively and invasively treated children completed the PedsQL and CDLQI at baseline and after follow-up of 6–8 weeks. At follow-up, change in health was assessed by a global rating of change (GRC) scale. Responsiveness was assessed by testing hypotheses on expected correlation strength between change scores of the PROMs and the GRC scale, and by calculating the area under the receiver operating characteristics curve (AUC). The PROMs were considered responsive if ≥75% of the hypotheses were confirmed or if the AUC was ≥0.7. Results: Twenty-nine children were recruited in three centers in the Netherlands and United States, of which 25 completed all baseline and follow-up measurements. For both the PedsQL and CDLQI, less than 75% of the hypotheses were confirmed and the AUC was

Programmatic assessment is both a philosophy and a method for assessment. It has been developed in medical education as a response to the limitation of the dominant testing or measurement approaches and to better align with changes in how medical competence was conceptualised. It is based on continual collection of assessment and feedback information which is collected in a portfolio and periodically discussed with a mentor or coach. Students produce reflections in which they analyse the assessment information triangulated across methods to meaningfully--rather than purely numerically-analyse their strengths and weakness in the different competency domains. Decisions about students' progress are made proportionally, in that low-stakes decisions are made when only limited assessment information is available but high-stakes decisions will and can be made on rich information. Quality of the process and credibility of the outcomes is determined by a combination of psychometric methods, qualitative rigour, and through organisational due processes. The method is still young, preliminary results are promising but more empirical justification research is needed.

Vocational education aims at preparing students for their future as professionals; thus, students are often required to undergo a form of on-the-job learning. Assessment of professional performance during on-the-job learning is not based on single observations, but on prolonged participation at the workplace culminating in judgement by a workplace educator. However, little is known about "how" educators reach a judgement. This paper, therefore, aims at exploring how educators inform their judgements about students' professional performance. We consider the workplace to be a participatory learning environment, which entails that judgement about professional performance should be made in relation to the community students participated in and that the educator from that community has a central role. Our study takes an explorative approach based on interviews and uses template analysis to generate insight. Findings revealed that educators use multiple strategies to gather and process information about students to inform their judgement. Findings also showed that assessment starts at the first meeting, when the educator forms a first impression of the student and judges whether or not the student is suitable for the job. Educators at the workplace also showed that they are aware of the context in which they form their judgement.

Abstract Background Several studies have shown that conceptions of teachers on teaching and learning can influence the teaching practices and behavior in higher education. This association is also found in undergraduate medical education but not yet established in postgraduate medical setting. An instrument, Conceptions of Learning and Teaching (COLT) was developed to measure conception of teachers in undergraduate medical education. COLT is a 3-factor 18-item questionnaire. The objective of this study is to evaluate if COLT is valid for postgraduate medical education. Methods We invited postgraduate clinical faculty from 3 hospitals in the Netherlands to fill out the COLT. Confirmatory and exploratory factor analysis were performed to evaluate the fit of the postgraduate clinical faculty data to the COLT. Analysis of variance was done to evaluate if there was difference among the 3 hospitals in terms of the response by the clinical faculty. Results Confirmatory factor analysis showed that the postgraduate faculty data had a 2 factor structure after removal of five items. These factors were Teacher Centeredness (TC) and combined Appreciation of Active Learning and Orientation to Professional Practice (A-P) and were considered as comparable to the factors in the original COLT, expressing the post-graduate learning and teaching setting. As several items were removed, the fit was suboptimal, yet did suggest validity for use of the COLT for postgraduate medical education. Conclusion The modified COLT can be used to measure conceptions of teaching and learning in postgraduate medical education. We recommend further study to improve the factor structure of the modified COLT.

Abstract Background Developing self-regulated learning in preclinical settings is important for future lifelong learning. Previous studies indicate professional identity formation, i.e., formation of self-identity with internalized values and norms of professionalism, might promote self-regulated learning. We designed a professional identity formation-oriented reflection and learning plan format, then tested effectiveness on raising self-regulated learning in a preclinical year curriculum. Methods A randomized controlled crossover trial was conducted using 112 students at Jichi Medical University. In six one-day problem-based learning sessions in a 7-month pre-clinical year curriculum, Groups A (n = 56, female 18, mean age 21.5y ± 0.7) and B (n = 56, female 11, mean age 21.7y ± 1.0) experienced professional identity formation-oriented format: Group A had three sessions with the intervention format in the first half, B in the second half. Between-group identity stages and self-regulated learning levels were compared using professional identity essays and the Motivated Strategies for Learning Questionnaire. Results Two-level regression analyses showed no improvement in questionnaire categories but moderate improvement of professional identity stages over time (R 2 = 0.069), regardless of timing of intervention. Conclusions Professional identity moderately forms during the pre-clinical year curriculum. However, neither identity nor self-regulated learning is raised significantly by limited intervention.