Search

Self-directed learning is generally considered a key competence in higher education. To enable self-directed learning, assessment practices increasingly embrace assessment for learning rather than the assessment of learning, shifting the focus from grades and scores to provision of rich, narrative, and personalized feedback. Students are expected to collect, interpret, and give meaning to this feedback, in order to self-assess their progress and to formulate new, appropriate learning goals and strategies. However, interpretation of aggregated, longitudinal narrative feedback has been proven to be very challenging, cognitively demanding, and time consuming. In this article, we, therefore, explored the applicability of existing, proven text mining techniques to support feedback interpretation. More specifically, we investigated whether it is possible to automatically generate meaningful information about prevailing topics and the emotional load of feedback provided in medical students' competence-based portfolios (N = 1500), taking into account the competence framework and the students' various performance levels. Our findings indicate that the text-mining techniques topic modeling and sentiment analysis make it feasible to automatically unveil the two principal aspects of narrative feedback, namely the most relevant topics in the feedback and their sentiment. This article, therefore, takes a valuable first step toward the automatic, online support of students, who are tasked with meaningful interpretation of complex narrative data in their portfolio as they develop into self-directed life-long learners.

Background Although Clinical Competency Committees (CCCs) were implemented to facilitate the goals of competency-based medical education, implementation has been variable, and we do not know if and how these committees affected programs and assessment in graduate medical education (GME). Objective To explore the roles CCCs fulfill in GME and their effect on trainees, faculty, and programs. Methods We conducted a narrative review of CCC primary research with the following inclusion criteria: all articles must be research in nature, focused on GME and specifically studying CCCs, and published in English language journals from January 2013 to November 2022. Results The main results are as follows: (1) The primary role of the CCC (decision-making on trainee progress) is mostly described in "snapshots" (ie, focusing on a single aspect of this role at a single point in time); (2) CCCs are taking on secondary roles, some of which were anticipated (eg, remediation, feedback) whereas others were "unanticipated" (eg, use of CCC data to validate trainee self-assessment, predict trainee performance in other settings such as certifying examinations, investigate gender bias in assessment); and (3) Articles briefly mentioned short-term outcomes of CCCs at the level of the trainees, faculty, and programs. However, most studies described interventions to aid CCC work and did not specifically aim at investigating short-term (eg, curriculum changes) or long-term outcomes (eg, improved patient outcomes). Conclusions CCCs fulfill a range of roles in assessment beyond their intended purpose. A more systematic approach is needed to investigate the outcomes of CCC implementation on GME. [ABSTRACT FROM AUTHOR]

Context Methods Conclusions Despite increasing discussion and scholarship, equity in assessment is rarely defined and distinguished in a way that allows for shared understanding in medical education. This paper seeks to clarify and expand the conversation about equity in assessment by critically reviewing three distinct and evolving orientations toward equity in assessment. Orientations refers to the positions, attitudes, interests or priorities individuals can hold toward equity in assessment. The three orientations include fairness‐oriented assessment, assessment for inclusion and justice‐oriented assessment. While fairness‐oriented assessment is a prevailing orientation in medical education, assessment for inclusion and justice‐oriented assessment, originally developed in other fields of education, deserve careful consideration.In this paper, the authors explore unique underpinning assumptions of each orientation by critically examining the foundational literature of each orientation. They reflect on the unique perspectives each orientation provides, including the actions one might take and what advantages and disadvantages might result from looking at equity in assessment from any one orientation.Informed by this reflection, the authors propose that to more effectively advance equity in assessment in medical education, those working in the field should clearly identify their respective orientations, intentionally choose methods, tools and measures aligned with their orientations and expand their work by exploring alternative orientations. [ABSTRACT FROM AUTHOR]

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments--rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically "the locus of regulation of learning" (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use--and, consequently, the effectiveness--of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Purpose: This study aims to investigate quality of feedback as offered by supervisor-assessors with varying levels of assessor expertise following assessment of performance in residency training in a health care setting. It furthermore investigates if and how different levels of assessor expertise influence feedback characteristics. Design/methodology/approach: Experienced (n = 18) and non-experienced (n = 16) supervisor-assessors with different levels of assessor expertise in general practice (GP) watched two videotapes, each presenting a trainee in a "real-life" patient encounter. After watching each videotape, participants documented performance ratings, wrote down narrative feedback comments and verbalized their feedback. Deductive content analysis of feedback protocols was used to explore quality of feedback. Between-group differences were assessed using qualitative-based quantitative analysis of feedback data. Findings: Overall, specificity and usefulness of both written and verbal feedback was limited. Differences in assessor expertise did not seem to affect feedback quality. Research limitations/implications: Results of the study are limited to a specific setting (GP) and assessment context. Further study in other settings and larger sample sizes may contribute to better understanding of the relation between assessor characteristics and feedback quality. Practical implications: Findings suggest that even with supervisor-assessors with varying levels of assessor expertise who are trained in performance assessment and the provision of performance feedback, high-quality feedback is not self-evident; coaching "on the job" of feedback providers and continuous evaluation of feedback processes in performance management systems is crucial. Instruments should facilitate provision of meaningful feedback in writing. Originality/value: The paper investigates quality of feedback immediately following assessment of performance, and links feedback quality to assessor expertise. Findings can contribute to improvement of performance management systems and assessments for developmental purposes. (Contains 2 tables and 2 figures.)

Context: In-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices. Purpose: The aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs. Discussion: The issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.

Processes involved in the regulation of learning have been researched for decades, because of its impact on academic and workplace performance. In fact, self‐regulated learning is the focus of countless studies in health professions education and higher education in general. While we will always need competent individuals who are able to regulate their own learning, developments in healthcare require a shift from a focus on the individual to the collective: collaboration within and between healthcare teams is at the heart of high‐quality patient care. Concepts of collaborative learning and collective competence challenge commonly held conceptualisations of regulatory learning and call for a focus on the social embeddedness of regulatory learning and processes regulating the learning of the collective. Therefore, this article questions the alignment of current conceptualisations of regulation of learning with demands for collaboration in current healthcare. We explore different conceptualisations of regulation of learning (self‐, co‐, and socially shared regulation of learning), and elaborate on how the integration of these conceptualisations adds to our understanding of regulatory learning in healthcare settings. Building on these insights, we furthermore suggest ways forward for research and educational practice. Bransen et al. explore the concept of 'self' to argue that the health professions need to shift from striving to optimize self‐regulation to understanding how to effectively regulate across self‐, co‐, and shared‐regulation. [ABSTRACT FROM AUTHOR]