Search

Educational change in higher education is challenging and complex, requiring engagement with a multitude of perspectives and contextual factors. In this paper, we present a case study based on our experiences of enacting a fundamental educational change in a medical program; namely, the steps taken in the transition to programmatic assessment. Specifically, we reflect on the successes and failures in embedding a coaching culture into programmatic assessment. To do this, we refer to the principles of programmatic assessment as they apply to this case and conclude with some key lessons that we have learnt from engaging in this change process. Fostering a culture of programmatic assessment that supports learners to thrive through coaching has required compromise and adaptability, particularly in light of the changes to teaching and learning necessitated by the global pandemic. We continue to inculcate this culture and enact the principles of programmatic assessment with a focus on continuous quality improvement.

The way quality of assessment has been perceived and assured has changed considerably in the recent 5 decades. Originally, assessment was mainly seen as a measurement problem with the aim to tell people apart, the competent from the not competent. Logically, reproducibility or reliability and construct validity were seen as necessary and sufficient for assessment quality and the role of human judgement was minimised. Later, assessment moved back into the authentic workplace with various workplace-based assessment (WBA) methods. Although originally approached from the same measurement framework, WBA and other assessments gradually became assessment processes that included or embraced human judgement but based on good support and assessment expertise. Currently, assessment is treated as a whole system problem in which competence is evaluated from an integrated rather than a reductionist perspective. Current research therefore focuses on how to support and improve human judgement, how to triangulate assessment information meaningfully and how to construct fairness, credibility and defensibility from a systems perspective. But, given the rapid changes in society, education and healthcare, yet another evolution in our thinking about good assessment is likely to lurk around the corner.

Arguably, constructive alignment has been the major challenge for assessment in the context of problem-based learning (PBL). PBL focuses on promoting abilities such as clinical reasoning, team skills and metacognition. PBL also aims to foster self-directed learning and deep learning as opposed to rote learning. This has incentivized researchers in assessment to find possible solutions. Originally, these solutions were sought in developing the right instruments to measure these PBL-related skills. The search for these instruments has been accelerated by the emergence of competency-based education. With competency-based education assessment moved away from purely standardized testing, relying more heavily on professional judgment of complex skills. Valuable lessons have been learned that are directly relevant for assessment in PBL. Later, solutions were sought in the development of new assessment strategies, initially again with individual instruments such as progress testing, but later through a more holistic approach to the assessment program as a whole. Programmatic assessment is such an integral approach to assessment. It focuses on optimizing learning through assessment, while at the same gathering rich information that can be used for rigorous decision-making about learner progression. Programmatic assessment comes very close to achieving the desired constructive alignment with PBL, but its wide adoption--just like PBL--will take many years ahead of us.

Programmatic assessment is both a philosophy and a method for assessment. It has been developed in medical education as a response to the limitation of the dominant testing or measurement approaches and to better align with changes in how medical competence was conceptualised. It is based on continual collection of assessment and feedback information which is collected in a portfolio and periodically discussed with a mentor or coach. Students produce reflections in which they analyse the assessment information triangulated across methods to meaningfully--rather than purely numerically-analyse their strengths and weakness in the different competency domains. Decisions about students' progress are made proportionally, in that low-stakes decisions are made when only limited assessment information is available but high-stakes decisions will and can be made on rich information. Quality of the process and credibility of the outcomes is determined by a combination of psychometric methods, qualitative rigour, and through organisational due processes. The method is still young, preliminary results are promising but more empirical justification research is needed.

Objectives Traditionally, evaluation is considered a measurement process that can be performed independently of the cultural context. However, more recently the importance of considering raters’ sense-making, that is, the process by which raters assign meaning to their collective experiences, is being recognised. Thus far, the majority of the discussion on this topic has originated from Western perspectives. Little is known about the potential influence of an Asian culture on raters’ sense-making. This study explored residents’ sense-making associated with evaluating their clinical teachers within an Asian setting to better understand contextual dependency of validity.Design A qualitative study using constructivist grounded theory.Setting The Japanese Ministry of Health, Labour and Welfare has implemented a system to monitor the quality of clinical teaching within its 2-year postgraduate training programme. An evaluation instrument was developed specifically for the Japanese setting through which residents can evaluate their clinical teachers.Participants 30 residents from 10 Japanese teaching hospitals with experience in evaluating their clinical teachers were sampled purposively and theoretically.Methods We conducted in-depth semistructured individual interviews. Sensitising concepts derived from Confucianism and principles of response process informed open, axial and selective coding.Results Two themes and four subthemes were constructed. Japanese residents emphasised the awareness of their relationship with their clinical teachers (1). This awareness was fuelled by their sense of hierarchy (1a) and being part of the collective society (1b). Residents described how the meaning of evaluation (2) was coloured by their perceived role as senior (2a) and their experienced responsibility for future generations (2b).Conclusions Japanese residents’ sense-making while evaluating their clinical teachers appears to be situated and affected by Japanese cultural values. These findings contribute to a better understanding of a culture’s influence on residents’ sense-making of evaluation instruments and the validity argument of evaluation.

Artificial intelligence (AI) can be a powerful tool for data analysis, but it can also mislead investigators, due in part to a fundamental difference between classic data analysis and data analysis using AI. A more or less limited data set is analyzed in classic data analysis, and a hypothesis is generated. That hypothesis is then tested using a separate data set, and the data are examined again. The premise is either accepted or rejected with a value p , indicating that any difference observed is due merely to chance. By contrast, a new hypothesis is generated in AI as each datum is added to the data set. We explore this discrepancy and suggest means to overcome it. [ABSTRACT FROM AUTHOR]

Educational change in higher education is challenging and complex, requiring engagement with a multitude of perspectives and contextual factors. In this paper, we present a case study based on our experiences of enacting a fundamental educational change in a medical program; namely, the steps taken in the transition to programmatic assessment. Specifically, we reflect on the successes and failures in embedding a coaching culture into programmatic assessment. To do this, we refer to the principles of programmatic assessment as they apply to this case and conclude with some key lessons that we have learnt from engaging in this change process. Fostering a culture of programmatic assessment that supports learners to thrive through coaching has required compromise and adaptability, particularly in light of the changes to teaching and learning necessitated by the global pandemic. We continue to inculcate this culture and enact the principles of programmatic assessment with a focus on continuous quality improvement.

"Helicopter research" refers to a practice where researchers from wealthier countries conduct studies in lower-income countries with little involvement of local researchers or community members. This practice also occurs domestically. In this Commentary, we outline strategies to curb domestic helicopter research and to foster equity-centered collaborations. [ABSTRACT FROM AUTHOR]

Business colleges must efficiently manage academic programs which lead to degrees in various majors. In this paper we illustrate how data envelopment analysis (DEA) can be applied to analyze the relative efficiency of how different academic programs transform students into graduates. Assuming a set of common inputs and outputs, we use this optimization technique to compare the programs, and identify those which assume a maximum level of relative efficiency. Our DEA analysis also enables the analyses of why some programs are relatively inefficient and provide specific, quantitative guidance on how to make them more efficient. [ABSTRACT FROM AUTHOR]

Abstract Background Despite growing evidence of the benefits of including assessment for learning strategies within programmes of assessment, practical implementation of these approaches is often problematical. Organisational culture change is often hindered by personal and collective beliefs which encourage adherence to the existing organisational paradigm. We aimed to explore how these beliefs influenced proposals to redesign a summative assessment culture in order to improve students’ use of assessment-related feedback. Methods Using the principles of participatory design, a mixed group comprising medical students, clinical teachers and senior faculty members was challenged to develop radical solutions to improve the use of post-assessment feedback. Follow-up interviews were conducted with individual members of the group to explore their personal beliefs about the proposed redesign. Data were analysed using a socio-cultural lens. Results Proposed changes were dominated by a shared belief in the primacy of the summative assessment paradigm, which prevented radical redesign solutions from being accepted by group members. Participants’ prior assessment experiences strongly influenced proposals for change. As participants had largely only experienced a summative assessment culture, they found it difficult to conceptualise radical change in the assessment culture. Although all group members participated, students were less successful at persuading the group to adopt their ideas. Faculty members and clinical teachers often used indirect techniques to close down discussions. The strength of individual beliefs became more apparent in the follow-up interviews. Conclusions Naïve epistemologies and prior personal experiences were influential in the assessment redesign but were usually not expressed explicitly in a group setting, perhaps because of cultural conventions of politeness. In order to successfully implement a change in assessment culture, firmly-held intuitive beliefs about summative assessment will need to be clearly understood as a first step.

Artificial Intelligence (AI) is a very powerful new tool that is destined to markedly advance many areas of dermatology, including cosmetic dermatology, oculoplastics, cancer detection and treatment, dermatopathlogy, and identification of pathogens. Along with these are some special new risks and concerns, however, including ethical considerations, data analysis, interpretation of scientific studies, and recognizing systematic failures and fraud, particularly in generative AI. Each of these issues is reviewed collectively and in turn in this special of Clinics in Dermatology. [ABSTRACT FROM AUTHOR]

To retain talented scientists in academia, there is a need for structural reform to postdoctoral researcher (postdoc) mentorship. These changes include mentorship training for postdocs and their mentors, formalizing postdoc mentorship networks and postdoc cohorts within a department, and incorporating mentorship development plans (MDPs) in funding decisions for principal investigators (PIs). To retain talented scientists in academia, there is a need for structural reform to postdoctoral researcher (postdoc) mentorship. These changes include mentorship training for postdocs and their mentors, formalizing postdoc mentorship networks and postdoc cohorts within a department, and incorporating mentorship development plans in funding decisions for principal investigators (PIs). [ABSTRACT FROM AUTHOR]

Amebelodon and Konobelodon are two members of a group of proboscideans called shovel-tusked gomphotheres. The phylogenetic relationship between them is uncertain, with some authors considering them to be closely related taxa while others consider them only distantly related. This uncertainty stems from confusing issues related to morphology, biochronology, and biogeography. Contributing to this confusion, Amebelodon consists of two distinct morphotypes that do not obviously resemble each other. One, typified by A. floridanus, possesses a conservative dentition and lower tusks, while the other, represented solely by A. fricki, possesses an advanced dentition and lower tusks that closely resemble those found in Konobelodon. In a traditional evolutionary scenario involving these taxa, conservative Amebelodon appeared in North America during the latest Miocene and gave rise to advanced Amebelodon. Advanced Amebelodon in turn gave rise to Konobelodon, with Konobelodon then dispersing into Eurasia. However, new discoveries of Konobelodon in the Late Miocene of Europe and Asia render this hypothesis implausible, and suggest rather that Amebelodon first appeared in Asia during the Late Miocene, where it gave rise to Konobelodon. These findings in turn suggest that A. fricki in North America represents an immigrant from Eurasia. If this new hypothesis is correct, then the status of the conservative species such as A. floridanus within Amebelodon becomes problematic, with the implication that Amebelodon is polyphyletic. On this basis as well as extensive morphological differences, one of these conservative species, Amebelodon floridanus, is here referred to a new genus, Stenobelodon. [ABSTRACT FROM AUTHOR]

The recent rise of interest among the medical education community in individual faculty making subjective judgments about medical trainee performance appears to be directly related to the introduction of notions of integrated competency-based education and assessment for learning. Although it is known that assessor expertise plays an important role in performance assessment, the roles played by different factors remain to be unraveled. We therefore conducted an exploratory study with the aim of building a preliminary model to gain a better understanding of assessor expertise. Using a grounded theory approach, we conducted seventeen semi-structured interviews with individual faculty members who differed in professional background and assessment experience. The interviews focused on participants' perceptions of how they arrived at judgments about student performance. The analysis resulted in three categories and three recurring themes within these categories: the categories "assessor characteristics," "assessors' perceptions of the assessment tasks," and the "assessment context," and the themes "perceived challenges," "coping strategies," and "personal development." Central to understanding the key processes in performance assessment appear to be the dynamic interrelatedness of the different factors and the developmental nature of the processes. The results are supported by literature from the field of expertise development and in line with findings from social cognition research. The conceptual framework has implications for faculty development and the design of programs of assessment.

It has become axiomatic that assessment impacts powerfully on student learning. However, surprisingly little research has been published emanating from authentic higher education settings about the nature and mechanism of the pre-assessment learning effects of summative assessment. Less still emanates from health sciences education settings. This study explored the pre-assessment learning effects of summative assessment in theoretical modules by exploring the variables at play in a multifaceted assessment system and the relationships between them. Using a grounded theory strategy, in-depth interviews were conducted with individual medical students and analyzed qualitatively. Respondents' learning was influenced by task demands and system design. Assessment impacted on respondents' cognitive processing activities and metacognitive regulation activities. Individually, our findings confirm findings from other studies in disparate non-medical settings and identify some new factors at play in this setting. Taken together, findings from this study provide, for the first time, some insight into how a whole assessment system influences student learning over time in a medical education setting. The findings from this authentic and complex setting paint a nuanced picture of how intricate and multifaceted interactions between various factors in an assessment system interact to influence student learning. A model linking the sources, mechanism and consequences of the pre-assessment learning effects of summative assessment is proposed that could help enhance the use of summative assessment as a tool to augment learning. (Contains 2 tables and 1 figure.)

It has become axiomatic that assessment impacts powerfully on student learning, but there is a surprising dearth of research on how. This study explored the mechanism of impact of summative assessment on the process of learning of theory in higher education. Individual, in-depth interviews were conducted with medical students and analyzed qualitatively. The impact of assessment on learning was mediated through various determinants of action. Respondents' learning behaviour was influenced by: appraising the impact of assessment; appraising their learning response; their perceptions of agency; and contextual factors. This study adds to scant extant evidence and proposes a mechanism to explain this impact. It should help enhance the use of assessment as a tool to augment learning. (Contains 1 table and 1 figure.)

Progress testing provides data on the growth of students' knowledge over the course of the curriculum obtained from the results of all students in the curriculum on periodical similar tests pitched at end-of-curriculum level. Since 2001, three medical schools have jointly constructed and administered four progress tests annually. All students in the 6-year undergraduate curricula of these schools take the same tests resulting in 24 distinct measurements per academic year (four tests for six student year groups), which may be used to compare performance between and within schools. Because single point measurements had proven unreliable, we devised a method to use cumulative information to compare schools' test performance. This cumulative deviation method involves calculation of the deviations of schools' scores from the cross-institutional average score for 24 measurement moments in 1 year. The current study shows that it appears to be feasible to use a combination of the cumulative deviation method and trend analysis for subdomains of medical knowledge to detect strengths and weaknesses in knowledge development in medical curricula. We illustrate the method by applying it to data from 16 consecutive progress tests administered to all students (4,300) of three medical schools in the academic years 2001/2002 through 2004/2005.

Context: In-training assessment (ITA), defined as multiple assessments of performance in the setting of day-to-day practice, is an invaluable tool in assessment programmes which aim to assess professional competence in a comprehensive and valid way. Research on clinical performance ratings, however, consistently shows weaknesses concerning accuracy, reliability and validity. Attempts to improve the psychometric characteristics of ITA focusing on standardisation and objectivity of measurement thus far result in limited improvement of ITA-practices. Purpose: The aim of the paper is to demonstrate that the psychometric framework may limit more meaningful educational approaches to performance assessment, because it does not take into account key issues in the mechanics of the assessment process. Based on insights from other disciplines, we propose an approach to ITA that takes a constructivist, social-psychological perspective and integrates elements of theories of cognition, motivation and decision making. A central assumption in the proposed framework is that performance assessment is a judgment and decision making process, in which rating outcomes are influenced by interactions between individuals and the social context in which assessment occurs. Discussion: The issues raised in the article and the proposed assessment framework bring forward a number of implications for current performance assessment practice. It is argued that focusing on the context of performance assessment may be more effective in improving ITA practices than focusing strictly on raters and rating instruments. Furthermore, the constructivist approach towards assessment has important implications for assessment procedures as well as the evaluation of assessment quality. Finally, it is argued that further research into performance assessment should contribute towards a better understanding of the factors that influence rating outcomes, such as rater motivation, assessment procedures and other contextual variables.

Introduction: In health science education clinical clerkships serve the twofold purpose of guiding student learning and assessment of performance. Evidently, both formative and summative assessment procedures are needed in clerkship assessment. In-training evaluation (ITE) has the potential to serve both assessment functions. Implementation of effective ITE, however, has been shown to be problematic, partly because integration of assessment functions may have negative consequences for teaching and learning. This study investigates student perceptions of the impact of an integrated assessment approach, seeking to refine criteria for effective ITE. Method: In the curriculum of Maastricht Midwifery School (MMS), clerkship assessment is based on ITE serving both assessment functions. The ITE model is based on principles of extensive work sampling, and frequent documentation of performance. A focus group technique was used to explore student perceptions on the impact of the ITE approach on student learning and supervisor teaching behaviour, and on the usefulness of information for decision making. Results: Results indicate that the assessment approach is effective in guidance of student learning. Furthermore, students consider the frequent performance documentation essential in clerkship grading. Acceptance and effectivity of ITE requires a learning environment which is safe and respectful. Transparency of assessment processes is the key to success. Suggestions for improvement focus on variation in evaluation formats, improvement of feedback (narrative, complete) and student involvement in assessment. Conclusion: ITE can fulfill both its formative and summative purposes when some crucial conditions are taken into account. Careful training of both supervisors and students in the use of ITE for student learning and performance measurement is essential.