Search

Objectives Given the absence of a common passing standard for students at UK medical schools, this paper compares independently set standards for common 'one from five' single-best-answer (multiple-choice) items used in graduation-level applied knowledge examinations and explores potential reasons for any differences. Methods A repeated cross-sectional study was conducted. Participating schools were sent a common set of graduation-level items (55 in 2013-2014; 60 in 2014-2015). Items were selected against a blueprint and subjected to a quality review process. Each school employed its own standard-setting process for the common items. The primary outcome was the passing standard for the common items by each medical school set using the Angoff or Ebel methods. Results Of 31 invited medical schools, 22 participated in 2013-2014 (71%) and 30 (97%) in 2014-2015. Schools used a mean of 49 and 53 common items in 2013-2014 and 2014-2015, respectively, representing around one-third of the items in the examinations in which they were embedded. Data from 19 (61%) and 26 (84%) schools, respectively, met the inclusion criteria for comparison of standards. There were statistically significant differences in the passing standards set by schools in both years (effect sizes ( f2): 0.041 in 2013-2014 and 0.218 in 2014-2015; both p < 0.001). The interquartile range of standards was 5.7 percentage points in 2013-2014 and 6.5 percentage points in 2014-2015. There was a positive correlation between the relative standards set by schools in the 2 years (Pearson's r = 0.57, n = 18, p = 0.014). Time allowed per item, method of standard setting and timing of examination in the curriculum did not have a statistically significant impact on standards. Conclusions Independently set standards for common single-best-answer items used in graduation-level examinations vary across UK medical schools. Further work to examine standard-setting processes in more detail is needed to help explain this variability and develop methods to reduce it. [ABSTRACT FROM AUTHOR]

Background: Inequity in assessment can lead to differential attainment. Degree classifications, such as 'Honours', are an assessment outcome used to differentiate students after graduation. However, there are no standardised criteria used to determine what constitutes these awards. Methods: We contacted all medical schools in the UK and collected data relating to classifications awarded, criteria used and percentage of students receiving classifications across the 5‐year period prior to the 2019/2020 academic year. Results: All 42 UK medical schools responded, and 36 universities provided usable data. Of these 36 universities, 30 (83%) awarded classifications above a 'Pass'. We identified four classifications above a 'Pass', and these were 'Commendation', 'Merit', 'Distinction' and 'Honours'. Sixteen (44%) universities awarded a single additional classification, and 14 (39%) universities awarded two or more. There was considerable variation in the criteria used by each university to award classifications. For example, 30 (67%) out of 45 classifications were dependent on all examined years, 9 (20%) for a combination of years and 6 (13%) for final year alone. Twenty‐five of 30 universities that awarded classifications provided data on the percentage of students awarded a classification, and a median of 15% of students received any type of classification from their university (range 5.3% to 38%). There was a wide range in the percentage of students awarded each classification type across the universities (e.g. Honours, range = 3.1%–24%). Conclusions: We demonstrate considerable variation in the way UK medical degree classifications are awarded—regarding terminology, criteria and percentage of students awarded classifications. We highlight that classifications are another form of inequity in medical education. There is a need to fully evaluate the value of hierarchical degree awards internationally as the consequential validity of these awards is understudied. Degree classifications may be another form of inequity in medical education given variation in terminology, criteria, and percentage of students awarded. Byrne et al. argue a need to better understand the consequential validity of these awards. [ABSTRACT FROM AUTHOR]

Introduction: Situational judgement tests (SJTs) have been widely adopted, internationally, into medical selection. It was hoped that such assessments could identify candidates likely to exhibit future professional behaviours. Understanding how performance on such tests may predict the risk of disciplinary action during medical school would provide evidence for the validity of such SJTs within student selection. It would also inform the implementation of such tests within student recruitment. Methods: This cohort study used data for 6910 medical students from 36 UK medical schools who sat the University Clinical Aptitude Test (UCAT) SJT in 2013. The relationship between SJT scores at application and the risk of subsequent disciplinary action during their studies was modelled. The incremental ability of the SJT scores to predict the risk of disciplinary action, above that already provided by UCAT cognitive test scores and secondary (high) school achievement, was also evaluated in 5535 of the students with information available on this latter metric. Results: Two hundred and ten (3.05%) of the students in the cohort experienced disciplinary action. The risk of disciplinary action reduced with increasing performance on the admissions SJT (odds ratio (OR) 0.80, 95% confidence interval (CI) 0.69 to 0.92, p = 0.002). This effect remained similar after adjusting for cognitive performance and prior academic attainment (OR 0.77, 95% CI 0.65 to 0.92, p = 0.004). The overall estimated effect‐size was small (Cohen's d = 0.08) and no evidence of 'threshold' effects were observed for the SJT scores and risk of disciplinary action. Conclusions: Performance on admissions SJTs can, at least modestly, incrementally predict the risk of subsequent disciplinary action, supporting their use in this context. However, for this SJT and outcome, there did not seem a distinct threshold score above which the risk of disciplinary action disproportionately increased. This should be considered when using the scores within medical selection. Performance on an admissions Situational Judgment Test is reported to predict risk of formal disciplinary action in medical students with implications for how such tests should be used to improve medical professionalism. [ABSTRACT FROM AUTHOR]

Context: Differential performance in postgraduate examinations between home medical graduates and those who qualified outside their country of practice is well recognised. This difference is especially marked in the practical component of the UK Membership of the Royal College of General Practitioners (MRCGP) examination. The potential causes of such disparities are not well understood. Methods: Data were available for 1874 international medical graduates who applied for general practice (GP) specialty training in the UK in 2008–2012. The primary outcome was performance in the Clinical Skills Assessment (CSA) OSCE component of the MRCGP. The main predictors were performance on a situational judgement test (SJT) and clinical problem‐solving test (CPST), a test of applied clinical knowledge, used in the selection for GP training. Data relating to demographic characteristics and English language fluency were also available. To better understand the relationship between the predictors, the selection measures and the outcome, a series of univariable and multivariable models were developed and tested, concluding with a structural equation model to explore causality. Results: The CSA rating was more strongly predicted by SJT scores (standardised beta, 0.26) than by performance on the CPST (standardised beta, 0.17). There was a relationship between English language fluency and CSA score that was mainly mediated via SJT performance. Conclusions: These findings demonstrate that performance on an SJT predicts performance in a high‐fidelity clinical simulation (the CSA) in international medical graduates. Although the constructs tested by SJTs are debated, and are likely to vary across settings, culturally appropriate knowledge of interpersonal competence is likely to be evaluated. Improving the confidence of doctors in this area through targeted educational interventions, rather than focusing on increased clinical knowledge, is likely to be more effective at reducing disparities observed in postgraduate examination performance. Thus, there are important implications for the design of specialty selection and licensing assessments globally. [ABSTRACT FROM AUTHOR]

Context Students who engage in self-regulated learning ( SRL) are more likely to achieve academic success compared with students who have deficits in SRL and tend to struggle with academic performance. Understanding how poor SRL affects the response to failure at assessment will inform the development of better remediation. Methods Semi-structured interviews were conducted with 55 students who had failed the final re-sit assessment at two medical schools in the UK to explore their use of SRL processes. A thematic analysis approach was used to identify the factors, from an SRL perspective, that prevented students from appropriately and adaptively overcoming failure, and confined them to a cycle of recurrent failure. Results Struggling students did not utilise key SRL processes, which caused them to make inappropriate choices of learning strategies for written and clinical formats of assessment, and to use maladaptive strategies for coping with failure. Their normalisation of the experience and external attribution of failure represented barriers to their taking up of formal support and seeking informal help from peers. Conclusions This study identified that struggling students had problems with SRL, which caused them to enter a cycle of failure as a result of their limited attempts to access formal and informal support. Implications for how medical schools can create a culture that supports the seeking of help and the development of SRL, and improves remediation for struggling students, are discussed. [ABSTRACT FROM AUTHOR]

Background In the United Kingdom medical students are selected predominantly on their academic merit. Their academic achievement marks are equated via the tariff point score structure administered by the Universities and Colleges Admissions Service (UCAS). We studied the applicant databases for 1998–2003 for one English medical school to determine the factors that predict high tariff point scores. Methods Complete demographic data and relative socio-economic status, educational institution attended and tariff point score was available for 8997 UK applicants aged 21 years or younger to the 5-year Bachelor of Medicine/Bachelor of Surgery (BM BS) course at Nottingham University medical school (and partially complete data for a further 1891 applicants). The data were subjected to standard univariate and multivariate analyses and to path analysis. Results In these samples, the independent predictors of a high tariff point score were being younger and male. The effect sizes were small, although significant. Higher tariff point scores were achieved by those from households less materially disadvantaged. Ethnicity was also a predictor with white, Chinese and those of mixed ethnic origin achieving higher tariff point scores than those from other groups. Finally, the type of school attended predicted academic achievement with applicants from further education colleges, independent schools and grant-maintained schools achieving higher tariff point scores. Conclusion Notwithstanding the relatively homogeneous (predominantly young, white, high academic achievers) applicant pool to a single UK medical school we identified consistent significant predictors of high tariff point scores. As high tariff point scores are still the major entry criterion to UK medical schools, our findings will be of value in informing policy decisions concerning ‘widening access’ schemes being established at government request. [ABSTRACT FROM AUTHOR]

To evaluate the development, validity and reliability of a multimodality objective structured clinical examination (OSCE) in undergraduate psychiatry, integrating interactive face-to-face and telephone history taking and communication skills stations, videotape mental state examinations and problem-oriented written stations.The development of the OSCE on a restricted budget is described. This study evaluates the validity and reliability of 4 15–18-station OSCEs for 128 students over 1 year. Face and content validity were assessed by a panel of clinicians and from feedback from OSCE participants. Correlations with consultant clinical‘firm grades’ were performed. Interrater reliability and internal consistency (interstation reliability) were assessed using generalisability theory.The OSCE was feasible to conduct and had a high level of high perceived face and content validity. Consultant firm grades correlated moderately with scores on interactive stations and poorly with written and video stations. Overall reliability was moderate to good, with G-coefficients in the range 0.55–0.68 for the 4 OSCEs.Integrating a range of modalities into an OSCE in psychiatry appears to represent a feasible, generally valid and reliable method of examination on a restricted budget. Different types of stations appear to have different advantages and disadvantages, supporting the integration of both interactive and written components into the OSCE format. [ABSTRACT FROM AUTHOR]