Search

Variation in examiner stringency is a recognised problem in many standardised summative assessments of performance such as the OSCE. The stated strength of the OSCE is that such error might largely balance out over the exam as a whole. This study uses linear mixed models to estimate the impact of different factors (examiner, station, candidate and exam) on station-level total domain score and, separately, on a single global grade. The exam data is from 442 separate administrations of an 18 station OSCE for international medical graduates who want to work in the National Health Service in the UK. We find that variation due to examiner is approximately twice as large for domain scores as it is for grades (16% vs. 8%), with smaller residual variance in the former (67% vs. 76%). Combined estimates of exam-level (relative) reliability across all data are 0.75 and 0.69 for domains scores and grades respectively. The correlation between two separate estimates of stringency for individual examiners (one for grades and one for domain scores) is relatively high (r=0.76) implying that examiners are generally quite consistent in their stringency between these two assessments of performance. Cluster analysis indicates that examiners fall into two broad groups characterised as hawks or doves on both measures. At the exam level, correcting for examiner stringency produces systematically lower cut-scores under borderline regression standard setting than using the raw marks. In turn, such a correction would produce higher pass rates--although meaningful direct comparisons are challenging to make. As in other studies, this work shows that OSCEs and other standardised performance assessments are subject to substantial variation in examiner stringency, and require sufficient domain sampling to ensure quality of pass/fail decision-making is at least adequate. More, perhaps qualitative, work is needed to understand better how examiners might score similarly (or differently) between the awarding of station-level domain scores and global grades. The issue of the potential systematic bias of borderline regression evidenced for the first time here, with sources of error producing cut-scores higher than they should be, also needs more investigation.

Continuing to study mathematics throughout schooling is considered important in most developed countries, where mathematics is incorporated within the curriculum until school-leaving age. By comparison, in England, relatively few post-16 (upper secondary) students study mathematics once it becomes optional. Core Maths qualifications, introduced in England in 2014, are intended to help increase post-16 mathematics participation. This paper uses data from a three-year, mixed-methods, longitudinal study to investigate a perceived benefit of the qualifications: the support Core Maths might give to other curriculum subjects in post-16 students' programmes. Amongst teachers and students, we find a widespread conviction that studying Core Maths benefits students' other subjects contemporaneously, whilst in national data from Core Maths students we find no evidence yet of enhanced examination attainment in other qualifications. We suggest that Core Maths impacts positively on students in ways which could be more usefully, and accurately, emphasised in promoting the course.

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/domain scores and global grades acting in combination. This complexity requires a more nuanced view of what stringency might mean when considering sources of variation of cut-scores in stations. This study uses data from 349 administrations of an 18-station, 36 candidate single circuit OSCE for international medical graduates wanting to practice in the UK (PLAB2). The station-level data was gathered over a 34-month period up to July 2019. Linear mixed models are used to estimate and then separate out examiner (n = 547), station (n = 330) and examination (n = 349) effects on borderline regression cut-scores. Examiners are the largest source of variation in cut-scores accounting for 56% of variance in cut-scores, compared to 6% for stations, < 1% for exam and 37% residual. Aggregating to the exam level tends to ameliorate this effect. For 96% of examinations, a 'fair' cut-score, equalising out variation in examiner stringency that candidates experience, is within one standard error of measurement (SEM) of the actual cut-score. The addition of the SEM to produce the final pass mark generally ensures the public is protected from almost all false positives in the examination caused by examiner cut-score stringency acting in candidates' favour.

A recent government move to increase numbers of students taking post-16 (post-compulsory) mathematics in England saw 2930 students being awarded the first Core Maths (CM) qualifications in 2016, the number rising to 9027 in 2019. This paper uses qualitative data, from a study investigating the successes and challenges of these new qualifications in their initial period of implementation, to explore perspectives on how the new qualification is being positioned within the existing post-16 curriculum structure. First- and second-order effects of the policy are considered, particularly in relation to the dichotomy between CM and the longstanding, highly academic, A-level Mathematics. Our findings reveal a positive regard for many aspects of CM. However, the systemic processes by which certain students are manoeuvred onto the CM course, and the resulting opportunities for progression of those students, could be interpreted as restrictive, despite the benefits for students of new opportunities for pursuing mathematics study.

Within research into children's persuasive writing, relatively little work has been done on the writing of advertisements, how such writing develops in the primary school years and the textual features that help to secure this development. Framed within rhetoric, writing and linguistics, an exploratory study was undertaken in which a standardised task and a repeat-measures design were used to investigate the writing of an advertisement by 112 9-10-year-old pupils from two English Local Authorities, undertaken in the spring term and again a year later. The scripts were first rated for five generic constituents of writing using the standardised task guidelines. The scripts were then rated for the use of specific textual features of advertisements. All constituents, and many textual features, showed increased use across the sample as a whole, even though further analysis showed that some children who had used certain features in Year 5 had not used them in Year 6. Qualitative analysis revealed common features within attainment subgroups in content, language use and overall effectiveness of the writing. There were indications that, if appropriately supported, experience of advertisement writing could contribute to children's abilities in tackling other forms of persuasive writing.

Background: Despite the large body of literature regarding student misconceptions, there has been relatively little cross-cultural research to directly compare the prevalence of common scientific misconceptions amongst students from different cultural backgrounds. Whilst previous research does suggest the international nature of many misconceptions, there is little evidence as to whether the prevalence of such common misconceptions varies from culture to culture. Purpose: To undertake a preliminary examination of the prevalence and reasons for some previously studied scientific misconceptions amongst English and Chinese undergraduate students so as to ascertain whether there is any evidence of cultural difference. Such a finding could help to identify teaching approaches in either country that are more effective in reducing the prevalence of common student misconceptions. Sample: The study involved a convenience sample of 40 undergraduate students--20 English and 20 Chinese drawn equally from two universities in the North of England--whose formal science education ended at ages 16 and 15 respectively. Design and methods: The study employed semi-structured interview schedule containing eight questions. Results: Whilst similar misconceptions existed amongst both English and Chinese undergraduates, their prevalence was significantly higher amongst the English students (Overall mean score for scientifically correct answers amongst Chinese students was 27.7% higher, p < 0.01, r = 0.64). Often when English and Chinese undergraduates had similar misconceptions, they tended to explain these by drawing upon very similar erroneous analogies and these appear to be only nominally culturally independent in that they are based on globally shared everyday experiences. Conclusion: Differences in the prevalence of misconceptions amongst English and Chinese undergraduates appear to arise from differences in the way in which specific areas of physics are taught in both countries. It might be possible to reduce the prevalence of misconceptions in both countries if a better understanding could be developed of how, and why, undergraduates use certain erroneous analogies, and why some teaching approaches seem more effective in reducing the prevalence of misconceptions than others.

In 2006 in England an innovative suite of science qualifications for 14-16-year-olds called Twenty-First Century Science (21CS) was introduced. These qualifications have a strong focus on developing scientific literacy in all students whilst simultaneously providing preparation for the study of post-compulsory science for a smaller proportion of students. Claims have been made that such an innovative qualification would impact significantly on post-compulsory science participation--either positively or negatively. Using national data in England to track one cohort of students over 2007-2011, this study compares progression rates to post-compulsory science qualifications in England between 21CS qualifications and more traditional non-21CS qualifications. Methods employed include simple comparisons of proportions progressing from each qualification, and more complex multi-level modelling approaches that take account of both students clustered in schools, and potentially differing demographic and achievement profiles of students in the 2 groups of qualifications. A simple descriptive analysis shows that there is very little difference in overall progression rates between the 2 types of 14-16 science qualification. More fine-grained descriptive analyses show that there are some important differences, based in particular on the interaction between the amount of science studied at ages 14-16, and on the post-16 science qualification chosen (biology, chemistry or physics). Furthermore, sophisticated modelling analyses indicate a consistently negative small to moderate impact on progression from the 21CS qualification. Overall, our findings suggest that the emphasis on scientific literacy within the 21CS qualification suite has not had a major impact on the uptake of post-compulsory science qualifications.

Increasing post-compulsory participation in science and science-related subjects is seen as a key education policy priority in England and more widely. This paper uses descriptive analysis of national data to investigate the effects of science attainment at 16, gender, socio-economic status, and school science pathway on progression into post-16 traditional and vocational science courses in state-funded schools in England. Comparisons are also made with progression into non-science subjects (history, mathematics and psychology). Multi-level statistical modelling is employed to provide independent estimates for all these effects, whilst also taking into account mathematics attainment at 16, and whether or not the 14-16 school also teaches to 18. The key findings of the descriptive analysis are that progression rates vary widely across post-16 sciences in terms of both gender and socio-economic status, but that other subjects too vary in these regards. Once prior attainment is accounted for, the gender differences across science and some other subjects largely remain but those due to socio-economic status are to some extent ameliorated. In terms of school science pathways, those students doing '"more" science at 14-16 are found to be more likely to progress to traditional science post-16. The statistical modelling further quantifies the relative importance of each of these effects in determining progression and shows how, in comparison to other courses, there is more variation at the school level in progression to vocational sciences. Further, the determinants of participation in these vocational sciences courses are of a different character to the non-vocational sciences.

We report on a three-year study of teachers' experiences of a major reform of the science National Curriculum for 14- to 16-year-olds in England. Teachers' responses to this curriculum reform were guided by: "personal" aims and biography; "internal" features of their workplace such as departmental collegiality; and "external" features such as educational policies outside of science. We argue that reforms should provide teachers with sufficient flexibility to allow them to adapt reforms appropriately to local contexts. Policy makers should also consider how different educational reforms might interact over time. Professional development activities should not be seen as opportunities to promote particular curriculum reforms; rather they should support teachers in developing an informed and critically reflective perspective on curriculum policy directives.

School science courses have widely varying participation rates across a range of student characteristics. One of the stated aims of the 2006 Key Stage 4 science curriculum reforms in England was to improve social mobility and inclusion. To encourage students to study more science, this reform was followed by the introduction in 2008 of an entitlement to study the three separate sciences at Key Stage 4 for the more highly attaining students. This paper uses longitudinal national data over a five-year period to investigate the extent and change of participation across science courses at KS4, focusing on student gender and socio-economic status. It finds that whilst there is some evidence of a move towards a more equitable gender balance for some courses, there is as yet little evidence of substantial change in differential participation rates by socio-economic status. (Contains 4 figures and 1 table.)

Over recent years, UK medical schools have moved to more integrated summative examinations. This paper analyses data from the written assessment of undergraduate medical students to investigate two key psychometric aspects of this type of high-stakes assessment. Firstly, the strength of the relationship between examiner predictions of item performance (as required under the Ebel standard setting method employed) and actual item performance ("facility") in the examination is explored. It is found that there is a systematic pattern of difference between these two measures, with examiners tending to underestimate the difficulty of items classified as relatively easy, and overestimating that of items classified harder. The implications of these differences for standard setting are considered. Secondly, the integration of the assessment raises the question as to whether the student total score in the exam can provide a single meaningful measure of student performance across a broad range of medical specialties. Therefore, Rasch measurement theory is employed to evaluate psychometric characteristics of the examination, including its dimensionality. Once adjustment is made for item interdependency, the examination is shown to be unidimensional with fit to the Rasch model implying that a single underlying trait, clinical knowledge, is being measured. (Contains 2 notes, 3 figures and 4 tables.)

This paper uses data from the National Pupil Database to investigate the differences in "performance" across the range of science courses available following the 2006 Key Stage 4 (KS4) science reforms in England. This is a value-added exploration (from Key Stage 3 [KS3] to KS4) aimed not at the student or the school level, but rather at that of the course. Different methodological approaches to carrying out such an analysis, ranging from simple non-contextualized techniques, to more complex fully contextualized multilevel models, are investigated and their limitations and benefits are evaluated. Important differences between courses are found in terms of the typical "value" they add to the students studying them with particular applied science courses producing higher mean KS4 outcomes for the same KS3 level compared with other courses. The implications of the emergence of such differences, in a context where schools are judged to a great extent on their value-added performance, are discussed. The relative importance of a variety of student characteristics in determining KS4 outcomes are also investigated. Substantive findings are that across all types of course, science prior attainment at KS3, rather than that of mathematics or English, is the most important predictor of KS4 performance in science, and that students of lower socio-economic status consistently make less progress over KS4 than might be expected, despite prior attainment being accounted for in the modelling. (Contains 4 tables, 2 figures, and 11 notes.)

This article is concerned with the impact of recent reforms to specifications for science exams taken by secondary school students in England at age 16+, and other policy shifts, particularly that towards "triple science". It draws on data from the National Pupil Database/Pupil Level Annual School Census, together with fieldwork in a range of schools. These data suggest that the increasing diversity of provision at key stage 4 (ages 14-16) may be leading to increasingly stratified student take-up, particularly according to attainment, and that schools and teachers themselves differ in their judgement of, and responses to, the new forms of provision. The resulting issues of equity and access are discussed. (Contains 5 figures and 2 tables.)

This article compares the examination performance of students following General Certificate of Secondary Education Applied Science and Double Award science on a value-added basis. It uses data from the National Pupil Database, which are analysed by a variety of methods, including multilevel modelling. It argues that the claims made within Office for Standards in Education reports on the impact of the new Applied Science course are inaccurate, and in some cases positively wrong. The authors' analysis, based on the second (2003-05) cohort of students following the course, shows that students of lower attainment at Key Stage 3 appear to perform better than would have been predicted from their Key Stage 3 attainment, but that higher attaining pupils perform less well. Schools offering Applied Science show greater diversity in the value-added effects they produce than they do with Double Award science. The article argues that analyses of this kind are of value in seeking to understand the impact of curriculum reform, but that they are dangerous as a generalized guide for policy. They must be complemented by work in schools in which the sources of the effects observed and of school diversity are examined. (Contains 6 tables, 1 figure and 16 notes.)

This paper explores gender differential performance in "gifted and talented" 9- and 13-year-olds in a mathematics assessment in England. Boys' and girls' attitudes to mathematics and their views about which gender is better at mathematics are also considered. The study employs the use of a matched sample of boys and girls so that school, age and previous achievement in mathematics can be controlled whilst exploring performance on World Class Test items. The main result of this research was that there was no significant gender difference in performance for the 9- or the 13-year-olds. However, attitudinal differences were found, including a seemingly commonly held stereotypical view of mathematics as a boys' subject. These results are important since the uptake of higher level mathematically-based courses by girls is poor. Further findings reveal that where "gifted" girls perform as well as "gifted" boys, their confidence in the subject is lower than their performance might suggest. This work is also discussed in the light of related research findings and in relation to stereotype threat theory. (Contains 10 tables and 8 figures.)

A cursory glance at 2008 might suggest it was the year in which Mormons firmly cemented their loyalty to the GOP. Former Massachusetts Governor Mitt Romney was the most serious [...]

Continuing to study mathematics throughout schooling is considered important in most developed countries, where mathematics is incorporated within the curriculum until school-leaving age. By comparison, in England, relatively few post-16 (upper secondary) students study mathematics once it becomes optional. Core Maths qualifications, introduced in England in 2014, are intended to help increase post-16 mathematics participation. This paper uses data from a three-year, mixed-methods, longitudinal study to investigate a perceived benefit of the qualifications: the support Core Maths might give to other curriculum subjects in post-16 students' programmes. Amongst teachers and students, we find a widespread conviction that studying Core Maths benefits students' other subjects contemporaneously, whilst in national data from Core Maths students we find no evidence yet of enhanced examination attainment in other qualifications. We suggest that Core Maths impacts positively on students in ways which could be more usefully, and accurately, emphasised in promoting the course. [ABSTRACT FROM AUTHOR]

Many institutions require candidates to achieve a minimum number of OSCE stations passed (MNSP) in addition to the aggregate pass mark. The stated rationale is usually that this conjunctive standard prevents excessive degrees of compensation across an assessment. However, there is a lack of consideration and discussion of this practice in the medical education literature. We consider the motivations for the adoption of the MNSP from the assessment designer perspective, outlining potential concerns about the complexity of what the OSCE is trying to achieve, particularly around the blueprinting process and the limitations of scoring instruments. We also introduce four potential methods for setting an examinee-centred MNSP standard, and highlight briefly the theoretical advantages and disadvantages of these approaches. There are psychometric arguments for and against the limiting of compensation in OSCEs, but it is clear that many stakeholders value the application of an MNSP standard. This paper adds to the limited literature on this important topic and notes that current MNSP practices are often problematic in high stakes settings. More empirical work is needed to develop understanding of the impact on pass/fail decision-making of the proposed standard setting methods developed in this paper. [ABSTRACT FROM AUTHOR]

In England, a relatively new set of post-16 qualifications has been developed under the umbrella term 'Core Maths', with a focus on the application of mathematics in context, including the kinds of mathematics needed to support other subjects, to provide a sound basis for the mathematical demands of higher education and employment and to develop problem-solving skills for use in life. The UK government has an ambition that all post-16 students should be studying some mathematics, and Core Maths was designed in part with this aspiration in mind. In this paper, longitudinal questionnaire data from over 100 Core Maths students in 13 case study institutions are analysed to measure students' views of teaching as transmissionist, their mathematical dispositions and self-identification and how these change over a year of studying Core Maths. We find some evidence that pedagogy in lessons is perceived as being less transmissionist than it was in school mathematics pre-16. There is also some evidence of a negative change in students' mathematical dispositions over a year of Core Maths. We conclude that supporting teachers in embedding new pedagogical approaches remains a challenge and that this issue could inhibit the growth of new qualifications like Core Maths. [ABSTRACT FROM AUTHOR]

Introduction: Borderline regression (BRM) is considered problematic in small cohort OSCEs (e.g. n < 50), with institutions often relying on item-centred standard setting approaches which can be resource intensive and lack defensibility in performance tests. Methods: Through an analysis of post-hoc station- and test-level metrics, we investigate the application of BRM in three different small-cohort OSCE contexts: the exam for international medical graduates wanting to practice in the UK, senior sequential undergraduate exams, and Physician associates exams in a large UK medical school. Results: We find that BRM provides robust metrics and concomitantly defensible cut scores in the majority of stations (percentage of problematic stations 5, 14, and 12%, respectively across our three contexts). Where problems occur, this is generally due to an insufficiently strong relationship between global grades and checklist scores to be confident in the standard set by BRM in these stations. Conclusion: This work challenges previous assumptions about the application of BRM in small test cohorts. Where there is sufficient spread of ability, BRM will generally provide defensible standards, assuming careful design of station-level scoring instruments. However, extant station cut-scores are preferred as a substitute where BRM standard setting problems do occur. [ABSTRACT FROM AUTHOR]