Search

Equating methods are designed to adjust between alternate versions of assessments targeting the same content at the same level, with the aim that scores from the different versions can be used interchangeably. The statistical processes used in equating have, however, been extended to statistically "link" assessments that differ, such as assessments of the same qualification type that assess different subjects. Despite careful debate on statistical linking in the literature, it can be tempting to apply equating methods and conclude that they have provided a definitive answer on whether a qualification is harder or easier than others. This article offers a novel demonstration of some limits of statistical equating by exploring how accurately various equating methods were able to equate between identical assessments. To do this, we made use of pairs of live assessments that are "cover sheet" versions of each other, that is, identical assessments with different assessment codes. The results showed that equating errors with real-world impact (e.g., an increase of 5-10 per cent in the proportion of students achieving a grade A) occurred even where equating conditions were apparently favourable. No single method consistently produced more accurate results than the others. The results emphasise the importance of considering multiple sources of information to make final grade boundary decisions. More broadly, the results are a reminder that if applied uncritically, equating methods can lead to incorrect conclusions about the relative difficulty of assessments.

The rationale for the use of comparative judgement (CJ) to help set grade boundaries is to provide a way of using expert judgement to identify and uphold certain minimum standards of performance rather than relying purely on statistical approaches such as comparable outcomes. This article summarises the results of recent trials of using CJ for this purpose in terms of how much difference it might have made to the positions of grade boundaries, the reported precision of estimates and the amount of time that was required from expert judges. The results show that estimated grade boundaries from a CJ approach tend to be fairly close to those that were set (using other forms of evidence) in practice. However, occasionally, CJ results displayed small but significant differences with existing boundary locations. This implies that adopting a CJ approach to awarding would have a noticeable impact on awarding decisions but not such a large one as to be implausible. This article also demonstrates that implementing CJ using simplified methods (described by Benton, Cunningham et al, 2020) achieves the same precision as alternative CJ approaches, but in less time. On average, each CJ exercise required roughly 30 judge-hours across all judges.

Assessment reliability can be affected by various types of unforeseen events. In any such circumstances where a concern is raised that the reliability of assessment is lower than usual, our natural inclination is to allow extra leniency in grading to reduce the chances of students missing out on a grade they deserve. This article shows how, by focusing on the risk for individual students, we might logically approach this situation in deciding exactly how much additional generosity is required. In particular, it shows how making progress with this problem requires an acceptance that no assessment system is perfect and transparency about the level of reliability that is achievable. Having developed an approach, this article also shows how this may lead to different outcomes than the competing desire to maintain assessment standards so that the group of students in question are not unfairly advantaged relative to previous and future cohorts.

Computer adaptive testing is intended to make assessment more reliable by tailoring the difficulty of the questions a student has to answer to their level of ability. Most commonly, this benefit is used to justify the length of tests being shortened whilst retaining the reliability of a longer, non-adaptive test. Improvements due to adaptive testing are often estimated using reliability coefficients based on item response theory (IRT). However, these coefficients assume that the underlying IRT model completely fits the data. This article takes a different approach, based on comparing the predictive value of shortened versions of real assessments based on adaptive and non-adaptive approaches. The results show that, when explored in this way, the benefits from adaptive testing may not always be quite a large as hoped.

This article reviews the evidence on the extent to which experts' perceptions of item difficulties, captured using comparative judgement, can predict empirical item difficulties. This evidence is drawn from existing published studies on this topic and also from statistical analysis of data held by Cambridge Assessment. Having reviewed the evidence, the article then proposes a simple mechanism by which such judgements can be used to equate different tests, and evaluates the likely accuracy of the method.

For many practical purposes, it is often assumed that the quality of a marker is directly related to their seniority. At its extreme, the assumption is that the most senior marker (the Principal Examiner) is always right even in cases where large numbers of junior markers have a collectively different opinion regarding the mark that should be awarded to a given script. To investigate this assumption, this article compares the predictive value of marks provided by the most senior marker to those made by simply taking the mean mark awarded by many junior markers. Predictive value was estimated via the correlation of the scores assigned to scripts with the overall achievement of the same candidates in other exam papers taken within the same month. By looking at the relative predictive value of the two different sources of marks, we can begin to make some inferences about the extent to which senior markers are genuinely more accurate than their junior colleagues.

The purpose of our project was to explore the explicit or implicit engagement of faculty members across the curriculum in teaching the entrepreneurial mindset. We begin by defining entrepreneurship on a spectrum, recognizing the contextual nature and psychological development associated with entrepreneurial thinking. We developed a self-report survey containing a combination of quantitative and qualitative items to determine faculty member knowledge of entrepreneurship and their engagement in teaching elements of the entrepreneurial mindset. We surveyed the faculty at a primarily teaching university in the western United States. Sixty-four faculty members (~20%) with representation from across the disciplines completed our survey. We found constrained knowledge of entrepreneurship, indications of teaching elements of the entrepreneurial mindset, and approaches to assignments that were limited in scope for fostering entrepreneurial thinking. The implications of our research are a need for professional development to enhance faculty members' knowledge of entrepreneurial thinking and support for instructional and content choices that could enhance student development of an entrepreneurial mindset.

This article demonstrates how meta-analytic techniques, that have typically been used to synthesize findings across numerous studies, can also be applied to examine the reasons why relationships between background characteristics and outcomes may vary across different locations in a single multi-site survey. This application is particularly relevant to the analysis of data from international surveys of student achievement. A brief introduction to the method of meta-regression is provided and the technique is demonstrated in an analysis of the extent to which the relationship between school autonomy and achievement varies depending upon the level of accountability in a country. The results show that the meta-regression approach to analysis is more accurate than combining data across all countries into a single simple model.

In recent years the use of expert judgement to set and maintain examination standards has been increasingly criticised in favour of approaches based on statistical modelling. This paper reviews existing research on this controversy and attempts to unify the evidence within a framework where expertise is utilised in the form of comparative judgement. Initially, the paper introduces a mathematical model for the way in which comparative judgement may operate. Data from existing studies of comparative judgement are then used to estimate suitable parameters for this model. Having derived a working mathematical model for the operation of expert judgement, the paper will then demonstrate that this model provides results that are broadly consistent with existing research, including both studies that are critical and studies that are supportive of the use of expert judgement. The model will then be applied to examine the required scale and design of expert-driven approaches to standard maintaining in order for these to have any chance of being as reliable as the currently favoured, purely statistical approaches. The paper also discusses the minimum conditions required for this approach to designing expert-driven approaches to be appropriate.

A key predictor of young people's future outcomes is their level of academic achievement whilst at school. In England this is most commonly measured by achievement in GCSEs. However, not all pupils will have taken the same set of GCSE examinations as, for example, they may make different subject choices. For this reason, GCSE performance is often aggregated into a simple measure such as "mean GCSE grade" before being used in statistical models. This paper investigates the merits of using an alternative method, based upon the relatively new technique of Generalised Boosting Models, which does not require for GCSE results in different subjects to be aggregated together. The importance of this research is that by evaluating the predictive performance of such a method we can ascertain how much useful information is lost in the process of GCSE aggregation. The results show that traditional predictions based upon simple aggregated measures of GCSE attainment are fairly similar to those based upon the more complex approach. This provides some confidence that, for the majority of outcomes, only a small amount of predictive information will be lost through the use of aggregated measures of GCSE performance.

Background: This article considers the impact of pupils studying for entry-level qualifications on their intentions of remaining in education. Such qualifications are intended to re-engage young people who are performing below expected levels and give them the opportunity to learn at a pace that suits them. Purpose: This article specifically attempts to estimate the impact of entry-level qualifications on a group of 14-16-year-olds approaching the end of compulsory schooling in England. The central research questions addressed in this article were; how do the aspirations and post-16 destinations of young people who have taken such qualifications compare with other similar young people who have taken more traditional qualifications, and is there any evidence of such qualifications being beneficial to those young people with lower levels of prior attainment? Sample: Roughly 10,000 students surveyed longitudinally within the Longitudinal Study of Young People in England (LSYPE). Design and methods: Initial data regarding the aspirations of young people was collected from the young people at age 13 or 14 in 2004. Details of the qualifications studied and achieved over the next two years were then collected from the National Pupil Database (NPD). At the end of two years, information regarding their aspirations was collected again. Data collections in subsequent years detailed whether young people had actually continued in education after the end of compulsory schooling at age 16. Multilevel modelling was used to examine the relationship between the qualifications studied by young people and their subsequent aspirations and destinations. Results: Overall, there were no differences between the outcomes of young people taking entry-level qualifications and similar young people who did not. However, the models revealed a statistically significant interaction between taking entry-level qualifications and prior attainment, suggesting a positive association between such qualifications and outcomes for those with low levels of prior attainment. Conclusions: There is some evidence that entry-level qualifications may be useful in re-engaging low-achieving young people in education.

In this paper, we present two studies examining how high school students learn to program in a mobile, social programming environment that we have developed and deployed ("IPRO"). IPRO is delivered, with an associated curriculum, as an iPod Touch app and is freely and publicly available. We find that the affordances of mobility and collaboration in IPRO deployed in a short enactment ([similarity]90 minutes) lead to improved performance on programming content targets, improved performance on computational content transfer targets, and improved attitudes towards programming and computer science more generally. (Contains 7 tables and 4 figures.)

Over the past decade, cohesion and integration have been a key concern for policymakers in countries across Europe and North America. The rapid increase in immigration, coupled with the perception of rapid social change and instability, is seen to have presented communities and governments with a range of "new" challenges. In the attendant debates, schools are often presented as part of the solution to these challenges, but much remains unknown about the relationship between schools and community cohesion. This article therefore explores the role of schools as a site of socialisation for children and young people and their role in fostering the attitudes, behaviours and norms that are typically associated with citizens in a cohesive society. This article focuses in particular on the role of school demographics and school climate and uses longitudinal data from students in England to examine these relationships. (Contains 4 tables and 18 notes.)

This article outlines an empirical investigation into equivalent forms reliability using a case study of a national curriculum reading test. Within the situation being studied, there has been a genuine attempt to create several equivalent forms and so it is of interest to compare the actual behaviour of the relationship between these forms to the relationship that would be expected theoretically. As such, this article not only reports the estimated levels of reliability based on several equivalent forms but also compares these to the levels of that would be expected from internal reliability estimates. Reliability is quantified both in terms of correlations and in terms of decision consistency. Results of analysis show a relatively good agreement between internal and equivalent forms estimates of reliability. The small discrepancies that exist tend to involve the internal estimates of reliability being higher than the equivalent forms estimates. Differences between the estimates are likely to be caused by sources of measurement error that are not captured in internal estimates. Within the situation under investigation, this may include marking and occasion-related errors. (Contains 7 tables, 1 figure, and 10 notes.)

Many have suggested that tinkering plays a critical role in novices learning to program, and recent work in learning analytics (Baker & Yacef, 2009 Blikstein, 2011) allows us to describe new relationships in the process. Using learning analytics, we explore how students progress from exploration, through tinkering, to refinement, a pathway that we term EXTIRE. The work contributes to learning sciences by: showing empirical support for previously theorized processes; identifying a role of tinkering in novices' learning; and presenting a data-driven approach to creating process descriptions. Furthermore, our findings illuminate how tinkering can be a valuable approach for novices.

Computer-based testing, or e-assessment, has the potential to deliver immediate results for the benefit of schools. This paper describes a project that aimed to exploit this potential by designing e-assessments where the results were intended for use by teachers in planning the next steps in teaching and learning: low-stakes, formative assessment. A pair of tests assessing early reading was developed, with the test items based on a range of distinct skills, including phonological segmentation, rhyming and word recognition. The tests were administered on a screen with the questions presented aurally and visually. In trials, a total of 1345 test results were obtained from pupils aged 5-7 years from 26 schools. Latent class analysis was used to identify patterns of performance within the data. Four latent classes were distinguished, each characterised by a pattern of responses related to the different test items. The strengths and weaknesses in early reading skills implied by each of these latent classes were described in terms of formative "profiles" provided for teachers in an online reporting package together with indicators for the next steps in teaching. The research resulted in an automated marking and analysis system that can be genuinely formative.

Background: Youth participation (or lack thereof) has been a subject of continued concern over the past 20 years. This decline has prompted huge interest in understanding how young people practice citizenship and in identifying measures that can help increase participation and interest. Purpose: The aim of this article is to examine how young people's citizenship practices change over the course of their adolescence, and to consider the implications for researching citizenship and citizenship education. Citizenship education has been identified as an important vehicle for shaping citizenship, but it is not the only variable shaping young people's practices and outcomes. In order to maximise the potential of citizenship education, it is therefore important to understand the ways in which age and lifestage can influence how young citizens engage with citizenship. Sources of evidence: This article is based on analysis of the longitudinal survey data from the Citizenship Education Longitudinal Survey in England (CELS), which ran from 2001 to 2009. Main findings: Trend analysis of the CELS data revealed notable increases in the cohort's participation in civic and political activities, marked changes in their attitudes, and an increase in levels of interest in and awareness of politics. This analysis also highlighted a "dip" in some of the cohort's citizenship practices as they progressed through Key Stage 4 of their compulsory secondary education (i.e. when the cohort were aged 14-16). Conclusions: Adolescence is a formative period in citizenship and educational development and young people's citizenship practices change markedly during this period. Age/life-stage is not the only variable shaping young people's citizenship practices but it has a clear impact, and can affect how young people engage with citizenship and, by extension, citizenship education. (Contains 6 figures and 15 notes.)

National value-added datasets have recently become available that record a pupil's progress from Key Stage 2 right through to GCSE. Such a dataset is clearly a useful tool for assessing the impact various characteristics of secondary schools have on pupil performance. This paper reports on a research project which involved the use of a variety of statistical techniques, including multi-level modelling, to explore the impact of single-sex education and school size on pupils' progress and opportunities. The analysis suggests that single-sex education is particularly beneficial to girls in comprehensive schools and to boys in selective schools. Furthermore, girls' schools were found to counter traditional sex-stereotyping in subject choices. In terms of school size, the results show that medium-sized schools obtain slightly better results than very large or very small schools. The paper concludes by exploring alternative explanations of the identified effects.

This paper presents two numerical approaches to modeling melt-wave erosion in conductors. The first uses a one-dimensional finite-difference formulation and provides a starting point for understanding the effects of applied field and material properties on erosion speed. The second approach involves manipulating material properties in the three-dimensional (3-D) finite-element analysis (FEA) code EMAP3D. Because EMAP3D cannot eject material from the computation in the same manner as the finite-difference code, a 'pseudoerosion' algorithm was devised to render cells above the melting temperature nonconducting. A comparison of the two approaches shows results that are in good agreement for depth of magnetic diffusion, current density, and temperature rise, as well as for depth and speed of erosion. These results are encouraging because they suggest that EMAP3D could be used to model melt-wave erosion in realistic 3-D armatures. Index Terms--Accuracy, finite difference, finite element, melt-wave erosion, time-step size.