Search

Inspired by the work of David Raffe and his co-authors who set out the positive benefits gained from comparing the policies of "the UK home nations" in an article published in 1999, researchers in the Education and Curriculum Team launched a project in early 2020 that we called Curriculum Watch. The aim of this project was to collate a literature and documents database of education and curriculum policies, research and analyses from across the four countries of the United Kingdom (UK). In this article, we draw on our literature database to make sense of the rapid changes in education policy that occurred in the early stages of the COVID-19 pandemic in the four UK nations of England, Scotland, Wales and Northern Ireland. We analyse some of the key areas of UK policy formation and content (in relation to curriculum, pedagogy and assessment) that we observed during the first six months of the unfolding pandemic. In addition, we reiterate the clear benefits of using comparative research methods in the UK context: our research findings support the idea that closeness of national contexts offers the opportunity for evidence exchange and policy learning in education.

Educational taxonomies are classification schemes which provide the terminology that educationalists need to describe and work with different areas of knowledge. It is good practice to use taxonomies to formulate and review curricula, learning objectives, and associated assessments. Demonstrating sufficient coverage of each of an adequate range of knowledge domains is critical for authenticity, for assessment reliability, and for transparency surrounding what students are learning. In this study we explored whether any educational taxonomies that were designed for general educational contexts (sometimes called 'academic' contexts) could be utilised in applied educational contexts (often called 'vocationally-related' in England). To do this, we identified nine published taxonomies with sufficient potential, and selected and combined the most appropriate. This process led us to develop a new model of demand. We then applied the selected taxonomies experimentally to existing curricula in a range of applied subjects which are taught at secondary and tertiary level in England. We also used the selected taxonomies to develop a tool for writing educational objectives. This article ends with suggestions for applying the selected taxonomies in other areas of assessment.

The Learning Passport (LP) is a collaborative project between the University of Cambridge, UNICEF and Microsoft, which aims to support the UNICEF goal of providing quality education provision for children and youth whose education has been disrupted by crisis or disaster. A core component of this project is a curriculum framework for Mathematics, Science and Literacy which supports educators working in emergency contexts. This framework provides a broad outline of the essential content progressions that should be incorporated into a curriculum to support quality learning in each subject area, and is intended to act as a blueprint for localised curriculum development across a variety of contexts. To support educators in the development of this localised curriculum an LP Adaptation Guidance document was also created. This document provides guidance on several factors that local curriculum developers should consider before using the LP Curriculum Framework for their own curriculum development process. This article discusses how key areas within the LP Adaptation Guidance have broader relevance beyond education in emergencies, highlighting that the challenges that exist within some of the most deprived educational contexts have applicability in all contexts.

Standardisation is a procedure used by Awarding Organisations to maximise marking reliability, by teaching examiners to consistently judge scripts using a mark scheme. However, research shows that people are better at comparing two objects than judging each object individually. Consequently, Oxford, Cambridge and RSA (OCR, a UK awarding organisation) proposed investigating a new procedure, involving ranking essays, where essay quality is judged in comparison to other essays. This study investigated the marking reliability yielded by traditional standardisation and ranking standardisation. The study entailed a marking experiment followed by examiners completing a questionnaire. In the control condition live procedures were emulated as authentically as possible within the confines of a study. The experimental condition involved ranking the quality of essays from the best to the worst and then assigning marks. After each standardisation procedure the examiners marked 50 essays from an AS History unit. All participants experienced both procedures, and marking reliability was measured. Additionally, the participants' questionnaire responses were analysed to gain an insight into examiners' experience. It is concluded that the Ranking Procedure is unsuitable for use in public examinations in its current form. The Traditional Procedure produced statistically significantly more reliable marking, whilst the Ranking Procedure involved a complex decision-making process. However, the Ranking Procedure produced slightly more reliable marking at the extremities of the mark range, where previous research has shown that marking tends to be less reliable.

As part of GCSE (General Certificate of Secondary Education) reforms in England, requirements for assessing application in science increased. Setting examination questions in context facilitates testing application as students need to apply what they know and understand to a particular situation. This research explored the nature of the contexts used in reformed GCSE combined science examinations and compared contexts used in a specification which specifically emphasises contextualised learning to those in other specifications. Eight combined science specimen examination papers were selected, including four from GCSE Twenty First Century Science (21C). A qualitative coding frame was used to code each contextualised item. Various strategies for testing in context were present. Contextual features that might risk introducing construct-irrelevant variance were infrequent but may suggest areas for attention in setter training. 21C papers included a higher proportion of items with detailed contexts and a higher proportion of items set in science-related adult/professional settings.

Background: "Thinking aloud" is a well-established method of data collection in education, assessment, and other fields of research. However, while many researchers have reported their views on its usage, the first-hand experiences of research participants have received less attention. Purpose: The aim of this exploratory study was to obtain the perspectives of GCSE examiners on thinking aloud as a means of investigating the cognitive strategies they use to mark examinations. Sample: The study focused on the experimental marking of two contrasting GCSE examination papers from 2004: mathematics and business studies. Twelve experienced examiners (six for each subject) participated in the study as paid volunteers. Design and methods: All examiners were individually asked to think aloud while concurrently marking up to five examination scripts each. Afterwards, they participated in one-to-one, semi-structured interviews about their experiences of the method. These interviews were recorded, transcribed and analysed qualitatively. Results: There were no clear differences between the mathematics and business studies examiners. However, views on the validity of thinking aloud, and advantages and disadvantages, varied considerably among individuals, possibly reflecting diverse personal marking techniques within each subject. Examiners varied in the kinds of questions they found it hardest to think aloud about. Conclusions: The findings have implications for further research and marking practice. They challenge two contrasting views reported in the literature: (1) that the simplest judgements are always hardest to verbalise; and (2) that it is always hardest to think aloud when engaging in complex thought processing. This indicates apparent variation in validity of the think aloud method among individuals. However, thinking aloud may also be useful in the training of new examiners. (Contains 2 notes.)

It is particularly important that GCSE and A-level marking is valid and reliable as it affects the life chances of many young people in England. Current developments in marking technology are coinciding with potential changes in procedures to ensure valid and reliable marking. In this research the effectiveness of procedures to facilitate the reliability of marking were evaluated using a design with three interventions. Two of the interventions, the standardisation meeting and personal feedback, are similar to the conventional approach in GCSE and A-level practice but within the constraints of a research study. The other intervention, pre-written feedback, is how examples of marked scripts might be presented for part of examiner standardisation or training in other situations; it is also similar to the approach to feedback used in earlier research by Shaw and Baird et al. Each participating examiner experienced two of the three types of intervention. Marking data were gathered before and after the first intervention as well as after the second intervention. Inter-examiner reliability was analysed using ANOVA models. Due to the different levels of inter-examiner reliability at the beginning of the experiment it was hard to compare the effectiveness of the different interventions. Nevertheless, this research highlights that: (1) in line with previous research, standardisation meetings "on their own" are not particularly effective for "experienced" examiners; (2) as has been found in previous research, more experienced examiners were more lenient; (3) most combinations of interventions (combinations of standardisation processes--including the conventional approach) led to a decline in the size of the marking error; and (4) sometimes feedback (personal or pre-written) was effective and sometimes it was not. (Contains 3 figures, 9 tables and 3 notes.)

The process of examination marking is complex, requiring examiners to engage in a variety of cognitive operations. While consideration has been given to marking practices in a few specific contexts, those of General Certificate of Secondary Education (GCSE) examiners have yet to receive serious attention. This study's aims, therefore, were: first, to investigate the cognitive strategies used when marking GCSEs; and second, to interpret them within the context of psychological theories of human judgement. Two contrasting GCSE examination papers were considered: Mathematics used a "points-based" marking scheme, while Business Studies relied on a "levels-based" scheme. Small groups of experienced examiners marked script samples, and using a concurrent "think aloud" method, verbal protocols were obtained. Using a semi-structured interview schedule, the examiners were then questioned retrospectively. Qualitative data analysis supported a model of five distinct cognitive marking strategies, which can be interpreted within dual-processing theories of judgement. The implications for GCSE marking are discussed. (Contains 1 table.)

Diverse strategies for marking GCSE examinations have been identified, ranging from simple automatic judgements to complex cognitive operations requiring considerable expertise. However, little is known about patterns of strategy usage or how such information could be utilised by examiners. We conducted a quantitative analysis of previous verbal protocol data collected from twelve experienced examiners using a concurrent "think aloud" method. Two contrasting examination papers had been marked: mathematics and business studies. Five senior examiners were interviewed about the findings. The analysis revealed differences in strategy usage among individual examiners, but most strikingly, between subjects and among question parts. For most questions, examiners used strategy combinations rather than single strategies. Business studies question parts could be categorised according to whether they entailed simple or complex strategies. However, most mathematics question parts entailed a mixture of diverse strategies. The examiners identified potential implications of the findings for examiner training and modernising marking systems. (Contains 1 table, 4 figures and 5 notes.)

Technological innovation undoubtedly offers many potential benefits for education and the assessment of learning, which have been acknowledged elsewhere. One area that is relatively under-researched relates to the practice of how assessors interact with longer texts that are presented on screen. This is an important area of study because there might be moves to scan or digitally capture longer texts, such as essays or portfolios that contain long textual chunks, and to deliver them online to assessors. This use of technology affords a number of benefits, including the opportunity to transfer data speedily. It also allows a number of assessors to view the same evidence simultaneously, enabling the timely checking of assessor reliability during the assessment process. This article explores the theoretical and limited empirical literature to discuss the potential impact that technology might have on assessors' judgments of texts presented on screen. The review cites literature from across a variety of educational settings where it is increasingly becoming common practice to capture and assess learners' work through digital methods. (Contains 2 notes.)

There is no published empirically based research on the new GCE synoptic assessment. Consequently no published research has been undertaken on its role, how it is taught or how pupils are prepared for the assessment. Neither is there a published evaluation of the effectiveness of teaching the synoptic units. It is these issues which are addressed in the present study. One hundred and twenty-five chemistry teachers replied to a questionnaire about the teaching strategies that they used and how they prepared candidates for A2 assessments. The analysis of the questionnaire responses found that it is the type of assessment--coursework or examination--that closely corresponded with the teaching strategies, perceived learning outcomes and approaches to preparing pupils for A2 assessments. The questionnaire responses were merged with GCSE and A level results. A series of multilevel models were fitted to identify any relationship between relative progress from mean GCSE scores to A level unit results and the questionnaire responses. Only significant results are presented and discussed. The synoptic unit and another A2 unit were assessed by a traditional written examination. Teaching the latter unit separately to other units and teachers' question spotting related to lower relative progress from mean GCSE to A level examination results. For the synoptic unit, teachers' question spotting related to a lower relative progress from mean GCSE to unit marks. For this unit the following related to higher relative progress from mean GCSE to synoptic unit results: (i) the teachers using grade descriptors to inform their preparation of pupils for the assessment; (ii) including the whole of the AS curriculum in revision sessions. For the coursework, the following related to higher relative progress from mean GCSE to unit marks: (i) pupils doing worked examples or exercises; (ii) teachers using statistical information about examination performance prepared by an external organisation for their centre. The following are appended: (1) The role of the synoptic unit in chemistry; and (2) Questions and responses to the chemistry questionnaire given as means (or % Yes) and frequencies. (Contains 8 tables and 5 notes.)

The research literature reveals that there are many factors that influence the consistency of assessors' or examiners' judgements. One issue that has not been considered is whether National Vocational Qualifications assessors' consistency of judgement is affected by different types of evidence. In this article, 15 Customer Service and 12 Assessor and Verifier assessors made assessment judgements about fictitious candidates' evidence. For the Customer Service and Assessor and Verifier units the assessors made judgements about different types of evidence (assessor observations, witness testimonies, personal statements, written underpinning knowledge questions and answers, and record of professional discussions). The assessors' judgements were statistically analysed. The internal consistency reliability was high, indicating that assessors tend to respond to each assessment criterion within a set in a similar way or that each set of criteria are measuring a common trait. The results are also used to discuss the consistency of assessors' judgements and whether they are affected by different types of evidence. When absolute differences were used to measure inter-rater reliability there was a statistically significant difference for the Assessor and Verifier Award--the assessors' judgements were in less agreement when using the witness testimony than when they assessed other types of evidence. Therefore, the effect of different types of evidence on the agreement between assessors' judgements about competence cannot be dismissed as an issue. Further research is required before we can say precisely which types of evidence affect assessor judgements and in which circumstances. (Contains 3 tables and 6 notes.)

Marking reliability is purported to be produced by having an effective community of practice. No experimental research has been identified which attempts to verify empirically the aspects of a community of practice that have been observed to produce marking reliability. This research outlines what that community of practice might entail and presents two experimental studies on the effects of particular aspects of community of practice on examiners' marking reliability. In the first study, the impact of exemplar work is investigated: examiners were provided with mark schemes and some examiners were provided with exemplar scripts and given feedback about the marking of those scripts. In the second study, the effects of discussion of the mark scheme are explored: all examiners received mark schemes and exemplar scripts, but some examiners did not attend a coordination meeting. Neither procedure (use of exemplar scripts or discussion between examiners) demonstrated an improvement in marking reliability, which called into question the predictive utility of the theory of community of practice.

For many years there has been concern about whether sex bias exists in various assessments. A literature review reveals little if any sex bias in UK national assessments at GCSE. Oxford, Cambridge and RSA examiners, for three case study subjects, completed the Bem Sex Role Inventory, a self-report inventory that measures the extent to which respondents are sex-typed. The responses were used to investigate (1) whether there was any sex bias in the marking, (2) the relation between the sex of the examiners and their marking. In the English examination one question was biased by half a mark in favour of girls, and it was found that the more senior the examiner the more generous the marking. These findings concur with results in other areas like testing English as a second language. It is concluded that sex bias should be monitored but is unlikely to be found.

The article discusses a study about the strategies that International General Certificate of Secondary Education (IGCSE) examiners apply to mark scripts of the candidates. The strategies include matching, scanning, evaluating, scrutinizing and no response. Accordingly, the research aims to understand more on the decision-making processes of the examiners.

Marking reliability is purported to be produced by having an effective community of practice. No experimental research has been identified which attempts to verify empirically the aspects of a community of practice that have been observed to produce marking reliability. This research outlines what that community of practice might entail and presents two experimental studies on the effects of particular aspects of community of practice on examiners' marking reliability. In the first study, the impact of exemplar work is investigated: examiners were provided with mark schemes and some examiners were provided with exemplar scripts and given feedback about the marking of those scripts. In the second study, the effects of discussion of the mark scheme are explored: all examiners received mark schemes and exemplar scripts, but some examiners did not attend a coordination meeting. Neither procedure (use of exemplar scripts or discussion between examiners) demonstrated an improvement in marking reliability, which called into question the predictive utility of the theory of community of practice. [ABSTRACT FROM AUTHOR]