Search

In the UK and elsewhere, large-scale educational assessment agencies are shifting the mode of school examination marking towards having examiners mark examination scripts on screen rather than on paper. This shift has prompted questions about whether the mode of marking might influence examiner marking accuracy, particularly in relation to extended essay responses. The study reported in this article investigated this issue empirically as part of a wider research project which looked broadly at the influence of marking mode on 12 examiners' marking outcomes and processes when assessing samples of extended essays. No association was found between marking mode and the magnitude of marking accuracy, and a practically and statistically negligible association was found between marking mode and the direction of marking accuracy. Given the mode-related influences on examiners' navigating and annotating behaviours that were also evident in the data, the article discusses the potential relationships between the findings. (Contains 9 tables, 2 figures and 3 notes.)

This research investigated whether features of examination questions influence students with dyslexia differently to others, potentially affecting whether they have a fair opportunity to show their knowledge, understanding and skills. A number of science examination questions were chosen. For some questions two slightly different versions were created. A total of 54 students considered by their teachers to have dyslexia and a matched control group of 51 students took the test under exam conditions. A dyslexia screening assessment was administered where possible and some students were interviewed. Facility values and Rasch analysis were used to compare performance between the versions of the same question and between those with and without dyslexia. Chi-square statistics found no statistically significant differences in performance between groups or between question versions. However, some tentative implications for good practice can be inferred (e.g. avoiding ambiguous pronouns, using bullet points). (Contains 1 note, 5 tables, and 4 figures.)

There is a growing body of research literature that considers how the mode of assessment, either computer-based or paper-based, might affect candidates' performances. Despite this, there is a fairly narrow literature that shifts the focus of attention to those making assessment judgements and which considers issues of assessor consistency when dealing with extended textual answers in different modes. This research project explored whether the mode in which a set of extended essay texts were accessed and read systematically influenced the assessment judgements made about them. During the project, 12 experienced English literature assessors marked two matched samples of 90 essay exam scripts on screen and on paper. A variety of statistical methods were used to compare the reliability of the essay marks given by the assessors across modes. It was found that mode did not present a systematic influence on marking reliability. The analyses also compared examiners' marks with a gold standard mark for each essay and found no shifts in the location of the standard of recognised attainment across modes.

The language of formative assessment, sometimes referred to as "assessment for learning" (AFL), is becoming ubiquitous within international educational discourse. Despite this, levels of common understanding might be considered questionable since the concept appears to have been subject to many different interpretations and to have resulted in varying implementations. The reason for this ambiguity partly lies in the way that the language of formative assessment or assessment "for" learning has increasingly been applied beyond the context of classroom practice; shifting the focus of assessment from explaining how classroom learning interactions can enhance individual development to considerations involving larger-scale, system-wide accountability purposes. This shift appears to contradict the original spirit of AFL and has led to a blurring of the formative and summative boundaries of assessment. This article presents some empirical data from a small-scale research project to suggest that this issue has a practical impact in the classroom. The article considers the variety of ways that the language of AFL is used by teachers in differing international contexts and suggests that a conflict of purpose might underlie such variations. The article goes on to suggest the need for further research addressing some of the fundamental paradoxes within assessment discourse caused by this conflation of purposes, and identifies areas for further research and investigation.

We investigate differences in what students learn about survey methodology in a class on public opinion presented in two critically different ways: with the inclusion or exclusion of an original research project using a random-digit-dial telephone survey. Using a quasi-experimental design and data obtained from pretests and posttests in two public opinion courses, we test the hypothesis that students who participate in an original survey research project will have a stronger understanding of survey research methods than students who do not. To better assess the effect of the active learning element of the course, we estimate average treatment effects on the students who participated in the original survey project using nearest neighbor matching (Abadie et al. 2004) with student scores on a pretest. We find evidence of modest improvement in learning of survey methods in the course featuring the original survey research project; however, the major finding here is that a course featuring this kind of opportunity appeals to a different kind of student than a course that allows participants to stay closer to the classroom and library instead of the social science research laboratory. This discovery may have important implications for our understanding of the effects of active learning opportunities in other types of elective courses.

Within large scale educational assessment agencies in the UK, there has been a shift towards assessors marking digitally scanned copies rather than the original paper scripts that were traditionally used. This project uses extended essay examination scripts to consider whether the mode in which an essay is read potentially influences the judgements made about it, employing qualitative and quantitative data-gathering approaches to focus on the annotation practices of readers who are assessing extended essays, and to explore whether this might be affected by mode. The project also gathers evidence about the spatial encoding ability of these readers to suggest that mode-related influences on annotation might influence the ability of readers to comprehend fully the material being read. Examiners had less spatial awareness about the location of script features on screen than on paper; however, annotating on screen might link to both the provision of appropriate annotations and the relative ease with which these are deployed whilst reading. There was also evidence that on-screen annotation could contribute to the development of good mental representations of texts on screen. (Contains 3 figures and 2 tables.)

Although the school assessment context in England relies heavily on externally imposed forms of testing, it also takes many different forms, including the use of examinations with pre-release materials. These examinations allow teachers and students early access to materials that will be used during assessments at a later date. This project uses a focus on validity and effects on motivation to explore this under-theorised assessment form. Data were gathered from six secondary schools that were using pre-release materials in advance of a General Certificate of Secondary Education (GCSE) geography examination. These data were collected via teacher interviews, lesson observations, and a questionnaire to students. Study findings suggest that the pre-release examination mode was beneficial in terms of: motivation; providing a positive learning experience; and facilitating the assessment of higher order types of skills to those often tested by other exams. (Contains 2 tables and 1 figure.)

Consistency of assessor judgement is a key concern for those charged with accrediting vocationally related learning outcomes. Literature suggests that the consistency of assessor judgements can be affected by the breadth of vocationally related contexts and concomitant learner and assessor experiences. It is important to understand how experts make judgements about learners' performances since this might help to make the assessment system more transparent and able to justify claims of fairness. This is particularly the case for vocationally related holistic assessments where assessors might attend to a variety of factors in different contexts when forming a judgement. This study uses qualitative data concerning the social and contextual features in which assessors' practice is situated. It uses Cultural Historical Activity Theory to explore a theoretical position suggesting that assessor judgments are influenced or framed within the context of their experience and differing perspectives. (Contains 2 figures and 2 tables.)

The UK educational assessment landscape is changing, characterised by attempts to create more flexible pathways through learning. This has led to attempts to formalise the comparative relationships between different general and vocational qualifications. The National Qualifications Framework (NQF), the Framework for Achievement and the Framework for Higher Education Qualifications in England, Wales and Northern Ireland all seek to explore the potential for comparing the value of different qualifications. These initiatives mirror other international trends, such as the implementation of the Australian Qualifications Framework. Grading is a potential enabling tool for comparing different qualifications. This is visible in the way that the NQF uses General Certificate of Secondary Education grades C and D to discriminate between Levels 2 and 3 of the framework. This article focuses on the challenges posed for competency-based qualifications in the light of such national and international moves towards developing unified qualifications frameworks. This is important given the strong traditional relationship between competency-based qualifications and binary (pass/fail or competent/not yet competent) reporting and its historically ambivalent relationship with grading techniques. The article also discusses some of the particular issues that are relevant when competency-based assessments are graded.

This study investigated the cognitive strategies that underpin assessors' holistic judgments of a school-based vocationally-related portfolio performance. Using a portfolio already identified as containing borderline qualities, quantitative data were gathered about features that six assessors attended to as they holistically evaluated the portfolio. This information was gathered through verbal protocols and supplemented with information from a modified Kelly's Repertory Grid interview technique (Kelly, 1955). This elicited assessors' perceptions about the characteristics of the assessment criteria, allowing the influence of each factor to be ranked. Another objective was to collect qualitative data about the socio-contextual features in which the assessors' practices were situated. The study uses Activity Theory to explore the relationship between the factors that potentially influence assessors' judgments, suggesting a theoretical position that assessor judgments are influenced or framed within the context of their experience and differing perspectives. (Contains 3 tables and 1 figure.)

Technological innovation undoubtedly offers many potential benefits for education and the assessment of learning, which have been acknowledged elsewhere. One area that is relatively under-researched relates to the practice of how assessors interact with longer texts that are presented on screen. This is an important area of study because there might be moves to scan or digitally capture longer texts, such as essays or portfolios that contain long textual chunks, and to deliver them online to assessors. This use of technology affords a number of benefits, including the opportunity to transfer data speedily. It also allows a number of assessors to view the same evidence simultaneously, enabling the timely checking of assessor reliability during the assessment process. This article explores the theoretical and limited empirical literature to discuss the potential impact that technology might have on assessors' judgments of texts presented on screen. The review cites literature from across a variety of educational settings where it is increasingly becoming common practice to capture and assess learners' work through digital methods. (Contains 2 notes.)

This study investigated the functions of annotations, the role of annotations in markers' decision-making processes, whether annotations conform to conventions, and whether these vary according to subject area. Across subjects a number of scripts were analysed to survey which annotations are subject specific and which are more general. Twelve examiners at different levels of responsibility marked the same General Certificate of Secondary Education Mathematics and Business Studies scripts as one another. The marking was analysed to identify whether there were any patterns in the use of annotations. Retrospective individual interviews were used to inform the analysis of the annotations. Findings showed that the frequency and type of annotations used differed between subjects and that within a subject team examiners drew from a common pool of annotations. Purposes for annotation use included supporting markers' judgements and explaining decisions. (Contains 7 tables.)

This study explores whether the anticipation of a graded (merit, pass, fail) test outcome rather than a binary (pass, fail) form of feedback influenced the motivation of learners taking a vocationally related test. Ninety-four students currently taking a vocationally related qualification were divided into control and experimental groups. Each student completed two matched tests, approximately one week apart, having been told in advance whether their results were to be reported in a binary or graded manner. Following each test a self-report measure of their motivation was collected through the use of a questionnaire based on an Expectancy-Value model of motivation. This paper reports on the statistical analysis of the questionnaire data exploring whether there were any motivational differences between participants' responses to the grading situations. It also investigates whether there were any differences in test scores between the groups, since theory suggests that motivation is an important factor in performance outcomes. (Contains 5 tables and 6 figures.)

This study follows from a previous study into children's attitudes to writing test stimulus features. In that study the views of 192 English eleven-year-olds were surveyed using a questionnaire. The survey found that the children were mainly influenced by features that they felt contributed to task difficulty. A qualitative study was designed in order to investigate children's views in more depth. Stimuli were constructed containing various features that children in the earlier study had suggested contributed to task difficulty. The children's ideas relating to the stimuli were elicited using a modified version of Kelly's repertory grid technique.